mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
docs: finalize scalable matchmaking plan
This commit is contained in:
+423
-196
@@ -1,243 +1,470 @@
|
||||
# Matchmaking — casual and ranked queues
|
||||
|
||||
Design scope for online casual and ranked play. This is a **1.0 launch
|
||||
blocker**, not a post-launch addition.
|
||||
Architecture and locked product policy for Phase 8. This is a **1.0 launch
|
||||
blocker**. The numbered, independently implementable tasks and their
|
||||
acceptance criteria live in [`multiplayer-todo.md`](../multiplayer-todo.md);
|
||||
the short live checklist is [`multiplayer-next.md`](../multiplayer-next.md).
|
||||
|
||||
Nothing described here is implemented yet. This doc exists to record the
|
||||
decisions and the reasoning before code is written; per-task implementation
|
||||
evidence belongs in `multiplayer-todo.md` once work starts, and the live
|
||||
checklist lives in [`multiplayer-next.md`](../multiplayer-next.md).
|
||||
Nothing in Phase 8 is implemented yet. This document records the decisions
|
||||
those tasks assume so an implementer does not have to redesign the system
|
||||
while building one part of it.
|
||||
|
||||
## The model change
|
||||
## 1. Model and non-negotiable constraints
|
||||
|
||||
The multiplayer that exists today is a **community-server** model. A
|
||||
dedicated server runs forever: it waits for `--min-players` by roster,
|
||||
counts down `--start-countdown`, loads the next arena from the rotation,
|
||||
plays a match, returns to the lobby, and repeats (`server_match_loop.gd`).
|
||||
Players reach it by direct IP, and after Phase 7 by a Steam server browser.
|
||||
The server is the durable thing and players come and go around it.
|
||||
The existing multiplayer is a community-server model: a dedicated server
|
||||
runs continuously, waits for players, rotates arenas, and can be reached by
|
||||
direct IP or the Phase 7 Steam browser. Matchmaking adds a second model:
|
||||
players queue, every selected human accepts a proposal, one server process is
|
||||
allocated for that match, and the process is destroyed after its result is
|
||||
durably recorded. Ranked selects six humans; relaxed casual selects two to six
|
||||
and discloses its bot-filled team composition before acceptance. Both models
|
||||
ship.
|
||||
|
||||
Queued matchmaking inverts that. Players are the durable thing: they enter a
|
||||
queue, a matchmaker groups them by rating and region, and a **server is
|
||||
allocated for that one match** and torn down afterwards. Both models can
|
||||
coexist — community servers via the browser, queues via the matchmaker — and
|
||||
they should, because the server browser is already most of the way to done.
|
||||
Locked constraints:
|
||||
|
||||
## Hard prerequisite: verified identity
|
||||
- One authoritative Godot process hosts exactly one match.
|
||||
- Solo 3v3 casual and ranked queues launch first. Party-shaped fields are
|
||||
reserved in contracts, but party formation is deferred.
|
||||
- Production matchmade traffic uses ticketed Steam Hosted Dedicated Server
|
||||
SDR. Direct ENet remains first-class for local development, CI, LAN,
|
||||
self-hosting, and community servers.
|
||||
- The control plane is Go, PostgreSQL, and Redis, deployed on Kubernetes.
|
||||
Agones owns game-server allocation and lifecycle.
|
||||
- Infrastructure is provider-portable. Provider-specific cluster, network,
|
||||
DNS, and secret-store configuration lives behind isolated deployment
|
||||
overlays; application code never calls a provider allocation API.
|
||||
- Launch game fleets run in Europe and North America. Placement is
|
||||
latency-first and never silently violates the ping ceiling to shorten a
|
||||
queue.
|
||||
- Phase 8 features are opt-in. With allocated mode disabled, `ServerConfig`,
|
||||
ENet, Docker Compose, and the existing community-server behavior remain
|
||||
unchanged.
|
||||
|
||||
Ranked cannot ship before Phase 7's Steam auth tickets.
|
||||
## 2. Steam and trust boundaries
|
||||
|
||||
Slot reclaim is currently keyed by **display name** (see
|
||||
`--slot-reservation-seconds`, and the known-issues list in
|
||||
`multiplayer-next.md`). A rating attached to a spoofable identity is worse
|
||||
than no rating at all: it is trivially farmed, and it invites players to
|
||||
invest in a ladder that cannot be defended. "Ranked is critical" therefore
|
||||
*raises* the priority of Steam identity rather than routing around it.
|
||||
Phase 7 verified Steam identity is a hard prerequisite. The current slot
|
||||
reservation is keyed by display name, so no public queue or rating may rely on
|
||||
it.
|
||||
|
||||
Casual queueing has a weaker requirement — it still needs stable identity for
|
||||
abandon penalties and ban enforcement, but the cost of a compromise is lower.
|
||||
Two Steam credentials have different purposes and must not be conflated:
|
||||
|
||||
## Architecture
|
||||
1. A client obtains a single-use Web API ticket for backend login. Only the
|
||||
secure backend calls `AuthenticateUserTicket`, checks the expected App ID
|
||||
and identity string, and turns the result into a revocable session. A
|
||||
client-supplied SteamID is never identity.
|
||||
2. For a formed match, the game coordinator creates a short-lived SDR relay
|
||||
ticket authorising one player to one hosted server. The client installs it
|
||||
before connecting. The server separately verifies the signed match join
|
||||
authorisation before admitting the player to the assigned roster slot.
|
||||
|
||||
Decided: **Steam for identity, a project-owned backend for everything else.**
|
||||
Steam authentication/session tickets are single-use and their lifecycle must
|
||||
include the appropriate cancel/end calls; identity is not valid until Steam's
|
||||
asynchronous validation succeeds. Hosted SDR relay tickets are different:
|
||||
they are short-lived, match/server/identity-scoped and deliberately cached for
|
||||
reconnect. See [Steam authentication](https://partner.steamgames.com/doc/features/auth)
|
||||
and the [`ISteamUserAuth` Web API](https://partner.steamgames.com/doc/webapi/isteamuserauth).
|
||||
|
||||
This reverses the "no backend" position stated in
|
||||
[`TECH_STACK.md`](TECH_STACK.md) and `README.md`, which described the state
|
||||
of the project before matchmaking was scoped. The dedicated server remains a
|
||||
Godot export; the new service is separate from it.
|
||||
Ticketed Hosted Dedicated Server SDR is the production transport because it
|
||||
hides player/server IP addresses and authenticates, encrypts, and rate-limits
|
||||
traffic. It also supplies relay routing that may improve the path. It requires
|
||||
a real App ID, coordinator SDK/signing approval, certificates, and hosted
|
||||
data-centre coordination with Valve; those are explicit release dependencies,
|
||||
not assumptions. See [Steam Datagram Relay](https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay).
|
||||
|
||||
The alternative — Steam-native matchmaking (lobbies plus Leaderboards or User
|
||||
Stats as the rating store) — was rejected on two grounds. Steam lobby
|
||||
matchmaking has no real concept of a skill distribution to match against, and
|
||||
Leaderboards are a display surface rather than a rating store with the
|
||||
transactional guarantees a ladder needs. It would also permanently bind the
|
||||
game to Steam, foreclosing other platforms.
|
||||
### Trust table
|
||||
|
||||
### Components
|
||||
|
||||
| Component | Runs where | Responsibility |
|
||||
| Input | Trusted only after | May affect |
|
||||
| --- | --- | --- |
|
||||
| Steam auth ticket validation | backend | Turn a client-supplied ticket into a verified SteamID via the Steamworks Web API. The only trusted source of identity. |
|
||||
| Queue / matchmaker | backend | Hold queued players per playlist and region; form matches on rating proximity with a widening tolerance over wait time. |
|
||||
| Rating store | backend (DB) | Per-identity, per-playlist rating and match history. Written only by the backend, never by a game client. |
|
||||
| Server allocator | backend | Start a dedicated-server instance per formed match, hand its address to the matched clients, reclaim it on exit. |
|
||||
| Dedicated server | Godot export | Unchanged simulation. Gains a mode where the roster is *assigned* rather than open, and reports a result at the end. |
|
||||
| Game client | Godot | Queue UI, estimated wait, accept/decline, connect-on-assignment, post-match rating delta. |
|
||||
| Steam Web API ticket | Backend validation for the expected App ID/identity and replay check | Backend session identity |
|
||||
| Steam ping location + active-probe evidence | Backend verifies nonce/freshness and computes estimates; later compares with observed RTT | Placement only, never results |
|
||||
| Queue/accept request | Auth, schema/rate-limit, revision and idempotency validation | That player's queue state |
|
||||
| Join authorisation | Server signature, expiry, match/server/SteamID/slot and connection-generation validation | Initial admission or idempotent reclaim of that same slot |
|
||||
| Gameplay input | Existing server framing, sequence, byte and rate validation | Authoritative simulation input only |
|
||||
| Match result | Assigned server workload identity plus match/server binding | Transactional result/rating commit |
|
||||
|
||||
### What already exists and gets reused
|
||||
Clients never submit ratings, outcome, penalty exemptions, server health, or
|
||||
allocation state. A dedicated server never holds the Steam publisher key or
|
||||
the coordinator root signing key. Keep the offline SDR CA separate from the
|
||||
online leaf ticket key. The online key is exposed only through a narrowly
|
||||
authorised signer backed by KMS/HSM-equivalent non-exportable storage; API,
|
||||
matcher, allocator and game-server pods cannot read it. The signer accepts
|
||||
only allocator-recorded assignments, audits every signature, and supports
|
||||
overlapping-key rotation.
|
||||
|
||||
The server side needs less new work than it looks:
|
||||
## 3. Control-plane architecture
|
||||
|
||||
- **`--max-matches=1`** already makes the server drain and `exit(0)` after a
|
||||
single match. That is precisely the lifecycle a per-match allocator wants;
|
||||
it was built for CI, and it generalises for free.
|
||||
- **`ServerConfig`** is a single-source-of-truth flag table with strict
|
||||
validation — new allocation flags are declared in one place and are
|
||||
automatically parsed, type-checked, config-file-backed and documented.
|
||||
- **`--min-players` / `--start-countdown` / `--slot-reservation-seconds`**
|
||||
are the match-formation primitives, and they already count *roster*
|
||||
members rather than raw peers.
|
||||
- **`MatchNet`'s roster** already survives the lobby→match transition, which
|
||||
is the structure an assigned roster slots into.
|
||||
- **`MatchState`** already has a legal-transition table with wire-stable
|
||||
integer values, so new lifecycle states append cleanly.
|
||||
Use one repository and shared domain packages, with independently runnable
|
||||
roles rather than independently designed microservices:
|
||||
|
||||
### What is genuinely new
|
||||
| Role | Responsibility |
|
||||
| --- | --- |
|
||||
| API | HTTPS/WebSocket auth, profile, queue commands, status resync |
|
||||
| Matcher | Atomic proposal formation from queue state |
|
||||
| Allocator | Agones allocation, server registration, assignment delivery |
|
||||
| Maintenance worker | Outbox delivery, season rollover, expiry, reconciliation |
|
||||
|
||||
- The backend service itself (process, deploy, DB, ops) — nothing like it
|
||||
exists in this repo today.
|
||||
- Server-authoritative **match results**: the dedicated server must report
|
||||
the outcome to the backend over a channel a client cannot forge. This is
|
||||
the first non-ENet/SDR network path in the project (see TECH_STACK's "no
|
||||
HTTP layer" note, which this supersedes).
|
||||
- An **assigned-roster** server mode: only the matched SteamIDs may take a
|
||||
slot, replacing the current first-come model.
|
||||
- Client-side queue UI and the accept/decline flow.
|
||||
API replicas are stateless. Redis sorted sets provide the fast candidate
|
||||
index, but Redis is never the durable allocation fence: asynchronous failover
|
||||
can lose an acknowledged write. A matcher claims a proposal in a PostgreSQL
|
||||
`SERIALIZABLE` transaction using a partial unique constraint that permits only
|
||||
one active proposal/match participation per player. The transaction records
|
||||
the proposal and participants before Redis cleanup; a stale Redis claim then
|
||||
loses at PostgreSQL and is repaired from the durable record. PostgreSQL is the
|
||||
source of truth for queue ownership, identities, sessions/revocations,
|
||||
seasons, ratings, matches, participants, penalties, result receipts, audits
|
||||
and the transactional outbox. Redis holds expiring presence, candidate
|
||||
indexes, latency evidence, session/revocation caches and transient fan-out.
|
||||
Losing the last acknowledged Redis write may delay/rematerialise a ticket or
|
||||
force a session cache miss, but can neither resurrect a revoked session, split
|
||||
a proposal nor corrupt a result/rating.
|
||||
|
||||
## Server orchestration and autoscaling
|
||||
For launch, run the horizontally scaled control plane in one primary
|
||||
Kubernetes region with a warm standby and tested restore path. Game fleets
|
||||
remain regional in EU and North America. This avoids a premature multi-writer
|
||||
database while keeping new-match control latency small relative to queue time.
|
||||
An outage may pause new queues/allocations, but live matches must continue.
|
||||
Targets are PostgreSQL RPO <= 5 minutes and control-plane RTO <= 30 minutes.
|
||||
|
||||
Requirement: game servers scale horizontally and automatically, spin up fast
|
||||
on demand, serve exactly one match, and shut down — so cost is incurred only
|
||||
while a match is being played. Docker and the existing CI gates must keep
|
||||
working unchanged.
|
||||
### Stable identifiers and state
|
||||
|
||||
### Why this is achievable: the server is already shaped for it
|
||||
Contracts define opaque `player_id`, `queue_ticket_id`, `proposal_id`,
|
||||
`match_id`, `server_id`, and `season_id`. Every mutating request has an
|
||||
idempotency key, expected revision, and versioned schema.
|
||||
|
||||
Two properties of the current build make per-match allocation practical
|
||||
rather than aspirational:
|
||||
The durable/transient state path is:
|
||||
|
||||
- **The container is small.** The `server` image target is a slim
|
||||
`ubuntu:24.04` runtime with three shared libraries and the exported
|
||||
binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image.
|
||||
Pulling it onto a fresh node is cheap.
|
||||
- **Boot to listening is sub-second.** Measured on this repo's
|
||||
`cosmicclash-server:latest`: **~870 ms** from container start to the
|
||||
`server_started` log line, averaged over three runs, read from the
|
||||
container's own clock. That measurement was taken under **x86_64 emulation
|
||||
on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux
|
||||
should be faster. Re-measure on the real target before setting timeouts.
|
||||
```
|
||||
QUEUED -> PROPOSED -> ACCEPTED -> ALLOCATING -> PROCESS_READY
|
||||
-> ASSIGNMENT_READY -> ASSIGNED -> CONNECTING -> LIVE
|
||||
-> RESULT_PENDING -> COMPLETED
|
||||
-> CANCELLED / EXPIRED / FAILED from the explicitly legal stages
|
||||
```
|
||||
|
||||
Combined with `--max-matches=1`, which already drains and `exit(0)`s after a
|
||||
single match, the lifecycle the allocator needs mostly exists: start
|
||||
container → serve one match → process exits → orchestrator reclaims.
|
||||
The API publishes revisioned changes over one authenticated WebSocket. REST
|
||||
GET endpoints are the recovery source after a disconnect or missed revision.
|
||||
Restarting the client resumes an unexpired ticket/assignment instead of
|
||||
creating another one.
|
||||
|
||||
### The cold-start tension, stated honestly
|
||||
Assignments include match/server IDs, protocol and client build, server image
|
||||
digest, playlist version, transport, region, expiry, a match-scoped join
|
||||
authorisation, and either the SDR hosted-server material or an ENet endpoint.
|
||||
The authorisation may be replayed only by the same Steam identity to reclaim
|
||||
the same match/server/slot before expiry. Each successful connection advances
|
||||
a server-owned generation and fences the prior connection; another identity,
|
||||
server or slot is always rejected. This deliberately supports reconnect when
|
||||
Steam or the control plane is temporarily unavailable. Incompatible
|
||||
protocol/build/playlist versions never enter one proposal.
|
||||
|
||||
"Only pay during a match" and "a player never waits" are in tension. A server
|
||||
must be listening *before* the matched players connect, so some cost always
|
||||
precedes the match. Sub-second boot makes the gap small enough that a pure
|
||||
scale-to-zero design is plausible — but the risk is not the container, it is
|
||||
everything around it: image pull on a cold node, scheduler placement, and
|
||||
network/port programming can each dwarf 870 ms.
|
||||
## 4. Queue and placement policy
|
||||
|
||||
Recommendation: **scale to zero at the node level is the wrong target; scale
|
||||
to zero at the match level is the right one.** Keep a small warm pool of
|
||||
nodes sized to the current queue depth, and start a per-match container on
|
||||
demand within it. The per-match process genuinely exists only for the match;
|
||||
the node pool absorbs the cold-start variance. Revisit only if measured
|
||||
allocation latency on real infrastructure shows the warm pool is unnecessary.
|
||||
Each verified player may own at most one active queue ticket. Heartbeats are
|
||||
sent every 10 seconds and queue presence expires after 30 seconds. Create,
|
||||
cancel, resume, accept, decline, and expiry are atomic and retry-safe.
|
||||
|
||||
### Findings that block a naive implementation
|
||||
The client submits its recent opaque Steam ping location plus nonce-bound
|
||||
active-probe responses from each regional endpoint; it does not submit the RTT
|
||||
used for placement. The backend validates a 30-second freshness window and
|
||||
nonce, then uses the Steam coordinator SDK and probe timings to compute the
|
||||
regional matrix. A predicted/observed discrepancy over 25 ms or 30% (whichever
|
||||
is larger) in three matches within 24 hours quarantines the account's samples:
|
||||
it may queue only in regions whose active probe independently remains under
|
||||
the ceiling until five clean matches clear the quarantine. The matchmaker:
|
||||
|
||||
**Readiness cannot be detected from the log line.** Godot's stdout is
|
||||
block-buffered when it is not attached to a TTY. Run the server image
|
||||
detached without `-t` and `docker logs` shows **nothing at all** — the
|
||||
`server_started` line does not appear even after 35 seconds, because the
|
||||
buffer never flushes. An orchestrator readiness probe that greps for that
|
||||
line will hang forever, and this was reproduced directly while measuring the
|
||||
boot time above. Either probe the UDP socket instead, or make the server
|
||||
flush explicitly. This also means container logs are not a reliable
|
||||
observability channel for a short-lived match server; treat log shipping as
|
||||
a separate problem.
|
||||
1. Finds regions in which every proposed player has predicted RTT <= 100 ms.
|
||||
2. Minimises the worst player's predicted RTT.
|
||||
3. Breaks ties by total predicted RTT, then ready server capacity.
|
||||
4. Widens rating tolerance with wait time, but never automatically widens the
|
||||
100 ms latency ceiling.
|
||||
|
||||
**One fixed port per container does not scale on a shared host.** `--port`
|
||||
defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing
|
||||
several matches onto one node needs either a port range allocated per
|
||||
container, or one address per container. This is a UDP service, so the usual
|
||||
HTTP ingress/L7 routing answers do not apply — the allocator must hand the
|
||||
client a concrete `host:port`.
|
||||
The target is regional observed p95 RTT <= 80 ms. Candidate formation is
|
||||
deterministic:
|
||||
|
||||
**The match cannot start on a schedule the players do not control.** Today
|
||||
the loop waits for `--min-players` then counts down. An allocated server is
|
||||
told *which* identities to expect, and needs a **no-show timeout**: if a
|
||||
matched player never connects, the server must abandon and exit rather than
|
||||
sit idle burning the cost this design is trying to avoid.
|
||||
- Anchor on the oldest compatible ticket (`enqueued_at`, then ticket ID).
|
||||
- A ticket's rating tolerance is `min(400, 100 + 25 * floor(wait_seconds/30))`
|
||||
rating points. A pair is compatible only when its absolute rating difference
|
||||
is within both tickets' tolerance. Provisional players use the same stored
|
||||
rating; their high RD changes ratings faster, not eligibility arithmetic.
|
||||
- Every candidate set contains the anchor. Choose the lexicographically
|
||||
smallest tuple `(worst_RTT, total_RTT, rating_range, -sum(wait_seconds),
|
||||
sorted_ticket_IDs)`, so latency dominates once the oldest player anchors
|
||||
fairness and every candidate satisfies the widening rule.
|
||||
- Partition humans exhaustively into legal teams and minimise absolute team
|
||||
mean-rating difference, then maximum opposing-player difference, then use
|
||||
lexical player IDs. Bots fill remaining casual slots after humans are
|
||||
assigned.
|
||||
|
||||
### Keeping Docker and CI green
|
||||
Every selected human receives a 10-second proposal before allocation. Ranked
|
||||
always selects six. Casual requires six before the anchor reaches 60 seconds;
|
||||
afterward it selects the largest compatible human count from six down to two,
|
||||
with at least one human per team, and displays teams/bot slots. Unanimous
|
||||
acceptance by the selected humans advances.
|
||||
|
||||
The existing gates must not regress. `make verify-phase6` builds the export,
|
||||
runs it in Compose, joins two headless clients and asserts both saw both
|
||||
goals and that the arena rotated between matches; `make verify-enet-integration`
|
||||
runs the source-build ENet matrix. Both depend on current behaviour:
|
||||
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
|
||||
assignment, and uses `--max-matches=2` to prove rotation.
|
||||
An explicit proposal decline cancels that player's ticket and applies a
|
||||
30-second casual or 2-minute ranked cooldown. A timeout applies 60 seconds in
|
||||
casual or 5 minutes in ranked. Three ranked proposal declines/timeouts within
|
||||
30 minutes apply 15 minutes. Players who accepted return with their original
|
||||
`enqueued_at` and precedence when another player declines, times out, or the
|
||||
allocation fails. Ordering ties use ticket ID. A ranked initial-connect
|
||||
no-show after accepting uses the ranked abandon cooldown ladder but never a
|
||||
rating loss because no rated match began.
|
||||
|
||||
The rule that keeps them passing: **every allocation feature is opt-in via a
|
||||
new `ServerConfig` flag whose default reproduces today's behaviour.** An
|
||||
assigned roster, a no-show timeout and result reporting must each be inert
|
||||
unless explicitly enabled. `ServerConfig` is built for exactly this — a flag
|
||||
declared once is parsed, validated, type-checked, config-file-backed and
|
||||
documented — and `tests/cases/` can cover the new parsing without a live
|
||||
server. A second Compose file should cover the allocated-match path rather
|
||||
than mutating the Phase 6 one, so the community-server model stays tested
|
||||
alongside the matchmade one.
|
||||
### Casual
|
||||
|
||||
### Open questions
|
||||
- Target six humans in 3v3.
|
||||
- After 60 seconds, a match may start with at least two humans, one on each
|
||||
team, and fill the other slots with server bots.
|
||||
- Human backfill may replace a bot/disconnected slot only at a kickoff
|
||||
boundary.
|
||||
- Backfill is a separate 10-second opt-in proposal showing score, time
|
||||
remaining, team and slot. Declining/timing out carries no cooldown. A
|
||||
backfilled player receives no hidden-rating update or abandon penalty for
|
||||
that match; acceptance removes their queue ticket only when assignment is
|
||||
ready. Choose the oldest ordinary casual ticket that meets the same build,
|
||||
region <=100 ms and current anchor-tolerance rules for the vacated human
|
||||
slot; ties use ticket ID.
|
||||
- An original casual participant gets 30 seconds to reconnect; leaving after
|
||||
that applies a 60-second queue cooldown. The match's ordinary hidden-rating
|
||||
result still applies, with no extra rating penalty.
|
||||
- An accepted casual initial-connect no-show gets the same 60-second cooldown.
|
||||
The match proceeds with a bot only if at least one human connected on each
|
||||
team; otherwise it cancels and restores every innocent ticket with original
|
||||
precedence.
|
||||
- Casual has a separate hidden Glicko-2 rating used only for matching.
|
||||
|
||||
- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this
|
||||
game-server lifecycle), Nomad, or direct cloud-API container starts. Not
|
||||
chosen. Agones is the strongest default because it models allocation,
|
||||
readiness and per-match lifetime natively.
|
||||
- **Port strategy** — port range per node versus one IP per match.
|
||||
- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 6–10 match
|
||||
processes per modern core and 150–250 MB RSS each. That estimate predates
|
||||
any allocation work and should be re-measured under real concurrency
|
||||
before it sizes a bill.
|
||||
- **Draining and deploys.** How a server version rolls out without killing
|
||||
matches in flight.
|
||||
### Ranked
|
||||
|
||||
## Casual vs ranked
|
||||
- Exactly six verified humans; never bots and never backfill.
|
||||
- Solo queue only at launch.
|
||||
- Only `ArenaRegistry` entries with `"random": true` are eligible. Elevated
|
||||
goals remain excluded until the trained-policy restriction is lifted.
|
||||
- A reconnecting player has 60 seconds to return using the existing
|
||||
assignment. After that, that player receives a loss regardless of the final
|
||||
team result and a rolling seven-day cooldown: 5 minutes, 15 minutes, 1
|
||||
hour, then 24 hours.
|
||||
- Delivery failure is not match-integrity failure. A healthy completed match
|
||||
remains rated while its result waits for the control plane. Rating is
|
||||
suppressed only when the authoritative roster, simulation or result is
|
||||
unavailable/corrupt, or a measured regional incident prevented fair play.
|
||||
|
||||
They are different playlists, not a difficulty toggle, and their rules
|
||||
diverge in ways that affect the server:
|
||||
## 5. Rating and seasons
|
||||
|
||||
| | Casual | Ranked |
|
||||
| --- | --- | --- |
|
||||
| Rating | Hidden, used only for matching | Visible, with tiers |
|
||||
| Backfill on disconnect | Yes — keep the match playable | No — the match is rating-bearing and must not change shape mid-way |
|
||||
| Bots filling empty slots | Acceptable (`--fill-bots` exists) | Never |
|
||||
| Abandon penalty | Light (short queue cooldown) | Real (rating loss, escalating cooldown) |
|
||||
| Arena selection | Full rotation | Restricted set, so a variant nobody has practised can't decide a ladder match |
|
||||
| Party / premade | Unrestricted | Constrained by size and rating spread |
|
||||
Use canonical Glicko-2 independently per playlist with initial rating 1500,
|
||||
RD 350, volatility 0.06, scale constant 173.7178 and tau 0.5. Updates are
|
||||
immediate per committed match rather than globally batched. Before an update,
|
||||
advance inactivity by whole 24-hour rating periods since the player's last
|
||||
rated match using `phi = min(350/173.7178, sqrt(phi^2 + sigma^2 * periods))`.
|
||||
|
||||
Note the arena constraint interacts with an existing rule: elevated-goal
|
||||
variants are Free-Play-only until a checkpoint trained on
|
||||
`training_elevated.tscn` is promoted (`arena_registry.gd`). Ranked's arena
|
||||
set should be drawn from `"random": true` arenas only.
|
||||
For each player `i`, transform every opposing human's pre-match rating/RD to
|
||||
`mu_j`/`phi_j` and use the canonical equations
|
||||
`g(phi)=1/sqrt(1+3*phi^2/pi^2)` and
|
||||
`E=1/(1+exp(-g(phi_j)*(mu_i-mu_j)))`. Ranked's three opponent contributions
|
||||
use `w=1/3`; casual uses `w=1/N` for the `N` opposing humans, ignoring bots.
|
||||
Thus human contributions total exactly one match in both sums:
|
||||
|
||||
## Open questions
|
||||
```
|
||||
v^-1 = sum(w * g(phi_j)^2 * E_j * (1-E_j))
|
||||
Delta = v * sum(w * g(phi_j) * (s-E_j))
|
||||
```
|
||||
|
||||
- **Rating algorithm.** Glicko-2 is the default recommendation over plain
|
||||
Elo: it models rating *uncertainty*, which matters enormously for a small
|
||||
launch population where most players have few games. Not yet decided.
|
||||
- **Team rating from individual ratings.** How a 3v3 match's outcome
|
||||
distributes across six players is a separate design problem from the
|
||||
rating system itself.
|
||||
- **Server cost.** Allocated servers cost real money per match, unlike
|
||||
community servers that players host themselves. `README.md`'s original
|
||||
note about a subscription to fund servers is suddenly load-bearing again.
|
||||
Population size and match length set the bill; this needs a number before
|
||||
launch, not after.
|
||||
- **Region / ping policy.** How much rating tolerance to trade for latency,
|
||||
and whether cross-region is ever allowed at low population.
|
||||
- **Placement matches** and whether ranked has a soft reset per season.
|
||||
- **Backend language and hosting.** Not chosen. It does *not* have to be C#
|
||||
despite the original README framing — that framing was aspirational and
|
||||
predates every real decision in this project.
|
||||
Then run the canonical Glicko-2 volatility iteration and rating/RD update. If
|
||||
a player has no opposing human, the match is unrated for that player. `s` is 1/0 for
|
||||
the authoritative winner/loser and 0.5 only for a completed draw. Overtime is
|
||||
an ordinary win/loss. A ranked abandoner gets `s=0` regardless of the final
|
||||
team result; non-abandoning players use the authoritative final result.
|
||||
Cancelled or integrity-failed matches do not update anyone.
|
||||
|
||||
## Explicitly out of scope
|
||||
Lock all six participant rows in lexical player-ID order and compute every
|
||||
new value from the same immutable pre-match snapshot inside one serializable
|
||||
transaction. This prevents update-order bias and concurrent double updates.
|
||||
Golden vectors include canonical one-player examples plus symmetric/asymmetric
|
||||
3v3, draw, overtime, abandon, inactivity and concurrent-result fixtures.
|
||||
|
||||
Tournaments, in-game leaderboards beyond a personal rank display,
|
||||
cross-platform play with non-Steam identity providers, and spectator/observer
|
||||
tooling for ranked matches. None are precluded by this design; none are
|
||||
launch scope.
|
||||
The first ten ranked matches are provisional. Ranked exposes
|
||||
backend-derived tiers; casual rating remains hidden. Stored Glicko values, not
|
||||
tier labels or client calculations, are authoritative. Ranked seasons last 12
|
||||
weeks. Ranked rollover sets
|
||||
`rating = 1500 + 0.75 * (rating - 1500)`, raises RD to at least 200 (capped at
|
||||
350), preserves volatility/history, and is an exactly-once idempotent
|
||||
transaction. Casual rating is continuous and never season-reset.
|
||||
|
||||
Rating, penalty, match completion, participant records and outbox events are
|
||||
committed in that transaction. Duplicate identical server results succeed
|
||||
idempotently. A conflicting result changes nothing and pages an operator.
|
||||
|
||||
## 6. Game-server allocation and lifecycle
|
||||
|
||||
Use provider-portable Kubernetes manifests and Agones. Create one versioned
|
||||
Fleet per compatible build and region; labels identify region, protocol,
|
||||
transport, and image/build. Allocate atomically with `GameServerAllocation`.
|
||||
Agones, rather than bespoke allocator code, owns selection and lifecycle. See
|
||||
[GameServerAllocation](https://agones.dev/site/docs/reference/gameserverallocation/).
|
||||
|
||||
The Godot process talks to the Agones REST sidecar through a small adapter
|
||||
that is a no-op when `AGONES_SDK_HTTP_PORT` is absent. This preserves native,
|
||||
Compose, and CI operation and works with the Agones local SDK emulator. See
|
||||
[Agones client SDKs](https://agones.dev/site/docs/guides/client-sdks/).
|
||||
|
||||
Agones has two distinct readiness points; conflating them is a deadlock because
|
||||
`GameServerAllocation` selects a Ready server and only then attaches the match
|
||||
metadata:
|
||||
|
||||
1. The PID-1 supervisor queries the local Agones SDK for the assigned dynamic
|
||||
port/address, exports `SDR_LISTEN_PORT` plus `SDR_IP=public-address:port`
|
||||
(or the ENet equivalent), and launches Godot. Godot validates static config,
|
||||
binds the socket and starts Health calls.
|
||||
2. Godot calls Agones `Ready()` after the process is genuinely listening.
|
||||
This is **process-ready** only; never infer it from detached stdout.
|
||||
3. `GameServerAllocation` atomically changes that Ready server to Allocated
|
||||
and supplies the signed roster/non-secret match configuration as metadata.
|
||||
4. The server watches the GameServer, observes Allocated metadata, verifies
|
||||
manifest signature/build/protocol/server binding, registers its hosted SDR
|
||||
address, and calls the backend `assignment_ready` endpoint.
|
||||
5. Only after `assignment_ready` does the allocator mint relay/join tickets and
|
||||
expose the assignment to clients. The server accepts only assigned
|
||||
identities/slots; each selected human has 30 seconds to connect.
|
||||
6. Run one authoritative match with Health calls independent of the simulation
|
||||
loop. Submit the canonical result using the bound workload identity.
|
||||
7. Write the signed result hash/payload to the backend and to a non-secret
|
||||
Agones annotation, remain Allocated, and retry until the backend durably
|
||||
acknowledges it. The maintenance worker reconciles the annotation after an
|
||||
API outage. `RESULT_PENDING` pages at 5 minutes and requires operator review
|
||||
at 30 minutes; it never silently becomes unrated.
|
||||
8. After acknowledgement call `Shutdown()` and exit. Invalid/empty allocations
|
||||
shut down immediately. An allocated GameServer is not recycled to Ready.
|
||||
|
||||
Agones supplies dynamic host ports so multiple isolated matches can share a
|
||||
node; HTTP ingress is not involved in gameplay routing. Hosted SDR additionally
|
||||
requires Valve approval for every provider/location, a valid `SDR_POPID`,
|
||||
public-IP/unsolicited-UDP reachability, provider firewall/NAT validation,
|
||||
per-location certificates and coordinator trust. Use an Agones dynamic or
|
||||
passthrough mapping whose externally reported port is the `SDR_IP` port while
|
||||
the process binds `SDR_LISTEN_PORT`; test SDR and ENet mappings separately.
|
||||
Credentials arrive through runtime secret mounts, never allocation metadata,
|
||||
arguments, logs or images.
|
||||
|
||||
Result authentication uses a projected, pod-bound service-account token with
|
||||
a dedicated audience and one service account per workload class. The backend
|
||||
validates the configured cluster issuer/JWKS, audience, expiry, namespace,
|
||||
service account, bound pod UID and allocator-recorded GameServer UID, then
|
||||
checks that GameServer/match binding in PostgreSQL. Issuers and trust roots are
|
||||
allowlisted and rotated explicitly for every cluster/provider. A one-match
|
||||
server credential issued after this attestation is an acceptable equivalent.
|
||||
|
||||
### Warm capacity and density
|
||||
|
||||
Both launch regions are active whenever their queues are enabled. Each active
|
||||
region maintains at least two Ready processes distributed across at least two
|
||||
on-demand nodes/failure domains; the minimum node floor is therefore two, not
|
||||
one. Pre-pull current and rollback images. A queue/proposal-aware
|
||||
FleetAutoscaler adjusts capacity above that Ready floor. **Allocated** process
|
||||
count may fall to zero; Ready processes do not. An administratively disabled
|
||||
region may scale both nodes and Fleet to zero and is excluded from placement.
|
||||
See [Agones FleetAutoscaler](https://agones.dev/site/docs/reference/fleetautoscaler/).
|
||||
|
||||
Do not run live matches on interruptible nodes. N+1 means loss of the largest
|
||||
single node still leaves two Ready slots plus sufficient headroom for already
|
||||
Allocated matches; certify it in the node-loss test. Set pod requests, limits
|
||||
and node caps only after native x86_64 benchmarks of boot time, p99
|
||||
CPU/RSS/network and 60 Hz tick behavior, retaining 30% headroom. The current
|
||||
6–10 processes/core and 150–250 MB estimates are not sizing data.
|
||||
|
||||
Godot/GDScript cannot intercept SIGTERM, so Phase 8 adds a small Go PID-1
|
||||
supervisor. It starts Godot, traps TERM, and sends a per-pod-token-authenticated
|
||||
localhost drain request to an allocated-mode-only Godot control socket. Godot
|
||||
refuses new admissions and reports completion to the supervisor. Kubernetes
|
||||
uses a 300-second termination grace period; at 285 seconds the supervisor
|
||||
forces an infrastructure-failure abort so the pod cannot hang forever in
|
||||
overtime. A PodDisruptionBudget and Agones-aware drain prevent voluntary
|
||||
eviction of Allocated servers. Planned releases create a new Fleet, route new
|
||||
allocations to it and wait for old Allocated count zero without sending TERM.
|
||||
Unexpected node failure cannot be made graceful and follows the integrity
|
||||
failure/refund path.
|
||||
|
||||
Live matches continue through an API/control-plane delivery outage. A valid
|
||||
result is retried/reconciled as above; only loss of authoritative match
|
||||
integrity suppresses rating.
|
||||
|
||||
## 7. Security and operational baseline
|
||||
|
||||
The threat model must cover forged clients, ticket replay, duplicate queueing,
|
||||
result forgery, queue manipulation, packet floods, botting, server compromise,
|
||||
insider access, DDoS, vulnerable dependencies, and denial-of-wallet attacks.
|
||||
|
||||
Minimum controls:
|
||||
|
||||
- Non-root containers, read-only root filesystems, dropped Linux
|
||||
capabilities, RuntimeDefault seccomp, resource limits, restricted Pod
|
||||
Security admission, and least-privilege service accounts/RBAC.
|
||||
- Private PostgreSQL/Redis; default-deny ingress and egress with explicit
|
||||
NetworkPolicy allowances. See the [Kubernetes application security
|
||||
checklist](https://kubernetes.io/docs/concepts/security/application-security-checklist/)
|
||||
and [NetworkPolicy](https://kubernetes.io/docs/reference/kubernetes-api/networking/network-policy-v1/).
|
||||
- Encrypted databases/backups, externally populated Kubernetes Secrets,
|
||||
documented rotation, and no credentials in Git, images, command lines, or
|
||||
telemetry.
|
||||
- Per-account/IP API limits, request/body/schema limits, replay and duplicate
|
||||
detection, generic public errors, allocation quotas, and budget alerts.
|
||||
- Put HTTPS/WebSocket traffic behind a provider-portable edge contract
|
||||
implemented by each infrastructure overlay: managed volumetric DDoS
|
||||
absorption, WAF/rate rules, TLS termination, origin-only ingress and health
|
||||
checks. Enforce authenticated queue admission, per-account connection caps,
|
||||
WebSocket handshake/message/idle limits, bounded fan-out and overload
|
||||
shedding. In degraded mode reject new login/queue/allocation work while
|
||||
preserving result ingestion and all live matches.
|
||||
- Images pinned by digest, SBOM generation, dependency/image scanning, signed
|
||||
releases, admission-time signature verification, and a critical-patch SLA.
|
||||
- Structured audit events for auth, queue transitions, allocation, roster
|
||||
rejection, result conflict, penalty, season rollover, and operator action.
|
||||
|
||||
## 8. SLOs, observability, and release gates
|
||||
|
||||
Launch SLOs:
|
||||
|
||||
| Measure | Target |
|
||||
| --- | --- |
|
||||
| Eligible predicted RTT | <= 100 ms for every player |
|
||||
| Regional observed RTT | p95 <= 80 ms |
|
||||
| Unanimous acceptance to `assignment_ready` | p95 <= 5 s, p99 <= 10 s with warm capacity |
|
||||
| Published assignment to successful connection | p95 <= 5 s |
|
||||
| Successful allocation and durable result | >= 99.9% |
|
||||
| Certified server density | No tick backlog, with 30% resource headroom |
|
||||
| Control-plane API under load | p95 <= 250 ms |
|
||||
|
||||
Dashboards and alerts cover queue depth/wait, rating spread, predicted versus
|
||||
observed RTT, proposals/declines, allocation latency/failure, Ready capacity,
|
||||
image pulls, connection/no-show, physics overruns, crashes, abnormal packet
|
||||
rates, result lag/conflicts, abandons, and cost per completed match. Correlate
|
||||
all components with queue/proposal/match/server IDs, but never log auth or
|
||||
relay tickets.
|
||||
|
||||
Testing layers:
|
||||
|
||||
- Go unit, race, fuzz, property, migration, and concurrency tests.
|
||||
- Fake Steam verifier and fake allocator for deterministic CI.
|
||||
- A second allocated-server Compose flow; never mutate
|
||||
`compose.phase6-smoke.yml`.
|
||||
- Disposable `kind` + Agones integration for readiness, dynamic ports,
|
||||
health/no-show shutdown, allocation races, multiple matches per node,
|
||||
draining, and rollback.
|
||||
- Network/chaos cases for 100 ms RTT, jitter/loss, client/backend/matcher
|
||||
restart, game-pod death, node drain, Redis failover, and control-plane loss.
|
||||
- Load test at least 10,000 queued clients, 100 proposals/second, and forecast
|
||||
launch concurrency x2 while preserving correctness and API latency.
|
||||
- Provider migration rehearsal: restore data and deploy both regional fleets
|
||||
on a second provider whose EU/NA locations already have Valve approval,
|
||||
POP/certificates, public UDP/firewall verification and coordinator trust;
|
||||
switch new allocations, drain the old provider, and terminate no live match.
|
||||
|
||||
Release order is development -> internal -> casual canary -> full casual ->
|
||||
provisional ranked -> full ranked. Each promotion requires its SLO/security
|
||||
gates, rollback rehearsal, EU and North America playtests, a measured cost
|
||||
model, and unchanged `make verify-phase6` and
|
||||
`make verify-enet-integration`.
|
||||
|
||||
## 9. Explicitly out of scope
|
||||
|
||||
Parties/premades, tournaments, ranked spectators, public global regions,
|
||||
non-Steam identity providers, and a global leaderboard beyond personal rank
|
||||
display are not launch scope. The contracts reserve party identity and avoid
|
||||
Steam-specific database primary keys so those additions do not require a
|
||||
destructive redesign.
|
||||
|
||||
Reference in New Issue
Block a user