docs: finalize scalable matchmaking plan

This commit is contained in:
Josh Creek
2026-08-31 19:57:49 +01:00
parent bcc12aad19
commit 8d4a0640e2
5 changed files with 647 additions and 343 deletions
+423 -196
View File
@@ -1,243 +1,470 @@
# Matchmaking — casual and ranked queues
Design scope for online casual and ranked play. This is a **1.0 launch
blocker**, not a post-launch addition.
Architecture and locked product policy for Phase 8. This is a **1.0 launch
blocker**. The numbered, independently implementable tasks and their
acceptance criteria live in [`multiplayer-todo.md`](../multiplayer-todo.md);
the short live checklist is [`multiplayer-next.md`](../multiplayer-next.md).
Nothing described here is implemented yet. This doc exists to record the
decisions and the reasoning before code is written; per-task implementation
evidence belongs in `multiplayer-todo.md` once work starts, and the live
checklist lives in [`multiplayer-next.md`](../multiplayer-next.md).
Nothing in Phase 8 is implemented yet. This document records the decisions
those tasks assume so an implementer does not have to redesign the system
while building one part of it.
## The model change
## 1. Model and non-negotiable constraints
The multiplayer that exists today is a **community-server** model. A
dedicated server runs forever: it waits for `--min-players` by roster,
counts down `--start-countdown`, loads the next arena from the rotation,
plays a match, returns to the lobby, and repeats (`server_match_loop.gd`).
Players reach it by direct IP, and after Phase 7 by a Steam server browser.
The server is the durable thing and players come and go around it.
The existing multiplayer is a community-server model: a dedicated server
runs continuously, waits for players, rotates arenas, and can be reached by
direct IP or the Phase 7 Steam browser. Matchmaking adds a second model:
players queue, every selected human accepts a proposal, one server process is
allocated for that match, and the process is destroyed after its result is
durably recorded. Ranked selects six humans; relaxed casual selects two to six
and discloses its bot-filled team composition before acceptance. Both models
ship.
Queued matchmaking inverts that. Players are the durable thing: they enter a
queue, a matchmaker groups them by rating and region, and a **server is
allocated for that one match** and torn down afterwards. Both models can
coexist — community servers via the browser, queues via the matchmaker — and
they should, because the server browser is already most of the way to done.
Locked constraints:
## Hard prerequisite: verified identity
- One authoritative Godot process hosts exactly one match.
- Solo 3v3 casual and ranked queues launch first. Party-shaped fields are
reserved in contracts, but party formation is deferred.
- Production matchmade traffic uses ticketed Steam Hosted Dedicated Server
SDR. Direct ENet remains first-class for local development, CI, LAN,
self-hosting, and community servers.
- The control plane is Go, PostgreSQL, and Redis, deployed on Kubernetes.
Agones owns game-server allocation and lifecycle.
- Infrastructure is provider-portable. Provider-specific cluster, network,
DNS, and secret-store configuration lives behind isolated deployment
overlays; application code never calls a provider allocation API.
- Launch game fleets run in Europe and North America. Placement is
latency-first and never silently violates the ping ceiling to shorten a
queue.
- Phase 8 features are opt-in. With allocated mode disabled, `ServerConfig`,
ENet, Docker Compose, and the existing community-server behavior remain
unchanged.
Ranked cannot ship before Phase 7's Steam auth tickets.
## 2. Steam and trust boundaries
Slot reclaim is currently keyed by **display name** (see
`--slot-reservation-seconds`, and the known-issues list in
`multiplayer-next.md`). A rating attached to a spoofable identity is worse
than no rating at all: it is trivially farmed, and it invites players to
invest in a ladder that cannot be defended. "Ranked is critical" therefore
*raises* the priority of Steam identity rather than routing around it.
Phase 7 verified Steam identity is a hard prerequisite. The current slot
reservation is keyed by display name, so no public queue or rating may rely on
it.
Casual queueing has a weaker requirement — it still needs stable identity for
abandon penalties and ban enforcement, but the cost of a compromise is lower.
Two Steam credentials have different purposes and must not be conflated:
## Architecture
1. A client obtains a single-use Web API ticket for backend login. Only the
secure backend calls `AuthenticateUserTicket`, checks the expected App ID
and identity string, and turns the result into a revocable session. A
client-supplied SteamID is never identity.
2. For a formed match, the game coordinator creates a short-lived SDR relay
ticket authorising one player to one hosted server. The client installs it
before connecting. The server separately verifies the signed match join
authorisation before admitting the player to the assigned roster slot.
Decided: **Steam for identity, a project-owned backend for everything else.**
Steam authentication/session tickets are single-use and their lifecycle must
include the appropriate cancel/end calls; identity is not valid until Steam's
asynchronous validation succeeds. Hosted SDR relay tickets are different:
they are short-lived, match/server/identity-scoped and deliberately cached for
reconnect. See [Steam authentication](https://partner.steamgames.com/doc/features/auth)
and the [`ISteamUserAuth` Web API](https://partner.steamgames.com/doc/webapi/isteamuserauth).
This reverses the "no backend" position stated in
[`TECH_STACK.md`](TECH_STACK.md) and `README.md`, which described the state
of the project before matchmaking was scoped. The dedicated server remains a
Godot export; the new service is separate from it.
Ticketed Hosted Dedicated Server SDR is the production transport because it
hides player/server IP addresses and authenticates, encrypts, and rate-limits
traffic. It also supplies relay routing that may improve the path. It requires
a real App ID, coordinator SDK/signing approval, certificates, and hosted
data-centre coordination with Valve; those are explicit release dependencies,
not assumptions. See [Steam Datagram Relay](https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay).
The alternative — Steam-native matchmaking (lobbies plus Leaderboards or User
Stats as the rating store) — was rejected on two grounds. Steam lobby
matchmaking has no real concept of a skill distribution to match against, and
Leaderboards are a display surface rather than a rating store with the
transactional guarantees a ladder needs. It would also permanently bind the
game to Steam, foreclosing other platforms.
### Trust table
### Components
| Component | Runs where | Responsibility |
| Input | Trusted only after | May affect |
| --- | --- | --- |
| Steam auth ticket validation | backend | Turn a client-supplied ticket into a verified SteamID via the Steamworks Web API. The only trusted source of identity. |
| Queue / matchmaker | backend | Hold queued players per playlist and region; form matches on rating proximity with a widening tolerance over wait time. |
| Rating store | backend (DB) | Per-identity, per-playlist rating and match history. Written only by the backend, never by a game client. |
| Server allocator | backend | Start a dedicated-server instance per formed match, hand its address to the matched clients, reclaim it on exit. |
| Dedicated server | Godot export | Unchanged simulation. Gains a mode where the roster is *assigned* rather than open, and reports a result at the end. |
| Game client | Godot | Queue UI, estimated wait, accept/decline, connect-on-assignment, post-match rating delta. |
| Steam Web API ticket | Backend validation for the expected App ID/identity and replay check | Backend session identity |
| Steam ping location + active-probe evidence | Backend verifies nonce/freshness and computes estimates; later compares with observed RTT | Placement only, never results |
| Queue/accept request | Auth, schema/rate-limit, revision and idempotency validation | That player's queue state |
| Join authorisation | Server signature, expiry, match/server/SteamID/slot and connection-generation validation | Initial admission or idempotent reclaim of that same slot |
| Gameplay input | Existing server framing, sequence, byte and rate validation | Authoritative simulation input only |
| Match result | Assigned server workload identity plus match/server binding | Transactional result/rating commit |
### What already exists and gets reused
Clients never submit ratings, outcome, penalty exemptions, server health, or
allocation state. A dedicated server never holds the Steam publisher key or
the coordinator root signing key. Keep the offline SDR CA separate from the
online leaf ticket key. The online key is exposed only through a narrowly
authorised signer backed by KMS/HSM-equivalent non-exportable storage; API,
matcher, allocator and game-server pods cannot read it. The signer accepts
only allocator-recorded assignments, audits every signature, and supports
overlapping-key rotation.
The server side needs less new work than it looks:
## 3. Control-plane architecture
- **`--max-matches=1`** already makes the server drain and `exit(0)` after a
single match. That is precisely the lifecycle a per-match allocator wants;
it was built for CI, and it generalises for free.
- **`ServerConfig`** is a single-source-of-truth flag table with strict
validation — new allocation flags are declared in one place and are
automatically parsed, type-checked, config-file-backed and documented.
- **`--min-players` / `--start-countdown` / `--slot-reservation-seconds`**
are the match-formation primitives, and they already count *roster*
members rather than raw peers.
- **`MatchNet`'s roster** already survives the lobby→match transition, which
is the structure an assigned roster slots into.
- **`MatchState`** already has a legal-transition table with wire-stable
integer values, so new lifecycle states append cleanly.
Use one repository and shared domain packages, with independently runnable
roles rather than independently designed microservices:
### What is genuinely new
| Role | Responsibility |
| --- | --- |
| API | HTTPS/WebSocket auth, profile, queue commands, status resync |
| Matcher | Atomic proposal formation from queue state |
| Allocator | Agones allocation, server registration, assignment delivery |
| Maintenance worker | Outbox delivery, season rollover, expiry, reconciliation |
- The backend service itself (process, deploy, DB, ops) — nothing like it
exists in this repo today.
- Server-authoritative **match results**: the dedicated server must report
the outcome to the backend over a channel a client cannot forge. This is
the first non-ENet/SDR network path in the project (see TECH_STACK's "no
HTTP layer" note, which this supersedes).
- An **assigned-roster** server mode: only the matched SteamIDs may take a
slot, replacing the current first-come model.
- Client-side queue UI and the accept/decline flow.
API replicas are stateless. Redis sorted sets provide the fast candidate
index, but Redis is never the durable allocation fence: asynchronous failover
can lose an acknowledged write. A matcher claims a proposal in a PostgreSQL
`SERIALIZABLE` transaction using a partial unique constraint that permits only
one active proposal/match participation per player. The transaction records
the proposal and participants before Redis cleanup; a stale Redis claim then
loses at PostgreSQL and is repaired from the durable record. PostgreSQL is the
source of truth for queue ownership, identities, sessions/revocations,
seasons, ratings, matches, participants, penalties, result receipts, audits
and the transactional outbox. Redis holds expiring presence, candidate
indexes, latency evidence, session/revocation caches and transient fan-out.
Losing the last acknowledged Redis write may delay/rematerialise a ticket or
force a session cache miss, but can neither resurrect a revoked session, split
a proposal nor corrupt a result/rating.
## Server orchestration and autoscaling
For launch, run the horizontally scaled control plane in one primary
Kubernetes region with a warm standby and tested restore path. Game fleets
remain regional in EU and North America. This avoids a premature multi-writer
database while keeping new-match control latency small relative to queue time.
An outage may pause new queues/allocations, but live matches must continue.
Targets are PostgreSQL RPO <= 5 minutes and control-plane RTO <= 30 minutes.
Requirement: game servers scale horizontally and automatically, spin up fast
on demand, serve exactly one match, and shut down — so cost is incurred only
while a match is being played. Docker and the existing CI gates must keep
working unchanged.
### Stable identifiers and state
### Why this is achievable: the server is already shaped for it
Contracts define opaque `player_id`, `queue_ticket_id`, `proposal_id`,
`match_id`, `server_id`, and `season_id`. Every mutating request has an
idempotency key, expected revision, and versioned schema.
Two properties of the current build make per-match allocation practical
rather than aspirational:
The durable/transient state path is:
- **The container is small.** The `server` image target is a slim
`ubuntu:24.04` runtime with three shared libraries and the exported
binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image.
Pulling it onto a fresh node is cheap.
- **Boot to listening is sub-second.** Measured on this repo's
`cosmicclash-server:latest`: **~870 ms** from container start to the
`server_started` log line, averaged over three runs, read from the
container's own clock. That measurement was taken under **x86_64 emulation
on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux
should be faster. Re-measure on the real target before setting timeouts.
```
QUEUED -> PROPOSED -> ACCEPTED -> ALLOCATING -> PROCESS_READY
-> ASSIGNMENT_READY -> ASSIGNED -> CONNECTING -> LIVE
-> RESULT_PENDING -> COMPLETED
-> CANCELLED / EXPIRED / FAILED from the explicitly legal stages
```
Combined with `--max-matches=1`, which already drains and `exit(0)`s after a
single match, the lifecycle the allocator needs mostly exists: start
container → serve one match → process exits → orchestrator reclaims.
The API publishes revisioned changes over one authenticated WebSocket. REST
GET endpoints are the recovery source after a disconnect or missed revision.
Restarting the client resumes an unexpired ticket/assignment instead of
creating another one.
### The cold-start tension, stated honestly
Assignments include match/server IDs, protocol and client build, server image
digest, playlist version, transport, region, expiry, a match-scoped join
authorisation, and either the SDR hosted-server material or an ENet endpoint.
The authorisation may be replayed only by the same Steam identity to reclaim
the same match/server/slot before expiry. Each successful connection advances
a server-owned generation and fences the prior connection; another identity,
server or slot is always rejected. This deliberately supports reconnect when
Steam or the control plane is temporarily unavailable. Incompatible
protocol/build/playlist versions never enter one proposal.
"Only pay during a match" and "a player never waits" are in tension. A server
must be listening *before* the matched players connect, so some cost always
precedes the match. Sub-second boot makes the gap small enough that a pure
scale-to-zero design is plausible — but the risk is not the container, it is
everything around it: image pull on a cold node, scheduler placement, and
network/port programming can each dwarf 870 ms.
## 4. Queue and placement policy
Recommendation: **scale to zero at the node level is the wrong target; scale
to zero at the match level is the right one.** Keep a small warm pool of
nodes sized to the current queue depth, and start a per-match container on
demand within it. The per-match process genuinely exists only for the match;
the node pool absorbs the cold-start variance. Revisit only if measured
allocation latency on real infrastructure shows the warm pool is unnecessary.
Each verified player may own at most one active queue ticket. Heartbeats are
sent every 10 seconds and queue presence expires after 30 seconds. Create,
cancel, resume, accept, decline, and expiry are atomic and retry-safe.
### Findings that block a naive implementation
The client submits its recent opaque Steam ping location plus nonce-bound
active-probe responses from each regional endpoint; it does not submit the RTT
used for placement. The backend validates a 30-second freshness window and
nonce, then uses the Steam coordinator SDK and probe timings to compute the
regional matrix. A predicted/observed discrepancy over 25 ms or 30% (whichever
is larger) in three matches within 24 hours quarantines the account's samples:
it may queue only in regions whose active probe independently remains under
the ceiling until five clean matches clear the quarantine. The matchmaker:
**Readiness cannot be detected from the log line.** Godot's stdout is
block-buffered when it is not attached to a TTY. Run the server image
detached without `-t` and `docker logs` shows **nothing at all** — the
`server_started` line does not appear even after 35 seconds, because the
buffer never flushes. An orchestrator readiness probe that greps for that
line will hang forever, and this was reproduced directly while measuring the
boot time above. Either probe the UDP socket instead, or make the server
flush explicitly. This also means container logs are not a reliable
observability channel for a short-lived match server; treat log shipping as
a separate problem.
1. Finds regions in which every proposed player has predicted RTT <= 100 ms.
2. Minimises the worst player's predicted RTT.
3. Breaks ties by total predicted RTT, then ready server capacity.
4. Widens rating tolerance with wait time, but never automatically widens the
100 ms latency ceiling.
**One fixed port per container does not scale on a shared host.** `--port`
defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing
several matches onto one node needs either a port range allocated per
container, or one address per container. This is a UDP service, so the usual
HTTP ingress/L7 routing answers do not apply — the allocator must hand the
client a concrete `host:port`.
The target is regional observed p95 RTT <= 80 ms. Candidate formation is
deterministic:
**The match cannot start on a schedule the players do not control.** Today
the loop waits for `--min-players` then counts down. An allocated server is
told *which* identities to expect, and needs a **no-show timeout**: if a
matched player never connects, the server must abandon and exit rather than
sit idle burning the cost this design is trying to avoid.
- Anchor on the oldest compatible ticket (`enqueued_at`, then ticket ID).
- A ticket's rating tolerance is `min(400, 100 + 25 * floor(wait_seconds/30))`
rating points. A pair is compatible only when its absolute rating difference
is within both tickets' tolerance. Provisional players use the same stored
rating; their high RD changes ratings faster, not eligibility arithmetic.
- Every candidate set contains the anchor. Choose the lexicographically
smallest tuple `(worst_RTT, total_RTT, rating_range, -sum(wait_seconds),
sorted_ticket_IDs)`, so latency dominates once the oldest player anchors
fairness and every candidate satisfies the widening rule.
- Partition humans exhaustively into legal teams and minimise absolute team
mean-rating difference, then maximum opposing-player difference, then use
lexical player IDs. Bots fill remaining casual slots after humans are
assigned.
### Keeping Docker and CI green
Every selected human receives a 10-second proposal before allocation. Ranked
always selects six. Casual requires six before the anchor reaches 60 seconds;
afterward it selects the largest compatible human count from six down to two,
with at least one human per team, and displays teams/bot slots. Unanimous
acceptance by the selected humans advances.
The existing gates must not regress. `make verify-phase6` builds the export,
runs it in Compose, joins two headless clients and asserts both saw both
goals and that the arena rotated between matches; `make verify-enet-integration`
runs the source-build ENet matrix. Both depend on current behaviour:
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
assignment, and uses `--max-matches=2` to prove rotation.
An explicit proposal decline cancels that player's ticket and applies a
30-second casual or 2-minute ranked cooldown. A timeout applies 60 seconds in
casual or 5 minutes in ranked. Three ranked proposal declines/timeouts within
30 minutes apply 15 minutes. Players who accepted return with their original
`enqueued_at` and precedence when another player declines, times out, or the
allocation fails. Ordering ties use ticket ID. A ranked initial-connect
no-show after accepting uses the ranked abandon cooldown ladder but never a
rating loss because no rated match began.
The rule that keeps them passing: **every allocation feature is opt-in via a
new `ServerConfig` flag whose default reproduces today's behaviour.** An
assigned roster, a no-show timeout and result reporting must each be inert
unless explicitly enabled. `ServerConfig` is built for exactly this — a flag
declared once is parsed, validated, type-checked, config-file-backed and
documented — and `tests/cases/` can cover the new parsing without a live
server. A second Compose file should cover the allocated-match path rather
than mutating the Phase 6 one, so the community-server model stays tested
alongside the matchmade one.
### Casual
### Open questions
- Target six humans in 3v3.
- After 60 seconds, a match may start with at least two humans, one on each
team, and fill the other slots with server bots.
- Human backfill may replace a bot/disconnected slot only at a kickoff
boundary.
- Backfill is a separate 10-second opt-in proposal showing score, time
remaining, team and slot. Declining/timing out carries no cooldown. A
backfilled player receives no hidden-rating update or abandon penalty for
that match; acceptance removes their queue ticket only when assignment is
ready. Choose the oldest ordinary casual ticket that meets the same build,
region <=100 ms and current anchor-tolerance rules for the vacated human
slot; ties use ticket ID.
- An original casual participant gets 30 seconds to reconnect; leaving after
that applies a 60-second queue cooldown. The match's ordinary hidden-rating
result still applies, with no extra rating penalty.
- An accepted casual initial-connect no-show gets the same 60-second cooldown.
The match proceeds with a bot only if at least one human connected on each
team; otherwise it cancels and restores every innocent ticket with original
precedence.
- Casual has a separate hidden Glicko-2 rating used only for matching.
- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this
game-server lifecycle), Nomad, or direct cloud-API container starts. Not
chosen. Agones is the strongest default because it models allocation,
readiness and per-match lifetime natively.
- **Port strategy** — port range per node versus one IP per match.
- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 610 match
processes per modern core and 150250 MB RSS each. That estimate predates
any allocation work and should be re-measured under real concurrency
before it sizes a bill.
- **Draining and deploys.** How a server version rolls out without killing
matches in flight.
### Ranked
## Casual vs ranked
- Exactly six verified humans; never bots and never backfill.
- Solo queue only at launch.
- Only `ArenaRegistry` entries with `"random": true` are eligible. Elevated
goals remain excluded until the trained-policy restriction is lifted.
- A reconnecting player has 60 seconds to return using the existing
assignment. After that, that player receives a loss regardless of the final
team result and a rolling seven-day cooldown: 5 minutes, 15 minutes, 1
hour, then 24 hours.
- Delivery failure is not match-integrity failure. A healthy completed match
remains rated while its result waits for the control plane. Rating is
suppressed only when the authoritative roster, simulation or result is
unavailable/corrupt, or a measured regional incident prevented fair play.
They are different playlists, not a difficulty toggle, and their rules
diverge in ways that affect the server:
## 5. Rating and seasons
| | Casual | Ranked |
| --- | --- | --- |
| Rating | Hidden, used only for matching | Visible, with tiers |
| Backfill on disconnect | Yes — keep the match playable | No — the match is rating-bearing and must not change shape mid-way |
| Bots filling empty slots | Acceptable (`--fill-bots` exists) | Never |
| Abandon penalty | Light (short queue cooldown) | Real (rating loss, escalating cooldown) |
| Arena selection | Full rotation | Restricted set, so a variant nobody has practised can't decide a ladder match |
| Party / premade | Unrestricted | Constrained by size and rating spread |
Use canonical Glicko-2 independently per playlist with initial rating 1500,
RD 350, volatility 0.06, scale constant 173.7178 and tau 0.5. Updates are
immediate per committed match rather than globally batched. Before an update,
advance inactivity by whole 24-hour rating periods since the player's last
rated match using `phi = min(350/173.7178, sqrt(phi^2 + sigma^2 * periods))`.
Note the arena constraint interacts with an existing rule: elevated-goal
variants are Free-Play-only until a checkpoint trained on
`training_elevated.tscn` is promoted (`arena_registry.gd`). Ranked's arena
set should be drawn from `"random": true` arenas only.
For each player `i`, transform every opposing human's pre-match rating/RD to
`mu_j`/`phi_j` and use the canonical equations
`g(phi)=1/sqrt(1+3*phi^2/pi^2)` and
`E=1/(1+exp(-g(phi_j)*(mu_i-mu_j)))`. Ranked's three opponent contributions
use `w=1/3`; casual uses `w=1/N` for the `N` opposing humans, ignoring bots.
Thus human contributions total exactly one match in both sums:
## Open questions
```
v^-1 = sum(w * g(phi_j)^2 * E_j * (1-E_j))
Delta = v * sum(w * g(phi_j) * (s-E_j))
```
- **Rating algorithm.** Glicko-2 is the default recommendation over plain
Elo: it models rating *uncertainty*, which matters enormously for a small
launch population where most players have few games. Not yet decided.
- **Team rating from individual ratings.** How a 3v3 match's outcome
distributes across six players is a separate design problem from the
rating system itself.
- **Server cost.** Allocated servers cost real money per match, unlike
community servers that players host themselves. `README.md`'s original
note about a subscription to fund servers is suddenly load-bearing again.
Population size and match length set the bill; this needs a number before
launch, not after.
- **Region / ping policy.** How much rating tolerance to trade for latency,
and whether cross-region is ever allowed at low population.
- **Placement matches** and whether ranked has a soft reset per season.
- **Backend language and hosting.** Not chosen. It does *not* have to be C#
despite the original README framing — that framing was aspirational and
predates every real decision in this project.
Then run the canonical Glicko-2 volatility iteration and rating/RD update. If
a player has no opposing human, the match is unrated for that player. `s` is 1/0 for
the authoritative winner/loser and 0.5 only for a completed draw. Overtime is
an ordinary win/loss. A ranked abandoner gets `s=0` regardless of the final
team result; non-abandoning players use the authoritative final result.
Cancelled or integrity-failed matches do not update anyone.
## Explicitly out of scope
Lock all six participant rows in lexical player-ID order and compute every
new value from the same immutable pre-match snapshot inside one serializable
transaction. This prevents update-order bias and concurrent double updates.
Golden vectors include canonical one-player examples plus symmetric/asymmetric
3v3, draw, overtime, abandon, inactivity and concurrent-result fixtures.
Tournaments, in-game leaderboards beyond a personal rank display,
cross-platform play with non-Steam identity providers, and spectator/observer
tooling for ranked matches. None are precluded by this design; none are
launch scope.
The first ten ranked matches are provisional. Ranked exposes
backend-derived tiers; casual rating remains hidden. Stored Glicko values, not
tier labels or client calculations, are authoritative. Ranked seasons last 12
weeks. Ranked rollover sets
`rating = 1500 + 0.75 * (rating - 1500)`, raises RD to at least 200 (capped at
350), preserves volatility/history, and is an exactly-once idempotent
transaction. Casual rating is continuous and never season-reset.
Rating, penalty, match completion, participant records and outbox events are
committed in that transaction. Duplicate identical server results succeed
idempotently. A conflicting result changes nothing and pages an operator.
## 6. Game-server allocation and lifecycle
Use provider-portable Kubernetes manifests and Agones. Create one versioned
Fleet per compatible build and region; labels identify region, protocol,
transport, and image/build. Allocate atomically with `GameServerAllocation`.
Agones, rather than bespoke allocator code, owns selection and lifecycle. See
[GameServerAllocation](https://agones.dev/site/docs/reference/gameserverallocation/).
The Godot process talks to the Agones REST sidecar through a small adapter
that is a no-op when `AGONES_SDK_HTTP_PORT` is absent. This preserves native,
Compose, and CI operation and works with the Agones local SDK emulator. See
[Agones client SDKs](https://agones.dev/site/docs/guides/client-sdks/).
Agones has two distinct readiness points; conflating them is a deadlock because
`GameServerAllocation` selects a Ready server and only then attaches the match
metadata:
1. The PID-1 supervisor queries the local Agones SDK for the assigned dynamic
port/address, exports `SDR_LISTEN_PORT` plus `SDR_IP=public-address:port`
(or the ENet equivalent), and launches Godot. Godot validates static config,
binds the socket and starts Health calls.
2. Godot calls Agones `Ready()` after the process is genuinely listening.
This is **process-ready** only; never infer it from detached stdout.
3. `GameServerAllocation` atomically changes that Ready server to Allocated
and supplies the signed roster/non-secret match configuration as metadata.
4. The server watches the GameServer, observes Allocated metadata, verifies
manifest signature/build/protocol/server binding, registers its hosted SDR
address, and calls the backend `assignment_ready` endpoint.
5. Only after `assignment_ready` does the allocator mint relay/join tickets and
expose the assignment to clients. The server accepts only assigned
identities/slots; each selected human has 30 seconds to connect.
6. Run one authoritative match with Health calls independent of the simulation
loop. Submit the canonical result using the bound workload identity.
7. Write the signed result hash/payload to the backend and to a non-secret
Agones annotation, remain Allocated, and retry until the backend durably
acknowledges it. The maintenance worker reconciles the annotation after an
API outage. `RESULT_PENDING` pages at 5 minutes and requires operator review
at 30 minutes; it never silently becomes unrated.
8. After acknowledgement call `Shutdown()` and exit. Invalid/empty allocations
shut down immediately. An allocated GameServer is not recycled to Ready.
Agones supplies dynamic host ports so multiple isolated matches can share a
node; HTTP ingress is not involved in gameplay routing. Hosted SDR additionally
requires Valve approval for every provider/location, a valid `SDR_POPID`,
public-IP/unsolicited-UDP reachability, provider firewall/NAT validation,
per-location certificates and coordinator trust. Use an Agones dynamic or
passthrough mapping whose externally reported port is the `SDR_IP` port while
the process binds `SDR_LISTEN_PORT`; test SDR and ENet mappings separately.
Credentials arrive through runtime secret mounts, never allocation metadata,
arguments, logs or images.
Result authentication uses a projected, pod-bound service-account token with
a dedicated audience and one service account per workload class. The backend
validates the configured cluster issuer/JWKS, audience, expiry, namespace,
service account, bound pod UID and allocator-recorded GameServer UID, then
checks that GameServer/match binding in PostgreSQL. Issuers and trust roots are
allowlisted and rotated explicitly for every cluster/provider. A one-match
server credential issued after this attestation is an acceptable equivalent.
### Warm capacity and density
Both launch regions are active whenever their queues are enabled. Each active
region maintains at least two Ready processes distributed across at least two
on-demand nodes/failure domains; the minimum node floor is therefore two, not
one. Pre-pull current and rollback images. A queue/proposal-aware
FleetAutoscaler adjusts capacity above that Ready floor. **Allocated** process
count may fall to zero; Ready processes do not. An administratively disabled
region may scale both nodes and Fleet to zero and is excluded from placement.
See [Agones FleetAutoscaler](https://agones.dev/site/docs/reference/fleetautoscaler/).
Do not run live matches on interruptible nodes. N+1 means loss of the largest
single node still leaves two Ready slots plus sufficient headroom for already
Allocated matches; certify it in the node-loss test. Set pod requests, limits
and node caps only after native x86_64 benchmarks of boot time, p99
CPU/RSS/network and 60 Hz tick behavior, retaining 30% headroom. The current
610 processes/core and 150250 MB estimates are not sizing data.
Godot/GDScript cannot intercept SIGTERM, so Phase 8 adds a small Go PID-1
supervisor. It starts Godot, traps TERM, and sends a per-pod-token-authenticated
localhost drain request to an allocated-mode-only Godot control socket. Godot
refuses new admissions and reports completion to the supervisor. Kubernetes
uses a 300-second termination grace period; at 285 seconds the supervisor
forces an infrastructure-failure abort so the pod cannot hang forever in
overtime. A PodDisruptionBudget and Agones-aware drain prevent voluntary
eviction of Allocated servers. Planned releases create a new Fleet, route new
allocations to it and wait for old Allocated count zero without sending TERM.
Unexpected node failure cannot be made graceful and follows the integrity
failure/refund path.
Live matches continue through an API/control-plane delivery outage. A valid
result is retried/reconciled as above; only loss of authoritative match
integrity suppresses rating.
## 7. Security and operational baseline
The threat model must cover forged clients, ticket replay, duplicate queueing,
result forgery, queue manipulation, packet floods, botting, server compromise,
insider access, DDoS, vulnerable dependencies, and denial-of-wallet attacks.
Minimum controls:
- Non-root containers, read-only root filesystems, dropped Linux
capabilities, RuntimeDefault seccomp, resource limits, restricted Pod
Security admission, and least-privilege service accounts/RBAC.
- Private PostgreSQL/Redis; default-deny ingress and egress with explicit
NetworkPolicy allowances. See the [Kubernetes application security
checklist](https://kubernetes.io/docs/concepts/security/application-security-checklist/)
and [NetworkPolicy](https://kubernetes.io/docs/reference/kubernetes-api/networking/network-policy-v1/).
- Encrypted databases/backups, externally populated Kubernetes Secrets,
documented rotation, and no credentials in Git, images, command lines, or
telemetry.
- Per-account/IP API limits, request/body/schema limits, replay and duplicate
detection, generic public errors, allocation quotas, and budget alerts.
- Put HTTPS/WebSocket traffic behind a provider-portable edge contract
implemented by each infrastructure overlay: managed volumetric DDoS
absorption, WAF/rate rules, TLS termination, origin-only ingress and health
checks. Enforce authenticated queue admission, per-account connection caps,
WebSocket handshake/message/idle limits, bounded fan-out and overload
shedding. In degraded mode reject new login/queue/allocation work while
preserving result ingestion and all live matches.
- Images pinned by digest, SBOM generation, dependency/image scanning, signed
releases, admission-time signature verification, and a critical-patch SLA.
- Structured audit events for auth, queue transitions, allocation, roster
rejection, result conflict, penalty, season rollover, and operator action.
## 8. SLOs, observability, and release gates
Launch SLOs:
| Measure | Target |
| --- | --- |
| Eligible predicted RTT | <= 100 ms for every player |
| Regional observed RTT | p95 <= 80 ms |
| Unanimous acceptance to `assignment_ready` | p95 <= 5 s, p99 <= 10 s with warm capacity |
| Published assignment to successful connection | p95 <= 5 s |
| Successful allocation and durable result | >= 99.9% |
| Certified server density | No tick backlog, with 30% resource headroom |
| Control-plane API under load | p95 <= 250 ms |
Dashboards and alerts cover queue depth/wait, rating spread, predicted versus
observed RTT, proposals/declines, allocation latency/failure, Ready capacity,
image pulls, connection/no-show, physics overruns, crashes, abnormal packet
rates, result lag/conflicts, abandons, and cost per completed match. Correlate
all components with queue/proposal/match/server IDs, but never log auth or
relay tickets.
Testing layers:
- Go unit, race, fuzz, property, migration, and concurrency tests.
- Fake Steam verifier and fake allocator for deterministic CI.
- A second allocated-server Compose flow; never mutate
`compose.phase6-smoke.yml`.
- Disposable `kind` + Agones integration for readiness, dynamic ports,
health/no-show shutdown, allocation races, multiple matches per node,
draining, and rollback.
- Network/chaos cases for 100 ms RTT, jitter/loss, client/backend/matcher
restart, game-pod death, node drain, Redis failover, and control-plane loss.
- Load test at least 10,000 queued clients, 100 proposals/second, and forecast
launch concurrency x2 while preserving correctness and API latency.
- Provider migration rehearsal: restore data and deploy both regional fleets
on a second provider whose EU/NA locations already have Valve approval,
POP/certificates, public UDP/firewall verification and coordinator trust;
switch new allocations, drain the old provider, and terminate no live match.
Release order is development -> internal -> casual canary -> full casual ->
provisional ranked -> full ranked. Each promotion requires its SLO/security
gates, rollback rehearsal, EU and North America playtests, a measured cost
model, and unchanged `make verify-phase6` and
`make verify-enet-integration`.
## 9. Explicitly out of scope
Parties/premades, tournaments, ranked spectators, public global regions,
non-Steam identity providers, and a global leaderboard beyond personal rank
display are not launch scope. The contracts reserve party identity and avoid
Steam-specific database primary keys so those additions do not require a
destructive redesign.