jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-14 02:22:06 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: finalize scalable matchmaking plan

Browse Source
This commit is contained in:
Josh Creek
2026-08-31 19:57:49 +01:00
parent bcc12aad19
commit 8d4a0640e2
5 changed files with 647 additions and 343 deletions
Download Patch File Download Diff File Expand all files Collapse all files
multiplayer-todo.md
+92 -68
View File
@@ -14,13 +14,13 @@ Everything below is written so an agent (or a person) can pick up a single numbe
The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 0–5 contain no unfinished tasks.
**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.20 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three items there are findings rather than plans, and each would break a naive implementation:
**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.53 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three findings would break a naive implementation:
| # | Finding | Why it bites |
|---|---|---|
| 8.8 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | An orchestrator readiness probe that greps the log hangs forever |
| 8.9 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches cannot share a host; being UDP, L7 ingress routing does not apply |
| 8.18 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | Allocation work trivially regresses `verify-phase6` unless every new feature defaults to today's behaviour |
| Task 8.28 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | Process-ready must be an explicit Agones call after static validation/listen; post-allocation assignment-ready is separate and neither uses a log grep |
| Task 8.29 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches need Agones dynamic UDP/SDR ports; L7 ingress does not route this traffic |
| Task 8.48 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | The allocated flow needs its own fixture so Phase 6 behavior and invocations stay unchanged |
### Blocking sign-off — the work exists, the verification does not
@@ -50,7 +50,7 @@ C is the one to plan around: it is fixed for free by task **7.4** (Steam auth ti
### Unstarted phases
- **Phase 6 external gate:** run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until **C** is fixed.
- **Phase 7 — Steam transport, browser, identity** (5 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, auth tickets, and bans await a project-owned Steamworks App ID. Carries the fix for **C**.
- **Phase 7 — Steam transport, browser, identity and production SDR** (8 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, verified tickets, bans, production credentials and ticketed Hosted Dedicated Server SDR await a project-owned Steamworks App ID and Valve coordination. Carries the fix for **C** and is the hard prerequisite for Phase 8.
Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into something another person can host. Phase 7 is the next block because Steam identity is required before public exposure.
@@ -69,7 +69,7 @@ Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into some
| 1 | **Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation.** | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. |
| 2 | **Dedicated servers only.** Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. |
| 3 | **ENet first**, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and **must never become the degraded path**. |
| 4 | **No custom backend.** | Steam's `ISteamGameServer` master-server listing covers discovery, `ISteamMatchmakingServers` covers the in-game browser, and Steam auth tickets cover identity and ban state. `README.md`'s C# backend stays unstarted. |
| 4 | **Community discovery uses no custom backend; superseded for queued play by Phase 8.** | Steam's server APIs remain enough for the community browser. Casual/ranked queues, durable ratings, allocation and authoritative results require the project-owned Go control plane specified in `docs/MATCHMAKING.md`; it does not replace the browser or direct-IP path. |
### 1.2 Rejected alternatives
@@ -1135,6 +1135,9 @@ New `--role=client-reconnect` grades the returning player: not a spectator, owns
| 7.3 `[D:7.2]` `[P]` | `server_browser.tscn` via `ISteamMatchmakingServers` | Internet, LAN, favourites and history lists all populate |
| 7.4 `[D:7.2]` `[P]` | Auth tickets in `hello` → `BeginAuthSession`; Steam identity in the roster; persistent ban list | Ownership, VAC and ban state verified server-side |
| 7.5 `[D:7.2]` `[P]` | Feature-gate every Steam call behind `OS.has_feature("steam") and ClassDB.class_exists("SteamMultiplayerPeer")`; verify the ENet path end to end | Non-Steam build is fully functional |
| 7.6 `[D:7.4]` | Separate single-use tickets for backend Web API login and game-server auth; wait for Steam's asynchronous validation and cancel/end every ticket session | Replayed, cancelled, wrong-App-ID and not-yet-validated identities cannot enter a roster or queue; no client-supplied SteamID is trusted |
| 7.7 `[D:7.1]` `[P]` | Obtain the production App ID, publisher key, SDR coordinator SDK/signing approval, certificates and hosted-data-centre support from Valve | Production prerequisites and rotation owners are recorded; no Spacewar credential or development certificate can reach a release build |
| 7.8 `[D:7.6,7.7]` | Ticketed Hosted Dedicated Server SDR: routing registration, coordinator-issued player→server relay tickets, client ticket installation, reconnect and expiry | Two real accounts complete and reconnect to an assigned dedicated match through SDR; server/player IPs are not exposed; ENet gates remain green |
> **The transport interface is written here, not in Phase 1.** Eight virtual methods (`begin_auth`, `advertise`, `get_identity`, `supports_server_browser`…) designed against an API nobody on the project has used will be wrong. Write `NetworkManager._make_peer()` concretely in Phase 1 and extract the boundary once there are two real implementations. Locked decision 3 guarantees the ENet path is never deleted, so there is no migration risk in waiting.
@@ -1155,86 +1158,107 @@ and players find it by IP or (7.3) the server browser. Matchmaking makes the
**allocated for that one match** and destroyed after. Both models ship; they
are different playlists, not a replacement.
**Hard dependency on 7.4.** Slot reclaim is keyed by display name today. A
rating attached to a spoofable identity is farmed trivially, so ranked cannot
ship before Steam auth tickets land. Casual queueing needs 7.4 too, for
abandon penalties and ban enforcement, but degrades more gracefully.
**Hard dependency on 7.6 and 7.8.** Slot reclaim is keyed by display name
today. A rating attached to a spoofable identity is farmed trivially, so no
queue ships before single-use verified identity lands. Production allocation
also depends on the ticketed Hosted Dedicated Server SDR route; ENet remains
the local/CI/community transport, not a silent production fallback.
#### 8a — Backend service
#### 8A — Architecture, contracts and data
| # | Task | Acceptance |
|---|---|---|
| 8.1 | Choose backend language, hosting and datastore. **Not C# by default** — that framing predates every real decision here | Written up with the rejected alternatives, as §1 does for the client decisions |
| 8.2 `[D:7.4]` | Steam auth ticket validation via the Steamworks Web API; a verified SteamID is the only trusted identity | A forged or replayed ticket is rejected; no client-supplied identity is ever trusted |
| 8.3 `[D:8.1]` | Rating store: per-identity, per-playlist rating plus match history, written only by the backend | A client cannot write its own rating by any path |
| 8.4 `[D:8.3]` | Rating algorithm. **Glicko-2 recommended over Elo** — it models rating *uncertainty*, which dominates at launch when most players have few games | Simulated against a synthetic population; placement behaviour is sane at n≈0 games |
| 8.5 `[D:8.4]` | Team-result → individual-rating distribution for 3v3 | A 3v3 outcome updates six ratings defensibly; documented, not folded into 8.4 |
| 8.6 `[D:8.3]` | Queue and matchmaker: per playlist and region, rating proximity with tolerance widening over wait time | Queue depth and wait time are observable; tolerance widening is tunable without redeploy |
| 8.1 | Add an ADR locking **Go + PostgreSQL + Redis**, provider-portable Kubernetes, Agones, ticketed Hosted Dedicated Server SDR, EU/NA fleets and independently runnable API, matcher, allocator and maintenance roles; keep `README.md`/`docs/TECH_STACK.md` consistent | The ADR names boundaries/rejected alternatives; current docs name the locked stack and replaceable provider; no application code calls a provider allocation API |
| 8.2 `[D:8.1]` | Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | Each SLO has a metric, numerator/denominator, percentile window, owner and alert threshold before implementation is judged against it |
| 8.3 `[D:8.1]` | Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | Generated contract tests cover every request, response, event and external error; clients can REST-resync after a missed WebSocket revision |
| 8.4 `[D:8.3]` | Define opaque `player_id`, `queue_ticket_id`, `proposal_id`, `match_id`, `server_id`, `season_id`, legal queue/match state transitions, revisions and idempotency keys | Duplicate/out-of-order commands converge; invalid transitions are rejected without partial state |
| 8.5 `[D:8.4]` | Add PostgreSQL migrations for durable queue ownership, active-participation fencing, identities, sessions/revocations, seasons, ratings/events, matches/participants, penalties, results, audits and outbox; document Redis caches/TTLs | A blank DB migrates up; lost Redis writes cannot resurrect revocation, split a proposal or corrupt durable state; rollback/forward compatibility is tested |
| 8.6 `[D:8.3,8.4]` | Lock assignment compatibility: protocol/client build, image digest, playlist version, transport, region, expiry and signed authorisation; add all allocated-mode `ServerConfig` flags as opt-in defaults | Incompatible builds never share a proposal; absent flags reproduce today's community server and existing config tests cover every new flag |
#### 8b — Server orchestration and autoscaling
Requirement: servers scale horizontally and automatically, spin up fast, serve
exactly one match, and shut down — cost incurred only while a match runs.
Two properties of the existing build make this practical rather than
aspirational, both **measured against `cosmicclash-server:latest`**, not
estimated:
- The runtime image (`server` target, slim `ubuntu:24.04`) is **~148 MB** of
content — not the ~2.6 GB `godot-ci` build image.
- Boot to the `server_started` line is **~870 ms**, container's own clock,
mean of three runs. **Taken under x86_64 emulation on an arm64 host, so it
is a pessimistic bound** — see 8.11.
`--max-matches=1` already drains and `exit(0)`s after one match. It was built
for CI and generalises to the allocator lifecycle for free.
#### 8B — Authentication and secure control plane
| # | Task | Acceptance |
|---|---|---|
| 8.7 | Choose the orchestrator. **Agones on Kubernetes is the recommended default** — it models allocation, readiness and per-match lifetime natively rather than making you rebuild them | Allocation, readiness and per-match teardown are all handled by the chosen system, not by bespoke glue |
| 8.8 | **Fix readiness detection — this blocks any naive implementation.** Godot's stdout is block-buffered off a TTY. Run the server image detached without `-t` and `docker logs` shows *nothing at all*; `server_started` does not appear even after 35 s. A readiness probe that greps the log hangs forever. Probe the UDP socket, or flush explicitly | A cold container is marked ready by a mechanism that does not depend on stdout; reproduced-and-fixed, not worked around by adding `-t` in one place |
| 8.9 | Multiple matches per host: a per-container port from a range, or one address per match. `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. **This is UDP — L7 ingress routing does not apply**, the allocator hands the client a concrete `host:port` | Two matches run concurrently on one node and neither can reach the other's traffic |
| 8.10 `[D:8.6]` | Assigned-roster server mode: only matched SteamIDs may claim a slot, replacing first-come. Plus a **no-show timeout** — an allocated server that never fills abandons and exits rather than idling at cost | An unmatched identity is refused a slot; a server nobody joins exits within the timeout |
| 8.11 | Re-measure boot-to-listening on **native x86_64 Linux** before it sets any timeout | A number from the real target platform replaces the ~870 ms emulated bound recorded above |
| 8.12 | Re-measure SERVER.md's sizing estimate (6–10 processes/core, 150–250 MB RSS) under real concurrency | A measured figure sizes the bill; the current estimate predates all allocation work |
| 8.13 `[D:8.10]` | Server-authoritative match result reporting to the backend over a channel a client cannot forge. **The project's first non-UDP network path** — simulation stays on ENet/SDR | A client cannot report, alter or suppress a result |
| 8.14 | Draining and deploys: roll out a server version without killing matches in flight | An in-flight match survives a deploy of the next server version |
| 8.7 `[D:7.6,8.3]` | Validate `AuthenticateUserTicket` only in the secure backend with the expected App ID and identity string; reject expiry, replay, wrong app, bans and malformed input | Publisher credentials exist only in the backend secret store; forged/replayed tickets and client-supplied SteamIDs never create a session |
| 8.8 `[D:8.7]` | Issue short-lived revocable sessions bound to verified Steam identity; add account/IP limits, body/schema limits, replay checks and generic public errors | Revocation takes effect across replicas; abuse cannot cause unbounded memory, work or response amplification |
| 8.9 `[D:8.4,8.7]` | Issue match-scoped join authorisations bound to SteamID/match/server/team/slot/protocol/expiry; allow same-identity slot reclaim while fencing prior connection generations | Altered/expired/wrong identity/server/slot is rejected; reconnect works without backend/Steam; a newer generation makes the old connection unable to send gameplay |
| 8.10 `[D:8.5,8.31]` | Authenticate results with pod-bound projected identity or one-match attested credential; validate issuer/audience/expiry, namespace/SA, pod UID, GameServer UID and allocator match binding | Another pod sharing a workload class cannot submit for the allocation; identical duplicates are idempotent; conflicting results are inert and alerting across all trusted clusters |
| 8.11 `[D:8.1]` | Write the threat model: forged clients/replay/queues/results, floods/bots, pod/insider compromise, gameplay and API DDoS, SDR signing-key theft, dependencies and denial-of-wallet | Every threat has prevention/detection/owner/verification; accepted residual risks are explicit; offline CA and online signer trust boundaries are separate |
| 8.12 `[D:8.11]` | Harden workloads and edge: restricted containers, least RBAC, private DB/Redis, default-deny networks, backups/secrets, volumetric DDoS/WAF/origin shielding, WebSocket limits and overload shedding | Policy/network tests enforce declared flows; edge load test preserves result ingress/live matches while rejecting new work; no credential appears in Git/images/args/telemetry |
| 8.13 `[D:8.12]` | Pin images by digest; generate SBOMs, scan dependencies/images, sign artifacts, verify signatures at admission and document a critical-fix SLA | CI blocks a vulnerable/disallowed or unsigned release artifact and records the exact provenance deployed |
#### 8c — Playlists and client
#### 8C — Queueing, matchmaking, playlists and rating
| # | Task | Acceptance |
|---|---|---|
| 8.15 `[D:8.6]` | Casual and ranked rulesets. They diverge on the server, not just in UI: backfill (casual yes / ranked never), bots filling slots (`--fill-bots` casual-only), abandon penalties, party size and rating spread | Ranked never backfills and never spawns a bot into a player slot |
| 8.16 `[D:8.15]` | Ranked arena restriction. Draw only from `"random": true` arenas — **elevated-goal variants stay Free-Play-only** until a checkpoint trained on `training_elevated.tscn` is promoted (`arena_registry.gd`), so a variant nobody has practised cannot decide a ladder match | Ranked cannot select an elevated-goal arena |
| 8.17 `[D:8.6]` | Client queue UI: playlist select, estimated wait, accept/decline, connect-on-assignment, post-match rating delta | A declined match returns the other players to the queue without penalty to them |
| 8.14 `[D:8.4,8.5,8.8]` | One durable PostgreSQL queue owner/player plus Redis candidate index: 10 s heartbeat, 30 s expiry, retry-safe create/cancel/resume and repair after cache loss | Replicas/duplicates never place a player twice; failover may delay/rematerialise an index but durable ownership and active-participation fences converge |
| 8.15 `[D:7.8,8.3]` | Submit opaque Steam ping location plus nonce-bound probes; backend computes estimates, enforces 30 s freshness and quarantines 3 discrepancies >25 ms or 30% until 5 clean matches | A client cannot directly choose its placement RTT; stale/forged evidence is rejected; quarantine behavior and server-observed comparison are deterministic |
| 8.16 `[D:8.14,8.15]` | Implement the locked candidate/team algorithm: <=100 ms, oldest anchor, `min(400,100+25*floor(wait/30))` mutual rating tolerance, documented set/region/team tie-breakers | Fixtures cover provisional players, EU/NA/no-common-region, widening caps, deterministic partitions and low population; no placement crosses 100 ms |
| 8.17 `[D:8.14,8.16]` | Ten-second proposal to **every selected human**: ranked 6; casual largest compatible 6→2 after 60 s with disclosed teams/bots; apply exact decline/timeout/no-show cooldown and queue-precedence rules | Allocation starts only after selected humans accept; 2–5-human casual is reachable; accepter timestamps restore exactly; ranked pre-match no-show has cooldown but no rating loss |
| 8.18 `[D:8.5,8.14,8.17]` | Horizontally replicated matcher: Redis candidates, then PostgreSQL serializable proposal/participant fence, then cache cleanup/repair | Forced loss of the last acknowledged Redis write, retries, worker death and failover cannot claim a player into two proposals/matches |
| 8.19 `[D:8.18]` | Casual policy: proposal composition above; >=1 human/team, exhaustive rating-balanced teams, bots after 60 s, opt-in kickoff-only backfill, 30 s reconnect and defined backfill/casual penalties | Every 2–6-human shape is tested; no mid-play replacement; declined/backfill participant gets no excluded rating/cooldown; original leaver gets only documented outcome/cooldown |
| 8.20 `[D:8.18]` | Ranked policy: exactly six verified solo humans, no bots/backfill, only `ArenaRegistry.random`; define initial no-show, proposal timeout and reconnect/abandon transitions | Ranked rejects parties/bots/backfill/elevated arenas; every pre-live failure returns five innocent players with original precedence and applies no rating |
| 8.21 `[D:8.5,8.20]` | Exact Glicko-2 equations from `docs/MATCHMAKING.md`: 1500/350/0.06/tau .5, ranked 1/3 and casual 1/N human-opponent weights, daily inactivity, immutable snapshot/lock order, draws/OT/abandons/cancellation | Canonical plus project 2–6-human/3v3 golden vectors pass; concurrent results serialize without order bias; clients have no rating-write path |
| 8.22 `[D:8.21]` | First ten ranked games provisional; casual rating hidden; ranked tiers derived from authoritative stored values | Matchmaking uses provisional rating/RD; UI visibility changes exactly on result ten without rewriting history |
| 8.23 `[D:8.21]` | Ranked-only 12-week exactly-once soft season: compress 25% toward 1500, RD >=200 capped 350, retain volatility/history; casual remains continuous | Retried/concurrent rollover applies once, never touches casual, and preserves every rating event |
| 8.24 `[D:8.9,8.20,8.21]` | Ranked reconnect/abandon: match-scoped authorisation, 60 s reclaim, server-owned connection generations, then abandoner loss and rolling 7-day 5 m/15 m/1 h/24 h cooldown | Reconnect works through backend outage and fences old peer; grace has no penalty; expiry outcome/escalation is deterministic and auditable |
| 8.25 `[D:8.10,8.24]` | Separate result delivery delay from match-integrity failure; signed Agones-annotation spool, retries, 5 m alert/30 m review, suppression only for lost/corrupt authority or measured unfair regional incident | API outage preserves rating/result; clients cannot request exemption; node/pod/integrity faults take the documented suppression/refund path |
#### 8d — Keeping Docker and CI green
`make verify-phase6` and `make verify-enet-integration` must not regress.
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
assignment, and uses `--max-matches=2` to prove arena rotation — all three are
things allocation work would otherwise trample.
#### 8D — Agones, allocation and regional scaling
| # | Task | Acceptance |
|---|---|---|
| 8.18 | **The rule: every allocation feature is opt-in via a `ServerConfig` flag whose default reproduces today's behaviour.** `ServerConfig` is built for exactly this — a flag declared once is parsed, validated, type-checked, config-file-backed and documented | `verify-phase6` and `verify-enet-integration` pass unchanged with no edits to their invocations |
| 8.19 `[D:8.18]` | A **second** Compose file for the allocated-match path rather than mutating `compose.phase6-smoke.yml`, so the community-server model stays tested alongside the matchmade one | Both models have a green CI gate; neither shares a fixture with the other |
| 8.20 `[D:8.18]` | `tests/cases/` coverage for the new flag parsing, per §10's no-live-server rule | New flags are unit-tested without a live server or a container |
| 8.26 `[D:8.1,8.6,8.12]` | Portable Helm/Kustomize Fleets per build/EU/NA region; isolate provider edge/network/DNS/secret and SDR POP/cert/public-UDP overlays | Two provider fixtures render; labels select region/build/protocol/transport; each fixture documents Valve approval and externally reachable UDP mapping |
| 8.27 `[D:8.26]` | Godot Agones REST adapter plus allocation-metadata watch and Go PID-1 supervisor scaffold; both bypass cloud behavior without SDK env; local SDK support | Native/existing Compose/CI remain functional; emulator exercises supervisor port discovery plus Get/Watch, Ready, Health, annotation and Shutdown |
| 8.28 `[D:8.6,8.27]` | **Process-ready stage:** supervisor obtains dynamic port, launches Godot; static config/listen/Health succeed, then explicit Agones Ready—no roster/backend-registration prerequisite and no stdout probe | A detached unallocated process reaches Ready; a broken listener/config never does; Health reclaims a hung process |
| 8.29 `[D:8.26,8.27]` | Separate ENet and Hosted-SDR dynamic/passthrough mappings; supervisor exports local `SDR_LISTEN_PORT` and external `SDR_IP`; validate POP/cert/firewall/NAT | Two isolated matches share a node; Agones-reported public endpoint receives relay traffic on the bound socket; ENet fixture remains independent |
| 8.30 `[D:8.18,8.26,8.28,8.29]` | Atomic `GameServerAllocation` from Ready filtered by region/build/protocol/transport, attaching signed roster/non-secret config with bounded race retry | Duplicate commands yield one Allocated server; exhaustion or retry leaves no orphan; no client assignment is exposed merely because process is Ready |
| 8.31 `[D:8.9,8.30]` | **Assignment-ready stage:** watch Allocated metadata, verify manifest/bindings, register hosted address, acknowledge backend; only then mint/expose client tickets | Modified/wrong manifest never reaches assignment-ready; clients cannot connect early; secrets never appear in metadata/args/logs |
| 8.32 `[D:8.2,8.26,8.30]` | FleetAutoscaler with >=2 Ready processes across >=2 on-demand nodes/failure domains per queue-enabled region; pre-pull current/rollback; scale **Allocated** count to zero, never the Ready floor | Warm allocation meets p95 5 s/p99 10 s; disabled regions alone scale fully to zero; one-node loss retains certified Ready/headroom |
| 8.33 `[D:8.26,8.32]` | On-demand-only live capacity and measured N+1: loss of largest node leaves two Ready slots plus headroom for surviving Allocated matches | Interruptible nodes cannot receive live matches; forced node loss neither overloads survivors nor prevents the next allocation |
| 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog |
| 8.35 `[D:8.17,8.19,8.20,8.30,8.31]` | Initial-connect no-show: 30 s after assignment-ready; ranked cancels/no-show cooldown, casual bot policy, empty allocation exits | No allocation idles indefinitely; innocent players regain original precedence; no pre-live failure changes rating |
| 8.36 `[D:8.10,8.25,8.28,8.30]` | Go PID-1 supervisor traps TERM and authenticates localhost drain; 300 s grace/285 s infrastructure abort; PDB + Agones-aware Fleet drain; planned releases never TERM Allocated pods | Rollout/rollback waits Allocated=0; TERM path is exercised; forced timeout is classified/refunded; unexpected node loss is not claimed graceful |
| 8.37 `[D:8.5,8.10,8.25,8.26,8.31]` | Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO <=5 m/RTO <=30 m; signed result annotation/reconciliation preserves delivery outages | Restore/failover meet targets; live simulation continues; valid delayed result commits exactly once after recovery |
| 8.38 `[D:7.7,8.26,8.36,8.37]` | Provider migration after Valve approves both providers' EU/NA POPs/certs and public UDP: restore, validate coordinator trust, switch allocations, drain old | Both geographies complete Hosted-SDR matches on new provider and no old-provider live match is terminated |
> **The cold-start tension is real and is not solved by fast boot.** "Only pay
> during a match" and "a player never waits" pull against each other: a server
> must be listening *before* the matched players connect. 870 ms makes the gap
> small, but the risk is not the container — image pull on a cold node,
> scheduler placement and network/port programming can each dwarf it.
> Recommendation: **match-level scale-to-zero over a small warm node pool**,
> not node-level scale-to-zero. The per-match process genuinely exists only for
> the match; the pool absorbs cold-start variance. Revisit only when measured
> allocation latency on real infrastructure says the pool is unnecessary.
#### 8E — Client experience and recovery
> **Server cost re-enters the design.** Community servers are paid for by
> whoever hosts them; allocated servers are paid for by the project, per match.
> `README.md`'s original note about a subscription to fund servers is suddenly
> load-bearing. 8.12 needs to produce a number before launch, not after.
| # | Task | Acceptance |
|---|---|---|
| 8.39 `[D:8.3,8.14,8.17]` | Queue UI: playlist/quality, elapsed and estimated wait, proposal countdown, allocation/connect state, cancel and latency/capacity explanations | Every backend state and terminal failure has a non-stuck visible state; cancel/decline is acknowledged authoritatively |
| 8.40 `[D:8.3,8.14]` | One authenticated revisioned WebSocket plus REST resync; resume valid queue/assignment after client restart | Missed/duplicate/out-of-order events converge and restart never creates a second ticket |
| 8.41 `[D:7.8,8.9,8.31,8.40]` | After assignment-ready, install SDR relay ticket before connect and send match-scoped join authorisation in `hello`; retain ENet assignments locally | Production connects/reconnects/fences old generation through SDR, never before assignment-ready; allocated/direct ENet and community flows remain compatible |
| 8.42 `[D:8.22,8.23,8.24,8.40]` | Backend-authoritative provisional/rank/tier/delta, abandon status and season countdown UI | Client performs no rating math and displays the committed revision after reconnect |
| 8.43 `[D:8.39,8.40,8.41]` | Recovery paths for decline, expiry, startup failure, version mismatch, auth expiry, regional outage and failed reconnect | Automated UI/state tests prove every case returns to a usable queue/menu or resumes the match without a duplicate action |
#### 8F — Observability, verification, cost and rollout
| # | Task | Acceptance |
|---|---|---|
| 8.44 `[D:8.3,8.4,8.28,8.31]` | Propagate queue/proposal/match/server IDs and process-ready/assignment-ready through logs, metrics, traces and replay metadata; redact credentials | One ID traces queue→result across components and automated secret-canary tests find no auth/relay ticket |
| 8.45 `[D:8.2,8.44]` | Dashboards/alerts for wait/MMR/RTT, proposals, allocation/Ready/image pull, connect/no-show, tick/crash/flood, result conflict/lag, abandons and cost | Each SLO and security/cost signal has an exercised alert and runbook |
| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | Go unit, race, fuzz, property, migration and concurrency suites | CI covers auth/join replay-reclaim, stale revisions, durable matcher fencing with lost Redis ack, rollover, result conflict/delivery retry and PostgreSQL retry |
| 8.47 `[D:8.7,8.30]` | Fake Steam verifier and fake allocator for deterministic CI | Normal CI needs no Steam/cloud secret or internet access and can force every success/failure deterministically |
| 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | Second Compose flow: fake backend → queue/proposal → process-ready/allocation/assignment-ready → ENet roster → result ack → shutdown; do not edit Phase 6 fixture | Both server models have independent green gates; existing Make invocations remain unchanged |
| 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | Disposable `kind` + Agones integration gate | CI covers dynamic ports, both readiness stages, roster/no-show, races, multi-match node, result-pending reconciliation, drain and rollback |
| 8.50 `[D:8.25,8.37,8.43,8.49]` | Network/chaos suite: 100 ms RTT, jitter/loss, client/API/matcher restart, game-pod death, node drain, Redis failover and control-plane loss | System recovers to a defined state; infrastructure-caused cases cannot penalise affected players |
| 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | Load test >=10,000 queued clients, >=100 proposals/s and forecast launch concurrency x2 | API p95 <=250 ms, durable matcher fence holds, both readiness/allocation SLOs are met and replicas scale without duplicate claims |
| 8.52 `[D:8.32,8.34,8.45,8.51]` | Per-region cost model from measured density, warm capacity, bandwidth, DB/Redis and telemetry; add budgets and allocation quotas | Cost per completed match and forecast monthly bands are recorded; a denial-of-wallet test triggers limits/alerts before budget breach |
| 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | Progressive release: development → internal → casual canary → casual → provisional ranked → ranked | Each promotion requires SLO/security/cost gates, rollback rehearsal, EU+NA playtests and unchanged legacy gates; rollback criteria and owner are explicit |
Implementation invariants for every task above:
- Matchmade mode is opt-in; every new `ServerConfig` default preserves the
existing community-server path.
- `compose.phase6-smoke.yml`, `make verify-phase6`, and
`make verify-enet-integration` are not repurposed or weakened.
- Production uses ticketed Hosted Dedicated Server SDR; ENet remains the
deterministic local/CI and direct-IP path.
- One process serves one match. Warm processes/nodes absorb startup variance;
capacity and cost are determined from 8.34 measurements, not old estimates.
- Implementation evidence is appended under the completed task as in earlier
phases; design changes first update `docs/MATCHMAKING.md` and dependencies.
---