diff --git a/README.md b/README.md index a1248ba7..3f11c5f3 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,11 @@ There will be default bots available, trained using reinforcement learning, and ## MVP -The first version of this game will be JUST the game, no server-side functionality at all. It will be a local only game where you can play against bots. Split-screen multiplayer could be added in a version 0.2 if demand is high enough. If there is sufficient interest then the server-side functionality can be added to enable online play, with a system in place to ensure that servers can be paid for (perhaps a cheap monthly subscription model?). +The original local-only milestone is complete; the 1.0 scope now includes +dedicated online play plus casual and ranked matchmaking. Community servers +remain self-hostable, while project-hosted match servers are allocated per +match through the provider-portable control plane described in +[`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Split-screen remains deferred. ## Monetisation diff --git a/docs/MATCHMAKING.md b/docs/MATCHMAKING.md index 2f5a7a9d..5a1c8aac 100644 --- a/docs/MATCHMAKING.md +++ b/docs/MATCHMAKING.md @@ -1,243 +1,470 @@ # Matchmaking — casual and ranked queues -Design scope for online casual and ranked play. This is a **1.0 launch -blocker**, not a post-launch addition. +Architecture and locked product policy for Phase 8. This is a **1.0 launch +blocker**. The numbered, independently implementable tasks and their +acceptance criteria live in [`multiplayer-todo.md`](../multiplayer-todo.md); +the short live checklist is [`multiplayer-next.md`](../multiplayer-next.md). -Nothing described here is implemented yet. This doc exists to record the -decisions and the reasoning before code is written; per-task implementation -evidence belongs in `multiplayer-todo.md` once work starts, and the live -checklist lives in [`multiplayer-next.md`](../multiplayer-next.md). +Nothing in Phase 8 is implemented yet. This document records the decisions +those tasks assume so an implementer does not have to redesign the system +while building one part of it. -## The model change +## 1. Model and non-negotiable constraints -The multiplayer that exists today is a **community-server** model. A -dedicated server runs forever: it waits for `--min-players` by roster, -counts down `--start-countdown`, loads the next arena from the rotation, -plays a match, returns to the lobby, and repeats (`server_match_loop.gd`). -Players reach it by direct IP, and after Phase 7 by a Steam server browser. -The server is the durable thing and players come and go around it. +The existing multiplayer is a community-server model: a dedicated server +runs continuously, waits for players, rotates arenas, and can be reached by +direct IP or the Phase 7 Steam browser. Matchmaking adds a second model: +players queue, every selected human accepts a proposal, one server process is +allocated for that match, and the process is destroyed after its result is +durably recorded. Ranked selects six humans; relaxed casual selects two to six +and discloses its bot-filled team composition before acceptance. Both models +ship. -Queued matchmaking inverts that. Players are the durable thing: they enter a -queue, a matchmaker groups them by rating and region, and a **server is -allocated for that one match** and torn down afterwards. Both models can -coexist — community servers via the browser, queues via the matchmaker — and -they should, because the server browser is already most of the way to done. +Locked constraints: -## Hard prerequisite: verified identity +- One authoritative Godot process hosts exactly one match. +- Solo 3v3 casual and ranked queues launch first. Party-shaped fields are + reserved in contracts, but party formation is deferred. +- Production matchmade traffic uses ticketed Steam Hosted Dedicated Server + SDR. Direct ENet remains first-class for local development, CI, LAN, + self-hosting, and community servers. +- The control plane is Go, PostgreSQL, and Redis, deployed on Kubernetes. + Agones owns game-server allocation and lifecycle. +- Infrastructure is provider-portable. Provider-specific cluster, network, + DNS, and secret-store configuration lives behind isolated deployment + overlays; application code never calls a provider allocation API. +- Launch game fleets run in Europe and North America. Placement is + latency-first and never silently violates the ping ceiling to shorten a + queue. +- Phase 8 features are opt-in. With allocated mode disabled, `ServerConfig`, + ENet, Docker Compose, and the existing community-server behavior remain + unchanged. -Ranked cannot ship before Phase 7's Steam auth tickets. +## 2. Steam and trust boundaries -Slot reclaim is currently keyed by **display name** (see -`--slot-reservation-seconds`, and the known-issues list in -`multiplayer-next.md`). A rating attached to a spoofable identity is worse -than no rating at all: it is trivially farmed, and it invites players to -invest in a ladder that cannot be defended. "Ranked is critical" therefore -*raises* the priority of Steam identity rather than routing around it. +Phase 7 verified Steam identity is a hard prerequisite. The current slot +reservation is keyed by display name, so no public queue or rating may rely on +it. -Casual queueing has a weaker requirement — it still needs stable identity for -abandon penalties and ban enforcement, but the cost of a compromise is lower. +Two Steam credentials have different purposes and must not be conflated: -## Architecture +1. A client obtains a single-use Web API ticket for backend login. Only the + secure backend calls `AuthenticateUserTicket`, checks the expected App ID + and identity string, and turns the result into a revocable session. A + client-supplied SteamID is never identity. +2. For a formed match, the game coordinator creates a short-lived SDR relay + ticket authorising one player to one hosted server. The client installs it + before connecting. The server separately verifies the signed match join + authorisation before admitting the player to the assigned roster slot. -Decided: **Steam for identity, a project-owned backend for everything else.** +Steam authentication/session tickets are single-use and their lifecycle must +include the appropriate cancel/end calls; identity is not valid until Steam's +asynchronous validation succeeds. Hosted SDR relay tickets are different: +they are short-lived, match/server/identity-scoped and deliberately cached for +reconnect. See [Steam authentication](https://partner.steamgames.com/doc/features/auth) +and the [`ISteamUserAuth` Web API](https://partner.steamgames.com/doc/webapi/isteamuserauth). -This reverses the "no backend" position stated in -[`TECH_STACK.md`](TECH_STACK.md) and `README.md`, which described the state -of the project before matchmaking was scoped. The dedicated server remains a -Godot export; the new service is separate from it. +Ticketed Hosted Dedicated Server SDR is the production transport because it +hides player/server IP addresses and authenticates, encrypts, and rate-limits +traffic. It also supplies relay routing that may improve the path. It requires +a real App ID, coordinator SDK/signing approval, certificates, and hosted +data-centre coordination with Valve; those are explicit release dependencies, +not assumptions. See [Steam Datagram Relay](https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay). -The alternative — Steam-native matchmaking (lobbies plus Leaderboards or User -Stats as the rating store) — was rejected on two grounds. Steam lobby -matchmaking has no real concept of a skill distribution to match against, and -Leaderboards are a display surface rather than a rating store with the -transactional guarantees a ladder needs. It would also permanently bind the -game to Steam, foreclosing other platforms. +### Trust table -### Components - -| Component | Runs where | Responsibility | +| Input | Trusted only after | May affect | | --- | --- | --- | -| Steam auth ticket validation | backend | Turn a client-supplied ticket into a verified SteamID via the Steamworks Web API. The only trusted source of identity. | -| Queue / matchmaker | backend | Hold queued players per playlist and region; form matches on rating proximity with a widening tolerance over wait time. | -| Rating store | backend (DB) | Per-identity, per-playlist rating and match history. Written only by the backend, never by a game client. | -| Server allocator | backend | Start a dedicated-server instance per formed match, hand its address to the matched clients, reclaim it on exit. | -| Dedicated server | Godot export | Unchanged simulation. Gains a mode where the roster is *assigned* rather than open, and reports a result at the end. | -| Game client | Godot | Queue UI, estimated wait, accept/decline, connect-on-assignment, post-match rating delta. | +| Steam Web API ticket | Backend validation for the expected App ID/identity and replay check | Backend session identity | +| Steam ping location + active-probe evidence | Backend verifies nonce/freshness and computes estimates; later compares with observed RTT | Placement only, never results | +| Queue/accept request | Auth, schema/rate-limit, revision and idempotency validation | That player's queue state | +| Join authorisation | Server signature, expiry, match/server/SteamID/slot and connection-generation validation | Initial admission or idempotent reclaim of that same slot | +| Gameplay input | Existing server framing, sequence, byte and rate validation | Authoritative simulation input only | +| Match result | Assigned server workload identity plus match/server binding | Transactional result/rating commit | -### What already exists and gets reused +Clients never submit ratings, outcome, penalty exemptions, server health, or +allocation state. A dedicated server never holds the Steam publisher key or +the coordinator root signing key. Keep the offline SDR CA separate from the +online leaf ticket key. The online key is exposed only through a narrowly +authorised signer backed by KMS/HSM-equivalent non-exportable storage; API, +matcher, allocator and game-server pods cannot read it. The signer accepts +only allocator-recorded assignments, audits every signature, and supports +overlapping-key rotation. -The server side needs less new work than it looks: +## 3. Control-plane architecture -- **`--max-matches=1`** already makes the server drain and `exit(0)` after a - single match. That is precisely the lifecycle a per-match allocator wants; - it was built for CI, and it generalises for free. -- **`ServerConfig`** is a single-source-of-truth flag table with strict - validation — new allocation flags are declared in one place and are - automatically parsed, type-checked, config-file-backed and documented. -- **`--min-players` / `--start-countdown` / `--slot-reservation-seconds`** - are the match-formation primitives, and they already count *roster* - members rather than raw peers. -- **`MatchNet`'s roster** already survives the lobby→match transition, which - is the structure an assigned roster slots into. -- **`MatchState`** already has a legal-transition table with wire-stable - integer values, so new lifecycle states append cleanly. +Use one repository and shared domain packages, with independently runnable +roles rather than independently designed microservices: -### What is genuinely new +| Role | Responsibility | +| --- | --- | +| API | HTTPS/WebSocket auth, profile, queue commands, status resync | +| Matcher | Atomic proposal formation from queue state | +| Allocator | Agones allocation, server registration, assignment delivery | +| Maintenance worker | Outbox delivery, season rollover, expiry, reconciliation | -- The backend service itself (process, deploy, DB, ops) — nothing like it - exists in this repo today. -- Server-authoritative **match results**: the dedicated server must report - the outcome to the backend over a channel a client cannot forge. This is - the first non-ENet/SDR network path in the project (see TECH_STACK's "no - HTTP layer" note, which this supersedes). -- An **assigned-roster** server mode: only the matched SteamIDs may take a - slot, replacing the current first-come model. -- Client-side queue UI and the accept/decline flow. +API replicas are stateless. Redis sorted sets provide the fast candidate +index, but Redis is never the durable allocation fence: asynchronous failover +can lose an acknowledged write. A matcher claims a proposal in a PostgreSQL +`SERIALIZABLE` transaction using a partial unique constraint that permits only +one active proposal/match participation per player. The transaction records +the proposal and participants before Redis cleanup; a stale Redis claim then +loses at PostgreSQL and is repaired from the durable record. PostgreSQL is the +source of truth for queue ownership, identities, sessions/revocations, +seasons, ratings, matches, participants, penalties, result receipts, audits +and the transactional outbox. Redis holds expiring presence, candidate +indexes, latency evidence, session/revocation caches and transient fan-out. +Losing the last acknowledged Redis write may delay/rematerialise a ticket or +force a session cache miss, but can neither resurrect a revoked session, split +a proposal nor corrupt a result/rating. -## Server orchestration and autoscaling +For launch, run the horizontally scaled control plane in one primary +Kubernetes region with a warm standby and tested restore path. Game fleets +remain regional in EU and North America. This avoids a premature multi-writer +database while keeping new-match control latency small relative to queue time. +An outage may pause new queues/allocations, but live matches must continue. +Targets are PostgreSQL RPO <= 5 minutes and control-plane RTO <= 30 minutes. -Requirement: game servers scale horizontally and automatically, spin up fast -on demand, serve exactly one match, and shut down — so cost is incurred only -while a match is being played. Docker and the existing CI gates must keep -working unchanged. +### Stable identifiers and state -### Why this is achievable: the server is already shaped for it +Contracts define opaque `player_id`, `queue_ticket_id`, `proposal_id`, +`match_id`, `server_id`, and `season_id`. Every mutating request has an +idempotency key, expected revision, and versioned schema. -Two properties of the current build make per-match allocation practical -rather than aspirational: +The durable/transient state path is: -- **The container is small.** The `server` image target is a slim - `ubuntu:24.04` runtime with three shared libraries and the exported - binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image. - Pulling it onto a fresh node is cheap. -- **Boot to listening is sub-second.** Measured on this repo's - `cosmicclash-server:latest`: **~870 ms** from container start to the - `server_started` log line, averaged over three runs, read from the - container's own clock. That measurement was taken under **x86_64 emulation - on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux - should be faster. Re-measure on the real target before setting timeouts. +``` +QUEUED -> PROPOSED -> ACCEPTED -> ALLOCATING -> PROCESS_READY + -> ASSIGNMENT_READY -> ASSIGNED -> CONNECTING -> LIVE + -> RESULT_PENDING -> COMPLETED + -> CANCELLED / EXPIRED / FAILED from the explicitly legal stages +``` -Combined with `--max-matches=1`, which already drains and `exit(0)`s after a -single match, the lifecycle the allocator needs mostly exists: start -container → serve one match → process exits → orchestrator reclaims. +The API publishes revisioned changes over one authenticated WebSocket. REST +GET endpoints are the recovery source after a disconnect or missed revision. +Restarting the client resumes an unexpired ticket/assignment instead of +creating another one. -### The cold-start tension, stated honestly +Assignments include match/server IDs, protocol and client build, server image +digest, playlist version, transport, region, expiry, a match-scoped join +authorisation, and either the SDR hosted-server material or an ENet endpoint. +The authorisation may be replayed only by the same Steam identity to reclaim +the same match/server/slot before expiry. Each successful connection advances +a server-owned generation and fences the prior connection; another identity, +server or slot is always rejected. This deliberately supports reconnect when +Steam or the control plane is temporarily unavailable. Incompatible +protocol/build/playlist versions never enter one proposal. -"Only pay during a match" and "a player never waits" are in tension. A server -must be listening *before* the matched players connect, so some cost always -precedes the match. Sub-second boot makes the gap small enough that a pure -scale-to-zero design is plausible — but the risk is not the container, it is -everything around it: image pull on a cold node, scheduler placement, and -network/port programming can each dwarf 870 ms. +## 4. Queue and placement policy -Recommendation: **scale to zero at the node level is the wrong target; scale -to zero at the match level is the right one.** Keep a small warm pool of -nodes sized to the current queue depth, and start a per-match container on -demand within it. The per-match process genuinely exists only for the match; -the node pool absorbs the cold-start variance. Revisit only if measured -allocation latency on real infrastructure shows the warm pool is unnecessary. +Each verified player may own at most one active queue ticket. Heartbeats are +sent every 10 seconds and queue presence expires after 30 seconds. Create, +cancel, resume, accept, decline, and expiry are atomic and retry-safe. -### Findings that block a naive implementation +The client submits its recent opaque Steam ping location plus nonce-bound +active-probe responses from each regional endpoint; it does not submit the RTT +used for placement. The backend validates a 30-second freshness window and +nonce, then uses the Steam coordinator SDK and probe timings to compute the +regional matrix. A predicted/observed discrepancy over 25 ms or 30% (whichever +is larger) in three matches within 24 hours quarantines the account's samples: +it may queue only in regions whose active probe independently remains under +the ceiling until five clean matches clear the quarantine. The matchmaker: -**Readiness cannot be detected from the log line.** Godot's stdout is -block-buffered when it is not attached to a TTY. Run the server image -detached without `-t` and `docker logs` shows **nothing at all** — the -`server_started` line does not appear even after 35 seconds, because the -buffer never flushes. An orchestrator readiness probe that greps for that -line will hang forever, and this was reproduced directly while measuring the -boot time above. Either probe the UDP socket instead, or make the server -flush explicitly. This also means container logs are not a reliable -observability channel for a short-lived match server; treat log shipping as -a separate problem. +1. Finds regions in which every proposed player has predicted RTT <= 100 ms. +2. Minimises the worst player's predicted RTT. +3. Breaks ties by total predicted RTT, then ready server capacity. +4. Widens rating tolerance with wait time, but never automatically widens the + 100 ms latency ceiling. -**One fixed port per container does not scale on a shared host.** `--port` -defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing -several matches onto one node needs either a port range allocated per -container, or one address per container. This is a UDP service, so the usual -HTTP ingress/L7 routing answers do not apply — the allocator must hand the -client a concrete `host:port`. +The target is regional observed p95 RTT <= 80 ms. Candidate formation is +deterministic: -**The match cannot start on a schedule the players do not control.** Today -the loop waits for `--min-players` then counts down. An allocated server is -told *which* identities to expect, and needs a **no-show timeout**: if a -matched player never connects, the server must abandon and exit rather than -sit idle burning the cost this design is trying to avoid. +- Anchor on the oldest compatible ticket (`enqueued_at`, then ticket ID). +- A ticket's rating tolerance is `min(400, 100 + 25 * floor(wait_seconds/30))` + rating points. A pair is compatible only when its absolute rating difference + is within both tickets' tolerance. Provisional players use the same stored + rating; their high RD changes ratings faster, not eligibility arithmetic. +- Every candidate set contains the anchor. Choose the lexicographically + smallest tuple `(worst_RTT, total_RTT, rating_range, -sum(wait_seconds), + sorted_ticket_IDs)`, so latency dominates once the oldest player anchors + fairness and every candidate satisfies the widening rule. +- Partition humans exhaustively into legal teams and minimise absolute team + mean-rating difference, then maximum opposing-player difference, then use + lexical player IDs. Bots fill remaining casual slots after humans are + assigned. -### Keeping Docker and CI green +Every selected human receives a 10-second proposal before allocation. Ranked +always selects six. Casual requires six before the anchor reaches 60 seconds; +afterward it selects the largest compatible human count from six down to two, +with at least one human per team, and displays teams/bot slots. Unanimous +acceptance by the selected humans advances. -The existing gates must not regress. `make verify-phase6` builds the export, -runs it in Compose, joins two headless clients and asserts both saw both -goals and that the arena rotated between matches; `make verify-enet-integration` -runs the source-build ENet matrix. Both depend on current behaviour: -`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot -assignment, and uses `--max-matches=2` to prove rotation. +An explicit proposal decline cancels that player's ticket and applies a +30-second casual or 2-minute ranked cooldown. A timeout applies 60 seconds in +casual or 5 minutes in ranked. Three ranked proposal declines/timeouts within +30 minutes apply 15 minutes. Players who accepted return with their original +`enqueued_at` and precedence when another player declines, times out, or the +allocation fails. Ordering ties use ticket ID. A ranked initial-connect +no-show after accepting uses the ranked abandon cooldown ladder but never a +rating loss because no rated match began. -The rule that keeps them passing: **every allocation feature is opt-in via a -new `ServerConfig` flag whose default reproduces today's behaviour.** An -assigned roster, a no-show timeout and result reporting must each be inert -unless explicitly enabled. `ServerConfig` is built for exactly this — a flag -declared once is parsed, validated, type-checked, config-file-backed and -documented — and `tests/cases/` can cover the new parsing without a live -server. A second Compose file should cover the allocated-match path rather -than mutating the Phase 6 one, so the community-server model stays tested -alongside the matchmade one. +### Casual -### Open questions +- Target six humans in 3v3. +- After 60 seconds, a match may start with at least two humans, one on each + team, and fill the other slots with server bots. +- Human backfill may replace a bot/disconnected slot only at a kickoff + boundary. +- Backfill is a separate 10-second opt-in proposal showing score, time + remaining, team and slot. Declining/timing out carries no cooldown. A + backfilled player receives no hidden-rating update or abandon penalty for + that match; acceptance removes their queue ticket only when assignment is + ready. Choose the oldest ordinary casual ticket that meets the same build, + region <=100 ms and current anchor-tolerance rules for the vacated human + slot; ties use ticket ID. +- An original casual participant gets 30 seconds to reconnect; leaving after + that applies a 60-second queue cooldown. The match's ordinary hidden-rating + result still applies, with no extra rating penalty. +- An accepted casual initial-connect no-show gets the same 60-second cooldown. + The match proceeds with a bot only if at least one human connected on each + team; otherwise it cancels and restores every innocent ticket with original + precedence. +- Casual has a separate hidden Glicko-2 rating used only for matching. -- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this - game-server lifecycle), Nomad, or direct cloud-API container starts. Not - chosen. Agones is the strongest default because it models allocation, - readiness and per-match lifetime natively. -- **Port strategy** — port range per node versus one IP per match. -- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 6–10 match - processes per modern core and 150–250 MB RSS each. That estimate predates - any allocation work and should be re-measured under real concurrency - before it sizes a bill. -- **Draining and deploys.** How a server version rolls out without killing - matches in flight. +### Ranked -## Casual vs ranked +- Exactly six verified humans; never bots and never backfill. +- Solo queue only at launch. +- Only `ArenaRegistry` entries with `"random": true` are eligible. Elevated + goals remain excluded until the trained-policy restriction is lifted. +- A reconnecting player has 60 seconds to return using the existing + assignment. After that, that player receives a loss regardless of the final + team result and a rolling seven-day cooldown: 5 minutes, 15 minutes, 1 + hour, then 24 hours. +- Delivery failure is not match-integrity failure. A healthy completed match + remains rated while its result waits for the control plane. Rating is + suppressed only when the authoritative roster, simulation or result is + unavailable/corrupt, or a measured regional incident prevented fair play. -They are different playlists, not a difficulty toggle, and their rules -diverge in ways that affect the server: +## 5. Rating and seasons -| | Casual | Ranked | -| --- | --- | --- | -| Rating | Hidden, used only for matching | Visible, with tiers | -| Backfill on disconnect | Yes — keep the match playable | No — the match is rating-bearing and must not change shape mid-way | -| Bots filling empty slots | Acceptable (`--fill-bots` exists) | Never | -| Abandon penalty | Light (short queue cooldown) | Real (rating loss, escalating cooldown) | -| Arena selection | Full rotation | Restricted set, so a variant nobody has practised can't decide a ladder match | -| Party / premade | Unrestricted | Constrained by size and rating spread | +Use canonical Glicko-2 independently per playlist with initial rating 1500, +RD 350, volatility 0.06, scale constant 173.7178 and tau 0.5. Updates are +immediate per committed match rather than globally batched. Before an update, +advance inactivity by whole 24-hour rating periods since the player's last +rated match using `phi = min(350/173.7178, sqrt(phi^2 + sigma^2 * periods))`. -Note the arena constraint interacts with an existing rule: elevated-goal -variants are Free-Play-only until a checkpoint trained on -`training_elevated.tscn` is promoted (`arena_registry.gd`). Ranked's arena -set should be drawn from `"random": true` arenas only. +For each player `i`, transform every opposing human's pre-match rating/RD to +`mu_j`/`phi_j` and use the canonical equations +`g(phi)=1/sqrt(1+3*phi^2/pi^2)` and +`E=1/(1+exp(-g(phi_j)*(mu_i-mu_j)))`. Ranked's three opponent contributions +use `w=1/3`; casual uses `w=1/N` for the `N` opposing humans, ignoring bots. +Thus human contributions total exactly one match in both sums: -## Open questions +``` +v^-1 = sum(w * g(phi_j)^2 * E_j * (1-E_j)) +Delta = v * sum(w * g(phi_j) * (s-E_j)) +``` -- **Rating algorithm.** Glicko-2 is the default recommendation over plain - Elo: it models rating *uncertainty*, which matters enormously for a small - launch population where most players have few games. Not yet decided. -- **Team rating from individual ratings.** How a 3v3 match's outcome - distributes across six players is a separate design problem from the - rating system itself. -- **Server cost.** Allocated servers cost real money per match, unlike - community servers that players host themselves. `README.md`'s original - note about a subscription to fund servers is suddenly load-bearing again. - Population size and match length set the bill; this needs a number before - launch, not after. -- **Region / ping policy.** How much rating tolerance to trade for latency, - and whether cross-region is ever allowed at low population. -- **Placement matches** and whether ranked has a soft reset per season. -- **Backend language and hosting.** Not chosen. It does *not* have to be C# - despite the original README framing — that framing was aspirational and - predates every real decision in this project. +Then run the canonical Glicko-2 volatility iteration and rating/RD update. If +a player has no opposing human, the match is unrated for that player. `s` is 1/0 for +the authoritative winner/loser and 0.5 only for a completed draw. Overtime is +an ordinary win/loss. A ranked abandoner gets `s=0` regardless of the final +team result; non-abandoning players use the authoritative final result. +Cancelled or integrity-failed matches do not update anyone. -## Explicitly out of scope +Lock all six participant rows in lexical player-ID order and compute every +new value from the same immutable pre-match snapshot inside one serializable +transaction. This prevents update-order bias and concurrent double updates. +Golden vectors include canonical one-player examples plus symmetric/asymmetric +3v3, draw, overtime, abandon, inactivity and concurrent-result fixtures. -Tournaments, in-game leaderboards beyond a personal rank display, -cross-platform play with non-Steam identity providers, and spectator/observer -tooling for ranked matches. None are precluded by this design; none are -launch scope. +The first ten ranked matches are provisional. Ranked exposes +backend-derived tiers; casual rating remains hidden. Stored Glicko values, not +tier labels or client calculations, are authoritative. Ranked seasons last 12 +weeks. Ranked rollover sets +`rating = 1500 + 0.75 * (rating - 1500)`, raises RD to at least 200 (capped at +350), preserves volatility/history, and is an exactly-once idempotent +transaction. Casual rating is continuous and never season-reset. + +Rating, penalty, match completion, participant records and outbox events are +committed in that transaction. Duplicate identical server results succeed +idempotently. A conflicting result changes nothing and pages an operator. + +## 6. Game-server allocation and lifecycle + +Use provider-portable Kubernetes manifests and Agones. Create one versioned +Fleet per compatible build and region; labels identify region, protocol, +transport, and image/build. Allocate atomically with `GameServerAllocation`. +Agones, rather than bespoke allocator code, owns selection and lifecycle. See +[GameServerAllocation](https://agones.dev/site/docs/reference/gameserverallocation/). + +The Godot process talks to the Agones REST sidecar through a small adapter +that is a no-op when `AGONES_SDK_HTTP_PORT` is absent. This preserves native, +Compose, and CI operation and works with the Agones local SDK emulator. See +[Agones client SDKs](https://agones.dev/site/docs/guides/client-sdks/). + +Agones has two distinct readiness points; conflating them is a deadlock because +`GameServerAllocation` selects a Ready server and only then attaches the match +metadata: + +1. The PID-1 supervisor queries the local Agones SDK for the assigned dynamic + port/address, exports `SDR_LISTEN_PORT` plus `SDR_IP=public-address:port` + (or the ENet equivalent), and launches Godot. Godot validates static config, + binds the socket and starts Health calls. +2. Godot calls Agones `Ready()` after the process is genuinely listening. + This is **process-ready** only; never infer it from detached stdout. +3. `GameServerAllocation` atomically changes that Ready server to Allocated + and supplies the signed roster/non-secret match configuration as metadata. +4. The server watches the GameServer, observes Allocated metadata, verifies + manifest signature/build/protocol/server binding, registers its hosted SDR + address, and calls the backend `assignment_ready` endpoint. +5. Only after `assignment_ready` does the allocator mint relay/join tickets and + expose the assignment to clients. The server accepts only assigned + identities/slots; each selected human has 30 seconds to connect. +6. Run one authoritative match with Health calls independent of the simulation + loop. Submit the canonical result using the bound workload identity. +7. Write the signed result hash/payload to the backend and to a non-secret + Agones annotation, remain Allocated, and retry until the backend durably + acknowledges it. The maintenance worker reconciles the annotation after an + API outage. `RESULT_PENDING` pages at 5 minutes and requires operator review + at 30 minutes; it never silently becomes unrated. +8. After acknowledgement call `Shutdown()` and exit. Invalid/empty allocations + shut down immediately. An allocated GameServer is not recycled to Ready. + +Agones supplies dynamic host ports so multiple isolated matches can share a +node; HTTP ingress is not involved in gameplay routing. Hosted SDR additionally +requires Valve approval for every provider/location, a valid `SDR_POPID`, +public-IP/unsolicited-UDP reachability, provider firewall/NAT validation, +per-location certificates and coordinator trust. Use an Agones dynamic or +passthrough mapping whose externally reported port is the `SDR_IP` port while +the process binds `SDR_LISTEN_PORT`; test SDR and ENet mappings separately. +Credentials arrive through runtime secret mounts, never allocation metadata, +arguments, logs or images. + +Result authentication uses a projected, pod-bound service-account token with +a dedicated audience and one service account per workload class. The backend +validates the configured cluster issuer/JWKS, audience, expiry, namespace, +service account, bound pod UID and allocator-recorded GameServer UID, then +checks that GameServer/match binding in PostgreSQL. Issuers and trust roots are +allowlisted and rotated explicitly for every cluster/provider. A one-match +server credential issued after this attestation is an acceptable equivalent. + +### Warm capacity and density + +Both launch regions are active whenever their queues are enabled. Each active +region maintains at least two Ready processes distributed across at least two +on-demand nodes/failure domains; the minimum node floor is therefore two, not +one. Pre-pull current and rollback images. A queue/proposal-aware +FleetAutoscaler adjusts capacity above that Ready floor. **Allocated** process +count may fall to zero; Ready processes do not. An administratively disabled +region may scale both nodes and Fleet to zero and is excluded from placement. +See [Agones FleetAutoscaler](https://agones.dev/site/docs/reference/fleetautoscaler/). + +Do not run live matches on interruptible nodes. N+1 means loss of the largest +single node still leaves two Ready slots plus sufficient headroom for already +Allocated matches; certify it in the node-loss test. Set pod requests, limits +and node caps only after native x86_64 benchmarks of boot time, p99 +CPU/RSS/network and 60 Hz tick behavior, retaining 30% headroom. The current +6–10 processes/core and 150–250 MB estimates are not sizing data. + +Godot/GDScript cannot intercept SIGTERM, so Phase 8 adds a small Go PID-1 +supervisor. It starts Godot, traps TERM, and sends a per-pod-token-authenticated +localhost drain request to an allocated-mode-only Godot control socket. Godot +refuses new admissions and reports completion to the supervisor. Kubernetes +uses a 300-second termination grace period; at 285 seconds the supervisor +forces an infrastructure-failure abort so the pod cannot hang forever in +overtime. A PodDisruptionBudget and Agones-aware drain prevent voluntary +eviction of Allocated servers. Planned releases create a new Fleet, route new +allocations to it and wait for old Allocated count zero without sending TERM. +Unexpected node failure cannot be made graceful and follows the integrity +failure/refund path. + +Live matches continue through an API/control-plane delivery outage. A valid +result is retried/reconciled as above; only loss of authoritative match +integrity suppresses rating. + +## 7. Security and operational baseline + +The threat model must cover forged clients, ticket replay, duplicate queueing, +result forgery, queue manipulation, packet floods, botting, server compromise, +insider access, DDoS, vulnerable dependencies, and denial-of-wallet attacks. + +Minimum controls: + +- Non-root containers, read-only root filesystems, dropped Linux + capabilities, RuntimeDefault seccomp, resource limits, restricted Pod + Security admission, and least-privilege service accounts/RBAC. +- Private PostgreSQL/Redis; default-deny ingress and egress with explicit + NetworkPolicy allowances. See the [Kubernetes application security + checklist](https://kubernetes.io/docs/concepts/security/application-security-checklist/) + and [NetworkPolicy](https://kubernetes.io/docs/reference/kubernetes-api/networking/network-policy-v1/). +- Encrypted databases/backups, externally populated Kubernetes Secrets, + documented rotation, and no credentials in Git, images, command lines, or + telemetry. +- Per-account/IP API limits, request/body/schema limits, replay and duplicate + detection, generic public errors, allocation quotas, and budget alerts. +- Put HTTPS/WebSocket traffic behind a provider-portable edge contract + implemented by each infrastructure overlay: managed volumetric DDoS + absorption, WAF/rate rules, TLS termination, origin-only ingress and health + checks. Enforce authenticated queue admission, per-account connection caps, + WebSocket handshake/message/idle limits, bounded fan-out and overload + shedding. In degraded mode reject new login/queue/allocation work while + preserving result ingestion and all live matches. +- Images pinned by digest, SBOM generation, dependency/image scanning, signed + releases, admission-time signature verification, and a critical-patch SLA. +- Structured audit events for auth, queue transitions, allocation, roster + rejection, result conflict, penalty, season rollover, and operator action. + +## 8. SLOs, observability, and release gates + +Launch SLOs: + +| Measure | Target | +| --- | --- | +| Eligible predicted RTT | <= 100 ms for every player | +| Regional observed RTT | p95 <= 80 ms | +| Unanimous acceptance to `assignment_ready` | p95 <= 5 s, p99 <= 10 s with warm capacity | +| Published assignment to successful connection | p95 <= 5 s | +| Successful allocation and durable result | >= 99.9% | +| Certified server density | No tick backlog, with 30% resource headroom | +| Control-plane API under load | p95 <= 250 ms | + +Dashboards and alerts cover queue depth/wait, rating spread, predicted versus +observed RTT, proposals/declines, allocation latency/failure, Ready capacity, +image pulls, connection/no-show, physics overruns, crashes, abnormal packet +rates, result lag/conflicts, abandons, and cost per completed match. Correlate +all components with queue/proposal/match/server IDs, but never log auth or +relay tickets. + +Testing layers: + +- Go unit, race, fuzz, property, migration, and concurrency tests. +- Fake Steam verifier and fake allocator for deterministic CI. +- A second allocated-server Compose flow; never mutate + `compose.phase6-smoke.yml`. +- Disposable `kind` + Agones integration for readiness, dynamic ports, + health/no-show shutdown, allocation races, multiple matches per node, + draining, and rollback. +- Network/chaos cases for 100 ms RTT, jitter/loss, client/backend/matcher + restart, game-pod death, node drain, Redis failover, and control-plane loss. +- Load test at least 10,000 queued clients, 100 proposals/second, and forecast + launch concurrency x2 while preserving correctness and API latency. +- Provider migration rehearsal: restore data and deploy both regional fleets + on a second provider whose EU/NA locations already have Valve approval, + POP/certificates, public UDP/firewall verification and coordinator trust; + switch new allocations, drain the old provider, and terminate no live match. + +Release order is development -> internal -> casual canary -> full casual -> +provisional ranked -> full ranked. Each promotion requires its SLO/security +gates, rollback rehearsal, EU and North America playtests, a measured cost +model, and unchanged `make verify-phase6` and +`make verify-enet-integration`. + +## 9. Explicitly out of scope + +Parties/premades, tournaments, ranked spectators, public global regions, +non-Steam identity providers, and a global leaderboard beyond personal rank +display are not launch scope. The contracts reserve party identity and avoid +Steam-specific database primary keys so those additions do not require a +destructive redesign. diff --git a/docs/TECH_STACK.md b/docs/TECH_STACK.md index 84cc34fb..9e5db897 100644 --- a/docs/TECH_STACK.md +++ b/docs/TECH_STACK.md @@ -189,8 +189,9 @@ Python, no .NET, no network." Keeping the shipped game GDScript-only (no ## Planned, not yet built -- **A matchmaking backend service** — Steam auth ticket validation, casual - and ranked queues, a rating store, and per-match dedicated-server - allocation. Language and hosting are undecided. This is a 1.0 launch - blocker and the single largest departure from "one Godot project, no - backend". See [`MATCHMAKING.md`](MATCHMAKING.md). +- **A Go matchmaking control plane** — independently runnable API, matcher, + allocator and maintenance roles backed by PostgreSQL and Redis, deployed on + provider-portable Kubernetes with Agones-managed game fleets. The cloud + provider remains deliberately replaceable; the application stack is locked. + This is a 1.0 launch blocker and the single largest departure from "one + Godot project, no backend". See [`MATCHMAKING.md`](MATCHMAKING.md). diff --git a/multiplayer-next.md b/multiplayer-next.md index 87d20f5a..7423ef04 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1,93 +1,141 @@ # Multiplayer — next work -Short, current checklist for online multiplayer. Historical design decisions, -implementation evidence, and completed work stay in -[`multiplayer-todo.md`](multiplayer-todo.md). +Short, current checklist for online multiplayer. Historical decisions, +implementation evidence and task-level acceptance criteria stay in +[`multiplayer-todo.md`](multiplayer-todo.md). Phase 8 architecture and locked +product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). -## Release blockers +## Existing release blockers -- [ ] **Phase 4 playtest:** a human playtest at roughly 100 ms RTT. Confirm - that ship and ball interaction feel local and contact corrections feel like - bumps rather than glitches. -- [ ] **Phase 5 session:** complete a real 3v3 match with a mid-match - disconnect and late joiner. -- [ ] **Phase 6 external check:** run the exported Docker server and clients - from separate machines over the internet, then play a full match. Keep this - controlled-only until Steam identity is complete. +- [ ] **Phase 4 playtest:** play at roughly 100 ms RTT; confirm ship/ball + interaction feels local and contact corrections read as bumps, not glitches. +- [ ] **Phase 5 session:** finish a real 3v3 match with a mid-match disconnect + and late joiner. +- [ ] **Phase 6 external check:** play the exported Docker server from separate + internet machines. Keep the check controlled until verified identity lands. -## Phase 7 — Steam, identity, discovery +## Phase 7 — production Steam prerequisite -- [ ] Obtain the pinned GodotSteam client/server builds and Steamworks SDK - access described in [`STEAM.md`](STEAM.md). -- [ ] Run `make verify-steam-templates` with the custom executables and fix - any custom-template failures. -- [ ] Validate a two-account Steam SDR host/join using the existing explicit - `NetworkManager` Steam transport. ENet direct-IP must keep passing its smoke - test. -- [ ] Build the Steam server browser: internet, LAN, favourites, and history. -- [ ] Add Steam auth tickets, verified Steam identity in the roster, and a - persistent ban list. This fixes the slot-reclaim security issue below. +- [ ] Obtain the pinned GodotSteam client/server builds and Steamworks SDK; + pass `make verify-steam-templates` without weakening ENet verification. +- [ ] Validate two real accounts through the explicit Steam transport and + build Internet/LAN/favourites/history server-browser views. +- [ ] Add single-use auth tickets, asynchronous server validation, verified + Steam identity, identity-keyed reconnect and persistent bans. +- [ ] Obtain the real App ID, publisher key, coordinator SDK/signing approval, + certificates and Hosted Dedicated Server data-centre support from Valve. +- [ ] Implement ticketed Hosted Dedicated Server SDR routing, ticket install, + reconnect and expiry. Preserve direct ENet for local/CI/community servers. -## Phase 8 — casual and ranked matchmaking (1.0 launch blocker) +## Phase 8 — architecture, contracts and durable data -Design and reasoning: [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). This is a -different server model from the community-server one that exists today — -players queue, a matchmaker groups them, and a server is allocated per match. -Phase 7's Steam auth tickets are a hard prerequisite: a rating attached to a -spoofable identity is worse than no rating. +- [ ] Lock the Go/PostgreSQL/Redis, Kubernetes/Agones, SDR, EU/NA and + provider-portability ADR; define measurable launch SLOs. +- [ ] Publish versioned OpenAPI/WebSocket contracts, stable IDs, legal state + transitions, revisions and idempotency semantics. +- [ ] Add PostgreSQL queue ownership/active-participation fences, durable + domain migrations/outbox and Redis indexes/TTLs; lost Redis writes must not + split a proposal or corrupt durable state. +- [ ] Define assignment compatibility and opt-in `ServerConfig` flags whose + defaults reproduce the community-server path. -- [ ] Decide the rating algorithm (Glicko-2 recommended over Elo for a small - launch population) and how a team result distributes across individuals. -- [ ] Choose the backend language and hosting, and cost out allocated servers - per match at expected population. -- [ ] Stand up the backend: Steam auth ticket validation via the Steamworks - Web API, queue, rating store, server allocator. -- [ ] Add an assigned-roster server mode so only matched SteamIDs may claim a - slot, replacing the first-come model. -- [ ] Add server-authoritative match result reporting to the backend over a - channel a client cannot forge. -- [ ] Client queue UI: playlist select, estimated wait, accept/decline, - connect-on-assignment, post-match rating delta. -- [ ] Casual and ranked playlist rulesets (backfill, bots, abandon penalties, - arena restriction — see the comparison table in the design doc). +## Phase 8 — identity and security -Server orchestration (same phase — the servers must autoscale and bill only -for the duration of a match): +- [ ] Validate Steam Web API tickets only in the secure backend; issue + revocable sessions and reconnect-safe match/identity/slot authorisations + with server-owned connection-generation fencing. +- [ ] Authenticate results with pod/GameServer-bound workload identity; make + identical duplicates idempotent and conflicting results inert/alerting. +- [ ] Complete the threat model for forgery, replay, queue/flood/bot abuse, + workload/insider compromise, DDoS, supply chain and denial-of-wallet. +- [ ] Enforce restricted workloads/RBAC/networks/private stores/backups/secrets; + isolate SDR signing behind an audited non-exportable signer and add + volumetric edge defense, WebSocket limits and overload shedding. +- [ ] Pin, scan, SBOM and sign artifacts; verify signatures at admission and + document the critical vulnerability SLA. -- [ ] Choose the orchestrator (Agones on Kubernetes is the recommended - default; it models allocation, readiness and per-match lifetime natively). -- [ ] Fix readiness detection. Godot's stdout is block-buffered off a TTY, so - `server_started` never appears in `docker logs` for a detached container — - a log-grep readiness probe hangs forever. Probe the UDP socket, or flush. -- [ ] Support more than one match per host: a per-container port from a range, - or one address per match. `--port` defaults to 7777 and the Dockerfile - hardcodes `EXPOSE 7777/udp`. -- [ ] Add a no-show timeout so an allocated server that never fills abandons - and exits instead of idling at cost. -- [ ] Re-measure boot-to-listening on native x86_64 Linux. The repo's current - figure is ~870 ms, measured under emulation on arm64 — a pessimistic bound. -- [ ] Re-measure the SERVER.md sizing estimate (6–10 processes/core, - 150–250 MB RSS) under real concurrency before it sizes a bill. -- [ ] Keep `make verify-phase6` and `make verify-enet-integration` green: - every allocation feature is opt-in via a `ServerConfig` flag defaulting to - today's behaviour, with a second Compose file for the allocated path rather - than mutating `compose.phase6-smoke.yml`. +## Phase 8 — queues, playlists and rating -## Known issues to resolve before public hosting +- [ ] Add one PostgreSQL-owned queue ticket/player with 10 s heartbeat, 30 s + expiry, Redis candidate cache and restart/failover repair. +- [ ] Validate opaque Steam ping locations and nonce-bound probes server-side; + require <=100 ms, enforce discrepancy quarantine and the locked widening/ + region/team tie-break rules. +- [ ] Send 10 s proposals to every selected human: ranked six, relaxed casual + two to six with disclosed bots; enforce exact cooldown and queue-precedence + behavior. +- [ ] Fence proposals/participants in a PostgreSQL serializable transaction; + prove loss of an acknowledged Redis write cannot split players. +- [ ] Casual: target 3v3 humans, after 60 s allow >=2 humans (one/team) plus + bots, kickoff-only human backfill and no backfill loss/decline penalty. +- [ ] Ranked: exactly six humans, solo-only, no bots/backfill, random-enabled + non-elevated arenas only, 60 s reconnect grace and escalating abandons. +- [ ] Implement the documented exact Glicko-2 equations, fractional 3v3 + weights, inactivity/update locking/golden vectors and ten provisional games. +- [ ] Add ranked-only exactly-once 12-week soft seasons; distinguish retryable + result-delivery outages from match-integrity failures and rating exemptions. -- [ ] Slot reclaim is currently keyed by display name, so someone can take a - disconnected player's reserved slot. Do not expose public servers before - verified Steam identity lands. -- [ ] Investigate occasional input loss during a long server stall; the - existing sequence resync recovers it, but transport delivery is variable. +## Phase 8 — Agones and regional server capacity + +- [ ] Add portable EU/NA Agones Fleets with provider edge/network/secret and + Valve-approved SDR POP/certificate/public-UDP overlays. +- [ ] Add the local-safe Agones adapter and separate process-ready (listen then + Ready) from assignment-ready (Allocated manifest verified and registered). +- [ ] Allocate from Ready by region/build/protocol/transport; use separately + verified ENet and SDR dynamic/passthrough port mappings. +- [ ] Deliver/verify the signed roster after allocation and expose client + tickets only after backend `assignment_ready`. +- [ ] Keep >=2 Ready processes across >=2 on-demand nodes/failure domains per + queue-enabled region; only Allocated count may fall to zero. +- [ ] Spread on-demand capacity across zones with N+1 headroom; do not place + live matches on interruptible nodes. +- [ ] Benchmark native x86_64 boot, p99 CPU/RSS/network and tick health; set + requests/limits and node density from measurements plus 30% headroom. +- [ ] Add 30 s no-show handling, Go PID-1 TERM/drain supervision, PDB/Fleet + drain, signed result annotation/retry, RPO <=5 m and RTO <=30 m. +- [ ] Rehearse migration only after the second provider's EU/NA locations have + Valve approval, POP/certs, public UDP/firewall and coordinator trust. + +## Phase 8 — client and recovery + +- [ ] Build queue/proposal/allocation/connect/rating UI with explicit latency, + capacity, expiry and recovery states. +- [ ] Use one authenticated revisioned WebSocket plus REST resync; resume a + valid ticket/assignment after restart rather than duplicating it. +- [ ] After assignment-ready, install SDR ticket and send reconnect-safe join + authorisation in `hello`; fence old connections and retain ENet behavior. +- [ ] Display only backend-authoritative provisional rank/tier/delta, abandon + status and season time; clients perform no rating calculation. + +## Phase 8 — operations and release gates + +- [ ] Correlate queue→result with IDs and add dashboards/alerts for SLOs, + security, failures and cost without logging credentials. +- [ ] Add Go race/fuzz/property/migration/concurrency coverage plus fake Steam + and fake allocation for offline deterministic CI. +- [ ] Add an independent allocated-server Compose flow; do not mutate + `compose.phase6-smoke.yml` or weaken either existing Make gate. +- [ ] Add disposable `kind`/Agones integration, 100 ms network/chaos cases and + proof that infrastructure failures cannot punish players. +- [ ] Load-test >=10,000 queued clients, >=100 proposals/s and forecast launch + concurrency x2 while holding API p95 <=250 ms and allocation correctness. +- [ ] Record cost per completed match, budget/denial-of-wallet controls and + deploy progressively: internal → casual canary → casual → provisional + ranked → ranked, with EU/NA playtests and rollback gates. + +## Known issues before public hosting + +- [ ] Replace display-name slot reclaim with verified Steam identity. +- [ ] Investigate occasional transport input loss during a long server stall. - [ ] Fix the remaining `_broadcast_snapshot` packet-send stderr race. ## Decide after the latency playtest -- [ ] Decide whether client-only, contact-cohort shadow physics is worthwhile - for the remaining prediction weakness. +- [ ] Decide whether client-only contact-cohort shadow physics is worthwhile. ## Explicitly deferred -120 Hz simulation, latency-gap measurement, audio hooks, and split-screen are -not part of the current multiplayer release path. +Parties/premades, tournaments, ranked spectators, non-Steam identity, +additional global regions, global leaderboards, 120 Hz simulation, +latency-gap measurement, audio hooks and split-screen are not in the launch +path. diff --git a/multiplayer-todo.md b/multiplayer-todo.md index f3d4d50f..1142736b 100644 --- a/multiplayer-todo.md +++ b/multiplayer-todo.md @@ -14,13 +14,13 @@ Everything below is written so an agent (or a person) can pick up a single numbe The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 0–5 contain no unfinished tasks. -**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.20 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three items there are findings rather than plans, and each would break a naive implementation: +**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.53 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three findings would break a naive implementation: | # | Finding | Why it bites | |---|---|---| -| 8.8 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | An orchestrator readiness probe that greps the log hangs forever | -| 8.9 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches cannot share a host; being UDP, L7 ingress routing does not apply | -| 8.18 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | Allocation work trivially regresses `verify-phase6` unless every new feature defaults to today's behaviour | +| Task 8.28 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | Process-ready must be an explicit Agones call after static validation/listen; post-allocation assignment-ready is separate and neither uses a log grep | +| Task 8.29 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches need Agones dynamic UDP/SDR ports; L7 ingress does not route this traffic | +| Task 8.48 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | The allocated flow needs its own fixture so Phase 6 behavior and invocations stay unchanged | ### Blocking sign-off — the work exists, the verification does not @@ -50,7 +50,7 @@ C is the one to plan around: it is fixed for free by task **7.4** (Steam auth ti ### Unstarted phases - **Phase 6 external gate:** run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until **C** is fixed. -- **Phase 7 — Steam transport, browser, identity** (5 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, auth tickets, and bans await a project-owned Steamworks App ID. Carries the fix for **C**. +- **Phase 7 — Steam transport, browser, identity and production SDR** (8 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, verified tickets, bans, production credentials and ticketed Hosted Dedicated Server SDR await a project-owned Steamworks App ID and Valve coordination. Carries the fix for **C** and is the hard prerequisite for Phase 8. Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into something another person can host. Phase 7 is the next block because Steam identity is required before public exposure. @@ -69,7 +69,7 @@ Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into some | 1 | **Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation.** | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. | | 2 | **Dedicated servers only.** Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. | | 3 | **ENet first**, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and **must never become the degraded path**. | -| 4 | **No custom backend.** | Steam's `ISteamGameServer` master-server listing covers discovery, `ISteamMatchmakingServers` covers the in-game browser, and Steam auth tickets cover identity and ban state. `README.md`'s C# backend stays unstarted. | +| 4 | **Community discovery uses no custom backend; superseded for queued play by Phase 8.** | Steam's server APIs remain enough for the community browser. Casual/ranked queues, durable ratings, allocation and authoritative results require the project-owned Go control plane specified in `docs/MATCHMAKING.md`; it does not replace the browser or direct-IP path. | ### 1.2 Rejected alternatives @@ -1135,6 +1135,9 @@ New `--role=client-reconnect` grades the returning player: not a spectator, owns | 7.3 `[D:7.2]` `[P]` | `server_browser.tscn` via `ISteamMatchmakingServers` | Internet, LAN, favourites and history lists all populate | | 7.4 `[D:7.2]` `[P]` | Auth tickets in `hello` → `BeginAuthSession`; Steam identity in the roster; persistent ban list | Ownership, VAC and ban state verified server-side | | 7.5 `[D:7.2]` `[P]` | Feature-gate every Steam call behind `OS.has_feature("steam") and ClassDB.class_exists("SteamMultiplayerPeer")`; verify the ENet path end to end | Non-Steam build is fully functional | +| 7.6 `[D:7.4]` | Separate single-use tickets for backend Web API login and game-server auth; wait for Steam's asynchronous validation and cancel/end every ticket session | Replayed, cancelled, wrong-App-ID and not-yet-validated identities cannot enter a roster or queue; no client-supplied SteamID is trusted | +| 7.7 `[D:7.1]` `[P]` | Obtain the production App ID, publisher key, SDR coordinator SDK/signing approval, certificates and hosted-data-centre support from Valve | Production prerequisites and rotation owners are recorded; no Spacewar credential or development certificate can reach a release build | +| 7.8 `[D:7.6,7.7]` | Ticketed Hosted Dedicated Server SDR: routing registration, coordinator-issued player→server relay tickets, client ticket installation, reconnect and expiry | Two real accounts complete and reconnect to an assigned dedicated match through SDR; server/player IPs are not exposed; ENet gates remain green | > **The transport interface is written here, not in Phase 1.** Eight virtual methods (`begin_auth`, `advertise`, `get_identity`, `supports_server_browser`…) designed against an API nobody on the project has used will be wrong. Write `NetworkManager._make_peer()` concretely in Phase 1 and extract the boundary once there are two real implementations. Locked decision 3 guarantees the ENet path is never deleted, so there is no migration risk in waiting. @@ -1155,86 +1158,107 @@ and players find it by IP or (7.3) the server browser. Matchmaking makes the **allocated for that one match** and destroyed after. Both models ship; they are different playlists, not a replacement. -**Hard dependency on 7.4.** Slot reclaim is keyed by display name today. A -rating attached to a spoofable identity is farmed trivially, so ranked cannot -ship before Steam auth tickets land. Casual queueing needs 7.4 too, for -abandon penalties and ban enforcement, but degrades more gracefully. +**Hard dependency on 7.6 and 7.8.** Slot reclaim is keyed by display name +today. A rating attached to a spoofable identity is farmed trivially, so no +queue ships before single-use verified identity lands. Production allocation +also depends on the ticketed Hosted Dedicated Server SDR route; ENet remains +the local/CI/community transport, not a silent production fallback. -#### 8a — Backend service +#### 8A — Architecture, contracts and data | # | Task | Acceptance | |---|---|---| -| 8.1 | Choose backend language, hosting and datastore. **Not C# by default** — that framing predates every real decision here | Written up with the rejected alternatives, as §1 does for the client decisions | -| 8.2 `[D:7.4]` | Steam auth ticket validation via the Steamworks Web API; a verified SteamID is the only trusted identity | A forged or replayed ticket is rejected; no client-supplied identity is ever trusted | -| 8.3 `[D:8.1]` | Rating store: per-identity, per-playlist rating plus match history, written only by the backend | A client cannot write its own rating by any path | -| 8.4 `[D:8.3]` | Rating algorithm. **Glicko-2 recommended over Elo** — it models rating *uncertainty*, which dominates at launch when most players have few games | Simulated against a synthetic population; placement behaviour is sane at n≈0 games | -| 8.5 `[D:8.4]` | Team-result → individual-rating distribution for 3v3 | A 3v3 outcome updates six ratings defensibly; documented, not folded into 8.4 | -| 8.6 `[D:8.3]` | Queue and matchmaker: per playlist and region, rating proximity with tolerance widening over wait time | Queue depth and wait time are observable; tolerance widening is tunable without redeploy | +| 8.1 | Add an ADR locking **Go + PostgreSQL + Redis**, provider-portable Kubernetes, Agones, ticketed Hosted Dedicated Server SDR, EU/NA fleets and independently runnable API, matcher, allocator and maintenance roles; keep `README.md`/`docs/TECH_STACK.md` consistent | The ADR names boundaries/rejected alternatives; current docs name the locked stack and replaceable provider; no application code calls a provider allocation API | +| 8.2 `[D:8.1]` | Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | Each SLO has a metric, numerator/denominator, percentile window, owner and alert threshold before implementation is judged against it | +| 8.3 `[D:8.1]` | Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | Generated contract tests cover every request, response, event and external error; clients can REST-resync after a missed WebSocket revision | +| 8.4 `[D:8.3]` | Define opaque `player_id`, `queue_ticket_id`, `proposal_id`, `match_id`, `server_id`, `season_id`, legal queue/match state transitions, revisions and idempotency keys | Duplicate/out-of-order commands converge; invalid transitions are rejected without partial state | +| 8.5 `[D:8.4]` | Add PostgreSQL migrations for durable queue ownership, active-participation fencing, identities, sessions/revocations, seasons, ratings/events, matches/participants, penalties, results, audits and outbox; document Redis caches/TTLs | A blank DB migrates up; lost Redis writes cannot resurrect revocation, split a proposal or corrupt durable state; rollback/forward compatibility is tested | +| 8.6 `[D:8.3,8.4]` | Lock assignment compatibility: protocol/client build, image digest, playlist version, transport, region, expiry and signed authorisation; add all allocated-mode `ServerConfig` flags as opt-in defaults | Incompatible builds never share a proposal; absent flags reproduce today's community server and existing config tests cover every new flag | -#### 8b — Server orchestration and autoscaling - -Requirement: servers scale horizontally and automatically, spin up fast, serve -exactly one match, and shut down — cost incurred only while a match runs. - -Two properties of the existing build make this practical rather than -aspirational, both **measured against `cosmicclash-server:latest`**, not -estimated: - -- The runtime image (`server` target, slim `ubuntu:24.04`) is **~148 MB** of - content — not the ~2.6 GB `godot-ci` build image. -- Boot to the `server_started` line is **~870 ms**, container's own clock, - mean of three runs. **Taken under x86_64 emulation on an arm64 host, so it - is a pessimistic bound** — see 8.11. - -`--max-matches=1` already drains and `exit(0)`s after one match. It was built -for CI and generalises to the allocator lifecycle for free. +#### 8B — Authentication and secure control plane | # | Task | Acceptance | |---|---|---| -| 8.7 | Choose the orchestrator. **Agones on Kubernetes is the recommended default** — it models allocation, readiness and per-match lifetime natively rather than making you rebuild them | Allocation, readiness and per-match teardown are all handled by the chosen system, not by bespoke glue | -| 8.8 | **Fix readiness detection — this blocks any naive implementation.** Godot's stdout is block-buffered off a TTY. Run the server image detached without `-t` and `docker logs` shows *nothing at all*; `server_started` does not appear even after 35 s. A readiness probe that greps the log hangs forever. Probe the UDP socket, or flush explicitly | A cold container is marked ready by a mechanism that does not depend on stdout; reproduced-and-fixed, not worked around by adding `-t` in one place | -| 8.9 | Multiple matches per host: a per-container port from a range, or one address per match. `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. **This is UDP — L7 ingress routing does not apply**, the allocator hands the client a concrete `host:port` | Two matches run concurrently on one node and neither can reach the other's traffic | -| 8.10 `[D:8.6]` | Assigned-roster server mode: only matched SteamIDs may claim a slot, replacing first-come. Plus a **no-show timeout** — an allocated server that never fills abandons and exits rather than idling at cost | An unmatched identity is refused a slot; a server nobody joins exits within the timeout | -| 8.11 | Re-measure boot-to-listening on **native x86_64 Linux** before it sets any timeout | A number from the real target platform replaces the ~870 ms emulated bound recorded above | -| 8.12 | Re-measure SERVER.md's sizing estimate (6–10 processes/core, 150–250 MB RSS) under real concurrency | A measured figure sizes the bill; the current estimate predates all allocation work | -| 8.13 `[D:8.10]` | Server-authoritative match result reporting to the backend over a channel a client cannot forge. **The project's first non-UDP network path** — simulation stays on ENet/SDR | A client cannot report, alter or suppress a result | -| 8.14 | Draining and deploys: roll out a server version without killing matches in flight | An in-flight match survives a deploy of the next server version | +| 8.7 `[D:7.6,8.3]` | Validate `AuthenticateUserTicket` only in the secure backend with the expected App ID and identity string; reject expiry, replay, wrong app, bans and malformed input | Publisher credentials exist only in the backend secret store; forged/replayed tickets and client-supplied SteamIDs never create a session | +| 8.8 `[D:8.7]` | Issue short-lived revocable sessions bound to verified Steam identity; add account/IP limits, body/schema limits, replay checks and generic public errors | Revocation takes effect across replicas; abuse cannot cause unbounded memory, work or response amplification | +| 8.9 `[D:8.4,8.7]` | Issue match-scoped join authorisations bound to SteamID/match/server/team/slot/protocol/expiry; allow same-identity slot reclaim while fencing prior connection generations | Altered/expired/wrong identity/server/slot is rejected; reconnect works without backend/Steam; a newer generation makes the old connection unable to send gameplay | +| 8.10 `[D:8.5,8.31]` | Authenticate results with pod-bound projected identity or one-match attested credential; validate issuer/audience/expiry, namespace/SA, pod UID, GameServer UID and allocator match binding | Another pod sharing a workload class cannot submit for the allocation; identical duplicates are idempotent; conflicting results are inert and alerting across all trusted clusters | +| 8.11 `[D:8.1]` | Write the threat model: forged clients/replay/queues/results, floods/bots, pod/insider compromise, gameplay and API DDoS, SDR signing-key theft, dependencies and denial-of-wallet | Every threat has prevention/detection/owner/verification; accepted residual risks are explicit; offline CA and online signer trust boundaries are separate | +| 8.12 `[D:8.11]` | Harden workloads and edge: restricted containers, least RBAC, private DB/Redis, default-deny networks, backups/secrets, volumetric DDoS/WAF/origin shielding, WebSocket limits and overload shedding | Policy/network tests enforce declared flows; edge load test preserves result ingress/live matches while rejecting new work; no credential appears in Git/images/args/telemetry | +| 8.13 `[D:8.12]` | Pin images by digest; generate SBOMs, scan dependencies/images, sign artifacts, verify signatures at admission and document a critical-fix SLA | CI blocks a vulnerable/disallowed or unsigned release artifact and records the exact provenance deployed | -#### 8c — Playlists and client +#### 8C — Queueing, matchmaking, playlists and rating | # | Task | Acceptance | |---|---|---| -| 8.15 `[D:8.6]` | Casual and ranked rulesets. They diverge on the server, not just in UI: backfill (casual yes / ranked never), bots filling slots (`--fill-bots` casual-only), abandon penalties, party size and rating spread | Ranked never backfills and never spawns a bot into a player slot | -| 8.16 `[D:8.15]` | Ranked arena restriction. Draw only from `"random": true` arenas — **elevated-goal variants stay Free-Play-only** until a checkpoint trained on `training_elevated.tscn` is promoted (`arena_registry.gd`), so a variant nobody has practised cannot decide a ladder match | Ranked cannot select an elevated-goal arena | -| 8.17 `[D:8.6]` | Client queue UI: playlist select, estimated wait, accept/decline, connect-on-assignment, post-match rating delta | A declined match returns the other players to the queue without penalty to them | +| 8.14 `[D:8.4,8.5,8.8]` | One durable PostgreSQL queue owner/player plus Redis candidate index: 10 s heartbeat, 30 s expiry, retry-safe create/cancel/resume and repair after cache loss | Replicas/duplicates never place a player twice; failover may delay/rematerialise an index but durable ownership and active-participation fences converge | +| 8.15 `[D:7.8,8.3]` | Submit opaque Steam ping location plus nonce-bound probes; backend computes estimates, enforces 30 s freshness and quarantines 3 discrepancies >25 ms or 30% until 5 clean matches | A client cannot directly choose its placement RTT; stale/forged evidence is rejected; quarantine behavior and server-observed comparison are deterministic | +| 8.16 `[D:8.14,8.15]` | Implement the locked candidate/team algorithm: <=100 ms, oldest anchor, `min(400,100+25*floor(wait/30))` mutual rating tolerance, documented set/region/team tie-breakers | Fixtures cover provisional players, EU/NA/no-common-region, widening caps, deterministic partitions and low population; no placement crosses 100 ms | +| 8.17 `[D:8.14,8.16]` | Ten-second proposal to **every selected human**: ranked 6; casual largest compatible 6→2 after 60 s with disclosed teams/bots; apply exact decline/timeout/no-show cooldown and queue-precedence rules | Allocation starts only after selected humans accept; 2–5-human casual is reachable; accepter timestamps restore exactly; ranked pre-match no-show has cooldown but no rating loss | +| 8.18 `[D:8.5,8.14,8.17]` | Horizontally replicated matcher: Redis candidates, then PostgreSQL serializable proposal/participant fence, then cache cleanup/repair | Forced loss of the last acknowledged Redis write, retries, worker death and failover cannot claim a player into two proposals/matches | +| 8.19 `[D:8.18]` | Casual policy: proposal composition above; >=1 human/team, exhaustive rating-balanced teams, bots after 60 s, opt-in kickoff-only backfill, 30 s reconnect and defined backfill/casual penalties | Every 2–6-human shape is tested; no mid-play replacement; declined/backfill participant gets no excluded rating/cooldown; original leaver gets only documented outcome/cooldown | +| 8.20 `[D:8.18]` | Ranked policy: exactly six verified solo humans, no bots/backfill, only `ArenaRegistry.random`; define initial no-show, proposal timeout and reconnect/abandon transitions | Ranked rejects parties/bots/backfill/elevated arenas; every pre-live failure returns five innocent players with original precedence and applies no rating | +| 8.21 `[D:8.5,8.20]` | Exact Glicko-2 equations from `docs/MATCHMAKING.md`: 1500/350/0.06/tau .5, ranked 1/3 and casual 1/N human-opponent weights, daily inactivity, immutable snapshot/lock order, draws/OT/abandons/cancellation | Canonical plus project 2–6-human/3v3 golden vectors pass; concurrent results serialize without order bias; clients have no rating-write path | +| 8.22 `[D:8.21]` | First ten ranked games provisional; casual rating hidden; ranked tiers derived from authoritative stored values | Matchmaking uses provisional rating/RD; UI visibility changes exactly on result ten without rewriting history | +| 8.23 `[D:8.21]` | Ranked-only 12-week exactly-once soft season: compress 25% toward 1500, RD >=200 capped 350, retain volatility/history; casual remains continuous | Retried/concurrent rollover applies once, never touches casual, and preserves every rating event | +| 8.24 `[D:8.9,8.20,8.21]` | Ranked reconnect/abandon: match-scoped authorisation, 60 s reclaim, server-owned connection generations, then abandoner loss and rolling 7-day 5 m/15 m/1 h/24 h cooldown | Reconnect works through backend outage and fences old peer; grace has no penalty; expiry outcome/escalation is deterministic and auditable | +| 8.25 `[D:8.10,8.24]` | Separate result delivery delay from match-integrity failure; signed Agones-annotation spool, retries, 5 m alert/30 m review, suppression only for lost/corrupt authority or measured unfair regional incident | API outage preserves rating/result; clients cannot request exemption; node/pod/integrity faults take the documented suppression/refund path | -#### 8d — Keeping Docker and CI green - -`make verify-phase6` and `make verify-enet-integration` must not regress. -`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot -assignment, and uses `--max-matches=2` to prove arena rotation — all three are -things allocation work would otherwise trample. +#### 8D — Agones, allocation and regional scaling | # | Task | Acceptance | |---|---|---| -| 8.18 | **The rule: every allocation feature is opt-in via a `ServerConfig` flag whose default reproduces today's behaviour.** `ServerConfig` is built for exactly this — a flag declared once is parsed, validated, type-checked, config-file-backed and documented | `verify-phase6` and `verify-enet-integration` pass unchanged with no edits to their invocations | -| 8.19 `[D:8.18]` | A **second** Compose file for the allocated-match path rather than mutating `compose.phase6-smoke.yml`, so the community-server model stays tested alongside the matchmade one | Both models have a green CI gate; neither shares a fixture with the other | -| 8.20 `[D:8.18]` | `tests/cases/` coverage for the new flag parsing, per §10's no-live-server rule | New flags are unit-tested without a live server or a container | +| 8.26 `[D:8.1,8.6,8.12]` | Portable Helm/Kustomize Fleets per build/EU/NA region; isolate provider edge/network/DNS/secret and SDR POP/cert/public-UDP overlays | Two provider fixtures render; labels select region/build/protocol/transport; each fixture documents Valve approval and externally reachable UDP mapping | +| 8.27 `[D:8.26]` | Godot Agones REST adapter plus allocation-metadata watch and Go PID-1 supervisor scaffold; both bypass cloud behavior without SDK env; local SDK support | Native/existing Compose/CI remain functional; emulator exercises supervisor port discovery plus Get/Watch, Ready, Health, annotation and Shutdown | +| 8.28 `[D:8.6,8.27]` | **Process-ready stage:** supervisor obtains dynamic port, launches Godot; static config/listen/Health succeed, then explicit Agones Ready—no roster/backend-registration prerequisite and no stdout probe | A detached unallocated process reaches Ready; a broken listener/config never does; Health reclaims a hung process | +| 8.29 `[D:8.26,8.27]` | Separate ENet and Hosted-SDR dynamic/passthrough mappings; supervisor exports local `SDR_LISTEN_PORT` and external `SDR_IP`; validate POP/cert/firewall/NAT | Two isolated matches share a node; Agones-reported public endpoint receives relay traffic on the bound socket; ENet fixture remains independent | +| 8.30 `[D:8.18,8.26,8.28,8.29]` | Atomic `GameServerAllocation` from Ready filtered by region/build/protocol/transport, attaching signed roster/non-secret config with bounded race retry | Duplicate commands yield one Allocated server; exhaustion or retry leaves no orphan; no client assignment is exposed merely because process is Ready | +| 8.31 `[D:8.9,8.30]` | **Assignment-ready stage:** watch Allocated metadata, verify manifest/bindings, register hosted address, acknowledge backend; only then mint/expose client tickets | Modified/wrong manifest never reaches assignment-ready; clients cannot connect early; secrets never appear in metadata/args/logs | +| 8.32 `[D:8.2,8.26,8.30]` | FleetAutoscaler with >=2 Ready processes across >=2 on-demand nodes/failure domains per queue-enabled region; pre-pull current/rollback; scale **Allocated** count to zero, never the Ready floor | Warm allocation meets p95 5 s/p99 10 s; disabled regions alone scale fully to zero; one-node loss retains certified Ready/headroom | +| 8.33 `[D:8.26,8.32]` | On-demand-only live capacity and measured N+1: loss of largest node leaves two Ready slots plus headroom for surviving Allocated matches | Interruptible nodes cannot receive live matches; forced node loss neither overloads survivors nor prevents the next allocation | +| 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog | +| 8.35 `[D:8.17,8.19,8.20,8.30,8.31]` | Initial-connect no-show: 30 s after assignment-ready; ranked cancels/no-show cooldown, casual bot policy, empty allocation exits | No allocation idles indefinitely; innocent players regain original precedence; no pre-live failure changes rating | +| 8.36 `[D:8.10,8.25,8.28,8.30]` | Go PID-1 supervisor traps TERM and authenticates localhost drain; 300 s grace/285 s infrastructure abort; PDB + Agones-aware Fleet drain; planned releases never TERM Allocated pods | Rollout/rollback waits Allocated=0; TERM path is exercised; forced timeout is classified/refunded; unexpected node loss is not claimed graceful | +| 8.37 `[D:8.5,8.10,8.25,8.26,8.31]` | Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO <=5 m/RTO <=30 m; signed result annotation/reconciliation preserves delivery outages | Restore/failover meet targets; live simulation continues; valid delayed result commits exactly once after recovery | +| 8.38 `[D:7.7,8.26,8.36,8.37]` | Provider migration after Valve approves both providers' EU/NA POPs/certs and public UDP: restore, validate coordinator trust, switch allocations, drain old | Both geographies complete Hosted-SDR matches on new provider and no old-provider live match is terminated | -> **The cold-start tension is real and is not solved by fast boot.** "Only pay -> during a match" and "a player never waits" pull against each other: a server -> must be listening *before* the matched players connect. 870 ms makes the gap -> small, but the risk is not the container — image pull on a cold node, -> scheduler placement and network/port programming can each dwarf it. -> Recommendation: **match-level scale-to-zero over a small warm node pool**, -> not node-level scale-to-zero. The per-match process genuinely exists only for -> the match; the pool absorbs cold-start variance. Revisit only when measured -> allocation latency on real infrastructure says the pool is unnecessary. +#### 8E — Client experience and recovery -> **Server cost re-enters the design.** Community servers are paid for by -> whoever hosts them; allocated servers are paid for by the project, per match. -> `README.md`'s original note about a subscription to fund servers is suddenly -> load-bearing. 8.12 needs to produce a number before launch, not after. +| # | Task | Acceptance | +|---|---|---| +| 8.39 `[D:8.3,8.14,8.17]` | Queue UI: playlist/quality, elapsed and estimated wait, proposal countdown, allocation/connect state, cancel and latency/capacity explanations | Every backend state and terminal failure has a non-stuck visible state; cancel/decline is acknowledged authoritatively | +| 8.40 `[D:8.3,8.14]` | One authenticated revisioned WebSocket plus REST resync; resume valid queue/assignment after client restart | Missed/duplicate/out-of-order events converge and restart never creates a second ticket | +| 8.41 `[D:7.8,8.9,8.31,8.40]` | After assignment-ready, install SDR relay ticket before connect and send match-scoped join authorisation in `hello`; retain ENet assignments locally | Production connects/reconnects/fences old generation through SDR, never before assignment-ready; allocated/direct ENet and community flows remain compatible | +| 8.42 `[D:8.22,8.23,8.24,8.40]` | Backend-authoritative provisional/rank/tier/delta, abandon status and season countdown UI | Client performs no rating math and displays the committed revision after reconnect | +| 8.43 `[D:8.39,8.40,8.41]` | Recovery paths for decline, expiry, startup failure, version mismatch, auth expiry, regional outage and failed reconnect | Automated UI/state tests prove every case returns to a usable queue/menu or resumes the match without a duplicate action | + +#### 8F — Observability, verification, cost and rollout + +| # | Task | Acceptance | +|---|---|---| +| 8.44 `[D:8.3,8.4,8.28,8.31]` | Propagate queue/proposal/match/server IDs and process-ready/assignment-ready through logs, metrics, traces and replay metadata; redact credentials | One ID traces queue→result across components and automated secret-canary tests find no auth/relay ticket | +| 8.45 `[D:8.2,8.44]` | Dashboards/alerts for wait/MMR/RTT, proposals, allocation/Ready/image pull, connect/no-show, tick/crash/flood, result conflict/lag, abandons and cost | Each SLO and security/cost signal has an exercised alert and runbook | +| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | Go unit, race, fuzz, property, migration and concurrency suites | CI covers auth/join replay-reclaim, stale revisions, durable matcher fencing with lost Redis ack, rollover, result conflict/delivery retry and PostgreSQL retry | +| 8.47 `[D:8.7,8.30]` | Fake Steam verifier and fake allocator for deterministic CI | Normal CI needs no Steam/cloud secret or internet access and can force every success/failure deterministically | +| 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | Second Compose flow: fake backend → queue/proposal → process-ready/allocation/assignment-ready → ENet roster → result ack → shutdown; do not edit Phase 6 fixture | Both server models have independent green gates; existing Make invocations remain unchanged | +| 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | Disposable `kind` + Agones integration gate | CI covers dynamic ports, both readiness stages, roster/no-show, races, multi-match node, result-pending reconciliation, drain and rollback | +| 8.50 `[D:8.25,8.37,8.43,8.49]` | Network/chaos suite: 100 ms RTT, jitter/loss, client/API/matcher restart, game-pod death, node drain, Redis failover and control-plane loss | System recovers to a defined state; infrastructure-caused cases cannot penalise affected players | +| 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | Load test >=10,000 queued clients, >=100 proposals/s and forecast launch concurrency x2 | API p95 <=250 ms, durable matcher fence holds, both readiness/allocation SLOs are met and replicas scale without duplicate claims | +| 8.52 `[D:8.32,8.34,8.45,8.51]` | Per-region cost model from measured density, warm capacity, bandwidth, DB/Redis and telemetry; add budgets and allocation quotas | Cost per completed match and forecast monthly bands are recorded; a denial-of-wallet test triggers limits/alerts before budget breach | +| 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | Progressive release: development → internal → casual canary → casual → provisional ranked → ranked | Each promotion requires SLO/security/cost gates, rollback rehearsal, EU+NA playtests and unchanged legacy gates; rollback criteria and owner are explicit | + +Implementation invariants for every task above: + +- Matchmade mode is opt-in; every new `ServerConfig` default preserves the + existing community-server path. +- `compose.phase6-smoke.yml`, `make verify-phase6`, and + `make verify-enet-integration` are not repurposed or weakened. +- Production uses ticketed Hosted Dedicated Server SDR; ENet remains the + deterministic local/CI and direct-IP path. +- One process serves one match. Warm processes/nodes absorb startup variance; + capacity and cost are determined from 8.34 measurements, not old estimates. +- Implementation evidence is appended under the completed task as in earlier + phases; design changes first update `docs/MATCHMAKING.md` and dependencies. ---