From bcc12aad1929090ae3e206a583df70cc0b289a75 Mon Sep 17 00:00:00 2001
From: Josh Creek <8179928+jcreek@users.noreply.github.com>
Date: Mon, 31 Aug 2026 18:49:59 +0100
Subject: [PATCH] docs(multiplayer-todo): add Phase 8 task breakdown for
 matchmaking and autoscaling

multiplayer-next.md carried the Phase 8 checklist but not the numbered
tasks with acceptance criteria that work actually gets picked up from.
That format lives in multiplayer-todo.md section 7, which already hosts
Phase 7 as in-progress, so Phase 8 goes there too.

Tasks 8.1-8.20 across four groups: backend service (identity, rating
store, Glicko-2, queue), server orchestration and autoscaling, playlists
and client UI, and keeping Docker/CI green. Section 0's short list gains
an index entry, and the status header now says Phase 8 is a 1.0 launch
blocker and the first phase to add a component outside the Godot
project.

Three entries are measured findings rather than plans, each of which
would break a naive implementation: stdout block-buffering making a
log-grep readiness probe hang forever, the hardcoded 7777/udp port
preventing more than one match per host, and compose.phase6-smoke.yml's
dependence on the exact behaviour allocation work would change.

Also fixes a now-false cross-reference: a Phase 4 note read 'not
Phase 8' meaning 'not a later phase', written when no Phase 8 existed.
CLAUDE.md's 'never add new work to multiplayer-todo.md' rule gains the
new-phase exception it always had in practice - Phase 7 was already
there.
---
 CLAUDE.md           |   2 +-
 multiplayer-todo.md | 108 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 107 insertions(+), 3 deletions(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index a0532eea..05a08d6b 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -15,7 +15,7 @@ Because the gameplay concept (vehicle soccer) can't be copyrighted but specific
 The prose docs carry far more design rationale than the code comments, and several are load-bearing:
 
 - `multiplayer-next.md` — **the current** multiplayer checklist. Short. Read this first for "what's left".
-- `multiplayer-todo.md` — 250 KB of historical design decisions, per-task implementation evidence, and §9's numbered "gotchas" list. Code comments cite it constantly by section/task number (`§2.4`, `task 5.10`); when a comment does, that section is the real explanation. Don't add new work here — it's the archive.
+- `multiplayer-todo.md` — 250 KB of historical design decisions, per-task implementation evidence, and §9's numbered "gotchas" list. Code comments cite it constantly by section/task number (`§2.4`, `task 5.10`); when a comment does, that section is the real explanation. Mostly an archive: Phases 0–6 are done, and day-to-day work is tracked in `multiplayer-next.md` instead. The exception is a *new phase* — Phase 7 (Steam) and Phase 8 (matchmaking) both keep their numbered task breakdown and acceptance criteria in §7, because that is the format tasks are picked up from.
 - `TRAINING.md` — the full RL workflow (training, curriculum generations, export, eval, difficulty tiers).
 - `SERVER.md` — dedicated-server build, config, systemd deploy, sizing.
 - `STEAM.md` — optional GodotSteam custom-build setup and the transport contract.
diff --git a/multiplayer-todo.md b/multiplayer-todo.md
index a48afa4f..f3d4d50f 100644
--- a/multiplayer-todo.md
+++ b/multiplayer-todo.md
@@ -6,7 +6,7 @@ points there too.
 
 Everything below is written so an agent (or a person) can pick up a single numbered task, do it, verify it against a stated acceptance criterion, and stop. Sections 1–6 are the decisions those tasks assume; read them before picking up work in Phase 2 or later.
 
-**Status: every task in Phases 0–6 is implemented and verified locally.** Both multiplayer milestones still need human verification — a playtest at ~100 ms RTT (Phase 4) and a real 3v3 session (Phase 5). Phase 6's public-internet gate is deliberately blocked by the display-name reclaim defect until Phase 7 identity work lands; its export, Docker, rotation/drain, and CI work are complete. Phase 7's Steam foundation is in progress. The client has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. See the outstanding list immediately below for what is left and why, §7 for the implemented work and its evidence, and §11 for what is deliberately deferred.
+**Status: every task in Phases 0–6 is implemented and verified locally.** Both multiplayer milestones still need human verification — a playtest at ~100 ms RTT (Phase 4) and a real 3v3 session (Phase 5). Phase 6's public-internet gate is deliberately blocked by the display-name reclaim defect until Phase 7 identity work lands; its export, Docker, rotation/drain, and CI work are complete. Phase 7's Steam foundation is in progress. **Phase 8 — matchmaking, ranked, and per-match server autoscaling — is a 1.0 launch blocker and is entirely unimplemented**; it is the first phase to add a component outside the Godot project, and its design lives in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). The client has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. See the outstanding list immediately below for what is left and why, §7 for the implemented work and its evidence, and §11 for what is deliberately deferred.
 
 ---
 
@@ -14,6 +14,14 @@ Everything below is written so an agent (or a person) can pick up a single numbe
 
 The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 0–5 contain no unfinished tasks.
 
+**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.20 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three items there are findings rather than plans, and each would break a naive implementation:
+
+| # | Finding | Why it bites |
+|---|---|---|
+| 8.8 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | An orchestrator readiness probe that greps the log hangs forever |
+| 8.9 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches cannot share a host; being UDP, L7 ingress routing does not apply |
+| 8.18 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | Allocation work trivially regresses `verify-phase6` unless every new feature defaults to today's behaviour |
+
 ### Blocking sign-off — the work exists, the verification does not
 
 | # | What | Why it is not done | Detail |
@@ -941,7 +949,7 @@ No own-ship prediction yet: the client renders everything, including its own shi
 | **4.9** `[D:4.4]` | **DONE.** Present-time remote visual extrapolation, angular integration, and render-only residual correction; delayed interpolation remains an A/B debug mode | Final two-bot present-time p99 ≤.208m / 3.146°, below .3m / 5° gate |
 | **4.10** `[D:4.9]` `[P]` | **DONE.** Signed starvation sentinel and client hysteresis/cooldown; headless `--test-bot` remains target depth 1 | Jitter run observed starvation fallback; stable runs preserve safe target behavior |
 
-> **Ball prediction is not optional and not Phase 8.** With §4.1 in place the touch registers correctly on the server, but the ball still *renders* a third of a beat late — your ship visibly passes through it before it moves. In a game whose entire point is hitting a ball, that is the difference between "networked" and "broken", and it is the same machinery as own-ship prediction applied to one more body. Do it while the prediction code is warm. Buffering server ball state into a *shadow* copy (rather than discarding it) is what lets you measure disagreement continuously instead of discovering a 3 m error at window end.
+> **Ball prediction is not optional and not deferrable to a later phase.** With §4.1 in place the touch registers correctly on the server, but the ball still *renders* a third of a beat late — your ship visibly passes through it before it moves. In a game whose entire point is hitting a ball, that is the difference between "networked" and "broken", and it is the same machinery as own-ship prediction applied to one more body. Do it while the prediction code is warm. Buffering server ball state into a *shadow* copy (rather than discarding it) is what lets you measure disagreement continuously instead of discovering a 3 m error at window end.
 
 | 4.11 `[D:4.2]` | **DONE.** Prediction history is filed under the **issuing** sequence, and a forced-input-transition trace gates the label | Marker mismatch 0.00–1.3% (was 9.3% LAN / 24% at 80±20ms); control run at the old label fails the same gate at 50% |
 | 4.12 `[D:4.11]` | **DONE.** Issued-but-unsimulated (attack-gap) sequences are recorded and skipped rather than diagnosed as history loss; the release path no longer re-files an already-issued sequence | Free-flight hard snaps 0 across all three 60 s conditions, down from 25/8/4 `missing_not_recorded` |
@@ -1132,6 +1140,102 @@ New `--role=client-reconnect` grades the returning player: not a spectator, owns
 
 > GodotSteam requires custom engine builds and export templates — **including for the headless server**. That is the part people discover three weeks in. Budget for it.
 
+### Phase 8 — Matchmaking, ranked ladder, per-match server autoscaling
+
+**1.0 launch blocker.** Full design and reasoning: [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md).
+Nothing here is implemented. Unlike Phases 0–7 this phase adds a component
+outside the Godot project — a backend service — and that is the largest
+architectural departure in the project's history, so read the design doc
+before picking up any task below.
+
+This inverts the server model. Phases 1–7 build a **community server**: it
+runs forever, waits for `--min-players`, plays a match, rotates arena, repeats,
+and players find it by IP or (7.3) the server browser. Matchmaking makes the
+*player* durable instead — queue, get grouped by rating, and a server is
+**allocated for that one match** and destroyed after. Both models ship; they
+are different playlists, not a replacement.
+
+**Hard dependency on 7.4.** Slot reclaim is keyed by display name today. A
+rating attached to a spoofable identity is farmed trivially, so ranked cannot
+ship before Steam auth tickets land. Casual queueing needs 7.4 too, for
+abandon penalties and ban enforcement, but degrades more gracefully.
+
+#### 8a — Backend service
+
+| # | Task | Acceptance |
+|---|---|---|
+| 8.1 | Choose backend language, hosting and datastore. **Not C# by default** — that framing predates every real decision here | Written up with the rejected alternatives, as §1 does for the client decisions |
+| 8.2 `[D:7.4]` | Steam auth ticket validation via the Steamworks Web API; a verified SteamID is the only trusted identity | A forged or replayed ticket is rejected; no client-supplied identity is ever trusted |
+| 8.3 `[D:8.1]` | Rating store: per-identity, per-playlist rating plus match history, written only by the backend | A client cannot write its own rating by any path |
+| 8.4 `[D:8.3]` | Rating algorithm. **Glicko-2 recommended over Elo** — it models rating *uncertainty*, which dominates at launch when most players have few games | Simulated against a synthetic population; placement behaviour is sane at n≈0 games |
+| 8.5 `[D:8.4]` | Team-result → individual-rating distribution for 3v3 | A 3v3 outcome updates six ratings defensibly; documented, not folded into 8.4 |
+| 8.6 `[D:8.3]` | Queue and matchmaker: per playlist and region, rating proximity with tolerance widening over wait time | Queue depth and wait time are observable; tolerance widening is tunable without redeploy |
+
+#### 8b — Server orchestration and autoscaling
+
+Requirement: servers scale horizontally and automatically, spin up fast, serve
+exactly one match, and shut down — cost incurred only while a match runs.
+
+Two properties of the existing build make this practical rather than
+aspirational, both **measured against `cosmicclash-server:latest`**, not
+estimated:
+
+- The runtime image (`server` target, slim `ubuntu:24.04`) is **~148 MB** of
+  content — not the ~2.6 GB `godot-ci` build image.
+- Boot to the `server_started` line is **~870 ms**, container's own clock,
+  mean of three runs. **Taken under x86_64 emulation on an arm64 host, so it
+  is a pessimistic bound** — see 8.11.
+
+`--max-matches=1` already drains and `exit(0)`s after one match. It was built
+for CI and generalises to the allocator lifecycle for free.
+
+| # | Task | Acceptance |
+|---|---|---|
+| 8.7 | Choose the orchestrator. **Agones on Kubernetes is the recommended default** — it models allocation, readiness and per-match lifetime natively rather than making you rebuild them | Allocation, readiness and per-match teardown are all handled by the chosen system, not by bespoke glue |
+| 8.8 | **Fix readiness detection — this blocks any naive implementation.** Godot's stdout is block-buffered off a TTY. Run the server image detached without `-t` and `docker logs` shows *nothing at all*; `server_started` does not appear even after 35 s. A readiness probe that greps the log hangs forever. Probe the UDP socket, or flush explicitly | A cold container is marked ready by a mechanism that does not depend on stdout; reproduced-and-fixed, not worked around by adding `-t` in one place |
+| 8.9 | Multiple matches per host: a per-container port from a range, or one address per match. `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. **This is UDP — L7 ingress routing does not apply**, the allocator hands the client a concrete `host:port` | Two matches run concurrently on one node and neither can reach the other's traffic |
+| 8.10 `[D:8.6]` | Assigned-roster server mode: only matched SteamIDs may claim a slot, replacing first-come. Plus a **no-show timeout** — an allocated server that never fills abandons and exits rather than idling at cost | An unmatched identity is refused a slot; a server nobody joins exits within the timeout |
+| 8.11 | Re-measure boot-to-listening on **native x86_64 Linux** before it sets any timeout | A number from the real target platform replaces the ~870 ms emulated bound recorded above |
+| 8.12 | Re-measure SERVER.md's sizing estimate (6–10 processes/core, 150–250 MB RSS) under real concurrency | A measured figure sizes the bill; the current estimate predates all allocation work |
+| 8.13 `[D:8.10]` | Server-authoritative match result reporting to the backend over a channel a client cannot forge. **The project's first non-UDP network path** — simulation stays on ENet/SDR | A client cannot report, alter or suppress a result |
+| 8.14 | Draining and deploys: roll out a server version without killing matches in flight | An in-flight match survives a deploy of the next server version |
+
+#### 8c — Playlists and client
+
+| # | Task | Acceptance |
+|---|---|---|
+| 8.15 `[D:8.6]` | Casual and ranked rulesets. They diverge on the server, not just in UI: backfill (casual yes / ranked never), bots filling slots (`--fill-bots` casual-only), abandon penalties, party size and rating spread | Ranked never backfills and never spawns a bot into a player slot |
+| 8.16 `[D:8.15]` | Ranked arena restriction. Draw only from `"random": true` arenas — **elevated-goal variants stay Free-Play-only** until a checkpoint trained on `training_elevated.tscn` is promoted (`arena_registry.gd`), so a variant nobody has practised cannot decide a ladder match | Ranked cannot select an elevated-goal arena |
+| 8.17 `[D:8.6]` | Client queue UI: playlist select, estimated wait, accept/decline, connect-on-assignment, post-match rating delta | A declined match returns the other players to the queue without penalty to them |
+
+#### 8d — Keeping Docker and CI green
+
+`make verify-phase6` and `make verify-enet-integration` must not regress.
+`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
+assignment, and uses `--max-matches=2` to prove arena rotation — all three are
+things allocation work would otherwise trample.
+
+| # | Task | Acceptance |
+|---|---|---|
+| 8.18 | **The rule: every allocation feature is opt-in via a `ServerConfig` flag whose default reproduces today's behaviour.** `ServerConfig` is built for exactly this — a flag declared once is parsed, validated, type-checked, config-file-backed and documented | `verify-phase6` and `verify-enet-integration` pass unchanged with no edits to their invocations |
+| 8.19 `[D:8.18]` | A **second** Compose file for the allocated-match path rather than mutating `compose.phase6-smoke.yml`, so the community-server model stays tested alongside the matchmade one | Both models have a green CI gate; neither shares a fixture with the other |
+| 8.20 `[D:8.18]` | `tests/cases/` coverage for the new flag parsing, per §10's no-live-server rule | New flags are unit-tested without a live server or a container |
+
+> **The cold-start tension is real and is not solved by fast boot.** "Only pay
+> during a match" and "a player never waits" pull against each other: a server
+> must be listening *before* the matched players connect. 870 ms makes the gap
+> small, but the risk is not the container — image pull on a cold node,
+> scheduler placement and network/port programming can each dwarf it.
+> Recommendation: **match-level scale-to-zero over a small warm node pool**,
+> not node-level scale-to-zero. The per-match process genuinely exists only for
+> the match; the pool absorbs cold-start variance. Revisit only when measured
+> allocation latency on real infrastructure says the pool is unnecessary.
+
+> **Server cost re-enters the design.** Community servers are paid for by
+> whoever hosts them; allocated servers are paid for by the project, per match.
+> `README.md`'s original note about a subscription to fund servers is suddenly
+> load-bearing. 8.12 needs to produce a number before launch, not after.
+
 ---
 
 ## 8. What needs refactoring, not extending