From ad9289fb218988feed0f186b87204c33dda9c0ba Mon Sep 17 00:00:00 2001
From: Josh Creek <8179928+jcreek@users.noreply.github.com>
Date: Fri, 4 Sep 2026 18:18:22 +0100
Subject: [PATCH] docs(multiplayer): flag the real root blocker of the
 allocation pipeline
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Found while investigating why the connect-wiring fix (§8.41,
ControlPlaneClient._connect_when_assigned) would still not work
end to end in a real deployment: nothing in production ever
publishes a player's signed match assignment.

store.SaveAssignment/SaveAssignments/SaveVerifiedAssignmentRoster --
the only functions that ever write the assignments table -- are
called only from tests, never from allocator/worker.go, cmd/allocator,
or anywhere else in the real service. allocator.Service.PublishRoster
(wired to store.PostgresRosterStore) is likewise never called from
production code.

allocation_match_sql.go's AdvanceServerRegistration SQL requires an
assignments row for every match participant before allowing the
ASSIGNMENT_READY transition. With nothing ever creating those rows, a
real match cannot advance past PROCESS_READY -- no player can ever
receive a real assignment or connect, regardless of how correct the
client-side connect-wiring fix from earlier this session is.

TestRealSupervisorRegistersAllocatedServerThroughControlPlane -- the
test that was supposed to prove this end to end -- manually seeds
store.SaveAssignment in its own setup rather than exercising the real
production write path, which is why this was never caught.

Per the user's explicit direction, this is flagged rather than fixed:
closing it needs new security-relevant design (a join-signing key
shared between the allocator, which would sign, and the game server,
which fleet.yaml already mounts a verification key for via
--join-authorisations-key-file but which no control-plane binary has
a matching signing flag for; roster-digest computation; per-player
domain.JoinAuthorisation construction from match_participants/identities
via the already-built domain.SignJoinAuthorisationHMAC), not a simple
wiring fix -- a wrong design choice here is a join-authorization
forgery risk, not just a UX gap, so it isn't something to build
unprompted the way the smaller fixes earlier this session were.

Recorded in three places for visibility: a new root-blocker callout
in §0 (the outstanding-work index), the top Status line, and expanded
detail in §8.31's own row, which previously undersold this as merely
'production signer... remain'.

No code changes. Verified the doc edit didn't touch anything else:
go build/vet/test -race clean, full Godot suite 220/220 clean
(unchanged, as expected).
---
 multiplayer-next.md | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/multiplayer-next.md b/multiplayer-next.md
index a95a0f33..d5df5941 100644
--- a/multiplayer-next.md
+++ b/multiplayer-next.md
@@ -12,7 +12,7 @@ those tasks assume; read them before picking up work in Phase 2 or later. §9
 is a running gotchas list — check it before debugging something that looks
 like a Godot/Jolt engine quirk, and add to it when you find a new one.
 
-**Status: every task in Phases 0–6 is implemented and verified locally.** Both multiplayer milestones still need human verification — a playtest at ~100 ms RTT (Phase 4) and a real 3v3 session (Phase 5). Phase 6's public-internet gate is no longer blocked for allocated reconnects, which use signed identity; direct/community reservations retain the documented display-name limitation. The export, Docker, rotation/drain, and CI work remain complete. Phase 7's Steam foundation is in progress. **Phase 8 — matchmaking, ranked, and per-match server autoscaling — remains a 1.0 launch blocker and is partially implemented:** the Go API/domain/store/Redis/allocator paths, migrations, supervisor, authenticated Kubernetes/Agones adapter, hardened Fleet baseline, testkit, and offline end-to-end path are in place. Production Steam identity/SDR, live cluster/public-network execution, release evidence, and human gates remain. It is the first phase to add a component outside the Godot project, and its design lives in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). The client has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. See the outstanding list immediately below for what is left and why, §7 for the implemented work and its evidence, and §11 for what is deliberately deferred.
+**Status: every task in Phases 0–6 is implemented and verified locally.** Both multiplayer milestones still need human verification — a playtest at ~100 ms RTT (Phase 4) and a real 3v3 session (Phase 5). Phase 6's public-internet gate is no longer blocked for allocated reconnects, which use signed identity; direct/community reservations retain the documented display-name limitation. The export, Docker, rotation/drain, and CI work remain complete. Phase 7's Steam foundation is in progress. **Phase 8 — matchmaking, ranked, and per-match server autoscaling — remains a 1.0 launch blocker and is partially implemented:** the Go API/domain/store/Redis/allocator paths, migrations, supervisor, authenticated Kubernetes/Agones adapter, hardened Fleet baseline, testkit, and offline end-to-end path are in place. Production Steam identity/SDR, live cluster/public-network execution, release evidence, and human gates remain. **A real deployment cannot complete a match end to end today**: the allocator never actually publishes a player's signed assignment in production (see §0's root-blocker callout, §8.31), so no match can advance past `PROCESS_READY` — this is flagged, not yet fixed. It is the first phase to add a component outside the Godot project, and its design lives in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). The client has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. See the outstanding list immediately below for what is left and why, §7 for the implemented work and its evidence, and §11 for what is deliberately deferred.
 
 ---
 
@@ -28,6 +28,8 @@ The one place to look before planning. Everything here is also written up where
 | Task 8.29 | ~~`--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`~~ **Fixed locally**: the allocated supervisor replaces the child port with the Agones-assigned endpoint and exports SDR variables only for Hosted-SDR | Live Agones passthrough/NAT and multi-match validation remain infrastructure gates |
 | Task 8.48 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | The allocated flow needs its own fixture so Phase 6 behavior and invocations stay unchanged |
 
+**The actual current root blocker (found 2026-09-04, not yet fixed)**: nothing in production ever publishes a player's signed match assignment. `store.SaveAssignment`/`SaveAssignments`/`SaveVerifiedAssignmentRoster` and `allocator.Service.PublishRoster` are fully built and tested in isolation, but no real code path (`allocator/worker.go`, `cmd/allocator`) ever calls them — only tests do, by seeding the table directly rather than exercising the real write path. Since `AdvanceServerRegistration`'s SQL requires an `assignments` row per participant before a match can reach `ASSIGNMENT_READY`, **a real deployment cannot advance any match past `PROCESS_READY`** — no player can ever receive a real assignment or connect, regardless of how correct every other piece (including §8.41's client-side connect-wiring fix) is. See §8.31 for the full detail. Closing it needs new security-relevant design (a join-signing key shared between allocator and game server, roster-digest computation, per-player authorisation construction) — flagged rather than built, at the user's explicit direction, pending a decision on that design.
+
 ### Blocking sign-off — the work exists, the verification does not
 
 | # | What | Why it is not done | Detail |
@@ -1226,7 +1228,7 @@ The allocated-runtime result reporter now keeps a completed match in `RESULTS` u
 | 8.28 `[D:8.6,8.27]` | **IN PROGRESS.** Supervisor separates explicit process-ready from Agones Ready and never scrapes stdout; allocated mode refuses to mark Ready without a configured readiness probe. The durable control-plane counterpart now exists: a workload-authenticated `POST /v1/servers/{id}/register` (and its `/api/v1` contract alias) advances a match's `ALLOCATING -> PROCESS_READY -> ASSIGNMENT_READY` state, and every participant's queue ticket with it, as one idempotent SERIALIZABLE transaction, gated on every participant already holding a live, unexpired assignment. The supervisor now calls it: once Agones Ready succeeds, it POSTs process-ready (`assignment_ready=false`) using a workload token read fresh from disk each call (matching kubelet's in-place rotation of a projected service-account token), and a registration failure kills the child rather than leaving an Agones-Ready-but-control-plane-unregistered process running; `ControlPlaneURL` unset (the default) is a total no-op. It then reports assignment-ready too: `server_boot.gd` already verifies its mounted roster synchronously before `/ready` is ever exposed (so process-ready implies the roster was valid), and the API's `ASSIGNMENT_READY` gate checks only durable `assignments` rows server-side — so no new Godot-side state was needed, correcting an earlier overcautious note here. The supervisor retries assignment-ready (default 5 attempts, 2s apart, configurable) since those durable rows may lag process-ready slightly; a persistent failure there does not kill the child, unlike process-ready. Per-allocation data (currently `match-id`) now has a real channel to an already-Ready pod: `server/agones.Client.Allocate` requests `cosmic-clash.io/match-id`/`cosmic-clash.io/allocation-id` as `GameServerAllocation.spec.metadata.annotations` (Agones applies these to the allocated GameServer's own `object_meta` — the only channel that exists post-allocation, since env vars are fixed at pod creation), and the supervisor reads them back from its existing `/gameserver` SDK call, falling back to them only when `MatchID` isn't explicitly configured. The image now exists: a new `Dockerfile` `game-server` target packages the supervisor as ENTRYPOINT alongside the same dedicated-server export `server` produces; the base Fleet now invokes that target with the control-plane URL, server/image Downward API identity, roster/signing/drain material, and exported Godot executable. | `server/supervisor/` tests prove Ready follows the probe and direct mode remains functional; `server_control.gd`, `agones_sdk.gd` and process-level smokes prove loopback `/ready`, `/health`, bearer-protected `/drain`, sidecar-shaped Health/Ready calls and drain admission fencing; `server/api/service.go`, `server/store/allocation_match_sql.go` and adversarial tests cover the registration route, digest/protocol validation, idempotent replay/conflict and the assignment-count gate; `server/supervisor/supervisor_test.go` covers opt-in registration, the workload-token/body/idempotency-key shape, the kill-on-failure path, the match-ID annotation fallback (both that it's used and that its absence fails closed before any HTTP call), the full process-ready-then-assignment-ready sequence and body shapes, and a retry test that fails assignment-ready twice with 409 before succeeding, asserting `Start()` still succeeds and the child is never killed; `server/agones/allocation_test.go` covers the requested annotations. `docker build --target game-server` verified for real: both binaries present, correct permissions, supervisor prints its usage; `server/store/stalled_allocation_sql.go`/`_test.go` and a live `TestPostgreSQLStalledAllocationsAreReclaimedWithoutPenalisingPlayers` cover the deadline boundary (a recent match must survive untouched), the no-penalty requeue and refreshed expiry, participant release, and idempotence against a second pass. **Fixed in passing**: `Dockerfile`'s `server` stage's `ubuntu` base digest had gone dead on Docker Hub (`docker pull` returned "not found", verified independently) — `make verify-phase6` was silently broken for a clean build before the re-pin; confirmed fixed with a full `make verify-phase6` run (arenas rotated, both goals observed, clean teardown). `deploy/cosmic-clash-server` now wraps its exec in `stdbuf -oL -eL` (falling back to unwrapped if unavailable), fixing a real, live-confirmed bug: a genuinely detached (`docker run -d`) container showed zero `docker logs` output — not even the startup line — for 20+ seconds while the process ran normally, and `docker stop`'s SIGTERM lost that buffered output permanently rather than delaying it; re-verified fixed against the real launcher script, then a full `make verify-phase6` re-run confirmed no regression. Health-reclaim now exists: `store.ExpireStalledAllocations` reclaims a match stuck in `ALLOCATING`/`PROCESS_READY`/`ASSIGNMENT_READY` past a deadline (server crashed, or was reclaimed by Agones as unhealthy, before ever registering) by failing the match and requeuing every participant to `QUEUED` with a fresh expiry rather than penalising them — task 8.50's own "infrastructure-caused cases cannot penalise affected players" criterion directly settles the requeue-vs-fail design question this had been blocked on. Wired into `cmd/maintenance` alongside the season-rollover sweep (`--stalled-allocation-deadline` default 2m, `--stalled-allocation-batch`). **Superseding the `fleet.yaml` framing above**: §8.10's `WorkloadVerify` blocker, and its delivery channel, are both now closed — a control-plane-self-issued signed token (not a Kubernetes JWT), minted by `cmd/allocator` into a `cosmic-clash.io/workload-token` annotation and read back by the supervisor, exactly the way `match-id` already worked — see §8.10. `/register` and `/result` no longer 503 unconditionally once every `--workload-secret` (control plane, allocator) is set consistently. The remaining gates are live Agones annotation/shutdown behavior and production cluster readiness; those are covered by §8.49 and remain explicitly open. |
 | 8.29 `[D:8.26,8.27]` | **IN PROGRESS.** Supervisor discovers and validates the Agones endpoint, propagates the actual dynamic `--port`, and exports `SDR_LISTEN_PORT`/`SDR_IP` only for Hosted-SDR while preserving an isolated ENet path | `server/supervisor/` tests cover invalid address/port rejection, dynamic port argument/env propagation and SDR-vs-ENet separation; real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT and multi-match fixture remain |
 | 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** PostgreSQL leases each `ALLOCATING` match under a deterministic allocation ID, derives immutable compatibility from its accepted roster, and atomically binds only a durably recorded provider allocation while advancing every ticket. Fresh and recovered provider results now share the same fail-closed validation of allocation/match/server identity, region, build, protocol, arena, transport, allocated state, and non-empty endpoint before persistence or binding. Workers bind the canonical allocation returned by durable reconciliation rather than the provider's pre-persistence object, preserving server-owned timestamps and normalization. Ambiguous provider outcomes retain the lease and recover by allocation ID before another external request. Agones request/response parsing and Fleet labels remain provider-portable | Unit/adversarial tests cover every fresh/recovered compatibility mismatch, empty endpoint, canonical durable result propagation, lease recovery, bind/release fencing, quota behavior, accepted-proposal gating, provider ambiguity, malformed responses, and immutable labels. PostgreSQL-tagged allocator/race/integration suites and the Agones-shaped HTTP runner remain committed; this provider-validation change awaits live database/cluster reruns while Docker storage, kind, and Helm are unavailable. Full unknown-outcome cluster recovery and signed roster metadata remain |
-| 8.31 `[D:8.9,8.30]` | **IN PROGRESS.** Assignment exposure requires Allocated state, exact allocation/match/server/region/build/protocol/transport compatibility, a hosted endpoint, and verified manifest/signature. Signed roster persistence now runs as one serializable transaction and proves the submitted set exactly equals the active durable match roster before writing any player row: allocation/server compatibility, player and Steam identity, canonical global slot, and team must all match. Partial rosters, unknown/substituted players, duplicate slots, mixed match/server/manifest batches, and zero revisions fail closed. Player recovery remains owner-, match-state-, server-, and expiry-scoped | Domain/store/allocator/API tests cover early exposure, tampered manifests/signatures, wrong compatibility, partial/mixed/duplicate rosters, durable Steam/team/slot mismatch, atomic no-row-on-failure behavior, expiry, and identical replay. PostgreSQL-tagged exact-roster regressions compile; live database and Agones reruns remain environment-dependent. Hosted-address registration, production signer, and client-ticket publication remain |
+| 8.31 `[D:8.9,8.30]` | **IN PROGRESS.** Assignment exposure requires Allocated state, exact allocation/match/server/region/build/protocol/transport compatibility, a hosted endpoint, and verified manifest/signature. Signed roster persistence now runs as one serializable transaction and proves the submitted set exactly equals the active durable match roster before writing any player row: allocation/server compatibility, player and Steam identity, canonical global slot, and team must all match. Partial rosters, unknown/substituted players, duplicate slots, mixed match/server/manifest batches, and zero revisions fail closed. Player recovery remains owner-, match-state-, server-, and expiry-scoped | Domain/store/allocator/API tests cover early exposure, tampered manifests/signatures, wrong compatibility, partial/mixed/duplicate rosters, durable Steam/team/slot mismatch, atomic no-row-on-failure behavior, expiry, and identical replay. PostgreSQL-tagged exact-roster regressions compile; live database and Agones reruns remain environment-dependent. **"Production signer... remain" understates this badly -- this is the actual root blocker of the whole allocation-to-connect pipeline, found 2026-09-04, flagged rather than fixed at the user's explicit direction (see §0)**: `store.SaveAssignment`/`SaveAssignments`/`SaveVerifiedAssignmentRoster` -- the only functions that ever write the `assignments` table this whole row describes -- are called only from tests, never from `allocator/worker.go`, `cmd/allocator`, or anywhere else in the real service; `allocator.Service.PublishRoster` (wired to `store.PostgresRosterStore`) is likewise never called from production code. `allocation_match_sql.go`'s `AdvanceServerRegistration` SQL requires an `assignments` row for every match participant before allowing the `ASSIGNMENT_READY` transition -- with nothing ever creating those rows, a real match cannot advance past `PROCESS_READY`, which also means §8.41's connect-wiring fix (`ControlPlaneClient._connect_when_assigned`) has nothing real to fetch in production even though it is itself correct. `TestRealSupervisorRegistersAllocatedServerThroughControlPlane` (the test that was supposed to prove this end to end) manually seeds `store.SaveAssignment` in its own setup rather than exercising the real write path, which is why this has never been caught. Closing it needs new security-relevant design, not just wiring: a join-signing key shared between the allocator (to sign) and the game server (`fleet.yaml` already mounts one for verification via `--join-authorisations-key-file`, but no control-plane binary has a matching signing flag), roster-digest computation, and per-player `domain.JoinAuthorisation` construction (slot/team from `match_participants`, `steam_id` from `identities`, reconnect generation) via the already-built `domain.SignJoinAuthorisationHMAC` |
 | 8.32 `[D:8.2,8.26,8.30]` | **IN PROGRESS.** Provider-neutral FleetAutoscaler baseline preserves a two-process Ready buffer, caps warm capacity, and leaves Allocated scale-down independent of the Ready floor; Fleet image references remain digest-pinned for current/rollback pre-pull | `deploy/k8s/base/fleet-autoscaler.yaml` and manifest tests cover Fleet ownership, Buffer policy and floor/cap invariants; regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99 and N+1 certification remain |
 | 8.33 `[D:8.26,8.32]` | **IN PROGRESS.** Fleet scheduling now requires on-demand capacity and spreads Ready processes across zones with skew 1; the autoscaler preserves the two-process Ready floor | `deploy/k8s/base/fleet.yaml` and manifest tests reject interruptible placement and single-zone concentration structurally; regional node pools, forced node-loss testing and measured N+1 headroom remain |
 | 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog |