diff --git a/multiplayer-next.md b/multiplayer-next.md index 023a66ed..df514f7e 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -66,7 +66,19 @@ and the Kubernetes base deployed a control-plane image nothing built while building a matcher image nothing deployed. What remains for a live deployment is external: a Steamworks App ID and publisher key ([#15](https://github.com/jcreek/CosmicClash/issues/15)), custom GodotSteam builds ([#16](https://github.com/jcreek/CosmicClash/issues/16)), -and a real cluster ([#17](https://github.com/jcreek/CosmicClash/issues/17)). +and a real cluster ([#17](https://github.com/jcreek/CosmicClash/issues/17)) and the images +to run there ([#31](https://github.com/jcreek/CosmicClash/issues/31)). + +**Rows were audited against the code on 2026-09-05.** Nine understated what +was already built — 8.6, 8.8, 8.13, 8.16, 8.19, 8.30, 8.42, 8.43, 8.52 — on +top of 7.4, 8.7, 8.20, 8.22 and 8.39 corrected while working on them. The +drift ran one way: rows kept listing work that had since landed, which makes +the backlog look larger than it is and invites rebuilding what exists. Twice +during this branch a task was picked up only to find one of its named parts +already complete (8.20's allocation wiring, 8.22's client UI). **When picking +up a row, verify its claim against the code before planning against it** — +and correct the row if it is stale, since an unverified row is a rumour, not +a backlog item. ### Blocking sign-off — the work exists, the verification does not @@ -164,8 +176,10 @@ a component outside the Godot project — a Go backend service — and that is the largest architectural departure in the project's history; read the design doc before picking up any task below. The local control-plane, durable-store, allocated-server, and verification paths are substantially -implemented; every row below lists only what's still open, not what's -built. **Task 8.31, formerly the critical path, is done — see §0.** What now +implemented; every row below is *intended* to list only what's still open, +not what's built — but see §0's audit note: rows drift toward understating +what has landed, so verify a row's claim against the code before planning +against it. **Task 8.31, formerly the critical path, is done — see §0.** What now gates a live deployment is external: an App ID ([#15](https://github.com/jcreek/CosmicClash/issues/15)), GodotSteam builds ([#16](https://github.com/jcreek/CosmicClash/issues/16)), and a cluster ([#17](https://github.com/jcreek/CosmicClash/issues/17)). @@ -192,18 +206,18 @@ are done; everything below is what's left on the tasks still open. | # | Task | Remaining | |---|---|---| | 8.5 `[D:8.4]` | PostgreSQL migrations 0001–0017 (idempotency, queue fencing, identities, ratings, matches, results, audits, outbox, allocator registry, proposal plans, leases, quotas, outbox dead-letter, retention indexes, allocation endpoints, probe challenges) | Verified against a live PostgreSQL; migrations now run to 0017. The local Docker storage exhaustion is a recurring symptom, not a one-off — see §9 gotcha on the integration scripts leaking anonymous volumes | -| 8.6 `[D:8.3,8.4]` | Allocated-mode `ServerConfig` fields | Signed-authorisation admission, dynamic endpoint wiring, full manifest/runtime tests remain | +| 8.6 `[D:8.3,8.4]` | Allocated-mode `ServerConfig` fields | Allocated-mode fields are all present in `ServerConfig` (`allocated-mode`, `match-id`, `server-id`, `playlist`, `client-build`, `assignment-expiry-unix`, `server-image-digest`, `transport`, `region`, the join-authorisation file/key pair, `readiness-port`, `drain-token-env`). Signed-authorisation admission is implemented in `MatchNet` and was hardened with key-set rotation; dynamic endpoint wiring exists via `AssignmentState` and `connect_to_assignment()`. Only live runtime verification against a real cluster remains ([#17](https://github.com/jcreek/CosmicClash/issues/17)) | #### 8B — Authentication and secure control plane | # | Task | Remaining | |---|---|---| | 8.7 `[D:7.6,8.3]` | Ticket policy binding expected App ID/identity | Adapter, bans and secret store landed: `server/steam` calls `ISteamUserAuth/AuthenticateUserTicket`, rejects family-shared and banned accounts, and separates a Valve outage (503) from a bad ticket (401); the publisher key is mounted into the control-plane Deployment alone from the `cosmic-clash-steam` Secret, asserted by a manifest test. Only verification against real Valve remains, which needs the App ID and key ([#15](https://github.com/jcreek/CosmicClash/issues/15)) | -| 8.8 `[D:8.7]` | Session policy (opaque tokens, digests, revocation) | Distributed revocation coordination, live Steam/session integration remain | +| 8.8 `[D:8.7]` | Session policy (opaque tokens, digests, revocation) | Distributed revocation coordination is done by construction: sessions are durable and `PostgresSessions.Authenticate` reads the row on every authenticated request, so a revocation takes effect immediately on every replica without any cross-replica protocol, and `ApplyIdentityBan` revokes an identity's sessions in the same transaction as the ban. Live Steam/session integration remains ([#15](https://github.com/jcreek/CosmicClash/issues/15)) | | 8.9 `[D:8.4,8.7]` | Join policy, durable reconnect leases | Live PostgreSQL/Godot process-restart and outage recovery verification remains | | 8.10 `[D:8.5,8.31]` | Workload credential policy (signed tokens, not Kubernetes JWTs), delivery channel, conflict alerting | Never run against a real Agones cluster; alert validated only statically, not against live Prometheus/Alertmanager traffic | | 8.12 `[D:8.11]` | Kubernetes hardening baseline, rate/quota limiting, degraded-mode gate | Private-store provisioning, distributed/global quotas, edge DDoS/WAF/origin shielding, encrypted backups, live policy/load tests remain | -| 8.13 `[D:8.12]` | Digest-pinned images, supply-chain policy checker | Registry SBOM/scan/sign/admission execution and a concrete production overlay remain | +| 8.13 `[D:8.12]` | Digest-pinned images, supply-chain policy checker | Registry SBOM/scan/sign/admission execution and a concrete production overlay remain — the build-and-pin half is tracked by [#31](https://github.com/jcreek/CosmicClash/issues/31) | #### 8C — Queueing, matchmaking, playlists and rating @@ -211,10 +225,10 @@ are done; everything below is what's left on the tasks still open. |---|---|---| | 8.14 `[D:8.4,8.5,8.8]` | Queue policy (ownership, heartbeat/expiry, candidate projection) | Live Redis failover-under-load and worker integration remain | | 8.15 `[D:7.8,8.3]` | Probe validation (RTT, nonce/freshness/region, quarantine), `POST /v1/probes/{region}/challenge`, durable single-use nonces, client probe collection before queueing, candidate-index refresh after probe | Steam coordinator ping-location source remains (a placeholder blob is sent without a Steam runtime); multi-region endpoint deployment remains | -| 8.16 `[D:8.14,8.15]` | Candidate/team formation, matcher worker | Long-running worker integration remains | +| 8.16 `[D:8.14,8.15]` | Candidate/team formation, matcher worker | The matcher is a real long-running poll loop and now has casual and ranked Deployments in `deploy/k8s/base`; what remains is live soak against a cluster rather than the integration itself ([#17](https://github.com/jcreek/CosmicClash/issues/17)) | | 8.17 `[D:8.14,8.16]` | Proposal policy (response window, cooldowns, offender/innocent split) | Live PostgreSQL execution and allocation integration remain | | 8.18 `[D:8.5,8.14,8.17]` | Store layer (serializable retries, claim SQL, atomic promotion) | Allocation runtime integration remains | -| 8.19 `[D:8.18]` | Casual lineup (2–6 humans, bot backfill) | Queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties, live integration remain | +| 8.19 `[D:8.18]` | Casual lineup (2–6 humans, bot backfill) | Casual lineup formation is built and wired (`domain.BuildCasualLineup`, called from `domain/formation.go` and `domain/noshow.go`), and all four penalty kinds are durably written (`PROPOSAL_DECLINED`, `PROPOSAL_TIMEOUT`, `INITIAL_CONNECT_NO_SHOW`, `MATCH_ABANDONED`). What genuinely remains is the opt-in 10 s **backfill proposal** path: `domain.CanCasualBackfill`/`CasualBackfillPenalty` exist as policy, but no matcher code path ever creates a proposal to fill a slot in an in-progress match. Live integration also remains | | 8.20 `[D:8.18]` | Ranked admission (six unique verified humans) | Done. Allocation wiring was already complete end to end (allocator sets the `cosmic-clash.io/arena-path` annotation → `supervisor.withAllocatedCompatibility` maps it to `--arena-path` → `server_boot.gd` → `ServerMatchLoop.allocated_arena_path`), with coverage at each hop. `ArenaRegistry` integration is now a cross-language guard rather than a shared list: `server/domain/ranked.go` must keep its own ranked-eligible subset (the choice is server-authoritative and made before any Godot process exists), so `arena_registry_sync_test.go` parses `arena_registry.gd` and fails if the two disagree in either direction, if rotation order diverges, or if a ranked path has no scene behind it. Verified against four drift scenarios including promoting an elevated variant, which the registry's own comment anticipates. Live ranked admission against a real cluster remains ([#17](https://github.com/jcreek/CosmicClash/issues/17)) | | 8.21 `[D:8.5,8.20]` | Rating core (Glicko-2, weights, transactional updates) | Live maintenance/DB execution remains | | 8.22 `[D:8.21]` | Ranked profile (provisional games, tiers) | Persisted tier policy done: bands live in `tier_bands`, seeded with the exact compiled launch policy so storage changed without behaviour changing, loaded at startup with a malformed policy failing startup rather than silently mis-tiering, and an empty table falling back to the compiled default so an operator can truncate back to known-good. Retuning is now a rolling restart rather than a rebuilt image. `PROVISIONAL` is rejected as a durable band, being derived from game count rather than rating. Client UI was already built (`RankedProfileState.display_text()` renders tier, provisional status, ranked games and the season countdown). Reconnect transport is tracked by 8.42 and depends on live auth/backend events | @@ -230,7 +244,7 @@ are done; everything below is what's left on the tasks still open. | 8.27 `[D:8.26]` | Supervisor package (Agones discovery, Ready transition) | Metadata watch, real Agones annotation/shutdown confirmation, emulator integration remain | | 8.28 `[D:8.6,8.27]` | Process-ready/Agones-Ready separation, control-plane registration | Remaining gates are live Agones annotation/shutdown behavior and production cluster readiness — see task 8.49 | | 8.29 `[D:8.26,8.27]` | Dynamic port/SDR env propagation | Real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT, multi-match fixture remain | -| 8.30 `[D:8.18,8.26,8.28,8.29]` | Allocation leasing, compatibility validation | Full unknown-outcome cluster recovery and signed roster metadata remain | +| 8.30 `[D:8.18,8.26,8.28,8.29]` | Allocation leasing, compatibility validation | Signed roster metadata landed with 8.31 — the allocator publishes one signed join authorisation per participant plus a manifest committing to a digest over the whole roster, and the supervisor materialises it before starting the game child. Full unknown-outcome cluster recovery remains ([#17](https://github.com/jcreek/CosmicClash/issues/17)) | | **8.31** `[D:8.9,8.30]` | Signed assignment/roster persistence, player recovery | **Done — this was the root blocker of the allocation-to-connect pipeline.** `allocator.Worker.RunOnce` now builds one signed join authorisation per durable participant and calls `PublishRoster` after binding; `cmd/allocator` takes `--join-authorisations-key-file`/`--join-authorisations-key-id` and refuses to start without them. The signing design is settled: HMAC-SHA256 over the canonical claim bytes with a key ID inside them, so servers hold a key *set* and rotation does not invalidate in-flight matches. The provider endpoint is now persisted on the allocation so a worker crashing between allocating and publishing can retry. Verified by an integration test that drives the real worker through the supervisor's own roster read path without seeding `assignments`. Live Agones verification remains ([#17](https://github.com/jcreek/CosmicClash/issues/17)) | | 8.32 `[D:8.2,8.26,8.30]` | FleetAutoscaler baseline, Ready buffer | Regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99, N+1 certification remain | | 8.33 `[D:8.26,8.32]` | Fleet scheduling, zone spread | Regional node pools, forced node-loss testing, measured N+1 headroom remain | @@ -247,8 +261,8 @@ are done; everything below is what's left on the tasks still open. | 8.39 `[D:8.3,8.14,8.17]` | `MatchmakingState`/`ControlPlaneClient`, queue/proposal UI, targeted revisioned events | Cross-replica fan-out landed: committed outbox events are published through PostgreSQL LISTEN/NOTIFY so the replica owning a subscriber's WebSocket delivers it, rather than whichever replica happened to drain the row. Verified against real PostgreSQL with two listeners. Live multi-replica verification under load remains | | 8.40 `[D:8.3,8.14]` | Revisioned event stream, REST resync, outbox dispatcher | Allocator and Redis fan-out live verification remain | | 8.41 `[D:7.8,8.9,8.31,8.40]` | Player-scoped assignment API, `connect_to_assignment()` wiring, join-authorisation verification in `MatchNet` | SDR relay-ticket installation and live Agones cluster integration remain | -| 8.42 `[D:8.22,8.23,8.24,8.40]` | `RankedProfileState`, backend-authoritative rating/tier display | Committed revision after reconnect, abandon status, season countdown remain dependent on live auth/backend events and Godot runtime verification | -| 8.43 `[D:8.39,8.40,8.41]` | Error/expiry UX, generic mutation retry, version-mismatch and failed-reconnect messaging | Long-running worker integration (§8.16) remains | +| 8.42 `[D:8.22,8.23,8.24,8.40]` | `RankedProfileState`, backend-authoritative rating/tier display | Season countdown is implemented (`RankedProfileState.display_text()` renders the remaining days alongside tier, provisional status and ranked games). Committed revision after reconnect and abandon status remain dependent on live auth/backend events and Godot runtime verification | +| 8.43 `[D:8.39,8.40,8.41]` | Error/expiry UX, generic mutation retry, version-mismatch and failed-reconnect messaging | Long-running worker soak (§8.16) remains; the worker itself is deployed | #### 8F — Observability, verification, cost and rollout @@ -262,7 +276,7 @@ are done; everything below is what's left on the tasks still open. | 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | Disposable kind+Agones cluster runner | CI/live evidence for production control-plane registration, roster/no-show, both readiness stages, races, multi-match node, result-pending reconciliation, drain, rollback remains open. Blocked locally on kind/Helm availability | | 8.50 `[D:8.25,8.37,8.43,8.49]` | Chaos recovery (stale allocation, no-penalty requeue) | **Local complete; production gate open** — 100 ms RTT/jitter/loss, matcher/client restart, game-pod death, node drain, Redis failover, control-plane loss, live chaos evidence remain | | 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | 10,000-client API load gate | **Local complete; production gate open** — PostgreSQL saturation, durable matcher fencing under load, forecast launch concurrency ×2, replica scaling remain live infrastructure gates | -| 8.52 `[D:8.32,8.34,8.45,8.51]` | Per-replica + shared regional allocator quota | Real image digest/secrets, measured regional cost model, threshold tuning, denial-of-wallet rehearsal remain | +| 8.52 `[D:8.32,8.34,8.45,8.51]` | Per-replica + shared regional allocator quota | Real image digest/secrets ([#31](https://github.com/jcreek/CosmicClash/issues/31)), measured regional cost model, threshold tuning, denial-of-wallet rehearsal remain | | 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | Fail-closed release-gate promotion validator | Actual reports, production rollback rehearsal, regional playtests, live promotion remain open | Implementation invariants for every task above: