CosmicClash/multiplayer-next.md

# Multiplayer — next work

Short, current checklist for online multiplayer. Historical decisions,
implementation evidence and task-level acceptance criteria stay in
[`multiplayer-todo.md`](multiplayer-todo.md). Phase 8 architecture and locked
product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md).

## Existing release blockers

- [ ] **Phase 4 playtest:** play at roughly 100 ms RTT; confirm ship/ball
  interaction feels local and contact corrections read as bumps, not glitches.
- [ ] **Phase 5 session:** finish a real 3v3 match with a mid-match disconnect
  and late joiner.
- [ ] **Phase 6 external check:** play the exported Docker server from separate
  internet machines. Keep the check controlled until verified identity lands.

## Phase 7 — production Steam prerequisite

- [ ] Obtain the pinned GodotSteam client/server builds and Steamworks SDK;
  pass `make verify-steam-templates` without weakening ENet verification.
- [ ] Validate two real accounts through the explicit Steam transport and
  build Internet/LAN/favourites/history server-browser views.
- [ ] Add single-use auth tickets, asynchronous server validation, verified
  Steam identity, identity-keyed reconnect and persistent bans.
- [ ] Obtain the real App ID, publisher key, coordinator SDK/signing approval,
  certificates and Hosted Dedicated Server data-centre support from Valve.
- [ ] Implement ticketed Hosted Dedicated Server SDR routing, ticket install,
  reconnect and expiry. Preserve direct ENet for local/CI/community servers.

## Phase 8 — architecture, contracts and durable data

- [x] Lock the Go/PostgreSQL/Redis, Kubernetes/Agones, SDR, EU/NA and
  provider-portability ADR ([ADR-001](docs/ADR-001-matchmaking-platform.md));
  measurable launch SLOs are defined in
  [MATCHMAKING-SLOs.md](docs/MATCHMAKING-SLOs.md).
- [x] Publish versioned OpenAPI/WebSocket contracts, stable IDs, legal state
  transitions, revisions and idempotency semantics ([v1 contracts](server/contracts/v1/)).
- [ ] **IN PROGRESS:** Add PostgreSQL queue ownership/active-participation
  fences, durable domain migrations/outbox and Redis indexes/TTLs; lost Redis
  writes must not split a proposal or corrupt durable state. Initial migration
  and serializable store boundaries are implemented, including durable queue
  create/heartbeat/cancel/recovery adapters; an opt-in pgx/Docker harness now
  executes the migrations and real queue create/idempotency/ownership/recovery
  path, an executable migration runner now serializes and records forward
  application, and a TTL-bound Redis candidate index now supports atomic rebuild,
  snapshot and removal with durable-source repair on partial/malformed cache
  state; proposal/result transactions and live Redis restart/failover gates
  remain. The matcher package now performs bounded candidate formation and
  delegates the final proposal claim to the durable transaction boundary;
  queue tickets now also retain server-derived probe RTT metadata for
  authoritative matcher reads; ranked metadata/provider wiring and live
  Redis repair remain. The authenticated probe API now records validated
  server-computed RTT values into the active player's durable queue ticket and
  fails closed when that write is unavailable; Steam/coordinator evidence
  acquisition and multi-region probe population remain.
- [ ] **IN PROGRESS:** Durable allocator registry now records READY GameServer
  projections and atomically claims compatible capacity with replay/conflict
  fencing; `server/agones` now submits and validates namespaced
  `GameServerAllocation` responses, including dynamic address/port data;
  `server/allocator` now requires provider allocation reconciliation into the
  durable registry before returning an endpoint; allocator-facing roster
  publication now requires an allocated endpoint and verifies canonical
  join-authorisation signatures before exposing player rows; live Agones
  integration remains.
- [ ] **IN PROGRESS:** Run the Go control plane against PostgreSQL/Redis with
  independently runnable API, matcher, allocator and maintenance roles. The
  `cmd/control-plane` API role now opens PostgreSQL, applies migrations, wires
  authenticated durable queue/proposal/assignment/session adapters, and shuts
  down gracefully; optional `--redis-addr` publishes queue mutations to a
  TTL-bound best-effort candidate projection without making Redis authoritative;
  a runnable casual `cmd/matcher` role now polls PostgreSQL and delegates
  proposal claims to the durable transaction; `cmd/maintenance` now runs
  bounded ranked-season rollover batches with signal-bound shutdown;
  provider-backed allocation, ranked provider wiring, Redis worker wiring and
  live service checks remain.
- [ ] **IN PROGRESS:** Define assignment compatibility and opt-in `ServerConfig`
  flags whose defaults reproduce the community-server path. Allocation manifest
  validation now covers client build and future expiry; allocated servers now
  expose loopback process-ready/drain control, an Agones REST bridge for
  health/lifecycle calls, and fence new admissions while draining; signed
  admission remains.

## Phase 8 — identity and security

- [ ] **IN PROGRESS:** Validate Steam Web API tickets only in the secure backend;
  issue revocable sessions and reconnect-safe match/identity/slot authorisations
  with server-owned connection-generation fencing. Pure Go ticket/session and
  reconnect policies exist, including canonical signed join-authorisation
  issuance/verification; production Steam/backend adapters and persistent
  lease fencing remain.
- [ ] **IN PROGRESS:** Authenticate results with pod/GameServer-bound workload
  identity; make identical duplicates idempotent and conflicting results
  inert/alerting. Pure Go credential-claim validation, binding, hashing,
  reconciliation, and the atomic receipt/completion/outbox SQL boundary exist;
  projected-token/JWT adapters, trusted-cluster verification, rating-lock
  integration, and production alerting remain. A dependency-free projected JWT
  adapter now verifies the compact-token signature through an injected trust
  boundary and delegates exact claim/time binding to the domain policy.
- [x] Complete the threat model for forgery, replay, queue/flood/bot abuse,
  workload/insider compromise, DDoS, supply chain and denial-of-wallet
  ([THREAT-MODEL.md](docs/THREAT-MODEL.md)).
- [ ] **IN PROGRESS:** Enforce restricted workloads/RBAC/networks/private
  stores/backups/secrets; isolate SDR signing behind an audited non-exportable
  signer and add volumetric edge defense, WebSocket limits and overload
  shedding. A provider-neutral restricted Kubernetes baseline and structural
  policy tests now exist; live edge/data-plane controls remain.
- [ ] **IN PROGRESS:** Pin, scan, SBOM and sign artifacts; verify signatures at
  admission and document the critical vulnerability SLA. Repository image
  references are now digest-pinned with a static secret-hygiene guard and a
  24-hour critical-fix policy; registry execution and concrete release
  provenance remain.

## Phase 8 — queues, playlists and rating

- [ ] **IN PROGRESS:** Add one PostgreSQL-owned queue ticket/player with 10 s
  heartbeat, 30 s expiry, Redis candidate cache and restart/failover repair.
  Authenticated queue creation now requires playlist, client build and
  protocol version and passes them to the server-owned candidate provider;
  PostgreSQL/Redis wiring remains.
- [ ] **IN PROGRESS:** Validate opaque Steam ping locations and nonce-bound probes server-side;
  require <=100 ms, enforce discrepancy quarantine and the locked widening/
  region/team tie-break rules. Authenticated probe transport now routes opaque
  location/nonce data through a server-owned evidence provider and refuses
  client RTT values; Steam/coordinator adapters remain.
- [ ] **IN PROGRESS:** Form deterministic candidate sets and balanced teams from
  the server-owned queue projection. Queue-backed oldest-anchor formation and
  duplicate-player fencing now exist; durable matcher claims remain.
- [ ] **IN PROGRESS:** Send 10 s proposals to every selected human: ranked six, relaxed casual
  two to six with disclosed bots; enforce exact cooldown and queue-precedence
  behavior. Playlist-aware proposal preparation now validates casual team humans
  and ranked identity/arena metadata before creating proposal state; durable
  queue precedence and allocation integration remain.
- [ ] **IN PROGRESS:** Fence proposals/participants in a PostgreSQL serializable transaction;
  prove loss of an acknowledged Redis write cannot split players. The Go store
  adapter now performs proposal insertion, participant insertion, and every
  queue-ticket promotion in one rollback-safe SERIALIZABLE callback; live DB/
  Redis failover testing remains.
- [ ] Casual: target 3v3 humans, after 60 s allow >=2 humans (one/team) plus
  bots, kickoff-only human backfill and no backfill loss/decline penalty.
- [ ] **IN PROGRESS:** Ranked: exactly six humans, solo-only, no bots/backfill,
  random-enabled non-elevated arenas only, 60 s reconnect grace and escalating abandons.
- [ ] **IN PROGRESS:** Implement the documented exact Glicko-2 equations, fractional 3v3
  weights, inactivity/update locking/golden vectors and ten provisional games.
  Backend-owned provisional status and validated ranked-tier derivation now exist;
  authenticated ranked-profile transport now exists; client display and
  persisted tier configuration remain.
- [ ] **IN PROGRESS:** Add ranked-only exactly-once 12-week soft seasons; distinguish retryable
  result-delivery outages from match-integrity failures and rating exemptions.
  A durable per-player/per-season rollover marker and SERIALIZABLE rating update
  boundary now exist; live scheduler/DB execution remains.

## Phase 8 — Agones and regional server capacity

- [ ] **IN PROGRESS:** Add portable EU/NA Agones Fleets with provider
  edge/network/secret and Valve-approved SDR POP/certificate/public-UDP
  overlays. A restricted provider-neutral Fleet base and distinct EU/NA
  Kustomize overlays now exist; live rendering and provider/Valve overlays
  remain.
- [ ] Add the local-safe Agones adapter and separate process-ready (listen then
  Ready) from assignment-ready (Allocated manifest verified and registered).
  The Go supervisor now validates dynamic address/port data and gates Ready on
  an explicit probe; Godot adapter and emulator integration remain.
- [ ] **IN PROGRESS:** Allocate from Ready by region/build/protocol/transport; use separately
  verified ENet and SDR dynamic/passthrough port mappings. The Go allocator
  now owns assignment publication with idempotent replay/conflict handling;
  Agones integration remains.
- [ ] Deliver/verify the signed roster after allocation and expose client
  tickets only after backend `assignment_ready`.
- [ ] **IN PROGRESS:** Keep >=2 Ready processes across >=2 on-demand
  nodes/failure domains per queue-enabled region; only Allocated count may
  fall to zero. A provider-neutral Agones FleetAutoscaler now encodes a
  two-process Ready buffer and six-process warm cap; regional node pools,
  pre-pull rollout and measured N+1 capacity remain.
- [ ] **IN PROGRESS:** Spread on-demand capacity across zones with N+1
  headroom; do not place live matches on interruptible nodes. The Fleet now
  requires the on-demand capacity label and uses a zone topology spread
  constraint; force-loss testing of the largest node and measured headroom
  remain.
- [ ] Benchmark native x86_64 boot, p99 CPU/RSS/network and tick health; set
  requests/limits and node density from measurements plus 30% headroom.
- [ ] Add 30 s no-show handling, Go PID-1 TERM/drain supervision, PDB/Fleet
  drain, signed result annotation/retry, RPO <=5 m and RTO <=30 m. The Go
  supervisor now owns bounded drain-before-kill orchestration, the drain
  boundary is authenticated and loopback-only, and the base PDB protects the
  two-Ready floor; lifecycle/PDB/Fleet integration remains.
- [ ] Rehearse migration only after the second provider's EU/NA locations have
  Valve approval, POP/certs, public UDP/firewall and coordinator trust.

## Phase 8 — client and recovery

- [ ] Build queue/proposal/allocation/connect/rating UI with explicit latency,
  capacity, expiry and recovery states.
- [ ] Use one authenticated revisioned WebSocket plus REST resync; resume a
  valid ticket/assignment after restart rather than duplicating it.
- [ ] After assignment-ready, install SDR ticket and send reconnect-safe join
  authorisation in `hello`; fence old connections and retain ENet behavior.
- [ ] Display only backend-authoritative provisional rank/tier/delta, abandon
  status and season time; clients perform no rating calculation.

## Phase 8 — operations and release gates

- [ ] Correlate queue→result with IDs and add dashboards/alerts for SLOs,
  security, failures and cost without logging credentials.
- [ ] Add Go race/fuzz/property/migration/concurrency coverage plus fake Steam
  and fake allocation for offline deterministic CI.
- [ ] Add an independent allocated-server Compose flow; do not mutate
  `compose.phase6-smoke.yml` or weaken either existing Make gate.
- [ ] Add disposable `kind`/Agones integration, 100 ms network/chaos cases and
  proof that infrastructure failures cannot punish players.
- [ ] Load-test >=10,000 queued clients, >=100 proposals/s and forecast launch
  concurrency x2 while holding API p95 <=250 ms and allocation correctness.
- [ ] Record cost per completed match, budget/denial-of-wallet controls and
  deploy progressively: internal → casual canary → casual → provisional
  ranked → ranked, with EU/NA playtests and rollback gates.

## Known issues before public hosting

- [ ] Replace display-name slot reclaim with verified Steam identity.
- [ ] Investigate occasional transport input loss during a long server stall.
- [x] Fix the remaining `_broadcast_snapshot` packet-send stderr race.

## Decide after the latency playtest

- [ ] Decide whether client-only contact-cohort shadow physics is worthwhile.

## Explicitly deferred

Parties/premades, tournaments, ranked spectators, non-Steam identity,
additional global regions, global leaderboards, 120 Hz simulation,
latency-gap measurement, audio hooks and split-screen are not in the launch
path.