Multiplayer — next work

Short, current checklist for online multiplayer. Historical decisions, implementation evidence and task-level acceptance criteria stay in multiplayer-todo.md. Phase 8 architecture and locked product policy are in docs/MATCHMAKING.md.

Existing release blockers

Phase 4 playtest: play at roughly 100 ms RTT; confirm ship/ball interaction feels local and contact corrections read as bumps, not glitches.
Phase 5 session: finish a real 3v3 match with a mid-match disconnect and late joiner.
Phase 6 external check: play the exported Docker server from separate internet machines. Keep the check controlled until verified identity lands.

Phase 7 — production Steam prerequisite

Obtain the pinned GodotSteam client/server builds and Steamworks SDK; pass make verify-steam-templates without weakening ENet verification.
Validate two real accounts through the explicit Steam transport and build Internet/LAN/favourites/history server-browser views.
Add single-use auth tickets, asynchronous server validation, verified Steam identity, identity-keyed reconnect and persistent bans.
Obtain the real App ID, publisher key, coordinator SDK/signing approval, certificates and Hosted Dedicated Server data-centre support from Valve.
Implement ticketed Hosted Dedicated Server SDR routing, ticket install, reconnect and expiry. Preserve direct ENet for local/CI/community servers.

Phase 8 — architecture, contracts and durable data

Lock the Go/PostgreSQL/Redis, Kubernetes/Agones, SDR, EU/NA and provider-portability ADR (ADR-001); measurable launch SLOs are defined in MATCHMAKING-SLOs.md.
Publish versioned OpenAPI/WebSocket contracts, stable IDs, legal state transitions, revisions and idempotency semantics (v1 contracts).
IN PROGRESS: Add PostgreSQL queue ownership/active-participation fences, durable domain migrations/outbox and Redis indexes/TTLs; lost Redis writes must not split a proposal or corrupt durable state. Initial migration and serializable store boundaries are implemented, including durable queue create/heartbeat/cancel/recovery adapters; an opt-in pgx/Docker harness now executes the migrations and real queue create/idempotency/ownership/recovery path, an executable migration runner now serializes and records forward application, and a TTL-bound Redis candidate index now supports atomic rebuild, snapshot and removal with durable-source repair on partial/malformed cache state; proposal/result transactions and live Redis restart/failover gates remain. The matcher package now performs bounded candidate formation and delegates the final proposal claim to the durable transaction boundary; the runnable matcher supports casual and explicitly enabled ranked six-player polling with durable Steam identity metadata lookup. Queue tickets now also retain server-derived probe RTT metadata for authoritative matcher reads; ranked arena selection and live Redis repair remain. The authenticated probe API now records validated server-computed RTT values into the active player's durable queue ticket and fails closed when that write is unavailable; Steam/coordinator evidence acquisition and multi-region probe population remain.
IN PROGRESS: Durable allocator registry now records READY GameServer projections and atomically claims compatible capacity with replay/conflict fencing; server/agones now submits and validates namespaced GameServerAllocation responses, including dynamic address/port data; server/allocator now requires provider allocation reconciliation into the durable registry before returning an endpoint and exposes an accepted-proposal gate that validates unanimous responses and playlist/participant invariants; allocator-facing roster publication now requires an allocated endpoint and verifies canonical join-authorisation signatures before exposing player rows; live Agones integration remains.
IN PROGRESS: Run the Go control plane against PostgreSQL/Redis with independently runnable API, matcher, allocator and maintenance roles. The cmd/control-plane API role now opens PostgreSQL, applies migrations, wires authenticated durable queue/proposal/assignment/session adapters, and shuts down gracefully; optional --redis-addr publishes queue mutations to a TTL-bound best-effort candidate projection without making Redis authoritative; a runnable casual cmd/matcher role now polls PostgreSQL and delegates proposal claims to the durable transaction; cmd/maintenance now runs bounded ranked-season rollover batches with signal-bound shutdown; provider-backed allocation, allocator runtime wiring, Redis failover and live service checks remain.
IN PROGRESS: Define assignment compatibility and opt-in ServerConfig flags whose defaults reproduce the community-server path. Allocation manifest validation now covers client build and future expiry; allocated servers now expose loopback process-ready/drain control, an Agones REST bridge for health/lifecycle calls, and fence new admissions while draining; signed admission remains.

Phase 8 — identity and security

IN PROGRESS: Validate Steam Web API tickets only in the secure backend; issue revocable sessions and reconnect-safe match/identity/slot authorisations with server-owned connection-generation fencing. Pure Go ticket/session and reconnect policies exist, including canonical signed join-authorisation issuance/verification; production Steam/backend adapters and persistent lease fencing remain.
IN PROGRESS: Authenticate results with pod/GameServer-bound workload identity; make identical duplicates idempotent and conflicting results inert/alerting. Pure Go credential-claim validation, binding, hashing, reconciliation, and the atomic receipt/completion/outbox SQL boundary exist; projected-token/JWT adapters, trusted-cluster verification, rating-lock integration, and production alerting remain. The API now exposes the workload-authenticated server result route and delegates completion to the durable receipt/outbox adapter. A dependency-free projected JWT adapter now verifies the compact-token signature through an injected trust boundary and delegates exact claim/time binding to the domain policy.
Complete the threat model for forgery, replay, queue/flood/bot abuse, workload/insider compromise, DDoS, supply chain and denial-of-wallet (THREAT-MODEL.md).
IN PROGRESS: Enforce restricted workloads/RBAC/networks/private stores/backups/secrets; isolate SDR signing behind an audited non-exportable signer and add volumetric edge defense, WebSocket limits and overload shedding. A provider-neutral restricted Kubernetes baseline and structural policy tests now exist; live edge/data-plane controls remain.
IN PROGRESS: Pin, scan, SBOM and sign artifacts; verify signatures at admission and document the critical vulnerability SLA. Repository image references are now digest-pinned with a static secret-hygiene guard and a 24-hour critical-fix policy; registry execution and concrete release provenance remain.

Phase 8 — queues, playlists and rating

IN PROGRESS: Add one PostgreSQL-owned queue ticket/player with 10 s heartbeat, 30 s expiry, Redis candidate cache and restart/failover repair. Authenticated queue creation now requires playlist, client build and protocol version and passes them to the server-owned candidate provider; PostgreSQL/Redis wiring remains.
IN PROGRESS: Validate opaque Steam ping locations and nonce-bound probes server-side; require <=100 ms, enforce discrepancy quarantine and the locked widening/ region/team tie-break rules. Authenticated probe transport now routes opaque location/nonce data through a server-owned evidence provider and refuses client RTT values; Steam/coordinator adapters remain.
IN PROGRESS: Form deterministic candidate sets and balanced teams from the server-owned queue projection. Queue-backed oldest-anchor formation and duplicate-player fencing now exist; durable matcher claims remain.
IN PROGRESS: Send 10 s proposals to every selected human: ranked six, relaxed casual two to six with disclosed bots; enforce exact cooldown and queue-precedence behavior. Playlist-aware proposal preparation now validates casual team humans and ranked identity/arena metadata before creating proposal state; durable queue precedence and allocation integration remain.
IN PROGRESS: Fence proposals/participants in a PostgreSQL serializable transaction; prove loss of an acknowledged Redis write cannot split players. The Go store adapter now performs proposal insertion, participant insertion, and every queue-ticket promotion in one rollback-safe SERIALIZABLE callback; accepted proposals now atomically promote their exact team/slot map and tickets into an ALLOCATING match; final unanimous proposal acceptance now invokes this replay-safe promotion through the API; the runnable casual matcher can optionally use a Redis candidate projection and repairs an empty/lost index from PostgreSQL before claiming durably; live DB/Redis failover testing remains.
Casual: target 3v3 humans, after 60 s allow >=2 humans (one/team) plus bots, kickoff-only human backfill and no backfill loss/decline penalty.
IN PROGRESS: Ranked: exactly six humans, solo-only, no bots/backfill, random-enabled non-elevated arenas only, 60 s reconnect grace and escalating abandons.
IN PROGRESS: Implement the documented exact Glicko-2 equations, fractional 3v3 weights, inactivity/update locking/golden vectors and ten provisional games. Certified result completion now applies the canonical per-player update inside the same durable transaction, with lexical rating locks and ranked-game revision increments. Backend-owned provisional status and validated ranked-tier derivation now exist; authenticated ranked-profile transport now exists; client display and persisted tier configuration remain.
IN PROGRESS: Add ranked-only exactly-once 12-week soft seasons; distinguish retryable result-delivery outages from match-integrity failures and rating exemptions. A durable per-player/per-season rollover marker and SERIALIZABLE rating update boundary now exist; live scheduler/DB execution remains.

Phase 8 — Agones and regional server capacity

IN PROGRESS: Add portable EU/NA Agones Fleets with provider edge/network/secret and Valve-approved SDR POP/certificate/public-UDP overlays. A restricted provider-neutral Fleet base and distinct EU/NA Kustomize overlays now exist; live rendering and provider/Valve overlays remain.
Add the local-safe Agones adapter and separate process-ready (listen then Ready) from assignment-ready (Allocated manifest verified and registered). The Go supervisor now validates dynamic address/port data and gates Ready on an explicit probe; Godot adapter and emulator integration remain.
IN PROGRESS: Allocate from Ready by region/build/protocol/transport; use separately verified ENet and SDR dynamic/passthrough port mappings. The Go allocator now owns assignment publication with idempotent replay/conflict handling; Agones integration remains.
Deliver/verify the signed roster after allocation and expose client tickets only after backend assignment_ready.
IN PROGRESS: Keep >=2 Ready processes across >=2 on-demand nodes/failure domains per queue-enabled region; only Allocated count may fall to zero. A provider-neutral Agones FleetAutoscaler now encodes a two-process Ready buffer and six-process warm cap; regional node pools, pre-pull rollout and measured N+1 capacity remain.
IN PROGRESS: Spread on-demand capacity across zones with N+1 headroom; do not place live matches on interruptible nodes. The Fleet now requires the on-demand capacity label and uses a zone topology spread constraint; force-loss testing of the largest node and measured headroom remain.
Benchmark native x86_64 boot, p99 CPU/RSS/network and tick health; set requests/limits and node density from measurements plus 30% headroom.
Add 30 s no-show handling, Go PID-1 TERM/drain supervision, PDB/Fleet drain, signed result annotation/retry, RPO <=5 m and RTO <=30 m. The Go supervisor now owns bounded drain-before-kill orchestration, the drain boundary is authenticated and loopback-only, and the base PDB protects the two-Ready floor; lifecycle/PDB/Fleet integration remains.
Rehearse migration only after the second provider's EU/NA locations have Valve approval, POP/certs, public UDP/firewall and coordinator trust.

Phase 8 — client and recovery

Build queue/proposal/allocation/connect/rating UI with explicit latency, capacity, expiry and recovery states.
Use one authenticated revisioned WebSocket plus REST resync; resume a valid ticket/assignment after restart rather than duplicating it.
After assignment-ready, install SDR ticket and send reconnect-safe join authorisation in hello; fence old connections and retain ENet behavior.
Display only backend-authoritative provisional rank/tier/delta, abandon status and season time; clients perform no rating calculation.

Phase 8 — operations and release gates

Correlate queue→result with IDs and add dashboards/alerts for SLOs, security, failures and cost without logging credentials.
Add Go race/fuzz/property/migration/concurrency coverage plus fake Steam and fake allocation for offline deterministic CI.
Add an independent allocated-server Compose flow; do not mutate compose.phase6-smoke.yml or weaken either existing Make gate.
Add disposable kind/Agones integration, 100 ms network/chaos cases and proof that infrastructure failures cannot punish players.
Load-test >=10,000 queued clients, >=100 proposals/s and forecast launch concurrency x2 while holding API p95 <=250 ms and allocation correctness.
Record cost per completed match, budget/denial-of-wallet controls and deploy progressively: internal → casual canary → casual → provisional ranked → ranked, with EU/NA playtests and rollback gates.

Known issues before public hosting

Replace display-name slot reclaim with verified Steam identity.
Investigate occasional transport input loss during a long server stall.
Fix the remaining _broadcast_snapshot packet-send stderr race.

Decide after the latency playtest

Decide whether client-only contact-cohort shadow physics is worthwhile.

Explicitly deferred

Parties/premades, tournaments, ranked spectators, non-Steam identity, additional global regions, global leaderboards, 120 Hz simulation, latency-gap measurement, audio hooks and split-screen are not in the launch path.

15 KiB Raw Blame History