docs: finalize scalable matchmaking plan

This commit is contained in:
Josh Creek
2026-08-31 19:57:49 +01:00
parent bcc12aad19
commit 8d4a0640e2
5 changed files with 647 additions and 343 deletions
+5 -1
View File
@@ -34,7 +34,11 @@ There will be default bots available, trained using reinforcement learning, and
## MVP ## MVP
The first version of this game will be JUST the game, no server-side functionality at all. It will be a local only game where you can play against bots. Split-screen multiplayer could be added in a version 0.2 if demand is high enough. If there is sufficient interest then the server-side functionality can be added to enable online play, with a system in place to ensure that servers can be paid for (perhaps a cheap monthly subscription model?). The original local-only milestone is complete; the 1.0 scope now includes
dedicated online play plus casual and ranked matchmaking. Community servers
remain self-hostable, while project-hosted match servers are allocated per
match through the provider-portable control plane described in
[`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Split-screen remains deferred.
## Monetisation ## Monetisation
+423 -196
View File
@@ -1,243 +1,470 @@
# Matchmaking — casual and ranked queues # Matchmaking — casual and ranked queues
Design scope for online casual and ranked play. This is a **1.0 launch Architecture and locked product policy for Phase 8. This is a **1.0 launch
blocker**, not a post-launch addition. blocker**. The numbered, independently implementable tasks and their
acceptance criteria live in [`multiplayer-todo.md`](../multiplayer-todo.md);
the short live checklist is [`multiplayer-next.md`](../multiplayer-next.md).
Nothing described here is implemented yet. This doc exists to record the Nothing in Phase 8 is implemented yet. This document records the decisions
decisions and the reasoning before code is written; per-task implementation those tasks assume so an implementer does not have to redesign the system
evidence belongs in `multiplayer-todo.md` once work starts, and the live while building one part of it.
checklist lives in [`multiplayer-next.md`](../multiplayer-next.md).
## The model change ## 1. Model and non-negotiable constraints
The multiplayer that exists today is a **community-server** model. A The existing multiplayer is a community-server model: a dedicated server
dedicated server runs forever: it waits for `--min-players` by roster, runs continuously, waits for players, rotates arenas, and can be reached by
counts down `--start-countdown`, loads the next arena from the rotation, direct IP or the Phase 7 Steam browser. Matchmaking adds a second model:
plays a match, returns to the lobby, and repeats (`server_match_loop.gd`). players queue, every selected human accepts a proposal, one server process is
Players reach it by direct IP, and after Phase 7 by a Steam server browser. allocated for that match, and the process is destroyed after its result is
The server is the durable thing and players come and go around it. durably recorded. Ranked selects six humans; relaxed casual selects two to six
and discloses its bot-filled team composition before acceptance. Both models
ship.
Queued matchmaking inverts that. Players are the durable thing: they enter a Locked constraints:
queue, a matchmaker groups them by rating and region, and a **server is
allocated for that one match** and torn down afterwards. Both models can
coexist — community servers via the browser, queues via the matchmaker — and
they should, because the server browser is already most of the way to done.
## Hard prerequisite: verified identity - One authoritative Godot process hosts exactly one match.
- Solo 3v3 casual and ranked queues launch first. Party-shaped fields are
reserved in contracts, but party formation is deferred.
- Production matchmade traffic uses ticketed Steam Hosted Dedicated Server
SDR. Direct ENet remains first-class for local development, CI, LAN,
self-hosting, and community servers.
- The control plane is Go, PostgreSQL, and Redis, deployed on Kubernetes.
Agones owns game-server allocation and lifecycle.
- Infrastructure is provider-portable. Provider-specific cluster, network,
DNS, and secret-store configuration lives behind isolated deployment
overlays; application code never calls a provider allocation API.
- Launch game fleets run in Europe and North America. Placement is
latency-first and never silently violates the ping ceiling to shorten a
queue.
- Phase 8 features are opt-in. With allocated mode disabled, `ServerConfig`,
ENet, Docker Compose, and the existing community-server behavior remain
unchanged.
Ranked cannot ship before Phase 7's Steam auth tickets. ## 2. Steam and trust boundaries
Slot reclaim is currently keyed by **display name** (see Phase 7 verified Steam identity is a hard prerequisite. The current slot
`--slot-reservation-seconds`, and the known-issues list in reservation is keyed by display name, so no public queue or rating may rely on
`multiplayer-next.md`). A rating attached to a spoofable identity is worse it.
than no rating at all: it is trivially farmed, and it invites players to
invest in a ladder that cannot be defended. "Ranked is critical" therefore
*raises* the priority of Steam identity rather than routing around it.
Casual queueing has a weaker requirement — it still needs stable identity for Two Steam credentials have different purposes and must not be conflated:
abandon penalties and ban enforcement, but the cost of a compromise is lower.
## Architecture 1. A client obtains a single-use Web API ticket for backend login. Only the
secure backend calls `AuthenticateUserTicket`, checks the expected App ID
and identity string, and turns the result into a revocable session. A
client-supplied SteamID is never identity.
2. For a formed match, the game coordinator creates a short-lived SDR relay
ticket authorising one player to one hosted server. The client installs it
before connecting. The server separately verifies the signed match join
authorisation before admitting the player to the assigned roster slot.
Decided: **Steam for identity, a project-owned backend for everything else.** Steam authentication/session tickets are single-use and their lifecycle must
include the appropriate cancel/end calls; identity is not valid until Steam's
asynchronous validation succeeds. Hosted SDR relay tickets are different:
they are short-lived, match/server/identity-scoped and deliberately cached for
reconnect. See [Steam authentication](https://partner.steamgames.com/doc/features/auth)
and the [`ISteamUserAuth` Web API](https://partner.steamgames.com/doc/webapi/isteamuserauth).
This reverses the "no backend" position stated in Ticketed Hosted Dedicated Server SDR is the production transport because it
[`TECH_STACK.md`](TECH_STACK.md) and `README.md`, which described the state hides player/server IP addresses and authenticates, encrypts, and rate-limits
of the project before matchmaking was scoped. The dedicated server remains a traffic. It also supplies relay routing that may improve the path. It requires
Godot export; the new service is separate from it. a real App ID, coordinator SDK/signing approval, certificates, and hosted
data-centre coordination with Valve; those are explicit release dependencies,
not assumptions. See [Steam Datagram Relay](https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay).
The alternative — Steam-native matchmaking (lobbies plus Leaderboards or User ### Trust table
Stats as the rating store) — was rejected on two grounds. Steam lobby
matchmaking has no real concept of a skill distribution to match against, and
Leaderboards are a display surface rather than a rating store with the
transactional guarantees a ladder needs. It would also permanently bind the
game to Steam, foreclosing other platforms.
### Components | Input | Trusted only after | May affect |
| Component | Runs where | Responsibility |
| --- | --- | --- | | --- | --- | --- |
| Steam auth ticket validation | backend | Turn a client-supplied ticket into a verified SteamID via the Steamworks Web API. The only trusted source of identity. | | Steam Web API ticket | Backend validation for the expected App ID/identity and replay check | Backend session identity |
| Queue / matchmaker | backend | Hold queued players per playlist and region; form matches on rating proximity with a widening tolerance over wait time. | | Steam ping location + active-probe evidence | Backend verifies nonce/freshness and computes estimates; later compares with observed RTT | Placement only, never results |
| Rating store | backend (DB) | Per-identity, per-playlist rating and match history. Written only by the backend, never by a game client. | | Queue/accept request | Auth, schema/rate-limit, revision and idempotency validation | That player's queue state |
| Server allocator | backend | Start a dedicated-server instance per formed match, hand its address to the matched clients, reclaim it on exit. | | Join authorisation | Server signature, expiry, match/server/SteamID/slot and connection-generation validation | Initial admission or idempotent reclaim of that same slot |
| Dedicated server | Godot export | Unchanged simulation. Gains a mode where the roster is *assigned* rather than open, and reports a result at the end. | | Gameplay input | Existing server framing, sequence, byte and rate validation | Authoritative simulation input only |
| Game client | Godot | Queue UI, estimated wait, accept/decline, connect-on-assignment, post-match rating delta. | | Match result | Assigned server workload identity plus match/server binding | Transactional result/rating commit |
### What already exists and gets reused Clients never submit ratings, outcome, penalty exemptions, server health, or
allocation state. A dedicated server never holds the Steam publisher key or
the coordinator root signing key. Keep the offline SDR CA separate from the
online leaf ticket key. The online key is exposed only through a narrowly
authorised signer backed by KMS/HSM-equivalent non-exportable storage; API,
matcher, allocator and game-server pods cannot read it. The signer accepts
only allocator-recorded assignments, audits every signature, and supports
overlapping-key rotation.
The server side needs less new work than it looks: ## 3. Control-plane architecture
- **`--max-matches=1`** already makes the server drain and `exit(0)` after a Use one repository and shared domain packages, with independently runnable
single match. That is precisely the lifecycle a per-match allocator wants; roles rather than independently designed microservices:
it was built for CI, and it generalises for free.
- **`ServerConfig`** is a single-source-of-truth flag table with strict
validation — new allocation flags are declared in one place and are
automatically parsed, type-checked, config-file-backed and documented.
- **`--min-players` / `--start-countdown` / `--slot-reservation-seconds`**
are the match-formation primitives, and they already count *roster*
members rather than raw peers.
- **`MatchNet`'s roster** already survives the lobby→match transition, which
is the structure an assigned roster slots into.
- **`MatchState`** already has a legal-transition table with wire-stable
integer values, so new lifecycle states append cleanly.
### What is genuinely new | Role | Responsibility |
| --- | --- |
| API | HTTPS/WebSocket auth, profile, queue commands, status resync |
| Matcher | Atomic proposal formation from queue state |
| Allocator | Agones allocation, server registration, assignment delivery |
| Maintenance worker | Outbox delivery, season rollover, expiry, reconciliation |
- The backend service itself (process, deploy, DB, ops) — nothing like it API replicas are stateless. Redis sorted sets provide the fast candidate
exists in this repo today. index, but Redis is never the durable allocation fence: asynchronous failover
- Server-authoritative **match results**: the dedicated server must report can lose an acknowledged write. A matcher claims a proposal in a PostgreSQL
the outcome to the backend over a channel a client cannot forge. This is `SERIALIZABLE` transaction using a partial unique constraint that permits only
the first non-ENet/SDR network path in the project (see TECH_STACK's "no one active proposal/match participation per player. The transaction records
HTTP layer" note, which this supersedes). the proposal and participants before Redis cleanup; a stale Redis claim then
- An **assigned-roster** server mode: only the matched SteamIDs may take a loses at PostgreSQL and is repaired from the durable record. PostgreSQL is the
slot, replacing the current first-come model. source of truth for queue ownership, identities, sessions/revocations,
- Client-side queue UI and the accept/decline flow. seasons, ratings, matches, participants, penalties, result receipts, audits
and the transactional outbox. Redis holds expiring presence, candidate
indexes, latency evidence, session/revocation caches and transient fan-out.
Losing the last acknowledged Redis write may delay/rematerialise a ticket or
force a session cache miss, but can neither resurrect a revoked session, split
a proposal nor corrupt a result/rating.
## Server orchestration and autoscaling For launch, run the horizontally scaled control plane in one primary
Kubernetes region with a warm standby and tested restore path. Game fleets
remain regional in EU and North America. This avoids a premature multi-writer
database while keeping new-match control latency small relative to queue time.
An outage may pause new queues/allocations, but live matches must continue.
Targets are PostgreSQL RPO <= 5 minutes and control-plane RTO <= 30 minutes.
Requirement: game servers scale horizontally and automatically, spin up fast ### Stable identifiers and state
on demand, serve exactly one match, and shut down — so cost is incurred only
while a match is being played. Docker and the existing CI gates must keep
working unchanged.
### Why this is achievable: the server is already shaped for it Contracts define opaque `player_id`, `queue_ticket_id`, `proposal_id`,
`match_id`, `server_id`, and `season_id`. Every mutating request has an
idempotency key, expected revision, and versioned schema.
Two properties of the current build make per-match allocation practical The durable/transient state path is:
rather than aspirational:
- **The container is small.** The `server` image target is a slim ```
`ubuntu:24.04` runtime with three shared libraries and the exported QUEUED -> PROPOSED -> ACCEPTED -> ALLOCATING -> PROCESS_READY
binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image. -> ASSIGNMENT_READY -> ASSIGNED -> CONNECTING -> LIVE
Pulling it onto a fresh node is cheap. -> RESULT_PENDING -> COMPLETED
- **Boot to listening is sub-second.** Measured on this repo's -> CANCELLED / EXPIRED / FAILED from the explicitly legal stages
`cosmicclash-server:latest`: **~870 ms** from container start to the ```
`server_started` log line, averaged over three runs, read from the
container's own clock. That measurement was taken under **x86_64 emulation
on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux
should be faster. Re-measure on the real target before setting timeouts.
Combined with `--max-matches=1`, which already drains and `exit(0)`s after a The API publishes revisioned changes over one authenticated WebSocket. REST
single match, the lifecycle the allocator needs mostly exists: start GET endpoints are the recovery source after a disconnect or missed revision.
container → serve one match → process exits → orchestrator reclaims. Restarting the client resumes an unexpired ticket/assignment instead of
creating another one.
### The cold-start tension, stated honestly Assignments include match/server IDs, protocol and client build, server image
digest, playlist version, transport, region, expiry, a match-scoped join
authorisation, and either the SDR hosted-server material or an ENet endpoint.
The authorisation may be replayed only by the same Steam identity to reclaim
the same match/server/slot before expiry. Each successful connection advances
a server-owned generation and fences the prior connection; another identity,
server or slot is always rejected. This deliberately supports reconnect when
Steam or the control plane is temporarily unavailable. Incompatible
protocol/build/playlist versions never enter one proposal.
"Only pay during a match" and "a player never waits" are in tension. A server ## 4. Queue and placement policy
must be listening *before* the matched players connect, so some cost always
precedes the match. Sub-second boot makes the gap small enough that a pure
scale-to-zero design is plausible — but the risk is not the container, it is
everything around it: image pull on a cold node, scheduler placement, and
network/port programming can each dwarf 870 ms.
Recommendation: **scale to zero at the node level is the wrong target; scale Each verified player may own at most one active queue ticket. Heartbeats are
to zero at the match level is the right one.** Keep a small warm pool of sent every 10 seconds and queue presence expires after 30 seconds. Create,
nodes sized to the current queue depth, and start a per-match container on cancel, resume, accept, decline, and expiry are atomic and retry-safe.
demand within it. The per-match process genuinely exists only for the match;
the node pool absorbs the cold-start variance. Revisit only if measured
allocation latency on real infrastructure shows the warm pool is unnecessary.
### Findings that block a naive implementation The client submits its recent opaque Steam ping location plus nonce-bound
active-probe responses from each regional endpoint; it does not submit the RTT
used for placement. The backend validates a 30-second freshness window and
nonce, then uses the Steam coordinator SDK and probe timings to compute the
regional matrix. A predicted/observed discrepancy over 25 ms or 30% (whichever
is larger) in three matches within 24 hours quarantines the account's samples:
it may queue only in regions whose active probe independently remains under
the ceiling until five clean matches clear the quarantine. The matchmaker:
**Readiness cannot be detected from the log line.** Godot's stdout is 1. Finds regions in which every proposed player has predicted RTT <= 100 ms.
block-buffered when it is not attached to a TTY. Run the server image 2. Minimises the worst player's predicted RTT.
detached without `-t` and `docker logs` shows **nothing at all** — the 3. Breaks ties by total predicted RTT, then ready server capacity.
`server_started` line does not appear even after 35 seconds, because the 4. Widens rating tolerance with wait time, but never automatically widens the
buffer never flushes. An orchestrator readiness probe that greps for that 100 ms latency ceiling.
line will hang forever, and this was reproduced directly while measuring the
boot time above. Either probe the UDP socket instead, or make the server
flush explicitly. This also means container logs are not a reliable
observability channel for a short-lived match server; treat log shipping as
a separate problem.
**One fixed port per container does not scale on a shared host.** `--port` The target is regional observed p95 RTT <= 80 ms. Candidate formation is
defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing deterministic:
several matches onto one node needs either a port range allocated per
container, or one address per container. This is a UDP service, so the usual
HTTP ingress/L7 routing answers do not apply — the allocator must hand the
client a concrete `host:port`.
**The match cannot start on a schedule the players do not control.** Today - Anchor on the oldest compatible ticket (`enqueued_at`, then ticket ID).
the loop waits for `--min-players` then counts down. An allocated server is - A ticket's rating tolerance is `min(400, 100 + 25 * floor(wait_seconds/30))`
told *which* identities to expect, and needs a **no-show timeout**: if a rating points. A pair is compatible only when its absolute rating difference
matched player never connects, the server must abandon and exit rather than is within both tickets' tolerance. Provisional players use the same stored
sit idle burning the cost this design is trying to avoid. rating; their high RD changes ratings faster, not eligibility arithmetic.
- Every candidate set contains the anchor. Choose the lexicographically
smallest tuple `(worst_RTT, total_RTT, rating_range, -sum(wait_seconds),
sorted_ticket_IDs)`, so latency dominates once the oldest player anchors
fairness and every candidate satisfies the widening rule.
- Partition humans exhaustively into legal teams and minimise absolute team
mean-rating difference, then maximum opposing-player difference, then use
lexical player IDs. Bots fill remaining casual slots after humans are
assigned.
### Keeping Docker and CI green Every selected human receives a 10-second proposal before allocation. Ranked
always selects six. Casual requires six before the anchor reaches 60 seconds;
afterward it selects the largest compatible human count from six down to two,
with at least one human per team, and displays teams/bot slots. Unanimous
acceptance by the selected humans advances.
The existing gates must not regress. `make verify-phase6` builds the export, An explicit proposal decline cancels that player's ticket and applies a
runs it in Compose, joins two headless clients and asserts both saw both 30-second casual or 2-minute ranked cooldown. A timeout applies 60 seconds in
goals and that the arena rotated between matches; `make verify-enet-integration` casual or 5 minutes in ranked. Three ranked proposal declines/timeouts within
runs the source-build ENet matrix. Both depend on current behaviour: 30 minutes apply 15 minutes. Players who accepted return with their original
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot `enqueued_at` and precedence when another player declines, times out, or the
assignment, and uses `--max-matches=2` to prove rotation. allocation fails. Ordering ties use ticket ID. A ranked initial-connect
no-show after accepting uses the ranked abandon cooldown ladder but never a
rating loss because no rated match began.
The rule that keeps them passing: **every allocation feature is opt-in via a ### Casual
new `ServerConfig` flag whose default reproduces today's behaviour.** An
assigned roster, a no-show timeout and result reporting must each be inert
unless explicitly enabled. `ServerConfig` is built for exactly this — a flag
declared once is parsed, validated, type-checked, config-file-backed and
documented — and `tests/cases/` can cover the new parsing without a live
server. A second Compose file should cover the allocated-match path rather
than mutating the Phase 6 one, so the community-server model stays tested
alongside the matchmade one.
### Open questions - Target six humans in 3v3.
- After 60 seconds, a match may start with at least two humans, one on each
team, and fill the other slots with server bots.
- Human backfill may replace a bot/disconnected slot only at a kickoff
boundary.
- Backfill is a separate 10-second opt-in proposal showing score, time
remaining, team and slot. Declining/timing out carries no cooldown. A
backfilled player receives no hidden-rating update or abandon penalty for
that match; acceptance removes their queue ticket only when assignment is
ready. Choose the oldest ordinary casual ticket that meets the same build,
region <=100 ms and current anchor-tolerance rules for the vacated human
slot; ties use ticket ID.
- An original casual participant gets 30 seconds to reconnect; leaving after
that applies a 60-second queue cooldown. The match's ordinary hidden-rating
result still applies, with no extra rating penalty.
- An accepted casual initial-connect no-show gets the same 60-second cooldown.
The match proceeds with a bot only if at least one human connected on each
team; otherwise it cancels and restores every innocent ticket with original
precedence.
- Casual has a separate hidden Glicko-2 rating used only for matching.
- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this ### Ranked
game-server lifecycle), Nomad, or direct cloud-API container starts. Not
chosen. Agones is the strongest default because it models allocation,
readiness and per-match lifetime natively.
- **Port strategy** — port range per node versus one IP per match.
- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 610 match
processes per modern core and 150250 MB RSS each. That estimate predates
any allocation work and should be re-measured under real concurrency
before it sizes a bill.
- **Draining and deploys.** How a server version rolls out without killing
matches in flight.
## Casual vs ranked - Exactly six verified humans; never bots and never backfill.
- Solo queue only at launch.
- Only `ArenaRegistry` entries with `"random": true` are eligible. Elevated
goals remain excluded until the trained-policy restriction is lifted.
- A reconnecting player has 60 seconds to return using the existing
assignment. After that, that player receives a loss regardless of the final
team result and a rolling seven-day cooldown: 5 minutes, 15 minutes, 1
hour, then 24 hours.
- Delivery failure is not match-integrity failure. A healthy completed match
remains rated while its result waits for the control plane. Rating is
suppressed only when the authoritative roster, simulation or result is
unavailable/corrupt, or a measured regional incident prevented fair play.
They are different playlists, not a difficulty toggle, and their rules ## 5. Rating and seasons
diverge in ways that affect the server:
| | Casual | Ranked | Use canonical Glicko-2 independently per playlist with initial rating 1500,
| --- | --- | --- | RD 350, volatility 0.06, scale constant 173.7178 and tau 0.5. Updates are
| Rating | Hidden, used only for matching | Visible, with tiers | immediate per committed match rather than globally batched. Before an update,
| Backfill on disconnect | Yes — keep the match playable | No — the match is rating-bearing and must not change shape mid-way | advance inactivity by whole 24-hour rating periods since the player's last
| Bots filling empty slots | Acceptable (`--fill-bots` exists) | Never | rated match using `phi = min(350/173.7178, sqrt(phi^2 + sigma^2 * periods))`.
| Abandon penalty | Light (short queue cooldown) | Real (rating loss, escalating cooldown) |
| Arena selection | Full rotation | Restricted set, so a variant nobody has practised can't decide a ladder match |
| Party / premade | Unrestricted | Constrained by size and rating spread |
Note the arena constraint interacts with an existing rule: elevated-goal For each player `i`, transform every opposing human's pre-match rating/RD to
variants are Free-Play-only until a checkpoint trained on `mu_j`/`phi_j` and use the canonical equations
`training_elevated.tscn` is promoted (`arena_registry.gd`). Ranked's arena `g(phi)=1/sqrt(1+3*phi^2/pi^2)` and
set should be drawn from `"random": true` arenas only. `E=1/(1+exp(-g(phi_j)*(mu_i-mu_j)))`. Ranked's three opponent contributions
use `w=1/3`; casual uses `w=1/N` for the `N` opposing humans, ignoring bots.
Thus human contributions total exactly one match in both sums:
## Open questions ```
v^-1 = sum(w * g(phi_j)^2 * E_j * (1-E_j))
Delta = v * sum(w * g(phi_j) * (s-E_j))
```
- **Rating algorithm.** Glicko-2 is the default recommendation over plain Then run the canonical Glicko-2 volatility iteration and rating/RD update. If
Elo: it models rating *uncertainty*, which matters enormously for a small a player has no opposing human, the match is unrated for that player. `s` is 1/0 for
launch population where most players have few games. Not yet decided. the authoritative winner/loser and 0.5 only for a completed draw. Overtime is
- **Team rating from individual ratings.** How a 3v3 match's outcome an ordinary win/loss. A ranked abandoner gets `s=0` regardless of the final
distributes across six players is a separate design problem from the team result; non-abandoning players use the authoritative final result.
rating system itself. Cancelled or integrity-failed matches do not update anyone.
- **Server cost.** Allocated servers cost real money per match, unlike
community servers that players host themselves. `README.md`'s original
note about a subscription to fund servers is suddenly load-bearing again.
Population size and match length set the bill; this needs a number before
launch, not after.
- **Region / ping policy.** How much rating tolerance to trade for latency,
and whether cross-region is ever allowed at low population.
- **Placement matches** and whether ranked has a soft reset per season.
- **Backend language and hosting.** Not chosen. It does *not* have to be C#
despite the original README framing — that framing was aspirational and
predates every real decision in this project.
## Explicitly out of scope Lock all six participant rows in lexical player-ID order and compute every
new value from the same immutable pre-match snapshot inside one serializable
transaction. This prevents update-order bias and concurrent double updates.
Golden vectors include canonical one-player examples plus symmetric/asymmetric
3v3, draw, overtime, abandon, inactivity and concurrent-result fixtures.
Tournaments, in-game leaderboards beyond a personal rank display, The first ten ranked matches are provisional. Ranked exposes
cross-platform play with non-Steam identity providers, and spectator/observer backend-derived tiers; casual rating remains hidden. Stored Glicko values, not
tooling for ranked matches. None are precluded by this design; none are tier labels or client calculations, are authoritative. Ranked seasons last 12
launch scope. weeks. Ranked rollover sets
`rating = 1500 + 0.75 * (rating - 1500)`, raises RD to at least 200 (capped at
350), preserves volatility/history, and is an exactly-once idempotent
transaction. Casual rating is continuous and never season-reset.
Rating, penalty, match completion, participant records and outbox events are
committed in that transaction. Duplicate identical server results succeed
idempotently. A conflicting result changes nothing and pages an operator.
## 6. Game-server allocation and lifecycle
Use provider-portable Kubernetes manifests and Agones. Create one versioned
Fleet per compatible build and region; labels identify region, protocol,
transport, and image/build. Allocate atomically with `GameServerAllocation`.
Agones, rather than bespoke allocator code, owns selection and lifecycle. See
[GameServerAllocation](https://agones.dev/site/docs/reference/gameserverallocation/).
The Godot process talks to the Agones REST sidecar through a small adapter
that is a no-op when `AGONES_SDK_HTTP_PORT` is absent. This preserves native,
Compose, and CI operation and works with the Agones local SDK emulator. See
[Agones client SDKs](https://agones.dev/site/docs/guides/client-sdks/).
Agones has two distinct readiness points; conflating them is a deadlock because
`GameServerAllocation` selects a Ready server and only then attaches the match
metadata:
1. The PID-1 supervisor queries the local Agones SDK for the assigned dynamic
port/address, exports `SDR_LISTEN_PORT` plus `SDR_IP=public-address:port`
(or the ENet equivalent), and launches Godot. Godot validates static config,
binds the socket and starts Health calls.
2. Godot calls Agones `Ready()` after the process is genuinely listening.
This is **process-ready** only; never infer it from detached stdout.
3. `GameServerAllocation` atomically changes that Ready server to Allocated
and supplies the signed roster/non-secret match configuration as metadata.
4. The server watches the GameServer, observes Allocated metadata, verifies
manifest signature/build/protocol/server binding, registers its hosted SDR
address, and calls the backend `assignment_ready` endpoint.
5. Only after `assignment_ready` does the allocator mint relay/join tickets and
expose the assignment to clients. The server accepts only assigned
identities/slots; each selected human has 30 seconds to connect.
6. Run one authoritative match with Health calls independent of the simulation
loop. Submit the canonical result using the bound workload identity.
7. Write the signed result hash/payload to the backend and to a non-secret
Agones annotation, remain Allocated, and retry until the backend durably
acknowledges it. The maintenance worker reconciles the annotation after an
API outage. `RESULT_PENDING` pages at 5 minutes and requires operator review
at 30 minutes; it never silently becomes unrated.
8. After acknowledgement call `Shutdown()` and exit. Invalid/empty allocations
shut down immediately. An allocated GameServer is not recycled to Ready.
Agones supplies dynamic host ports so multiple isolated matches can share a
node; HTTP ingress is not involved in gameplay routing. Hosted SDR additionally
requires Valve approval for every provider/location, a valid `SDR_POPID`,
public-IP/unsolicited-UDP reachability, provider firewall/NAT validation,
per-location certificates and coordinator trust. Use an Agones dynamic or
passthrough mapping whose externally reported port is the `SDR_IP` port while
the process binds `SDR_LISTEN_PORT`; test SDR and ENet mappings separately.
Credentials arrive through runtime secret mounts, never allocation metadata,
arguments, logs or images.
Result authentication uses a projected, pod-bound service-account token with
a dedicated audience and one service account per workload class. The backend
validates the configured cluster issuer/JWKS, audience, expiry, namespace,
service account, bound pod UID and allocator-recorded GameServer UID, then
checks that GameServer/match binding in PostgreSQL. Issuers and trust roots are
allowlisted and rotated explicitly for every cluster/provider. A one-match
server credential issued after this attestation is an acceptable equivalent.
### Warm capacity and density
Both launch regions are active whenever their queues are enabled. Each active
region maintains at least two Ready processes distributed across at least two
on-demand nodes/failure domains; the minimum node floor is therefore two, not
one. Pre-pull current and rollback images. A queue/proposal-aware
FleetAutoscaler adjusts capacity above that Ready floor. **Allocated** process
count may fall to zero; Ready processes do not. An administratively disabled
region may scale both nodes and Fleet to zero and is excluded from placement.
See [Agones FleetAutoscaler](https://agones.dev/site/docs/reference/fleetautoscaler/).
Do not run live matches on interruptible nodes. N+1 means loss of the largest
single node still leaves two Ready slots plus sufficient headroom for already
Allocated matches; certify it in the node-loss test. Set pod requests, limits
and node caps only after native x86_64 benchmarks of boot time, p99
CPU/RSS/network and 60 Hz tick behavior, retaining 30% headroom. The current
610 processes/core and 150250 MB estimates are not sizing data.
Godot/GDScript cannot intercept SIGTERM, so Phase 8 adds a small Go PID-1
supervisor. It starts Godot, traps TERM, and sends a per-pod-token-authenticated
localhost drain request to an allocated-mode-only Godot control socket. Godot
refuses new admissions and reports completion to the supervisor. Kubernetes
uses a 300-second termination grace period; at 285 seconds the supervisor
forces an infrastructure-failure abort so the pod cannot hang forever in
overtime. A PodDisruptionBudget and Agones-aware drain prevent voluntary
eviction of Allocated servers. Planned releases create a new Fleet, route new
allocations to it and wait for old Allocated count zero without sending TERM.
Unexpected node failure cannot be made graceful and follows the integrity
failure/refund path.
Live matches continue through an API/control-plane delivery outage. A valid
result is retried/reconciled as above; only loss of authoritative match
integrity suppresses rating.
## 7. Security and operational baseline
The threat model must cover forged clients, ticket replay, duplicate queueing,
result forgery, queue manipulation, packet floods, botting, server compromise,
insider access, DDoS, vulnerable dependencies, and denial-of-wallet attacks.
Minimum controls:
- Non-root containers, read-only root filesystems, dropped Linux
capabilities, RuntimeDefault seccomp, resource limits, restricted Pod
Security admission, and least-privilege service accounts/RBAC.
- Private PostgreSQL/Redis; default-deny ingress and egress with explicit
NetworkPolicy allowances. See the [Kubernetes application security
checklist](https://kubernetes.io/docs/concepts/security/application-security-checklist/)
and [NetworkPolicy](https://kubernetes.io/docs/reference/kubernetes-api/networking/network-policy-v1/).
- Encrypted databases/backups, externally populated Kubernetes Secrets,
documented rotation, and no credentials in Git, images, command lines, or
telemetry.
- Per-account/IP API limits, request/body/schema limits, replay and duplicate
detection, generic public errors, allocation quotas, and budget alerts.
- Put HTTPS/WebSocket traffic behind a provider-portable edge contract
implemented by each infrastructure overlay: managed volumetric DDoS
absorption, WAF/rate rules, TLS termination, origin-only ingress and health
checks. Enforce authenticated queue admission, per-account connection caps,
WebSocket handshake/message/idle limits, bounded fan-out and overload
shedding. In degraded mode reject new login/queue/allocation work while
preserving result ingestion and all live matches.
- Images pinned by digest, SBOM generation, dependency/image scanning, signed
releases, admission-time signature verification, and a critical-patch SLA.
- Structured audit events for auth, queue transitions, allocation, roster
rejection, result conflict, penalty, season rollover, and operator action.
## 8. SLOs, observability, and release gates
Launch SLOs:
| Measure | Target |
| --- | --- |
| Eligible predicted RTT | <= 100 ms for every player |
| Regional observed RTT | p95 <= 80 ms |
| Unanimous acceptance to `assignment_ready` | p95 <= 5 s, p99 <= 10 s with warm capacity |
| Published assignment to successful connection | p95 <= 5 s |
| Successful allocation and durable result | >= 99.9% |
| Certified server density | No tick backlog, with 30% resource headroom |
| Control-plane API under load | p95 <= 250 ms |
Dashboards and alerts cover queue depth/wait, rating spread, predicted versus
observed RTT, proposals/declines, allocation latency/failure, Ready capacity,
image pulls, connection/no-show, physics overruns, crashes, abnormal packet
rates, result lag/conflicts, abandons, and cost per completed match. Correlate
all components with queue/proposal/match/server IDs, but never log auth or
relay tickets.
Testing layers:
- Go unit, race, fuzz, property, migration, and concurrency tests.
- Fake Steam verifier and fake allocator for deterministic CI.
- A second allocated-server Compose flow; never mutate
`compose.phase6-smoke.yml`.
- Disposable `kind` + Agones integration for readiness, dynamic ports,
health/no-show shutdown, allocation races, multiple matches per node,
draining, and rollback.
- Network/chaos cases for 100 ms RTT, jitter/loss, client/backend/matcher
restart, game-pod death, node drain, Redis failover, and control-plane loss.
- Load test at least 10,000 queued clients, 100 proposals/second, and forecast
launch concurrency x2 while preserving correctness and API latency.
- Provider migration rehearsal: restore data and deploy both regional fleets
on a second provider whose EU/NA locations already have Valve approval,
POP/certificates, public UDP/firewall verification and coordinator trust;
switch new allocations, drain the old provider, and terminate no live match.
Release order is development -> internal -> casual canary -> full casual ->
provisional ranked -> full ranked. Each promotion requires its SLO/security
gates, rollback rehearsal, EU and North America playtests, a measured cost
model, and unchanged `make verify-phase6` and
`make verify-enet-integration`.
## 9. Explicitly out of scope
Parties/premades, tournaments, ranked spectators, public global regions,
non-Steam identity providers, and a global leaderboard beyond personal rank
display are not launch scope. The contracts reserve party identity and avoid
Steam-specific database primary keys so those additions do not require a
destructive redesign.
+6 -5
View File
@@ -189,8 +189,9 @@ Python, no .NET, no network." Keeping the shipped game GDScript-only (no
## Planned, not yet built ## Planned, not yet built
- **A matchmaking backend service** — Steam auth ticket validation, casual - **A Go matchmaking control plane** — independently runnable API, matcher,
and ranked queues, a rating store, and per-match dedicated-server allocator and maintenance roles backed by PostgreSQL and Redis, deployed on
allocation. Language and hosting are undecided. This is a 1.0 launch provider-portable Kubernetes with Agones-managed game fleets. The cloud
blocker and the single largest departure from "one Godot project, no provider remains deliberately replaceable; the application stack is locked.
backend". See [`MATCHMAKING.md`](MATCHMAKING.md). This is a 1.0 launch blocker and the single largest departure from "one
Godot project, no backend". See [`MATCHMAKING.md`](MATCHMAKING.md).
+121 -73
View File
@@ -1,93 +1,141 @@
# Multiplayer — next work # Multiplayer — next work
Short, current checklist for online multiplayer. Historical design decisions, Short, current checklist for online multiplayer. Historical decisions,
implementation evidence, and completed work stay in implementation evidence and task-level acceptance criteria stay in
[`multiplayer-todo.md`](multiplayer-todo.md). [`multiplayer-todo.md`](multiplayer-todo.md). Phase 8 architecture and locked
product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md).
## Release blockers ## Existing release blockers
- [ ] **Phase 4 playtest:** a human playtest at roughly 100 ms RTT. Confirm - [ ] **Phase 4 playtest:** play at roughly 100 ms RTT; confirm ship/ball
that ship and ball interaction feel local and contact corrections feel like interaction feels local and contact corrections read as bumps, not glitches.
bumps rather than glitches. - [ ] **Phase 5 session:** finish a real 3v3 match with a mid-match disconnect
- [ ] **Phase 5 session:** complete a real 3v3 match with a mid-match and late joiner.
disconnect and late joiner. - [ ] **Phase 6 external check:** play the exported Docker server from separate
- [ ] **Phase 6 external check:** run the exported Docker server and clients internet machines. Keep the check controlled until verified identity lands.
from separate machines over the internet, then play a full match. Keep this
controlled-only until Steam identity is complete.
## Phase 7 — Steam, identity, discovery ## Phase 7 — production Steam prerequisite
- [ ] Obtain the pinned GodotSteam client/server builds and Steamworks SDK - [ ] Obtain the pinned GodotSteam client/server builds and Steamworks SDK;
access described in [`STEAM.md`](STEAM.md). pass `make verify-steam-templates` without weakening ENet verification.
- [ ] Run `make verify-steam-templates` with the custom executables and fix - [ ] Validate two real accounts through the explicit Steam transport and
any custom-template failures. build Internet/LAN/favourites/history server-browser views.
- [ ] Validate a two-account Steam SDR host/join using the existing explicit - [ ] Add single-use auth tickets, asynchronous server validation, verified
`NetworkManager` Steam transport. ENet direct-IP must keep passing its smoke Steam identity, identity-keyed reconnect and persistent bans.
test. - [ ] Obtain the real App ID, publisher key, coordinator SDK/signing approval,
- [ ] Build the Steam server browser: internet, LAN, favourites, and history. certificates and Hosted Dedicated Server data-centre support from Valve.
- [ ] Add Steam auth tickets, verified Steam identity in the roster, and a - [ ] Implement ticketed Hosted Dedicated Server SDR routing, ticket install,
persistent ban list. This fixes the slot-reclaim security issue below. reconnect and expiry. Preserve direct ENet for local/CI/community servers.
## Phase 8 — casual and ranked matchmaking (1.0 launch blocker) ## Phase 8 — architecture, contracts and durable data
Design and reasoning: [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). This is a - [ ] Lock the Go/PostgreSQL/Redis, Kubernetes/Agones, SDR, EU/NA and
different server model from the community-server one that exists today — provider-portability ADR; define measurable launch SLOs.
players queue, a matchmaker groups them, and a server is allocated per match. - [ ] Publish versioned OpenAPI/WebSocket contracts, stable IDs, legal state
Phase 7's Steam auth tickets are a hard prerequisite: a rating attached to a transitions, revisions and idempotency semantics.
spoofable identity is worse than no rating. - [ ] Add PostgreSQL queue ownership/active-participation fences, durable
domain migrations/outbox and Redis indexes/TTLs; lost Redis writes must not
split a proposal or corrupt durable state.
- [ ] Define assignment compatibility and opt-in `ServerConfig` flags whose
defaults reproduce the community-server path.
- [ ] Decide the rating algorithm (Glicko-2 recommended over Elo for a small ## Phase 8 — identity and security
launch population) and how a team result distributes across individuals.
- [ ] Choose the backend language and hosting, and cost out allocated servers
per match at expected population.
- [ ] Stand up the backend: Steam auth ticket validation via the Steamworks
Web API, queue, rating store, server allocator.
- [ ] Add an assigned-roster server mode so only matched SteamIDs may claim a
slot, replacing the first-come model.
- [ ] Add server-authoritative match result reporting to the backend over a
channel a client cannot forge.
- [ ] Client queue UI: playlist select, estimated wait, accept/decline,
connect-on-assignment, post-match rating delta.
- [ ] Casual and ranked playlist rulesets (backfill, bots, abandon penalties,
arena restriction — see the comparison table in the design doc).
Server orchestration (same phase — the servers must autoscale and bill only - [ ] Validate Steam Web API tickets only in the secure backend; issue
for the duration of a match): revocable sessions and reconnect-safe match/identity/slot authorisations
with server-owned connection-generation fencing.
- [ ] Authenticate results with pod/GameServer-bound workload identity; make
identical duplicates idempotent and conflicting results inert/alerting.
- [ ] Complete the threat model for forgery, replay, queue/flood/bot abuse,
workload/insider compromise, DDoS, supply chain and denial-of-wallet.
- [ ] Enforce restricted workloads/RBAC/networks/private stores/backups/secrets;
isolate SDR signing behind an audited non-exportable signer and add
volumetric edge defense, WebSocket limits and overload shedding.
- [ ] Pin, scan, SBOM and sign artifacts; verify signatures at admission and
document the critical vulnerability SLA.
- [ ] Choose the orchestrator (Agones on Kubernetes is the recommended ## Phase 8 — queues, playlists and rating
default; it models allocation, readiness and per-match lifetime natively).
- [ ] Fix readiness detection. Godot's stdout is block-buffered off a TTY, so
`server_started` never appears in `docker logs` for a detached container —
a log-grep readiness probe hangs forever. Probe the UDP socket, or flush.
- [ ] Support more than one match per host: a per-container port from a range,
or one address per match. `--port` defaults to 7777 and the Dockerfile
hardcodes `EXPOSE 7777/udp`.
- [ ] Add a no-show timeout so an allocated server that never fills abandons
and exits instead of idling at cost.
- [ ] Re-measure boot-to-listening on native x86_64 Linux. The repo's current
figure is ~870 ms, measured under emulation on arm64 — a pessimistic bound.
- [ ] Re-measure the SERVER.md sizing estimate (610 processes/core,
150250 MB RSS) under real concurrency before it sizes a bill.
- [ ] Keep `make verify-phase6` and `make verify-enet-integration` green:
every allocation feature is opt-in via a `ServerConfig` flag defaulting to
today's behaviour, with a second Compose file for the allocated path rather
than mutating `compose.phase6-smoke.yml`.
## Known issues to resolve before public hosting - [ ] Add one PostgreSQL-owned queue ticket/player with 10 s heartbeat, 30 s
expiry, Redis candidate cache and restart/failover repair.
- [ ] Validate opaque Steam ping locations and nonce-bound probes server-side;
require <=100 ms, enforce discrepancy quarantine and the locked widening/
region/team tie-break rules.
- [ ] Send 10 s proposals to every selected human: ranked six, relaxed casual
two to six with disclosed bots; enforce exact cooldown and queue-precedence
behavior.
- [ ] Fence proposals/participants in a PostgreSQL serializable transaction;
prove loss of an acknowledged Redis write cannot split players.
- [ ] Casual: target 3v3 humans, after 60 s allow >=2 humans (one/team) plus
bots, kickoff-only human backfill and no backfill loss/decline penalty.
- [ ] Ranked: exactly six humans, solo-only, no bots/backfill, random-enabled
non-elevated arenas only, 60 s reconnect grace and escalating abandons.
- [ ] Implement the documented exact Glicko-2 equations, fractional 3v3
weights, inactivity/update locking/golden vectors and ten provisional games.
- [ ] Add ranked-only exactly-once 12-week soft seasons; distinguish retryable
result-delivery outages from match-integrity failures and rating exemptions.
- [ ] Slot reclaim is currently keyed by display name, so someone can take a ## Phase 8 — Agones and regional server capacity
disconnected player's reserved slot. Do not expose public servers before
verified Steam identity lands. - [ ] Add portable EU/NA Agones Fleets with provider edge/network/secret and
- [ ] Investigate occasional input loss during a long server stall; the Valve-approved SDR POP/certificate/public-UDP overlays.
existing sequence resync recovers it, but transport delivery is variable. - [ ] Add the local-safe Agones adapter and separate process-ready (listen then
Ready) from assignment-ready (Allocated manifest verified and registered).
- [ ] Allocate from Ready by region/build/protocol/transport; use separately
verified ENet and SDR dynamic/passthrough port mappings.
- [ ] Deliver/verify the signed roster after allocation and expose client
tickets only after backend `assignment_ready`.
- [ ] Keep >=2 Ready processes across >=2 on-demand nodes/failure domains per
queue-enabled region; only Allocated count may fall to zero.
- [ ] Spread on-demand capacity across zones with N+1 headroom; do not place
live matches on interruptible nodes.
- [ ] Benchmark native x86_64 boot, p99 CPU/RSS/network and tick health; set
requests/limits and node density from measurements plus 30% headroom.
- [ ] Add 30 s no-show handling, Go PID-1 TERM/drain supervision, PDB/Fleet
drain, signed result annotation/retry, RPO <=5 m and RTO <=30 m.
- [ ] Rehearse migration only after the second provider's EU/NA locations have
Valve approval, POP/certs, public UDP/firewall and coordinator trust.
## Phase 8 — client and recovery
- [ ] Build queue/proposal/allocation/connect/rating UI with explicit latency,
capacity, expiry and recovery states.
- [ ] Use one authenticated revisioned WebSocket plus REST resync; resume a
valid ticket/assignment after restart rather than duplicating it.
- [ ] After assignment-ready, install SDR ticket and send reconnect-safe join
authorisation in `hello`; fence old connections and retain ENet behavior.
- [ ] Display only backend-authoritative provisional rank/tier/delta, abandon
status and season time; clients perform no rating calculation.
## Phase 8 — operations and release gates
- [ ] Correlate queue→result with IDs and add dashboards/alerts for SLOs,
security, failures and cost without logging credentials.
- [ ] Add Go race/fuzz/property/migration/concurrency coverage plus fake Steam
and fake allocation for offline deterministic CI.
- [ ] Add an independent allocated-server Compose flow; do not mutate
`compose.phase6-smoke.yml` or weaken either existing Make gate.
- [ ] Add disposable `kind`/Agones integration, 100 ms network/chaos cases and
proof that infrastructure failures cannot punish players.
- [ ] Load-test >=10,000 queued clients, >=100 proposals/s and forecast launch
concurrency x2 while holding API p95 <=250 ms and allocation correctness.
- [ ] Record cost per completed match, budget/denial-of-wallet controls and
deploy progressively: internal → casual canary → casual → provisional
ranked → ranked, with EU/NA playtests and rollback gates.
## Known issues before public hosting
- [ ] Replace display-name slot reclaim with verified Steam identity.
- [ ] Investigate occasional transport input loss during a long server stall.
- [ ] Fix the remaining `_broadcast_snapshot` packet-send stderr race. - [ ] Fix the remaining `_broadcast_snapshot` packet-send stderr race.
## Decide after the latency playtest ## Decide after the latency playtest
- [ ] Decide whether client-only, contact-cohort shadow physics is worthwhile - [ ] Decide whether client-only contact-cohort shadow physics is worthwhile.
for the remaining prediction weakness.
## Explicitly deferred ## Explicitly deferred
120 Hz simulation, latency-gap measurement, audio hooks, and split-screen are Parties/premades, tournaments, ranked spectators, non-Steam identity,
not part of the current multiplayer release path. additional global regions, global leaderboards, 120 Hz simulation,
latency-gap measurement, audio hooks and split-screen are not in the launch
path.
+92 -68
View File
@@ -14,13 +14,13 @@ Everything below is written so an agent (or a person) can pick up a single numbe
The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 05 contain no unfinished tasks. The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 05 contain no unfinished tasks.
**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.18.20 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three items there are findings rather than plans, and each would break a naive implementation: **Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.18.53 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three findings would break a naive implementation:
| # | Finding | Why it bites | | # | Finding | Why it bites |
|---|---|---| |---|---|---|
| 8.8 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | An orchestrator readiness probe that greps the log hangs forever | | Task 8.28 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | Process-ready must be an explicit Agones call after static validation/listen; post-allocation assignment-ready is separate and neither uses a log grep |
| 8.9 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches cannot share a host; being UDP, L7 ingress routing does not apply | | Task 8.29 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches need Agones dynamic UDP/SDR ports; L7 ingress does not route this traffic |
| 8.18 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | Allocation work trivially regresses `verify-phase6` unless every new feature defaults to today's behaviour | | Task 8.48 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | The allocated flow needs its own fixture so Phase 6 behavior and invocations stay unchanged |
### Blocking sign-off — the work exists, the verification does not ### Blocking sign-off — the work exists, the verification does not
@@ -50,7 +50,7 @@ C is the one to plan around: it is fixed for free by task **7.4** (Steam auth ti
### Unstarted phases ### Unstarted phases
- **Phase 6 external gate:** run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until **C** is fixed. - **Phase 6 external gate:** run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until **C** is fixed.
- **Phase 7 — Steam transport, browser, identity** (5 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, auth tickets, and bans await a project-owned Steamworks App ID. Carries the fix for **C**. - **Phase 7 — Steam transport, browser, identity and production SDR** (8 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, verified tickets, bans, production credentials and ticketed Hosted Dedicated Server SDR await a project-owned Steamworks App ID and Valve coordination. Carries the fix for **C** and is the hard prerequisite for Phase 8.
Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into something another person can host. Phase 7 is the next block because Steam identity is required before public exposure. Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into something another person can host. Phase 7 is the next block because Steam identity is required before public exposure.
@@ -69,7 +69,7 @@ Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into some
| 1 | **Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation.** | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. | | 1 | **Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation.** | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. |
| 2 | **Dedicated servers only.** Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. | | 2 | **Dedicated servers only.** Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. |
| 3 | **ENet first**, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and **must never become the degraded path**. | | 3 | **ENet first**, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and **must never become the degraded path**. |
| 4 | **No custom backend.** | Steam's `ISteamGameServer` master-server listing covers discovery, `ISteamMatchmakingServers` covers the in-game browser, and Steam auth tickets cover identity and ban state. `README.md`'s C# backend stays unstarted. | | 4 | **Community discovery uses no custom backend; superseded for queued play by Phase 8.** | Steam's server APIs remain enough for the community browser. Casual/ranked queues, durable ratings, allocation and authoritative results require the project-owned Go control plane specified in `docs/MATCHMAKING.md`; it does not replace the browser or direct-IP path. |
### 1.2 Rejected alternatives ### 1.2 Rejected alternatives
@@ -1135,6 +1135,9 @@ New `--role=client-reconnect` grades the returning player: not a spectator, owns
| 7.3 `[D:7.2]` `[P]` | `server_browser.tscn` via `ISteamMatchmakingServers` | Internet, LAN, favourites and history lists all populate | | 7.3 `[D:7.2]` `[P]` | `server_browser.tscn` via `ISteamMatchmakingServers` | Internet, LAN, favourites and history lists all populate |
| 7.4 `[D:7.2]` `[P]` | Auth tickets in `hello``BeginAuthSession`; Steam identity in the roster; persistent ban list | Ownership, VAC and ban state verified server-side | | 7.4 `[D:7.2]` `[P]` | Auth tickets in `hello``BeginAuthSession`; Steam identity in the roster; persistent ban list | Ownership, VAC and ban state verified server-side |
| 7.5 `[D:7.2]` `[P]` | Feature-gate every Steam call behind `OS.has_feature("steam") and ClassDB.class_exists("SteamMultiplayerPeer")`; verify the ENet path end to end | Non-Steam build is fully functional | | 7.5 `[D:7.2]` `[P]` | Feature-gate every Steam call behind `OS.has_feature("steam") and ClassDB.class_exists("SteamMultiplayerPeer")`; verify the ENet path end to end | Non-Steam build is fully functional |
| 7.6 `[D:7.4]` | Separate single-use tickets for backend Web API login and game-server auth; wait for Steam's asynchronous validation and cancel/end every ticket session | Replayed, cancelled, wrong-App-ID and not-yet-validated identities cannot enter a roster or queue; no client-supplied SteamID is trusted |
| 7.7 `[D:7.1]` `[P]` | Obtain the production App ID, publisher key, SDR coordinator SDK/signing approval, certificates and hosted-data-centre support from Valve | Production prerequisites and rotation owners are recorded; no Spacewar credential or development certificate can reach a release build |
| 7.8 `[D:7.6,7.7]` | Ticketed Hosted Dedicated Server SDR: routing registration, coordinator-issued player→server relay tickets, client ticket installation, reconnect and expiry | Two real accounts complete and reconnect to an assigned dedicated match through SDR; server/player IPs are not exposed; ENet gates remain green |
> **The transport interface is written here, not in Phase 1.** Eight virtual methods (`begin_auth`, `advertise`, `get_identity`, `supports_server_browser`…) designed against an API nobody on the project has used will be wrong. Write `NetworkManager._make_peer()` concretely in Phase 1 and extract the boundary once there are two real implementations. Locked decision 3 guarantees the ENet path is never deleted, so there is no migration risk in waiting. > **The transport interface is written here, not in Phase 1.** Eight virtual methods (`begin_auth`, `advertise`, `get_identity`, `supports_server_browser`…) designed against an API nobody on the project has used will be wrong. Write `NetworkManager._make_peer()` concretely in Phase 1 and extract the boundary once there are two real implementations. Locked decision 3 guarantees the ENet path is never deleted, so there is no migration risk in waiting.
@@ -1155,86 +1158,107 @@ and players find it by IP or (7.3) the server browser. Matchmaking makes the
**allocated for that one match** and destroyed after. Both models ship; they **allocated for that one match** and destroyed after. Both models ship; they
are different playlists, not a replacement. are different playlists, not a replacement.
**Hard dependency on 7.4.** Slot reclaim is keyed by display name today. A **Hard dependency on 7.6 and 7.8.** Slot reclaim is keyed by display name
rating attached to a spoofable identity is farmed trivially, so ranked cannot today. A rating attached to a spoofable identity is farmed trivially, so no
ship before Steam auth tickets land. Casual queueing needs 7.4 too, for queue ships before single-use verified identity lands. Production allocation
abandon penalties and ban enforcement, but degrades more gracefully. also depends on the ticketed Hosted Dedicated Server SDR route; ENet remains
the local/CI/community transport, not a silent production fallback.
#### 8aBackend service #### 8AArchitecture, contracts and data
| # | Task | Acceptance | | # | Task | Acceptance |
|---|---|---| |---|---|---|
| 8.1 | Choose backend language, hosting and datastore. **Not C# by default** — that framing predates every real decision here | Written up with the rejected alternatives, as §1 does for the client decisions | | 8.1 | Add an ADR locking **Go + PostgreSQL + Redis**, provider-portable Kubernetes, Agones, ticketed Hosted Dedicated Server SDR, EU/NA fleets and independently runnable API, matcher, allocator and maintenance roles; keep `README.md`/`docs/TECH_STACK.md` consistent | The ADR names boundaries/rejected alternatives; current docs name the locked stack and replaceable provider; no application code calls a provider allocation API |
| 8.2 `[D:7.4]` | Steam auth ticket validation via the Steamworks Web API; a verified SteamID is the only trusted identity | A forged or replayed ticket is rejected; no client-supplied identity is ever trusted | | 8.2 `[D:8.1]` | Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | Each SLO has a metric, numerator/denominator, percentile window, owner and alert threshold before implementation is judged against it |
| 8.3 `[D:8.1]` | Rating store: per-identity, per-playlist rating plus match history, written only by the backend | A client cannot write its own rating by any path | | 8.3 `[D:8.1]` | Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | Generated contract tests cover every request, response, event and external error; clients can REST-resync after a missed WebSocket revision |
| 8.4 `[D:8.3]` | Rating algorithm. **Glicko-2 recommended over Elo** — it models rating *uncertainty*, which dominates at launch when most players have few games | Simulated against a synthetic population; placement behaviour is sane at n≈0 games | | 8.4 `[D:8.3]` | Define opaque `player_id`, `queue_ticket_id`, `proposal_id`, `match_id`, `server_id`, `season_id`, legal queue/match state transitions, revisions and idempotency keys | Duplicate/out-of-order commands converge; invalid transitions are rejected without partial state |
| 8.5 `[D:8.4]` | Team-result → individual-rating distribution for 3v3 | A 3v3 outcome updates six ratings defensibly; documented, not folded into 8.4 | | 8.5 `[D:8.4]` | Add PostgreSQL migrations for durable queue ownership, active-participation fencing, identities, sessions/revocations, seasons, ratings/events, matches/participants, penalties, results, audits and outbox; document Redis caches/TTLs | A blank DB migrates up; lost Redis writes cannot resurrect revocation, split a proposal or corrupt durable state; rollback/forward compatibility is tested |
| 8.6 `[D:8.3]` | Queue and matchmaker: per playlist and region, rating proximity with tolerance widening over wait time | Queue depth and wait time are observable; tolerance widening is tunable without redeploy | | 8.6 `[D:8.3,8.4]` | Lock assignment compatibility: protocol/client build, image digest, playlist version, transport, region, expiry and signed authorisation; add all allocated-mode `ServerConfig` flags as opt-in defaults | Incompatible builds never share a proposal; absent flags reproduce today's community server and existing config tests cover every new flag |
#### 8bServer orchestration and autoscaling #### 8BAuthentication and secure control plane
Requirement: servers scale horizontally and automatically, spin up fast, serve
exactly one match, and shut down — cost incurred only while a match runs.
Two properties of the existing build make this practical rather than
aspirational, both **measured against `cosmicclash-server:latest`**, not
estimated:
- The runtime image (`server` target, slim `ubuntu:24.04`) is **~148 MB** of
content — not the ~2.6 GB `godot-ci` build image.
- Boot to the `server_started` line is **~870 ms**, container's own clock,
mean of three runs. **Taken under x86_64 emulation on an arm64 host, so it
is a pessimistic bound** — see 8.11.
`--max-matches=1` already drains and `exit(0)`s after one match. It was built
for CI and generalises to the allocator lifecycle for free.
| # | Task | Acceptance | | # | Task | Acceptance |
|---|---|---| |---|---|---|
| 8.7 | Choose the orchestrator. **Agones on Kubernetes is the recommended default** — it models allocation, readiness and per-match lifetime natively rather than making you rebuild them | Allocation, readiness and per-match teardown are all handled by the chosen system, not by bespoke glue | | 8.7 `[D:7.6,8.3]` | Validate `AuthenticateUserTicket` only in the secure backend with the expected App ID and identity string; reject expiry, replay, wrong app, bans and malformed input | Publisher credentials exist only in the backend secret store; forged/replayed tickets and client-supplied SteamIDs never create a session |
| 8.8 | **Fix readiness detection — this blocks any naive implementation.** Godot's stdout is block-buffered off a TTY. Run the server image detached without `-t` and `docker logs` shows *nothing at all*; `server_started` does not appear even after 35 s. A readiness probe that greps the log hangs forever. Probe the UDP socket, or flush explicitly | A cold container is marked ready by a mechanism that does not depend on stdout; reproduced-and-fixed, not worked around by adding `-t` in one place | | 8.8 `[D:8.7]` | Issue short-lived revocable sessions bound to verified Steam identity; add account/IP limits, body/schema limits, replay checks and generic public errors | Revocation takes effect across replicas; abuse cannot cause unbounded memory, work or response amplification |
| 8.9 | Multiple matches per host: a per-container port from a range, or one address per match. `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. **This is UDP — L7 ingress routing does not apply**, the allocator hands the client a concrete `host:port` | Two matches run concurrently on one node and neither can reach the other's traffic | | 8.9 `[D:8.4,8.7]` | Issue match-scoped join authorisations bound to SteamID/match/server/team/slot/protocol/expiry; allow same-identity slot reclaim while fencing prior connection generations | Altered/expired/wrong identity/server/slot is rejected; reconnect works without backend/Steam; a newer generation makes the old connection unable to send gameplay |
| 8.10 `[D:8.6]` | Assigned-roster server mode: only matched SteamIDs may claim a slot, replacing first-come. Plus a **no-show timeout** an allocated server that never fills abandons and exits rather than idling at cost | An unmatched identity is refused a slot; a server nobody joins exits within the timeout | | 8.10 `[D:8.5,8.31]` | Authenticate results with pod-bound projected identity or one-match attested credential; validate issuer/audience/expiry, namespace/SA, pod UID, GameServer UID and allocator match binding | Another pod sharing a workload class cannot submit for the allocation; identical duplicates are idempotent; conflicting results are inert and alerting across all trusted clusters |
| 8.11 | Re-measure boot-to-listening on **native x86_64 Linux** before it sets any timeout | A number from the real target platform replaces the ~870 ms emulated bound recorded above | | 8.11 `[D:8.1]` | Write the threat model: forged clients/replay/queues/results, floods/bots, pod/insider compromise, gameplay and API DDoS, SDR signing-key theft, dependencies and denial-of-wallet | Every threat has prevention/detection/owner/verification; accepted residual risks are explicit; offline CA and online signer trust boundaries are separate |
| 8.12 | Re-measure SERVER.md's sizing estimate (610 processes/core, 150250 MB RSS) under real concurrency | A measured figure sizes the bill; the current estimate predates all allocation work | | 8.12 `[D:8.11]` | Harden workloads and edge: restricted containers, least RBAC, private DB/Redis, default-deny networks, backups/secrets, volumetric DDoS/WAF/origin shielding, WebSocket limits and overload shedding | Policy/network tests enforce declared flows; edge load test preserves result ingress/live matches while rejecting new work; no credential appears in Git/images/args/telemetry |
| 8.13 `[D:8.10]` | Server-authoritative match result reporting to the backend over a channel a client cannot forge. **The project's first non-UDP network path** — simulation stays on ENet/SDR | A client cannot report, alter or suppress a result | | 8.13 `[D:8.12]` | Pin images by digest; generate SBOMs, scan dependencies/images, sign artifacts, verify signatures at admission and document a critical-fix SLA | CI blocks a vulnerable/disallowed or unsigned release artifact and records the exact provenance deployed |
| 8.14 | Draining and deploys: roll out a server version without killing matches in flight | An in-flight match survives a deploy of the next server version |
#### 8cPlaylists and client #### 8CQueueing, matchmaking, playlists and rating
| # | Task | Acceptance | | # | Task | Acceptance |
|---|---|---| |---|---|---|
| 8.15 `[D:8.6]` | Casual and ranked rulesets. They diverge on the server, not just in UI: backfill (casual yes / ranked never), bots filling slots (`--fill-bots` casual-only), abandon penalties, party size and rating spread | Ranked never backfills and never spawns a bot into a player slot | | 8.14 `[D:8.4,8.5,8.8]` | One durable PostgreSQL queue owner/player plus Redis candidate index: 10 s heartbeat, 30 s expiry, retry-safe create/cancel/resume and repair after cache loss | Replicas/duplicates never place a player twice; failover may delay/rematerialise an index but durable ownership and active-participation fences converge |
| 8.16 `[D:8.15]` | Ranked arena restriction. Draw only from `"random": true` arenas — **elevated-goal variants stay Free-Play-only** until a checkpoint trained on `training_elevated.tscn` is promoted (`arena_registry.gd`), so a variant nobody has practised cannot decide a ladder match | Ranked cannot select an elevated-goal arena | | 8.15 `[D:7.8,8.3]` | Submit opaque Steam ping location plus nonce-bound probes; backend computes estimates, enforces 30 s freshness and quarantines 3 discrepancies >25 ms or 30% until 5 clean matches | A client cannot directly choose its placement RTT; stale/forged evidence is rejected; quarantine behavior and server-observed comparison are deterministic |
| 8.17 `[D:8.6]` | Client queue UI: playlist select, estimated wait, accept/decline, connect-on-assignment, post-match rating delta | A declined match returns the other players to the queue without penalty to them | | 8.16 `[D:8.14,8.15]` | Implement the locked candidate/team algorithm: <=100 ms, oldest anchor, `min(400,100+25*floor(wait/30))` mutual rating tolerance, documented set/region/team tie-breakers | Fixtures cover provisional players, EU/NA/no-common-region, widening caps, deterministic partitions and low population; no placement crosses 100 ms |
| 8.17 `[D:8.14,8.16]` | Ten-second proposal to **every selected human**: ranked 6; casual largest compatible 6→2 after 60 s with disclosed teams/bots; apply exact decline/timeout/no-show cooldown and queue-precedence rules | Allocation starts only after selected humans accept; 25-human casual is reachable; accepter timestamps restore exactly; ranked pre-match no-show has cooldown but no rating loss |
| 8.18 `[D:8.5,8.14,8.17]` | Horizontally replicated matcher: Redis candidates, then PostgreSQL serializable proposal/participant fence, then cache cleanup/repair | Forced loss of the last acknowledged Redis write, retries, worker death and failover cannot claim a player into two proposals/matches |
| 8.19 `[D:8.18]` | Casual policy: proposal composition above; >=1 human/team, exhaustive rating-balanced teams, bots after 60 s, opt-in kickoff-only backfill, 30 s reconnect and defined backfill/casual penalties | Every 26-human shape is tested; no mid-play replacement; declined/backfill participant gets no excluded rating/cooldown; original leaver gets only documented outcome/cooldown |
| 8.20 `[D:8.18]` | Ranked policy: exactly six verified solo humans, no bots/backfill, only `ArenaRegistry.random`; define initial no-show, proposal timeout and reconnect/abandon transitions | Ranked rejects parties/bots/backfill/elevated arenas; every pre-live failure returns five innocent players with original precedence and applies no rating |
| 8.21 `[D:8.5,8.20]` | Exact Glicko-2 equations from `docs/MATCHMAKING.md`: 1500/350/0.06/tau .5, ranked 1/3 and casual 1/N human-opponent weights, daily inactivity, immutable snapshot/lock order, draws/OT/abandons/cancellation | Canonical plus project 26-human/3v3 golden vectors pass; concurrent results serialize without order bias; clients have no rating-write path |
| 8.22 `[D:8.21]` | First ten ranked games provisional; casual rating hidden; ranked tiers derived from authoritative stored values | Matchmaking uses provisional rating/RD; UI visibility changes exactly on result ten without rewriting history |
| 8.23 `[D:8.21]` | Ranked-only 12-week exactly-once soft season: compress 25% toward 1500, RD >=200 capped 350, retain volatility/history; casual remains continuous | Retried/concurrent rollover applies once, never touches casual, and preserves every rating event |
| 8.24 `[D:8.9,8.20,8.21]` | Ranked reconnect/abandon: match-scoped authorisation, 60 s reclaim, server-owned connection generations, then abandoner loss and rolling 7-day 5 m/15 m/1 h/24 h cooldown | Reconnect works through backend outage and fences old peer; grace has no penalty; expiry outcome/escalation is deterministic and auditable |
| 8.25 `[D:8.10,8.24]` | Separate result delivery delay from match-integrity failure; signed Agones-annotation spool, retries, 5 m alert/30 m review, suppression only for lost/corrupt authority or measured unfair regional incident | API outage preserves rating/result; clients cannot request exemption; node/pod/integrity faults take the documented suppression/refund path |
#### 8dKeeping Docker and CI green #### 8DAgones, allocation and regional scaling
`make verify-phase6` and `make verify-enet-integration` must not regress.
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
assignment, and uses `--max-matches=2` to prove arena rotation — all three are
things allocation work would otherwise trample.
| # | Task | Acceptance | | # | Task | Acceptance |
|---|---|---| |---|---|---|
| 8.18 | **The rule: every allocation feature is opt-in via a `ServerConfig` flag whose default reproduces today's behaviour.** `ServerConfig` is built for exactly this — a flag declared once is parsed, validated, type-checked, config-file-backed and documented | `verify-phase6` and `verify-enet-integration` pass unchanged with no edits to their invocations | | 8.26 `[D:8.1,8.6,8.12]` | Portable Helm/Kustomize Fleets per build/EU/NA region; isolate provider edge/network/DNS/secret and SDR POP/cert/public-UDP overlays | Two provider fixtures render; labels select region/build/protocol/transport; each fixture documents Valve approval and externally reachable UDP mapping |
| 8.19 `[D:8.18]` | A **second** Compose file for the allocated-match path rather than mutating `compose.phase6-smoke.yml`, so the community-server model stays tested alongside the matchmade one | Both models have a green CI gate; neither shares a fixture with the other | | 8.27 `[D:8.26]` | Godot Agones REST adapter plus allocation-metadata watch and Go PID-1 supervisor scaffold; both bypass cloud behavior without SDK env; local SDK support | Native/existing Compose/CI remain functional; emulator exercises supervisor port discovery plus Get/Watch, Ready, Health, annotation and Shutdown |
| 8.20 `[D:8.18]` | `tests/cases/` coverage for the new flag parsing, per §10's no-live-server rule | New flags are unit-tested without a live server or a container | | 8.28 `[D:8.6,8.27]` | **Process-ready stage:** supervisor obtains dynamic port, launches Godot; static config/listen/Health succeed, then explicit Agones Ready—no roster/backend-registration prerequisite and no stdout probe | A detached unallocated process reaches Ready; a broken listener/config never does; Health reclaims a hung process |
| 8.29 `[D:8.26,8.27]` | Separate ENet and Hosted-SDR dynamic/passthrough mappings; supervisor exports local `SDR_LISTEN_PORT` and external `SDR_IP`; validate POP/cert/firewall/NAT | Two isolated matches share a node; Agones-reported public endpoint receives relay traffic on the bound socket; ENet fixture remains independent |
| 8.30 `[D:8.18,8.26,8.28,8.29]` | Atomic `GameServerAllocation` from Ready filtered by region/build/protocol/transport, attaching signed roster/non-secret config with bounded race retry | Duplicate commands yield one Allocated server; exhaustion or retry leaves no orphan; no client assignment is exposed merely because process is Ready |
| 8.31 `[D:8.9,8.30]` | **Assignment-ready stage:** watch Allocated metadata, verify manifest/bindings, register hosted address, acknowledge backend; only then mint/expose client tickets | Modified/wrong manifest never reaches assignment-ready; clients cannot connect early; secrets never appear in metadata/args/logs |
| 8.32 `[D:8.2,8.26,8.30]` | FleetAutoscaler with >=2 Ready processes across >=2 on-demand nodes/failure domains per queue-enabled region; pre-pull current/rollback; scale **Allocated** count to zero, never the Ready floor | Warm allocation meets p95 5 s/p99 10 s; disabled regions alone scale fully to zero; one-node loss retains certified Ready/headroom |
| 8.33 `[D:8.26,8.32]` | On-demand-only live capacity and measured N+1: loss of largest node leaves two Ready slots plus headroom for surviving Allocated matches | Interruptible nodes cannot receive live matches; forced node loss neither overloads survivors nor prevents the next allocation |
| 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog |
| 8.35 `[D:8.17,8.19,8.20,8.30,8.31]` | Initial-connect no-show: 30 s after assignment-ready; ranked cancels/no-show cooldown, casual bot policy, empty allocation exits | No allocation idles indefinitely; innocent players regain original precedence; no pre-live failure changes rating |
| 8.36 `[D:8.10,8.25,8.28,8.30]` | Go PID-1 supervisor traps TERM and authenticates localhost drain; 300 s grace/285 s infrastructure abort; PDB + Agones-aware Fleet drain; planned releases never TERM Allocated pods | Rollout/rollback waits Allocated=0; TERM path is exercised; forced timeout is classified/refunded; unexpected node loss is not claimed graceful |
| 8.37 `[D:8.5,8.10,8.25,8.26,8.31]` | Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO <=5 m/RTO <=30 m; signed result annotation/reconciliation preserves delivery outages | Restore/failover meet targets; live simulation continues; valid delayed result commits exactly once after recovery |
| 8.38 `[D:7.7,8.26,8.36,8.37]` | Provider migration after Valve approves both providers' EU/NA POPs/certs and public UDP: restore, validate coordinator trust, switch allocations, drain old | Both geographies complete Hosted-SDR matches on new provider and no old-provider live match is terminated |
> **The cold-start tension is real and is not solved by fast boot.** "Only pay #### 8E — Client experience and recovery
> during a match" and "a player never waits" pull against each other: a server
> must be listening *before* the matched players connect. 870 ms makes the gap
> small, but the risk is not the container — image pull on a cold node,
> scheduler placement and network/port programming can each dwarf it.
> Recommendation: **match-level scale-to-zero over a small warm node pool**,
> not node-level scale-to-zero. The per-match process genuinely exists only for
> the match; the pool absorbs cold-start variance. Revisit only when measured
> allocation latency on real infrastructure says the pool is unnecessary.
> **Server cost re-enters the design.** Community servers are paid for by | # | Task | Acceptance |
> whoever hosts them; allocated servers are paid for by the project, per match. |---|---|---|
> `README.md`'s original note about a subscription to fund servers is suddenly | 8.39 `[D:8.3,8.14,8.17]` | Queue UI: playlist/quality, elapsed and estimated wait, proposal countdown, allocation/connect state, cancel and latency/capacity explanations | Every backend state and terminal failure has a non-stuck visible state; cancel/decline is acknowledged authoritatively |
> load-bearing. 8.12 needs to produce a number before launch, not after. | 8.40 `[D:8.3,8.14]` | One authenticated revisioned WebSocket plus REST resync; resume valid queue/assignment after client restart | Missed/duplicate/out-of-order events converge and restart never creates a second ticket |
| 8.41 `[D:7.8,8.9,8.31,8.40]` | After assignment-ready, install SDR relay ticket before connect and send match-scoped join authorisation in `hello`; retain ENet assignments locally | Production connects/reconnects/fences old generation through SDR, never before assignment-ready; allocated/direct ENet and community flows remain compatible |
| 8.42 `[D:8.22,8.23,8.24,8.40]` | Backend-authoritative provisional/rank/tier/delta, abandon status and season countdown UI | Client performs no rating math and displays the committed revision after reconnect |
| 8.43 `[D:8.39,8.40,8.41]` | Recovery paths for decline, expiry, startup failure, version mismatch, auth expiry, regional outage and failed reconnect | Automated UI/state tests prove every case returns to a usable queue/menu or resumes the match without a duplicate action |
#### 8F — Observability, verification, cost and rollout
| # | Task | Acceptance |
|---|---|---|
| 8.44 `[D:8.3,8.4,8.28,8.31]` | Propagate queue/proposal/match/server IDs and process-ready/assignment-ready through logs, metrics, traces and replay metadata; redact credentials | One ID traces queue→result across components and automated secret-canary tests find no auth/relay ticket |
| 8.45 `[D:8.2,8.44]` | Dashboards/alerts for wait/MMR/RTT, proposals, allocation/Ready/image pull, connect/no-show, tick/crash/flood, result conflict/lag, abandons and cost | Each SLO and security/cost signal has an exercised alert and runbook |
| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | Go unit, race, fuzz, property, migration and concurrency suites | CI covers auth/join replay-reclaim, stale revisions, durable matcher fencing with lost Redis ack, rollover, result conflict/delivery retry and PostgreSQL retry |
| 8.47 `[D:8.7,8.30]` | Fake Steam verifier and fake allocator for deterministic CI | Normal CI needs no Steam/cloud secret or internet access and can force every success/failure deterministically |
| 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | Second Compose flow: fake backend → queue/proposal → process-ready/allocation/assignment-ready → ENet roster → result ack → shutdown; do not edit Phase 6 fixture | Both server models have independent green gates; existing Make invocations remain unchanged |
| 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | Disposable `kind` + Agones integration gate | CI covers dynamic ports, both readiness stages, roster/no-show, races, multi-match node, result-pending reconciliation, drain and rollback |
| 8.50 `[D:8.25,8.37,8.43,8.49]` | Network/chaos suite: 100 ms RTT, jitter/loss, client/API/matcher restart, game-pod death, node drain, Redis failover and control-plane loss | System recovers to a defined state; infrastructure-caused cases cannot penalise affected players |
| 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | Load test >=10,000 queued clients, >=100 proposals/s and forecast launch concurrency x2 | API p95 <=250 ms, durable matcher fence holds, both readiness/allocation SLOs are met and replicas scale without duplicate claims |
| 8.52 `[D:8.32,8.34,8.45,8.51]` | Per-region cost model from measured density, warm capacity, bandwidth, DB/Redis and telemetry; add budgets and allocation quotas | Cost per completed match and forecast monthly bands are recorded; a denial-of-wallet test triggers limits/alerts before budget breach |
| 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | Progressive release: development → internal → casual canary → casual → provisional ranked → ranked | Each promotion requires SLO/security/cost gates, rollback rehearsal, EU+NA playtests and unchanged legacy gates; rollback criteria and owner are explicit |
Implementation invariants for every task above:
- Matchmade mode is opt-in; every new `ServerConfig` default preserves the
existing community-server path.
- `compose.phase6-smoke.yml`, `make verify-phase6`, and
`make verify-enet-integration` are not repurposed or weakened.
- Production uses ticketed Hosted Dedicated Server SDR; ENet remains the
deterministic local/CI and direct-IP path.
- One process serves one match. Warm processes/nodes absorb startup variance;
capacity and cost are determined from 8.34 measurements, not old estimates.
- Implementation evidence is appended under the completed task as in earlier
phases; design changes first update `docs/MATCHMAKING.md` and dependencies.
--- ---