mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
docs: finalize scalable matchmaking plan
This commit is contained in:
Josh Creek
@@ -34,7 +34,11 @@ There will be default bots available, trained using reinforcement learning, and
|
||||
|
||||
## MVP
|
||||
|
||||
The first version of this game will be JUST the game, no server-side functionality at all. It will be a local only game where you can play against bots. Split-screen multiplayer could be added in a version 0.2 if demand is high enough. If there is sufficient interest then the server-side functionality can be added to enable online play, with a system in place to ensure that servers can be paid for (perhaps a cheap monthly subscription model?).
|
||||
The original local-only milestone is complete; the 1.0 scope now includes
|
||||
dedicated online play plus casual and ranked matchmaking. Community servers
|
||||
remain self-hostable, while project-hosted match servers are allocated per
|
||||
match through the provider-portable control plane described in
|
||||
[`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Split-screen remains deferred.
|
||||
|
||||
## Monetisation
|
||||
|
||||
|
||||
+423
-196
@@ -1,243 +1,470 @@
|
||||
# Matchmaking — casual and ranked queues
|
||||
|
||||
Design scope for online casual and ranked play. This is a **1.0 launch
|
||||
blocker**, not a post-launch addition.
|
||||
Architecture and locked product policy for Phase 8. This is a **1.0 launch
|
||||
blocker**. The numbered, independently implementable tasks and their
|
||||
acceptance criteria live in [`multiplayer-todo.md`](../multiplayer-todo.md);
|
||||
the short live checklist is [`multiplayer-next.md`](../multiplayer-next.md).
|
||||
|
||||
Nothing described here is implemented yet. This doc exists to record the
|
||||
decisions and the reasoning before code is written; per-task implementation
|
||||
evidence belongs in `multiplayer-todo.md` once work starts, and the live
|
||||
checklist lives in [`multiplayer-next.md`](../multiplayer-next.md).
|
||||
Nothing in Phase 8 is implemented yet. This document records the decisions
|
||||
those tasks assume so an implementer does not have to redesign the system
|
||||
while building one part of it.
|
||||
|
||||
## The model change
|
||||
## 1. Model and non-negotiable constraints
|
||||
|
||||
The multiplayer that exists today is a **community-server** model. A
|
||||
dedicated server runs forever: it waits for `--min-players` by roster,
|
||||
counts down `--start-countdown`, loads the next arena from the rotation,
|
||||
plays a match, returns to the lobby, and repeats (`server_match_loop.gd`).
|
||||
Players reach it by direct IP, and after Phase 7 by a Steam server browser.
|
||||
The server is the durable thing and players come and go around it.
|
||||
The existing multiplayer is a community-server model: a dedicated server
|
||||
runs continuously, waits for players, rotates arenas, and can be reached by
|
||||
direct IP or the Phase 7 Steam browser. Matchmaking adds a second model:
|
||||
players queue, every selected human accepts a proposal, one server process is
|
||||
allocated for that match, and the process is destroyed after its result is
|
||||
durably recorded. Ranked selects six humans; relaxed casual selects two to six
|
||||
and discloses its bot-filled team composition before acceptance. Both models
|
||||
ship.
|
||||
|
||||
Queued matchmaking inverts that. Players are the durable thing: they enter a
|
||||
queue, a matchmaker groups them by rating and region, and a **server is
|
||||
allocated for that one match** and torn down afterwards. Both models can
|
||||
coexist — community servers via the browser, queues via the matchmaker — and
|
||||
they should, because the server browser is already most of the way to done.
|
||||
Locked constraints:
|
||||
|
||||
## Hard prerequisite: verified identity
|
||||
- One authoritative Godot process hosts exactly one match.
|
||||
- Solo 3v3 casual and ranked queues launch first. Party-shaped fields are
|
||||
reserved in contracts, but party formation is deferred.
|
||||
- Production matchmade traffic uses ticketed Steam Hosted Dedicated Server
|
||||
SDR. Direct ENet remains first-class for local development, CI, LAN,
|
||||
self-hosting, and community servers.
|
||||
- The control plane is Go, PostgreSQL, and Redis, deployed on Kubernetes.
|
||||
Agones owns game-server allocation and lifecycle.
|
||||
- Infrastructure is provider-portable. Provider-specific cluster, network,
|
||||
DNS, and secret-store configuration lives behind isolated deployment
|
||||
overlays; application code never calls a provider allocation API.
|
||||
- Launch game fleets run in Europe and North America. Placement is
|
||||
latency-first and never silently violates the ping ceiling to shorten a
|
||||
queue.
|
||||
- Phase 8 features are opt-in. With allocated mode disabled, `ServerConfig`,
|
||||
ENet, Docker Compose, and the existing community-server behavior remain
|
||||
unchanged.
|
||||
|
||||
Ranked cannot ship before Phase 7's Steam auth tickets.
|
||||
## 2. Steam and trust boundaries
|
||||
|
||||
Slot reclaim is currently keyed by **display name** (see
|
||||
`--slot-reservation-seconds`, and the known-issues list in
|
||||
`multiplayer-next.md`). A rating attached to a spoofable identity is worse
|
||||
than no rating at all: it is trivially farmed, and it invites players to
|
||||
invest in a ladder that cannot be defended. "Ranked is critical" therefore
|
||||
*raises* the priority of Steam identity rather than routing around it.
|
||||
Phase 7 verified Steam identity is a hard prerequisite. The current slot
|
||||
reservation is keyed by display name, so no public queue or rating may rely on
|
||||
it.
|
||||
|
||||
Casual queueing has a weaker requirement — it still needs stable identity for
|
||||
abandon penalties and ban enforcement, but the cost of a compromise is lower.
|
||||
Two Steam credentials have different purposes and must not be conflated:
|
||||
|
||||
## Architecture
|
||||
1. A client obtains a single-use Web API ticket for backend login. Only the
|
||||
secure backend calls `AuthenticateUserTicket`, checks the expected App ID
|
||||
and identity string, and turns the result into a revocable session. A
|
||||
client-supplied SteamID is never identity.
|
||||
2. For a formed match, the game coordinator creates a short-lived SDR relay
|
||||
ticket authorising one player to one hosted server. The client installs it
|
||||
before connecting. The server separately verifies the signed match join
|
||||
authorisation before admitting the player to the assigned roster slot.
|
||||
|
||||
Decided: **Steam for identity, a project-owned backend for everything else.**
|
||||
Steam authentication/session tickets are single-use and their lifecycle must
|
||||
include the appropriate cancel/end calls; identity is not valid until Steam's
|
||||
asynchronous validation succeeds. Hosted SDR relay tickets are different:
|
||||
they are short-lived, match/server/identity-scoped and deliberately cached for
|
||||
reconnect. See [Steam authentication](https://partner.steamgames.com/doc/features/auth)
|
||||
and the [`ISteamUserAuth` Web API](https://partner.steamgames.com/doc/webapi/isteamuserauth).
|
||||
|
||||
This reverses the "no backend" position stated in
|
||||
[`TECH_STACK.md`](TECH_STACK.md) and `README.md`, which described the state
|
||||
of the project before matchmaking was scoped. The dedicated server remains a
|
||||
Godot export; the new service is separate from it.
|
||||
Ticketed Hosted Dedicated Server SDR is the production transport because it
|
||||
hides player/server IP addresses and authenticates, encrypts, and rate-limits
|
||||
traffic. It also supplies relay routing that may improve the path. It requires
|
||||
a real App ID, coordinator SDK/signing approval, certificates, and hosted
|
||||
data-centre coordination with Valve; those are explicit release dependencies,
|
||||
not assumptions. See [Steam Datagram Relay](https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay).
|
||||
|
||||
The alternative — Steam-native matchmaking (lobbies plus Leaderboards or User
|
||||
Stats as the rating store) — was rejected on two grounds. Steam lobby
|
||||
matchmaking has no real concept of a skill distribution to match against, and
|
||||
Leaderboards are a display surface rather than a rating store with the
|
||||
transactional guarantees a ladder needs. It would also permanently bind the
|
||||
game to Steam, foreclosing other platforms.
|
||||
### Trust table
|
||||
|
||||
### Components
|
||||
|
||||
| Component | Runs where | Responsibility |
|
||||
| Input | Trusted only after | May affect |
|
||||
| --- | --- | --- |
|
||||
| Steam auth ticket validation | backend | Turn a client-supplied ticket into a verified SteamID via the Steamworks Web API. The only trusted source of identity. |
|
||||
| Queue / matchmaker | backend | Hold queued players per playlist and region; form matches on rating proximity with a widening tolerance over wait time. |
|
||||
| Rating store | backend (DB) | Per-identity, per-playlist rating and match history. Written only by the backend, never by a game client. |
|
||||
| Server allocator | backend | Start a dedicated-server instance per formed match, hand its address to the matched clients, reclaim it on exit. |
|
||||
| Dedicated server | Godot export | Unchanged simulation. Gains a mode where the roster is *assigned* rather than open, and reports a result at the end. |
|
||||
| Game client | Godot | Queue UI, estimated wait, accept/decline, connect-on-assignment, post-match rating delta. |
|
||||
| Steam Web API ticket | Backend validation for the expected App ID/identity and replay check | Backend session identity |
|
||||
| Steam ping location + active-probe evidence | Backend verifies nonce/freshness and computes estimates; later compares with observed RTT | Placement only, never results |
|
||||
| Queue/accept request | Auth, schema/rate-limit, revision and idempotency validation | That player's queue state |
|
||||
| Join authorisation | Server signature, expiry, match/server/SteamID/slot and connection-generation validation | Initial admission or idempotent reclaim of that same slot |
|
||||
| Gameplay input | Existing server framing, sequence, byte and rate validation | Authoritative simulation input only |
|
||||
| Match result | Assigned server workload identity plus match/server binding | Transactional result/rating commit |
|
||||
|
||||
### What already exists and gets reused
|
||||
Clients never submit ratings, outcome, penalty exemptions, server health, or
|
||||
allocation state. A dedicated server never holds the Steam publisher key or
|
||||
the coordinator root signing key. Keep the offline SDR CA separate from the
|
||||
online leaf ticket key. The online key is exposed only through a narrowly
|
||||
authorised signer backed by KMS/HSM-equivalent non-exportable storage; API,
|
||||
matcher, allocator and game-server pods cannot read it. The signer accepts
|
||||
only allocator-recorded assignments, audits every signature, and supports
|
||||
overlapping-key rotation.
|
||||
|
||||
The server side needs less new work than it looks:
|
||||
## 3. Control-plane architecture
|
||||
|
||||
- **`--max-matches=1`** already makes the server drain and `exit(0)` after a
|
||||
single match. That is precisely the lifecycle a per-match allocator wants;
|
||||
it was built for CI, and it generalises for free.
|
||||
- **`ServerConfig`** is a single-source-of-truth flag table with strict
|
||||
validation — new allocation flags are declared in one place and are
|
||||
automatically parsed, type-checked, config-file-backed and documented.
|
||||
- **`--min-players` / `--start-countdown` / `--slot-reservation-seconds`**
|
||||
are the match-formation primitives, and they already count *roster*
|
||||
members rather than raw peers.
|
||||
- **`MatchNet`'s roster** already survives the lobby→match transition, which
|
||||
is the structure an assigned roster slots into.
|
||||
- **`MatchState`** already has a legal-transition table with wire-stable
|
||||
integer values, so new lifecycle states append cleanly.
|
||||
Use one repository and shared domain packages, with independently runnable
|
||||
roles rather than independently designed microservices:
|
||||
|
||||
### What is genuinely new
|
||||
| Role | Responsibility |
|
||||
| --- | --- |
|
||||
| API | HTTPS/WebSocket auth, profile, queue commands, status resync |
|
||||
| Matcher | Atomic proposal formation from queue state |
|
||||
| Allocator | Agones allocation, server registration, assignment delivery |
|
||||
| Maintenance worker | Outbox delivery, season rollover, expiry, reconciliation |
|
||||
|
||||
- The backend service itself (process, deploy, DB, ops) — nothing like it
|
||||
exists in this repo today.
|
||||
- Server-authoritative **match results**: the dedicated server must report
|
||||
the outcome to the backend over a channel a client cannot forge. This is
|
||||
the first non-ENet/SDR network path in the project (see TECH_STACK's "no
|
||||
HTTP layer" note, which this supersedes).
|
||||
- An **assigned-roster** server mode: only the matched SteamIDs may take a
|
||||
slot, replacing the current first-come model.
|
||||
- Client-side queue UI and the accept/decline flow.
|
||||
API replicas are stateless. Redis sorted sets provide the fast candidate
|
||||
index, but Redis is never the durable allocation fence: asynchronous failover
|
||||
can lose an acknowledged write. A matcher claims a proposal in a PostgreSQL
|
||||
`SERIALIZABLE` transaction using a partial unique constraint that permits only
|
||||
one active proposal/match participation per player. The transaction records
|
||||
the proposal and participants before Redis cleanup; a stale Redis claim then
|
||||
loses at PostgreSQL and is repaired from the durable record. PostgreSQL is the
|
||||
source of truth for queue ownership, identities, sessions/revocations,
|
||||
seasons, ratings, matches, participants, penalties, result receipts, audits
|
||||
and the transactional outbox. Redis holds expiring presence, candidate
|
||||
indexes, latency evidence, session/revocation caches and transient fan-out.
|
||||
Losing the last acknowledged Redis write may delay/rematerialise a ticket or
|
||||
force a session cache miss, but can neither resurrect a revoked session, split
|
||||
a proposal nor corrupt a result/rating.
|
||||
|
||||
## Server orchestration and autoscaling
|
||||
For launch, run the horizontally scaled control plane in one primary
|
||||
Kubernetes region with a warm standby and tested restore path. Game fleets
|
||||
remain regional in EU and North America. This avoids a premature multi-writer
|
||||
database while keeping new-match control latency small relative to queue time.
|
||||
An outage may pause new queues/allocations, but live matches must continue.
|
||||
Targets are PostgreSQL RPO <= 5 minutes and control-plane RTO <= 30 minutes.
|
||||
|
||||
Requirement: game servers scale horizontally and automatically, spin up fast
|
||||
on demand, serve exactly one match, and shut down — so cost is incurred only
|
||||
while a match is being played. Docker and the existing CI gates must keep
|
||||
working unchanged.
|
||||
### Stable identifiers and state
|
||||
|
||||
### Why this is achievable: the server is already shaped for it
|
||||
Contracts define opaque `player_id`, `queue_ticket_id`, `proposal_id`,
|
||||
`match_id`, `server_id`, and `season_id`. Every mutating request has an
|
||||
idempotency key, expected revision, and versioned schema.
|
||||
|
||||
Two properties of the current build make per-match allocation practical
|
||||
rather than aspirational:
|
||||
The durable/transient state path is:
|
||||
|
||||
- **The container is small.** The `server` image target is a slim
|
||||
`ubuntu:24.04` runtime with three shared libraries and the exported
|
||||
binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image.
|
||||
Pulling it onto a fresh node is cheap.
|
||||
- **Boot to listening is sub-second.** Measured on this repo's
|
||||
`cosmicclash-server:latest`: **~870 ms** from container start to the
|
||||
`server_started` log line, averaged over three runs, read from the
|
||||
container's own clock. That measurement was taken under **x86_64 emulation
|
||||
on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux
|
||||
should be faster. Re-measure on the real target before setting timeouts.
|
||||
```
|
||||
QUEUED -> PROPOSED -> ACCEPTED -> ALLOCATING -> PROCESS_READY
|
||||
-> ASSIGNMENT_READY -> ASSIGNED -> CONNECTING -> LIVE
|
||||
-> RESULT_PENDING -> COMPLETED
|
||||
-> CANCELLED / EXPIRED / FAILED from the explicitly legal stages
|
||||
```
|
||||
|
||||
Combined with `--max-matches=1`, which already drains and `exit(0)`s after a
|
||||
single match, the lifecycle the allocator needs mostly exists: start
|
||||
container → serve one match → process exits → orchestrator reclaims.
|
||||
The API publishes revisioned changes over one authenticated WebSocket. REST
|
||||
GET endpoints are the recovery source after a disconnect or missed revision.
|
||||
Restarting the client resumes an unexpired ticket/assignment instead of
|
||||
creating another one.
|
||||
|
||||
### The cold-start tension, stated honestly
|
||||
Assignments include match/server IDs, protocol and client build, server image
|
||||
digest, playlist version, transport, region, expiry, a match-scoped join
|
||||
authorisation, and either the SDR hosted-server material or an ENet endpoint.
|
||||
The authorisation may be replayed only by the same Steam identity to reclaim
|
||||
the same match/server/slot before expiry. Each successful connection advances
|
||||
a server-owned generation and fences the prior connection; another identity,
|
||||
server or slot is always rejected. This deliberately supports reconnect when
|
||||
Steam or the control plane is temporarily unavailable. Incompatible
|
||||
protocol/build/playlist versions never enter one proposal.
|
||||
|
||||
"Only pay during a match" and "a player never waits" are in tension. A server
|
||||
must be listening *before* the matched players connect, so some cost always
|
||||
precedes the match. Sub-second boot makes the gap small enough that a pure
|
||||
scale-to-zero design is plausible — but the risk is not the container, it is
|
||||
everything around it: image pull on a cold node, scheduler placement, and
|
||||
network/port programming can each dwarf 870 ms.
|
||||
## 4. Queue and placement policy
|
||||
|
||||
Recommendation: **scale to zero at the node level is the wrong target; scale
|
||||
to zero at the match level is the right one.** Keep a small warm pool of
|
||||
nodes sized to the current queue depth, and start a per-match container on
|
||||
demand within it. The per-match process genuinely exists only for the match;
|
||||
the node pool absorbs the cold-start variance. Revisit only if measured
|
||||
allocation latency on real infrastructure shows the warm pool is unnecessary.
|
||||
Each verified player may own at most one active queue ticket. Heartbeats are
|
||||
sent every 10 seconds and queue presence expires after 30 seconds. Create,
|
||||
cancel, resume, accept, decline, and expiry are atomic and retry-safe.
|
||||
|
||||
### Findings that block a naive implementation
|
||||
The client submits its recent opaque Steam ping location plus nonce-bound
|
||||
active-probe responses from each regional endpoint; it does not submit the RTT
|
||||
used for placement. The backend validates a 30-second freshness window and
|
||||
nonce, then uses the Steam coordinator SDK and probe timings to compute the
|
||||
regional matrix. A predicted/observed discrepancy over 25 ms or 30% (whichever
|
||||
is larger) in three matches within 24 hours quarantines the account's samples:
|
||||
it may queue only in regions whose active probe independently remains under
|
||||
the ceiling until five clean matches clear the quarantine. The matchmaker:
|
||||
|
||||
**Readiness cannot be detected from the log line.** Godot's stdout is
|
||||
block-buffered when it is not attached to a TTY. Run the server image
|
||||
detached without `-t` and `docker logs` shows **nothing at all** — the
|
||||
`server_started` line does not appear even after 35 seconds, because the
|
||||
buffer never flushes. An orchestrator readiness probe that greps for that
|
||||
line will hang forever, and this was reproduced directly while measuring the
|
||||
boot time above. Either probe the UDP socket instead, or make the server
|
||||
flush explicitly. This also means container logs are not a reliable
|
||||
observability channel for a short-lived match server; treat log shipping as
|
||||
a separate problem.
|
||||
1. Finds regions in which every proposed player has predicted RTT <= 100 ms.
|
||||
2. Minimises the worst player's predicted RTT.
|
||||
3. Breaks ties by total predicted RTT, then ready server capacity.
|
||||
4. Widens rating tolerance with wait time, but never automatically widens the
|
||||
100 ms latency ceiling.
|
||||
|
||||
**One fixed port per container does not scale on a shared host.** `--port`
|
||||
defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing
|
||||
several matches onto one node needs either a port range allocated per
|
||||
container, or one address per container. This is a UDP service, so the usual
|
||||
HTTP ingress/L7 routing answers do not apply — the allocator must hand the
|
||||
client a concrete `host:port`.
|
||||
The target is regional observed p95 RTT <= 80 ms. Candidate formation is
|
||||
deterministic:
|
||||
|
||||
**The match cannot start on a schedule the players do not control.** Today
|
||||
the loop waits for `--min-players` then counts down. An allocated server is
|
||||
told *which* identities to expect, and needs a **no-show timeout**: if a
|
||||
matched player never connects, the server must abandon and exit rather than
|
||||
sit idle burning the cost this design is trying to avoid.
|
||||
- Anchor on the oldest compatible ticket (`enqueued_at`, then ticket ID).
|
||||
- A ticket's rating tolerance is `min(400, 100 + 25 * floor(wait_seconds/30))`
|
||||
rating points. A pair is compatible only when its absolute rating difference
|
||||
is within both tickets' tolerance. Provisional players use the same stored
|
||||
rating; their high RD changes ratings faster, not eligibility arithmetic.
|
||||
- Every candidate set contains the anchor. Choose the lexicographically
|
||||
smallest tuple `(worst_RTT, total_RTT, rating_range, -sum(wait_seconds),
|
||||
sorted_ticket_IDs)`, so latency dominates once the oldest player anchors
|
||||
fairness and every candidate satisfies the widening rule.
|
||||
- Partition humans exhaustively into legal teams and minimise absolute team
|
||||
mean-rating difference, then maximum opposing-player difference, then use
|
||||
lexical player IDs. Bots fill remaining casual slots after humans are
|
||||
assigned.
|
||||
|
||||
### Keeping Docker and CI green
|
||||
Every selected human receives a 10-second proposal before allocation. Ranked
|
||||
always selects six. Casual requires six before the anchor reaches 60 seconds;
|
||||
afterward it selects the largest compatible human count from six down to two,
|
||||
with at least one human per team, and displays teams/bot slots. Unanimous
|
||||
acceptance by the selected humans advances.
|
||||
|
||||
The existing gates must not regress. `make verify-phase6` builds the export,
|
||||
runs it in Compose, joins two headless clients and asserts both saw both
|
||||
goals and that the arena rotated between matches; `make verify-enet-integration`
|
||||
runs the source-build ENet matrix. Both depend on current behaviour:
|
||||
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
|
||||
assignment, and uses `--max-matches=2` to prove rotation.
|
||||
An explicit proposal decline cancels that player's ticket and applies a
|
||||
30-second casual or 2-minute ranked cooldown. A timeout applies 60 seconds in
|
||||
casual or 5 minutes in ranked. Three ranked proposal declines/timeouts within
|
||||
30 minutes apply 15 minutes. Players who accepted return with their original
|
||||
`enqueued_at` and precedence when another player declines, times out, or the
|
||||
allocation fails. Ordering ties use ticket ID. A ranked initial-connect
|
||||
no-show after accepting uses the ranked abandon cooldown ladder but never a
|
||||
rating loss because no rated match began.
|
||||
|
||||
The rule that keeps them passing: **every allocation feature is opt-in via a
|
||||
new `ServerConfig` flag whose default reproduces today's behaviour.** An
|
||||
assigned roster, a no-show timeout and result reporting must each be inert
|
||||
unless explicitly enabled. `ServerConfig` is built for exactly this — a flag
|
||||
declared once is parsed, validated, type-checked, config-file-backed and
|
||||
documented — and `tests/cases/` can cover the new parsing without a live
|
||||
server. A second Compose file should cover the allocated-match path rather
|
||||
than mutating the Phase 6 one, so the community-server model stays tested
|
||||
alongside the matchmade one.
|
||||
### Casual
|
||||
|
||||
### Open questions
|
||||
- Target six humans in 3v3.
|
||||
- After 60 seconds, a match may start with at least two humans, one on each
|
||||
team, and fill the other slots with server bots.
|
||||
- Human backfill may replace a bot/disconnected slot only at a kickoff
|
||||
boundary.
|
||||
- Backfill is a separate 10-second opt-in proposal showing score, time
|
||||
remaining, team and slot. Declining/timing out carries no cooldown. A
|
||||
backfilled player receives no hidden-rating update or abandon penalty for
|
||||
that match; acceptance removes their queue ticket only when assignment is
|
||||
ready. Choose the oldest ordinary casual ticket that meets the same build,
|
||||
region <=100 ms and current anchor-tolerance rules for the vacated human
|
||||
slot; ties use ticket ID.
|
||||
- An original casual participant gets 30 seconds to reconnect; leaving after
|
||||
that applies a 60-second queue cooldown. The match's ordinary hidden-rating
|
||||
result still applies, with no extra rating penalty.
|
||||
- An accepted casual initial-connect no-show gets the same 60-second cooldown.
|
||||
The match proceeds with a bot only if at least one human connected on each
|
||||
team; otherwise it cancels and restores every innocent ticket with original
|
||||
precedence.
|
||||
- Casual has a separate hidden Glicko-2 rating used only for matching.
|
||||
|
||||
- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this
|
||||
game-server lifecycle), Nomad, or direct cloud-API container starts. Not
|
||||
chosen. Agones is the strongest default because it models allocation,
|
||||
readiness and per-match lifetime natively.
|
||||
- **Port strategy** — port range per node versus one IP per match.
|
||||
- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 6–10 match
|
||||
processes per modern core and 150–250 MB RSS each. That estimate predates
|
||||
any allocation work and should be re-measured under real concurrency
|
||||
before it sizes a bill.
|
||||
- **Draining and deploys.** How a server version rolls out without killing
|
||||
matches in flight.
|
||||
### Ranked
|
||||
|
||||
## Casual vs ranked
|
||||
- Exactly six verified humans; never bots and never backfill.
|
||||
- Solo queue only at launch.
|
||||
- Only `ArenaRegistry` entries with `"random": true` are eligible. Elevated
|
||||
goals remain excluded until the trained-policy restriction is lifted.
|
||||
- A reconnecting player has 60 seconds to return using the existing
|
||||
assignment. After that, that player receives a loss regardless of the final
|
||||
team result and a rolling seven-day cooldown: 5 minutes, 15 minutes, 1
|
||||
hour, then 24 hours.
|
||||
- Delivery failure is not match-integrity failure. A healthy completed match
|
||||
remains rated while its result waits for the control plane. Rating is
|
||||
suppressed only when the authoritative roster, simulation or result is
|
||||
unavailable/corrupt, or a measured regional incident prevented fair play.
|
||||
|
||||
They are different playlists, not a difficulty toggle, and their rules
|
||||
diverge in ways that affect the server:
|
||||
## 5. Rating and seasons
|
||||
|
||||
| | Casual | Ranked |
|
||||
| --- | --- | --- |
|
||||
| Rating | Hidden, used only for matching | Visible, with tiers |
|
||||
| Backfill on disconnect | Yes — keep the match playable | No — the match is rating-bearing and must not change shape mid-way |
|
||||
| Bots filling empty slots | Acceptable (`--fill-bots` exists) | Never |
|
||||
| Abandon penalty | Light (short queue cooldown) | Real (rating loss, escalating cooldown) |
|
||||
| Arena selection | Full rotation | Restricted set, so a variant nobody has practised can't decide a ladder match |
|
||||
| Party / premade | Unrestricted | Constrained by size and rating spread |
|
||||
Use canonical Glicko-2 independently per playlist with initial rating 1500,
|
||||
RD 350, volatility 0.06, scale constant 173.7178 and tau 0.5. Updates are
|
||||
immediate per committed match rather than globally batched. Before an update,
|
||||
advance inactivity by whole 24-hour rating periods since the player's last
|
||||
rated match using `phi = min(350/173.7178, sqrt(phi^2 + sigma^2 * periods))`.
|
||||
|
||||
Note the arena constraint interacts with an existing rule: elevated-goal
|
||||
variants are Free-Play-only until a checkpoint trained on
|
||||
`training_elevated.tscn` is promoted (`arena_registry.gd`). Ranked's arena
|
||||
set should be drawn from `"random": true` arenas only.
|
||||
For each player `i`, transform every opposing human's pre-match rating/RD to
|
||||
`mu_j`/`phi_j` and use the canonical equations
|
||||
`g(phi)=1/sqrt(1+3*phi^2/pi^2)` and
|
||||
`E=1/(1+exp(-g(phi_j)*(mu_i-mu_j)))`. Ranked's three opponent contributions
|
||||
use `w=1/3`; casual uses `w=1/N` for the `N` opposing humans, ignoring bots.
|
||||
Thus human contributions total exactly one match in both sums:
|
||||
|
||||
## Open questions
|
||||
```
|
||||
v^-1 = sum(w * g(phi_j)^2 * E_j * (1-E_j))
|
||||
Delta = v * sum(w * g(phi_j) * (s-E_j))
|
||||
```
|
||||
|
||||
- **Rating algorithm.** Glicko-2 is the default recommendation over plain
|
||||
Elo: it models rating *uncertainty*, which matters enormously for a small
|
||||
launch population where most players have few games. Not yet decided.
|
||||
- **Team rating from individual ratings.** How a 3v3 match's outcome
|
||||
distributes across six players is a separate design problem from the
|
||||
rating system itself.
|
||||
- **Server cost.** Allocated servers cost real money per match, unlike
|
||||
community servers that players host themselves. `README.md`'s original
|
||||
note about a subscription to fund servers is suddenly load-bearing again.
|
||||
Population size and match length set the bill; this needs a number before
|
||||
launch, not after.
|
||||
- **Region / ping policy.** How much rating tolerance to trade for latency,
|
||||
and whether cross-region is ever allowed at low population.
|
||||
- **Placement matches** and whether ranked has a soft reset per season.
|
||||
- **Backend language and hosting.** Not chosen. It does *not* have to be C#
|
||||
despite the original README framing — that framing was aspirational and
|
||||
predates every real decision in this project.
|
||||
Then run the canonical Glicko-2 volatility iteration and rating/RD update. If
|
||||
a player has no opposing human, the match is unrated for that player. `s` is 1/0 for
|
||||
the authoritative winner/loser and 0.5 only for a completed draw. Overtime is
|
||||
an ordinary win/loss. A ranked abandoner gets `s=0` regardless of the final
|
||||
team result; non-abandoning players use the authoritative final result.
|
||||
Cancelled or integrity-failed matches do not update anyone.
|
||||
|
||||
## Explicitly out of scope
|
||||
Lock all six participant rows in lexical player-ID order and compute every
|
||||
new value from the same immutable pre-match snapshot inside one serializable
|
||||
transaction. This prevents update-order bias and concurrent double updates.
|
||||
Golden vectors include canonical one-player examples plus symmetric/asymmetric
|
||||
3v3, draw, overtime, abandon, inactivity and concurrent-result fixtures.
|
||||
|
||||
Tournaments, in-game leaderboards beyond a personal rank display,
|
||||
cross-platform play with non-Steam identity providers, and spectator/observer
|
||||
tooling for ranked matches. None are precluded by this design; none are
|
||||
launch scope.
|
||||
The first ten ranked matches are provisional. Ranked exposes
|
||||
backend-derived tiers; casual rating remains hidden. Stored Glicko values, not
|
||||
tier labels or client calculations, are authoritative. Ranked seasons last 12
|
||||
weeks. Ranked rollover sets
|
||||
`rating = 1500 + 0.75 * (rating - 1500)`, raises RD to at least 200 (capped at
|
||||
350), preserves volatility/history, and is an exactly-once idempotent
|
||||
transaction. Casual rating is continuous and never season-reset.
|
||||
|
||||
Rating, penalty, match completion, participant records and outbox events are
|
||||
committed in that transaction. Duplicate identical server results succeed
|
||||
idempotently. A conflicting result changes nothing and pages an operator.
|
||||
|
||||
## 6. Game-server allocation and lifecycle
|
||||
|
||||
Use provider-portable Kubernetes manifests and Agones. Create one versioned
|
||||
Fleet per compatible build and region; labels identify region, protocol,
|
||||
transport, and image/build. Allocate atomically with `GameServerAllocation`.
|
||||
Agones, rather than bespoke allocator code, owns selection and lifecycle. See
|
||||
[GameServerAllocation](https://agones.dev/site/docs/reference/gameserverallocation/).
|
||||
|
||||
The Godot process talks to the Agones REST sidecar through a small adapter
|
||||
that is a no-op when `AGONES_SDK_HTTP_PORT` is absent. This preserves native,
|
||||
Compose, and CI operation and works with the Agones local SDK emulator. See
|
||||
[Agones client SDKs](https://agones.dev/site/docs/guides/client-sdks/).
|
||||
|
||||
Agones has two distinct readiness points; conflating them is a deadlock because
|
||||
`GameServerAllocation` selects a Ready server and only then attaches the match
|
||||
metadata:
|
||||
|
||||
1. The PID-1 supervisor queries the local Agones SDK for the assigned dynamic
|
||||
port/address, exports `SDR_LISTEN_PORT` plus `SDR_IP=public-address:port`
|
||||
(or the ENet equivalent), and launches Godot. Godot validates static config,
|
||||
binds the socket and starts Health calls.
|
||||
2. Godot calls Agones `Ready()` after the process is genuinely listening.
|
||||
This is **process-ready** only; never infer it from detached stdout.
|
||||
3. `GameServerAllocation` atomically changes that Ready server to Allocated
|
||||
and supplies the signed roster/non-secret match configuration as metadata.
|
||||
4. The server watches the GameServer, observes Allocated metadata, verifies
|
||||
manifest signature/build/protocol/server binding, registers its hosted SDR
|
||||
address, and calls the backend `assignment_ready` endpoint.
|
||||
5. Only after `assignment_ready` does the allocator mint relay/join tickets and
|
||||
expose the assignment to clients. The server accepts only assigned
|
||||
identities/slots; each selected human has 30 seconds to connect.
|
||||
6. Run one authoritative match with Health calls independent of the simulation
|
||||
loop. Submit the canonical result using the bound workload identity.
|
||||
7. Write the signed result hash/payload to the backend and to a non-secret
|
||||
Agones annotation, remain Allocated, and retry until the backend durably
|
||||
acknowledges it. The maintenance worker reconciles the annotation after an
|
||||
API outage. `RESULT_PENDING` pages at 5 minutes and requires operator review
|
||||
at 30 minutes; it never silently becomes unrated.
|
||||
8. After acknowledgement call `Shutdown()` and exit. Invalid/empty allocations
|
||||
shut down immediately. An allocated GameServer is not recycled to Ready.
|
||||
|
||||
Agones supplies dynamic host ports so multiple isolated matches can share a
|
||||
node; HTTP ingress is not involved in gameplay routing. Hosted SDR additionally
|
||||
requires Valve approval for every provider/location, a valid `SDR_POPID`,
|
||||
public-IP/unsolicited-UDP reachability, provider firewall/NAT validation,
|
||||
per-location certificates and coordinator trust. Use an Agones dynamic or
|
||||
passthrough mapping whose externally reported port is the `SDR_IP` port while
|
||||
the process binds `SDR_LISTEN_PORT`; test SDR and ENet mappings separately.
|
||||
Credentials arrive through runtime secret mounts, never allocation metadata,
|
||||
arguments, logs or images.
|
||||
|
||||
Result authentication uses a projected, pod-bound service-account token with
|
||||
a dedicated audience and one service account per workload class. The backend
|
||||
validates the configured cluster issuer/JWKS, audience, expiry, namespace,
|
||||
service account, bound pod UID and allocator-recorded GameServer UID, then
|
||||
checks that GameServer/match binding in PostgreSQL. Issuers and trust roots are
|
||||
allowlisted and rotated explicitly for every cluster/provider. A one-match
|
||||
server credential issued after this attestation is an acceptable equivalent.
|
||||
|
||||
### Warm capacity and density
|
||||
|
||||
Both launch regions are active whenever their queues are enabled. Each active
|
||||
region maintains at least two Ready processes distributed across at least two
|
||||
on-demand nodes/failure domains; the minimum node floor is therefore two, not
|
||||
one. Pre-pull current and rollback images. A queue/proposal-aware
|
||||
FleetAutoscaler adjusts capacity above that Ready floor. **Allocated** process
|
||||
count may fall to zero; Ready processes do not. An administratively disabled
|
||||
region may scale both nodes and Fleet to zero and is excluded from placement.
|
||||
See [Agones FleetAutoscaler](https://agones.dev/site/docs/reference/fleetautoscaler/).
|
||||
|
||||
Do not run live matches on interruptible nodes. N+1 means loss of the largest
|
||||
single node still leaves two Ready slots plus sufficient headroom for already
|
||||
Allocated matches; certify it in the node-loss test. Set pod requests, limits
|
||||
and node caps only after native x86_64 benchmarks of boot time, p99
|
||||
CPU/RSS/network and 60 Hz tick behavior, retaining 30% headroom. The current
|
||||
6–10 processes/core and 150–250 MB estimates are not sizing data.
|
||||
|
||||
Godot/GDScript cannot intercept SIGTERM, so Phase 8 adds a small Go PID-1
|
||||
supervisor. It starts Godot, traps TERM, and sends a per-pod-token-authenticated
|
||||
localhost drain request to an allocated-mode-only Godot control socket. Godot
|
||||
refuses new admissions and reports completion to the supervisor. Kubernetes
|
||||
uses a 300-second termination grace period; at 285 seconds the supervisor
|
||||
forces an infrastructure-failure abort so the pod cannot hang forever in
|
||||
overtime. A PodDisruptionBudget and Agones-aware drain prevent voluntary
|
||||
eviction of Allocated servers. Planned releases create a new Fleet, route new
|
||||
allocations to it and wait for old Allocated count zero without sending TERM.
|
||||
Unexpected node failure cannot be made graceful and follows the integrity
|
||||
failure/refund path.
|
||||
|
||||
Live matches continue through an API/control-plane delivery outage. A valid
|
||||
result is retried/reconciled as above; only loss of authoritative match
|
||||
integrity suppresses rating.
|
||||
|
||||
## 7. Security and operational baseline
|
||||
|
||||
The threat model must cover forged clients, ticket replay, duplicate queueing,
|
||||
result forgery, queue manipulation, packet floods, botting, server compromise,
|
||||
insider access, DDoS, vulnerable dependencies, and denial-of-wallet attacks.
|
||||
|
||||
Minimum controls:
|
||||
|
||||
- Non-root containers, read-only root filesystems, dropped Linux
|
||||
capabilities, RuntimeDefault seccomp, resource limits, restricted Pod
|
||||
Security admission, and least-privilege service accounts/RBAC.
|
||||
- Private PostgreSQL/Redis; default-deny ingress and egress with explicit
|
||||
NetworkPolicy allowances. See the [Kubernetes application security
|
||||
checklist](https://kubernetes.io/docs/concepts/security/application-security-checklist/)
|
||||
and [NetworkPolicy](https://kubernetes.io/docs/reference/kubernetes-api/networking/network-policy-v1/).
|
||||
- Encrypted databases/backups, externally populated Kubernetes Secrets,
|
||||
documented rotation, and no credentials in Git, images, command lines, or
|
||||
telemetry.
|
||||
- Per-account/IP API limits, request/body/schema limits, replay and duplicate
|
||||
detection, generic public errors, allocation quotas, and budget alerts.
|
||||
- Put HTTPS/WebSocket traffic behind a provider-portable edge contract
|
||||
implemented by each infrastructure overlay: managed volumetric DDoS
|
||||
absorption, WAF/rate rules, TLS termination, origin-only ingress and health
|
||||
checks. Enforce authenticated queue admission, per-account connection caps,
|
||||
WebSocket handshake/message/idle limits, bounded fan-out and overload
|
||||
shedding. In degraded mode reject new login/queue/allocation work while
|
||||
preserving result ingestion and all live matches.
|
||||
- Images pinned by digest, SBOM generation, dependency/image scanning, signed
|
||||
releases, admission-time signature verification, and a critical-patch SLA.
|
||||
- Structured audit events for auth, queue transitions, allocation, roster
|
||||
rejection, result conflict, penalty, season rollover, and operator action.
|
||||
|
||||
## 8. SLOs, observability, and release gates
|
||||
|
||||
Launch SLOs:
|
||||
|
||||
| Measure | Target |
|
||||
| --- | --- |
|
||||
| Eligible predicted RTT | <= 100 ms for every player |
|
||||
| Regional observed RTT | p95 <= 80 ms |
|
||||
| Unanimous acceptance to `assignment_ready` | p95 <= 5 s, p99 <= 10 s with warm capacity |
|
||||
| Published assignment to successful connection | p95 <= 5 s |
|
||||
| Successful allocation and durable result | >= 99.9% |
|
||||
| Certified server density | No tick backlog, with 30% resource headroom |
|
||||
| Control-plane API under load | p95 <= 250 ms |
|
||||
|
||||
Dashboards and alerts cover queue depth/wait, rating spread, predicted versus
|
||||
observed RTT, proposals/declines, allocation latency/failure, Ready capacity,
|
||||
image pulls, connection/no-show, physics overruns, crashes, abnormal packet
|
||||
rates, result lag/conflicts, abandons, and cost per completed match. Correlate
|
||||
all components with queue/proposal/match/server IDs, but never log auth or
|
||||
relay tickets.
|
||||
|
||||
Testing layers:
|
||||
|
||||
- Go unit, race, fuzz, property, migration, and concurrency tests.
|
||||
- Fake Steam verifier and fake allocator for deterministic CI.
|
||||
- A second allocated-server Compose flow; never mutate
|
||||
`compose.phase6-smoke.yml`.
|
||||
- Disposable `kind` + Agones integration for readiness, dynamic ports,
|
||||
health/no-show shutdown, allocation races, multiple matches per node,
|
||||
draining, and rollback.
|
||||
- Network/chaos cases for 100 ms RTT, jitter/loss, client/backend/matcher
|
||||
restart, game-pod death, node drain, Redis failover, and control-plane loss.
|
||||
- Load test at least 10,000 queued clients, 100 proposals/second, and forecast
|
||||
launch concurrency x2 while preserving correctness and API latency.
|
||||
- Provider migration rehearsal: restore data and deploy both regional fleets
|
||||
on a second provider whose EU/NA locations already have Valve approval,
|
||||
POP/certificates, public UDP/firewall verification and coordinator trust;
|
||||
switch new allocations, drain the old provider, and terminate no live match.
|
||||
|
||||
Release order is development -> internal -> casual canary -> full casual ->
|
||||
provisional ranked -> full ranked. Each promotion requires its SLO/security
|
||||
gates, rollback rehearsal, EU and North America playtests, a measured cost
|
||||
model, and unchanged `make verify-phase6` and
|
||||
`make verify-enet-integration`.
|
||||
|
||||
## 9. Explicitly out of scope
|
||||
|
||||
Parties/premades, tournaments, ranked spectators, public global regions,
|
||||
non-Steam identity providers, and a global leaderboard beyond personal rank
|
||||
display are not launch scope. The contracts reserve party identity and avoid
|
||||
Steam-specific database primary keys so those additions do not require a
|
||||
destructive redesign.
|
||||
|
||||
+6
-5
@@ -189,8 +189,9 @@ Python, no .NET, no network." Keeping the shipped game GDScript-only (no
|
||||
|
||||
## Planned, not yet built
|
||||
|
||||
- **A matchmaking backend service** — Steam auth ticket validation, casual
|
||||
and ranked queues, a rating store, and per-match dedicated-server
|
||||
allocation. Language and hosting are undecided. This is a 1.0 launch
|
||||
blocker and the single largest departure from "one Godot project, no
|
||||
backend". See [`MATCHMAKING.md`](MATCHMAKING.md).
|
||||
- **A Go matchmaking control plane** — independently runnable API, matcher,
|
||||
allocator and maintenance roles backed by PostgreSQL and Redis, deployed on
|
||||
provider-portable Kubernetes with Agones-managed game fleets. The cloud
|
||||
provider remains deliberately replaceable; the application stack is locked.
|
||||
This is a 1.0 launch blocker and the single largest departure from "one
|
||||
Godot project, no backend". See [`MATCHMAKING.md`](MATCHMAKING.md).
|
||||
|
||||
+121
-73
@@ -1,93 +1,141 @@
|
||||
# Multiplayer — next work
|
||||
|
||||
Short, current checklist for online multiplayer. Historical design decisions,
|
||||
implementation evidence, and completed work stay in
|
||||
[`multiplayer-todo.md`](multiplayer-todo.md).
|
||||
Short, current checklist for online multiplayer. Historical decisions,
|
||||
implementation evidence and task-level acceptance criteria stay in
|
||||
[`multiplayer-todo.md`](multiplayer-todo.md). Phase 8 architecture and locked
|
||||
product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md).
|
||||
|
||||
## Release blockers
|
||||
## Existing release blockers
|
||||
|
||||
- [ ] **Phase 4 playtest:** a human playtest at roughly 100 ms RTT. Confirm
|
||||
that ship and ball interaction feel local and contact corrections feel like
|
||||
bumps rather than glitches.
|
||||
- [ ] **Phase 5 session:** complete a real 3v3 match with a mid-match
|
||||
disconnect and late joiner.
|
||||
- [ ] **Phase 6 external check:** run the exported Docker server and clients
|
||||
from separate machines over the internet, then play a full match. Keep this
|
||||
controlled-only until Steam identity is complete.
|
||||
- [ ] **Phase 4 playtest:** play at roughly 100 ms RTT; confirm ship/ball
|
||||
interaction feels local and contact corrections read as bumps, not glitches.
|
||||
- [ ] **Phase 5 session:** finish a real 3v3 match with a mid-match disconnect
|
||||
and late joiner.
|
||||
- [ ] **Phase 6 external check:** play the exported Docker server from separate
|
||||
internet machines. Keep the check controlled until verified identity lands.
|
||||
|
||||
## Phase 7 — Steam, identity, discovery
|
||||
## Phase 7 — production Steam prerequisite
|
||||
|
||||
- [ ] Obtain the pinned GodotSteam client/server builds and Steamworks SDK
|
||||
access described in [`STEAM.md`](STEAM.md).
|
||||
- [ ] Run `make verify-steam-templates` with the custom executables and fix
|
||||
any custom-template failures.
|
||||
- [ ] Validate a two-account Steam SDR host/join using the existing explicit
|
||||
`NetworkManager` Steam transport. ENet direct-IP must keep passing its smoke
|
||||
test.
|
||||
- [ ] Build the Steam server browser: internet, LAN, favourites, and history.
|
||||
- [ ] Add Steam auth tickets, verified Steam identity in the roster, and a
|
||||
persistent ban list. This fixes the slot-reclaim security issue below.
|
||||
- [ ] Obtain the pinned GodotSteam client/server builds and Steamworks SDK;
|
||||
pass `make verify-steam-templates` without weakening ENet verification.
|
||||
- [ ] Validate two real accounts through the explicit Steam transport and
|
||||
build Internet/LAN/favourites/history server-browser views.
|
||||
- [ ] Add single-use auth tickets, asynchronous server validation, verified
|
||||
Steam identity, identity-keyed reconnect and persistent bans.
|
||||
- [ ] Obtain the real App ID, publisher key, coordinator SDK/signing approval,
|
||||
certificates and Hosted Dedicated Server data-centre support from Valve.
|
||||
- [ ] Implement ticketed Hosted Dedicated Server SDR routing, ticket install,
|
||||
reconnect and expiry. Preserve direct ENet for local/CI/community servers.
|
||||
|
||||
## Phase 8 — casual and ranked matchmaking (1.0 launch blocker)
|
||||
## Phase 8 — architecture, contracts and durable data
|
||||
|
||||
Design and reasoning: [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). This is a
|
||||
different server model from the community-server one that exists today —
|
||||
players queue, a matchmaker groups them, and a server is allocated per match.
|
||||
Phase 7's Steam auth tickets are a hard prerequisite: a rating attached to a
|
||||
spoofable identity is worse than no rating.
|
||||
- [ ] Lock the Go/PostgreSQL/Redis, Kubernetes/Agones, SDR, EU/NA and
|
||||
provider-portability ADR; define measurable launch SLOs.
|
||||
- [ ] Publish versioned OpenAPI/WebSocket contracts, stable IDs, legal state
|
||||
transitions, revisions and idempotency semantics.
|
||||
- [ ] Add PostgreSQL queue ownership/active-participation fences, durable
|
||||
domain migrations/outbox and Redis indexes/TTLs; lost Redis writes must not
|
||||
split a proposal or corrupt durable state.
|
||||
- [ ] Define assignment compatibility and opt-in `ServerConfig` flags whose
|
||||
defaults reproduce the community-server path.
|
||||
|
||||
- [ ] Decide the rating algorithm (Glicko-2 recommended over Elo for a small
|
||||
launch population) and how a team result distributes across individuals.
|
||||
- [ ] Choose the backend language and hosting, and cost out allocated servers
|
||||
per match at expected population.
|
||||
- [ ] Stand up the backend: Steam auth ticket validation via the Steamworks
|
||||
Web API, queue, rating store, server allocator.
|
||||
- [ ] Add an assigned-roster server mode so only matched SteamIDs may claim a
|
||||
slot, replacing the first-come model.
|
||||
- [ ] Add server-authoritative match result reporting to the backend over a
|
||||
channel a client cannot forge.
|
||||
- [ ] Client queue UI: playlist select, estimated wait, accept/decline,
|
||||
connect-on-assignment, post-match rating delta.
|
||||
- [ ] Casual and ranked playlist rulesets (backfill, bots, abandon penalties,
|
||||
arena restriction — see the comparison table in the design doc).
|
||||
## Phase 8 — identity and security
|
||||
|
||||
Server orchestration (same phase — the servers must autoscale and bill only
|
||||
for the duration of a match):
|
||||
- [ ] Validate Steam Web API tickets only in the secure backend; issue
|
||||
revocable sessions and reconnect-safe match/identity/slot authorisations
|
||||
with server-owned connection-generation fencing.
|
||||
- [ ] Authenticate results with pod/GameServer-bound workload identity; make
|
||||
identical duplicates idempotent and conflicting results inert/alerting.
|
||||
- [ ] Complete the threat model for forgery, replay, queue/flood/bot abuse,
|
||||
workload/insider compromise, DDoS, supply chain and denial-of-wallet.
|
||||
- [ ] Enforce restricted workloads/RBAC/networks/private stores/backups/secrets;
|
||||
isolate SDR signing behind an audited non-exportable signer and add
|
||||
volumetric edge defense, WebSocket limits and overload shedding.
|
||||
- [ ] Pin, scan, SBOM and sign artifacts; verify signatures at admission and
|
||||
document the critical vulnerability SLA.
|
||||
|
||||
- [ ] Choose the orchestrator (Agones on Kubernetes is the recommended
|
||||
default; it models allocation, readiness and per-match lifetime natively).
|
||||
- [ ] Fix readiness detection. Godot's stdout is block-buffered off a TTY, so
|
||||
`server_started` never appears in `docker logs` for a detached container —
|
||||
a log-grep readiness probe hangs forever. Probe the UDP socket, or flush.
|
||||
- [ ] Support more than one match per host: a per-container port from a range,
|
||||
or one address per match. `--port` defaults to 7777 and the Dockerfile
|
||||
hardcodes `EXPOSE 7777/udp`.
|
||||
- [ ] Add a no-show timeout so an allocated server that never fills abandons
|
||||
and exits instead of idling at cost.
|
||||
- [ ] Re-measure boot-to-listening on native x86_64 Linux. The repo's current
|
||||
figure is ~870 ms, measured under emulation on arm64 — a pessimistic bound.
|
||||
- [ ] Re-measure the SERVER.md sizing estimate (6–10 processes/core,
|
||||
150–250 MB RSS) under real concurrency before it sizes a bill.
|
||||
- [ ] Keep `make verify-phase6` and `make verify-enet-integration` green:
|
||||
every allocation feature is opt-in via a `ServerConfig` flag defaulting to
|
||||
today's behaviour, with a second Compose file for the allocated path rather
|
||||
than mutating `compose.phase6-smoke.yml`.
|
||||
## Phase 8 — queues, playlists and rating
|
||||
|
||||
## Known issues to resolve before public hosting
|
||||
- [ ] Add one PostgreSQL-owned queue ticket/player with 10 s heartbeat, 30 s
|
||||
expiry, Redis candidate cache and restart/failover repair.
|
||||
- [ ] Validate opaque Steam ping locations and nonce-bound probes server-side;
|
||||
require <=100 ms, enforce discrepancy quarantine and the locked widening/
|
||||
region/team tie-break rules.
|
||||
- [ ] Send 10 s proposals to every selected human: ranked six, relaxed casual
|
||||
two to six with disclosed bots; enforce exact cooldown and queue-precedence
|
||||
behavior.
|
||||
- [ ] Fence proposals/participants in a PostgreSQL serializable transaction;
|
||||
prove loss of an acknowledged Redis write cannot split players.
|
||||
- [ ] Casual: target 3v3 humans, after 60 s allow >=2 humans (one/team) plus
|
||||
bots, kickoff-only human backfill and no backfill loss/decline penalty.
|
||||
- [ ] Ranked: exactly six humans, solo-only, no bots/backfill, random-enabled
|
||||
non-elevated arenas only, 60 s reconnect grace and escalating abandons.
|
||||
- [ ] Implement the documented exact Glicko-2 equations, fractional 3v3
|
||||
weights, inactivity/update locking/golden vectors and ten provisional games.
|
||||
- [ ] Add ranked-only exactly-once 12-week soft seasons; distinguish retryable
|
||||
result-delivery outages from match-integrity failures and rating exemptions.
|
||||
|
||||
- [ ] Slot reclaim is currently keyed by display name, so someone can take a
|
||||
disconnected player's reserved slot. Do not expose public servers before
|
||||
verified Steam identity lands.
|
||||
- [ ] Investigate occasional input loss during a long server stall; the
|
||||
existing sequence resync recovers it, but transport delivery is variable.
|
||||
## Phase 8 — Agones and regional server capacity
|
||||
|
||||
- [ ] Add portable EU/NA Agones Fleets with provider edge/network/secret and
|
||||
Valve-approved SDR POP/certificate/public-UDP overlays.
|
||||
- [ ] Add the local-safe Agones adapter and separate process-ready (listen then
|
||||
Ready) from assignment-ready (Allocated manifest verified and registered).
|
||||
- [ ] Allocate from Ready by region/build/protocol/transport; use separately
|
||||
verified ENet and SDR dynamic/passthrough port mappings.
|
||||
- [ ] Deliver/verify the signed roster after allocation and expose client
|
||||
tickets only after backend `assignment_ready`.
|
||||
- [ ] Keep >=2 Ready processes across >=2 on-demand nodes/failure domains per
|
||||
queue-enabled region; only Allocated count may fall to zero.
|
||||
- [ ] Spread on-demand capacity across zones with N+1 headroom; do not place
|
||||
live matches on interruptible nodes.
|
||||
- [ ] Benchmark native x86_64 boot, p99 CPU/RSS/network and tick health; set
|
||||
requests/limits and node density from measurements plus 30% headroom.
|
||||
- [ ] Add 30 s no-show handling, Go PID-1 TERM/drain supervision, PDB/Fleet
|
||||
drain, signed result annotation/retry, RPO <=5 m and RTO <=30 m.
|
||||
- [ ] Rehearse migration only after the second provider's EU/NA locations have
|
||||
Valve approval, POP/certs, public UDP/firewall and coordinator trust.
|
||||
|
||||
## Phase 8 — client and recovery
|
||||
|
||||
- [ ] Build queue/proposal/allocation/connect/rating UI with explicit latency,
|
||||
capacity, expiry and recovery states.
|
||||
- [ ] Use one authenticated revisioned WebSocket plus REST resync; resume a
|
||||
valid ticket/assignment after restart rather than duplicating it.
|
||||
- [ ] After assignment-ready, install SDR ticket and send reconnect-safe join
|
||||
authorisation in `hello`; fence old connections and retain ENet behavior.
|
||||
- [ ] Display only backend-authoritative provisional rank/tier/delta, abandon
|
||||
status and season time; clients perform no rating calculation.
|
||||
|
||||
## Phase 8 — operations and release gates
|
||||
|
||||
- [ ] Correlate queue→result with IDs and add dashboards/alerts for SLOs,
|
||||
security, failures and cost without logging credentials.
|
||||
- [ ] Add Go race/fuzz/property/migration/concurrency coverage plus fake Steam
|
||||
and fake allocation for offline deterministic CI.
|
||||
- [ ] Add an independent allocated-server Compose flow; do not mutate
|
||||
`compose.phase6-smoke.yml` or weaken either existing Make gate.
|
||||
- [ ] Add disposable `kind`/Agones integration, 100 ms network/chaos cases and
|
||||
proof that infrastructure failures cannot punish players.
|
||||
- [ ] Load-test >=10,000 queued clients, >=100 proposals/s and forecast launch
|
||||
concurrency x2 while holding API p95 <=250 ms and allocation correctness.
|
||||
- [ ] Record cost per completed match, budget/denial-of-wallet controls and
|
||||
deploy progressively: internal → casual canary → casual → provisional
|
||||
ranked → ranked, with EU/NA playtests and rollback gates.
|
||||
|
||||
## Known issues before public hosting
|
||||
|
||||
- [ ] Replace display-name slot reclaim with verified Steam identity.
|
||||
- [ ] Investigate occasional transport input loss during a long server stall.
|
||||
- [ ] Fix the remaining `_broadcast_snapshot` packet-send stderr race.
|
||||
|
||||
## Decide after the latency playtest
|
||||
|
||||
- [ ] Decide whether client-only, contact-cohort shadow physics is worthwhile
|
||||
for the remaining prediction weakness.
|
||||
- [ ] Decide whether client-only contact-cohort shadow physics is worthwhile.
|
||||
|
||||
## Explicitly deferred
|
||||
|
||||
120 Hz simulation, latency-gap measurement, audio hooks, and split-screen are
|
||||
not part of the current multiplayer release path.
|
||||
Parties/premades, tournaments, ranked spectators, non-Steam identity,
|
||||
additional global regions, global leaderboards, 120 Hz simulation,
|
||||
latency-gap measurement, audio hooks and split-screen are not in the launch
|
||||
path.
|
||||
|
||||
+92
-68
@@ -14,13 +14,13 @@ Everything below is written so an agent (or a person) can pick up a single numbe
|
||||
|
||||
The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 0–5 contain no unfinished tasks.
|
||||
|
||||
**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.20 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three items there are findings rather than plans, and each would break a naive implementation:
|
||||
**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is not started.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.53 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three findings would break a naive implementation:
|
||||
|
||||
| # | Finding | Why it bites |
|
||||
|---|---|---|
|
||||
| 8.8 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | An orchestrator readiness probe that greps the log hangs forever |
|
||||
| 8.9 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches cannot share a host; being UDP, L7 ingress routing does not apply |
|
||||
| 8.18 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | Allocation work trivially regresses `verify-phase6` unless every new feature defaults to today's behaviour |
|
||||
| Task 8.28 | Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears | Process-ready must be an explicit Agones call after static validation/listen; post-allocation assignment-ready is separate and neither uses a log grep |
|
||||
| Task 8.29 | `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp` | Several matches need Agones dynamic UDP/SDR ports; L7 ingress does not route this traffic |
|
||||
| Task 8.48 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | The allocated flow needs its own fixture so Phase 6 behavior and invocations stay unchanged |
|
||||
|
||||
### Blocking sign-off — the work exists, the verification does not
|
||||
|
||||
@@ -50,7 +50,7 @@ C is the one to plan around: it is fixed for free by task **7.4** (Steam auth ti
|
||||
### Unstarted phases
|
||||
|
||||
- **Phase 6 external gate:** run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until **C** is fixed.
|
||||
- **Phase 7 — Steam transport, browser, identity** (5 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, auth tickets, and bans await a project-owned Steamworks App ID. Carries the fix for **C**.
|
||||
- **Phase 7 — Steam transport, browser, identity and production SDR** (8 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, verified tickets, bans, production credentials and ticketed Hosted Dedicated Server SDR await a project-owned Steamworks App ID and Valve coordination. Carries the fix for **C** and is the hard prerequisite for Phase 8.
|
||||
|
||||
Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into something another person can host. Phase 7 is the next block because Steam identity is required before public exposure.
|
||||
|
||||
@@ -69,7 +69,7 @@ Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into some
|
||||
| 1 | **Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation.** | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. |
|
||||
| 2 | **Dedicated servers only.** Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. |
|
||||
| 3 | **ENet first**, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and **must never become the degraded path**. |
|
||||
| 4 | **No custom backend.** | Steam's `ISteamGameServer` master-server listing covers discovery, `ISteamMatchmakingServers` covers the in-game browser, and Steam auth tickets cover identity and ban state. `README.md`'s C# backend stays unstarted. |
|
||||
| 4 | **Community discovery uses no custom backend; superseded for queued play by Phase 8.** | Steam's server APIs remain enough for the community browser. Casual/ranked queues, durable ratings, allocation and authoritative results require the project-owned Go control plane specified in `docs/MATCHMAKING.md`; it does not replace the browser or direct-IP path. |
|
||||
|
||||
### 1.2 Rejected alternatives
|
||||
|
||||
@@ -1135,6 +1135,9 @@ New `--role=client-reconnect` grades the returning player: not a spectator, owns
|
||||
| 7.3 `[D:7.2]` `[P]` | `server_browser.tscn` via `ISteamMatchmakingServers` | Internet, LAN, favourites and history lists all populate |
|
||||
| 7.4 `[D:7.2]` `[P]` | Auth tickets in `hello` → `BeginAuthSession`; Steam identity in the roster; persistent ban list | Ownership, VAC and ban state verified server-side |
|
||||
| 7.5 `[D:7.2]` `[P]` | Feature-gate every Steam call behind `OS.has_feature("steam") and ClassDB.class_exists("SteamMultiplayerPeer")`; verify the ENet path end to end | Non-Steam build is fully functional |
|
||||
| 7.6 `[D:7.4]` | Separate single-use tickets for backend Web API login and game-server auth; wait for Steam's asynchronous validation and cancel/end every ticket session | Replayed, cancelled, wrong-App-ID and not-yet-validated identities cannot enter a roster or queue; no client-supplied SteamID is trusted |
|
||||
| 7.7 `[D:7.1]` `[P]` | Obtain the production App ID, publisher key, SDR coordinator SDK/signing approval, certificates and hosted-data-centre support from Valve | Production prerequisites and rotation owners are recorded; no Spacewar credential or development certificate can reach a release build |
|
||||
| 7.8 `[D:7.6,7.7]` | Ticketed Hosted Dedicated Server SDR: routing registration, coordinator-issued player→server relay tickets, client ticket installation, reconnect and expiry | Two real accounts complete and reconnect to an assigned dedicated match through SDR; server/player IPs are not exposed; ENet gates remain green |
|
||||
|
||||
> **The transport interface is written here, not in Phase 1.** Eight virtual methods (`begin_auth`, `advertise`, `get_identity`, `supports_server_browser`…) designed against an API nobody on the project has used will be wrong. Write `NetworkManager._make_peer()` concretely in Phase 1 and extract the boundary once there are two real implementations. Locked decision 3 guarantees the ENet path is never deleted, so there is no migration risk in waiting.
|
||||
|
||||
@@ -1155,86 +1158,107 @@ and players find it by IP or (7.3) the server browser. Matchmaking makes the
|
||||
**allocated for that one match** and destroyed after. Both models ship; they
|
||||
are different playlists, not a replacement.
|
||||
|
||||
**Hard dependency on 7.4.** Slot reclaim is keyed by display name today. A
|
||||
rating attached to a spoofable identity is farmed trivially, so ranked cannot
|
||||
ship before Steam auth tickets land. Casual queueing needs 7.4 too, for
|
||||
abandon penalties and ban enforcement, but degrades more gracefully.
|
||||
**Hard dependency on 7.6 and 7.8.** Slot reclaim is keyed by display name
|
||||
today. A rating attached to a spoofable identity is farmed trivially, so no
|
||||
queue ships before single-use verified identity lands. Production allocation
|
||||
also depends on the ticketed Hosted Dedicated Server SDR route; ENet remains
|
||||
the local/CI/community transport, not a silent production fallback.
|
||||
|
||||
#### 8a — Backend service
|
||||
#### 8A — Architecture, contracts and data
|
||||
|
||||
| # | Task | Acceptance |
|
||||
|---|---|---|
|
||||
| 8.1 | Choose backend language, hosting and datastore. **Not C# by default** — that framing predates every real decision here | Written up with the rejected alternatives, as §1 does for the client decisions |
|
||||
| 8.2 `[D:7.4]` | Steam auth ticket validation via the Steamworks Web API; a verified SteamID is the only trusted identity | A forged or replayed ticket is rejected; no client-supplied identity is ever trusted |
|
||||
| 8.3 `[D:8.1]` | Rating store: per-identity, per-playlist rating plus match history, written only by the backend | A client cannot write its own rating by any path |
|
||||
| 8.4 `[D:8.3]` | Rating algorithm. **Glicko-2 recommended over Elo** — it models rating *uncertainty*, which dominates at launch when most players have few games | Simulated against a synthetic population; placement behaviour is sane at n≈0 games |
|
||||
| 8.5 `[D:8.4]` | Team-result → individual-rating distribution for 3v3 | A 3v3 outcome updates six ratings defensibly; documented, not folded into 8.4 |
|
||||
| 8.6 `[D:8.3]` | Queue and matchmaker: per playlist and region, rating proximity with tolerance widening over wait time | Queue depth and wait time are observable; tolerance widening is tunable without redeploy |
|
||||
| 8.1 | Add an ADR locking **Go + PostgreSQL + Redis**, provider-portable Kubernetes, Agones, ticketed Hosted Dedicated Server SDR, EU/NA fleets and independently runnable API, matcher, allocator and maintenance roles; keep `README.md`/`docs/TECH_STACK.md` consistent | The ADR names boundaries/rejected alternatives; current docs name the locked stack and replaceable provider; no application code calls a provider allocation API |
|
||||
| 8.2 `[D:8.1]` | Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | Each SLO has a metric, numerator/denominator, percentile window, owner and alert threshold before implementation is judged against it |
|
||||
| 8.3 `[D:8.1]` | Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | Generated contract tests cover every request, response, event and external error; clients can REST-resync after a missed WebSocket revision |
|
||||
| 8.4 `[D:8.3]` | Define opaque `player_id`, `queue_ticket_id`, `proposal_id`, `match_id`, `server_id`, `season_id`, legal queue/match state transitions, revisions and idempotency keys | Duplicate/out-of-order commands converge; invalid transitions are rejected without partial state |
|
||||
| 8.5 `[D:8.4]` | Add PostgreSQL migrations for durable queue ownership, active-participation fencing, identities, sessions/revocations, seasons, ratings/events, matches/participants, penalties, results, audits and outbox; document Redis caches/TTLs | A blank DB migrates up; lost Redis writes cannot resurrect revocation, split a proposal or corrupt durable state; rollback/forward compatibility is tested |
|
||||
| 8.6 `[D:8.3,8.4]` | Lock assignment compatibility: protocol/client build, image digest, playlist version, transport, region, expiry and signed authorisation; add all allocated-mode `ServerConfig` flags as opt-in defaults | Incompatible builds never share a proposal; absent flags reproduce today's community server and existing config tests cover every new flag |
|
||||
|
||||
#### 8b — Server orchestration and autoscaling
|
||||
|
||||
Requirement: servers scale horizontally and automatically, spin up fast, serve
|
||||
exactly one match, and shut down — cost incurred only while a match runs.
|
||||
|
||||
Two properties of the existing build make this practical rather than
|
||||
aspirational, both **measured against `cosmicclash-server:latest`**, not
|
||||
estimated:
|
||||
|
||||
- The runtime image (`server` target, slim `ubuntu:24.04`) is **~148 MB** of
|
||||
content — not the ~2.6 GB `godot-ci` build image.
|
||||
- Boot to the `server_started` line is **~870 ms**, container's own clock,
|
||||
mean of three runs. **Taken under x86_64 emulation on an arm64 host, so it
|
||||
is a pessimistic bound** — see 8.11.
|
||||
|
||||
`--max-matches=1` already drains and `exit(0)`s after one match. It was built
|
||||
for CI and generalises to the allocator lifecycle for free.
|
||||
#### 8B — Authentication and secure control plane
|
||||
|
||||
| # | Task | Acceptance |
|
||||
|---|---|---|
|
||||
| 8.7 | Choose the orchestrator. **Agones on Kubernetes is the recommended default** — it models allocation, readiness and per-match lifetime natively rather than making you rebuild them | Allocation, readiness and per-match teardown are all handled by the chosen system, not by bespoke glue |
|
||||
| 8.8 | **Fix readiness detection — this blocks any naive implementation.** Godot's stdout is block-buffered off a TTY. Run the server image detached without `-t` and `docker logs` shows *nothing at all*; `server_started` does not appear even after 35 s. A readiness probe that greps the log hangs forever. Probe the UDP socket, or flush explicitly | A cold container is marked ready by a mechanism that does not depend on stdout; reproduced-and-fixed, not worked around by adding `-t` in one place |
|
||||
| 8.9 | Multiple matches per host: a per-container port from a range, or one address per match. `--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. **This is UDP — L7 ingress routing does not apply**, the allocator hands the client a concrete `host:port` | Two matches run concurrently on one node and neither can reach the other's traffic |
|
||||
| 8.10 `[D:8.6]` | Assigned-roster server mode: only matched SteamIDs may claim a slot, replacing first-come. Plus a **no-show timeout** — an allocated server that never fills abandons and exits rather than idling at cost | An unmatched identity is refused a slot; a server nobody joins exits within the timeout |
|
||||
| 8.11 | Re-measure boot-to-listening on **native x86_64 Linux** before it sets any timeout | A number from the real target platform replaces the ~870 ms emulated bound recorded above |
|
||||
| 8.12 | Re-measure SERVER.md's sizing estimate (6–10 processes/core, 150–250 MB RSS) under real concurrency | A measured figure sizes the bill; the current estimate predates all allocation work |
|
||||
| 8.13 `[D:8.10]` | Server-authoritative match result reporting to the backend over a channel a client cannot forge. **The project's first non-UDP network path** — simulation stays on ENet/SDR | A client cannot report, alter or suppress a result |
|
||||
| 8.14 | Draining and deploys: roll out a server version without killing matches in flight | An in-flight match survives a deploy of the next server version |
|
||||
| 8.7 `[D:7.6,8.3]` | Validate `AuthenticateUserTicket` only in the secure backend with the expected App ID and identity string; reject expiry, replay, wrong app, bans and malformed input | Publisher credentials exist only in the backend secret store; forged/replayed tickets and client-supplied SteamIDs never create a session |
|
||||
| 8.8 `[D:8.7]` | Issue short-lived revocable sessions bound to verified Steam identity; add account/IP limits, body/schema limits, replay checks and generic public errors | Revocation takes effect across replicas; abuse cannot cause unbounded memory, work or response amplification |
|
||||
| 8.9 `[D:8.4,8.7]` | Issue match-scoped join authorisations bound to SteamID/match/server/team/slot/protocol/expiry; allow same-identity slot reclaim while fencing prior connection generations | Altered/expired/wrong identity/server/slot is rejected; reconnect works without backend/Steam; a newer generation makes the old connection unable to send gameplay |
|
||||
| 8.10 `[D:8.5,8.31]` | Authenticate results with pod-bound projected identity or one-match attested credential; validate issuer/audience/expiry, namespace/SA, pod UID, GameServer UID and allocator match binding | Another pod sharing a workload class cannot submit for the allocation; identical duplicates are idempotent; conflicting results are inert and alerting across all trusted clusters |
|
||||
| 8.11 `[D:8.1]` | Write the threat model: forged clients/replay/queues/results, floods/bots, pod/insider compromise, gameplay and API DDoS, SDR signing-key theft, dependencies and denial-of-wallet | Every threat has prevention/detection/owner/verification; accepted residual risks are explicit; offline CA and online signer trust boundaries are separate |
|
||||
| 8.12 `[D:8.11]` | Harden workloads and edge: restricted containers, least RBAC, private DB/Redis, default-deny networks, backups/secrets, volumetric DDoS/WAF/origin shielding, WebSocket limits and overload shedding | Policy/network tests enforce declared flows; edge load test preserves result ingress/live matches while rejecting new work; no credential appears in Git/images/args/telemetry |
|
||||
| 8.13 `[D:8.12]` | Pin images by digest; generate SBOMs, scan dependencies/images, sign artifacts, verify signatures at admission and document a critical-fix SLA | CI blocks a vulnerable/disallowed or unsigned release artifact and records the exact provenance deployed |
|
||||
|
||||
#### 8c — Playlists and client
|
||||
#### 8C — Queueing, matchmaking, playlists and rating
|
||||
|
||||
| # | Task | Acceptance |
|
||||
|---|---|---|
|
||||
| 8.15 `[D:8.6]` | Casual and ranked rulesets. They diverge on the server, not just in UI: backfill (casual yes / ranked never), bots filling slots (`--fill-bots` casual-only), abandon penalties, party size and rating spread | Ranked never backfills and never spawns a bot into a player slot |
|
||||
| 8.16 `[D:8.15]` | Ranked arena restriction. Draw only from `"random": true` arenas — **elevated-goal variants stay Free-Play-only** until a checkpoint trained on `training_elevated.tscn` is promoted (`arena_registry.gd`), so a variant nobody has practised cannot decide a ladder match | Ranked cannot select an elevated-goal arena |
|
||||
| 8.17 `[D:8.6]` | Client queue UI: playlist select, estimated wait, accept/decline, connect-on-assignment, post-match rating delta | A declined match returns the other players to the queue without penalty to them |
|
||||
| 8.14 `[D:8.4,8.5,8.8]` | One durable PostgreSQL queue owner/player plus Redis candidate index: 10 s heartbeat, 30 s expiry, retry-safe create/cancel/resume and repair after cache loss | Replicas/duplicates never place a player twice; failover may delay/rematerialise an index but durable ownership and active-participation fences converge |
|
||||
| 8.15 `[D:7.8,8.3]` | Submit opaque Steam ping location plus nonce-bound probes; backend computes estimates, enforces 30 s freshness and quarantines 3 discrepancies >25 ms or 30% until 5 clean matches | A client cannot directly choose its placement RTT; stale/forged evidence is rejected; quarantine behavior and server-observed comparison are deterministic |
|
||||
| 8.16 `[D:8.14,8.15]` | Implement the locked candidate/team algorithm: <=100 ms, oldest anchor, `min(400,100+25*floor(wait/30))` mutual rating tolerance, documented set/region/team tie-breakers | Fixtures cover provisional players, EU/NA/no-common-region, widening caps, deterministic partitions and low population; no placement crosses 100 ms |
|
||||
| 8.17 `[D:8.14,8.16]` | Ten-second proposal to **every selected human**: ranked 6; casual largest compatible 6→2 after 60 s with disclosed teams/bots; apply exact decline/timeout/no-show cooldown and queue-precedence rules | Allocation starts only after selected humans accept; 2–5-human casual is reachable; accepter timestamps restore exactly; ranked pre-match no-show has cooldown but no rating loss |
|
||||
| 8.18 `[D:8.5,8.14,8.17]` | Horizontally replicated matcher: Redis candidates, then PostgreSQL serializable proposal/participant fence, then cache cleanup/repair | Forced loss of the last acknowledged Redis write, retries, worker death and failover cannot claim a player into two proposals/matches |
|
||||
| 8.19 `[D:8.18]` | Casual policy: proposal composition above; >=1 human/team, exhaustive rating-balanced teams, bots after 60 s, opt-in kickoff-only backfill, 30 s reconnect and defined backfill/casual penalties | Every 2–6-human shape is tested; no mid-play replacement; declined/backfill participant gets no excluded rating/cooldown; original leaver gets only documented outcome/cooldown |
|
||||
| 8.20 `[D:8.18]` | Ranked policy: exactly six verified solo humans, no bots/backfill, only `ArenaRegistry.random`; define initial no-show, proposal timeout and reconnect/abandon transitions | Ranked rejects parties/bots/backfill/elevated arenas; every pre-live failure returns five innocent players with original precedence and applies no rating |
|
||||
| 8.21 `[D:8.5,8.20]` | Exact Glicko-2 equations from `docs/MATCHMAKING.md`: 1500/350/0.06/tau .5, ranked 1/3 and casual 1/N human-opponent weights, daily inactivity, immutable snapshot/lock order, draws/OT/abandons/cancellation | Canonical plus project 2–6-human/3v3 golden vectors pass; concurrent results serialize without order bias; clients have no rating-write path |
|
||||
| 8.22 `[D:8.21]` | First ten ranked games provisional; casual rating hidden; ranked tiers derived from authoritative stored values | Matchmaking uses provisional rating/RD; UI visibility changes exactly on result ten without rewriting history |
|
||||
| 8.23 `[D:8.21]` | Ranked-only 12-week exactly-once soft season: compress 25% toward 1500, RD >=200 capped 350, retain volatility/history; casual remains continuous | Retried/concurrent rollover applies once, never touches casual, and preserves every rating event |
|
||||
| 8.24 `[D:8.9,8.20,8.21]` | Ranked reconnect/abandon: match-scoped authorisation, 60 s reclaim, server-owned connection generations, then abandoner loss and rolling 7-day 5 m/15 m/1 h/24 h cooldown | Reconnect works through backend outage and fences old peer; grace has no penalty; expiry outcome/escalation is deterministic and auditable |
|
||||
| 8.25 `[D:8.10,8.24]` | Separate result delivery delay from match-integrity failure; signed Agones-annotation spool, retries, 5 m alert/30 m review, suppression only for lost/corrupt authority or measured unfair regional incident | API outage preserves rating/result; clients cannot request exemption; node/pod/integrity faults take the documented suppression/refund path |
|
||||
|
||||
#### 8d — Keeping Docker and CI green
|
||||
|
||||
`make verify-phase6` and `make verify-enet-integration` must not regress.
|
||||
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
|
||||
assignment, and uses `--max-matches=2` to prove arena rotation — all three are
|
||||
things allocation work would otherwise trample.
|
||||
#### 8D — Agones, allocation and regional scaling
|
||||
|
||||
| # | Task | Acceptance |
|
||||
|---|---|---|
|
||||
| 8.18 | **The rule: every allocation feature is opt-in via a `ServerConfig` flag whose default reproduces today's behaviour.** `ServerConfig` is built for exactly this — a flag declared once is parsed, validated, type-checked, config-file-backed and documented | `verify-phase6` and `verify-enet-integration` pass unchanged with no edits to their invocations |
|
||||
| 8.19 `[D:8.18]` | A **second** Compose file for the allocated-match path rather than mutating `compose.phase6-smoke.yml`, so the community-server model stays tested alongside the matchmade one | Both models have a green CI gate; neither shares a fixture with the other |
|
||||
| 8.20 `[D:8.18]` | `tests/cases/` coverage for the new flag parsing, per §10's no-live-server rule | New flags are unit-tested without a live server or a container |
|
||||
| 8.26 `[D:8.1,8.6,8.12]` | Portable Helm/Kustomize Fleets per build/EU/NA region; isolate provider edge/network/DNS/secret and SDR POP/cert/public-UDP overlays | Two provider fixtures render; labels select region/build/protocol/transport; each fixture documents Valve approval and externally reachable UDP mapping |
|
||||
| 8.27 `[D:8.26]` | Godot Agones REST adapter plus allocation-metadata watch and Go PID-1 supervisor scaffold; both bypass cloud behavior without SDK env; local SDK support | Native/existing Compose/CI remain functional; emulator exercises supervisor port discovery plus Get/Watch, Ready, Health, annotation and Shutdown |
|
||||
| 8.28 `[D:8.6,8.27]` | **Process-ready stage:** supervisor obtains dynamic port, launches Godot; static config/listen/Health succeed, then explicit Agones Ready—no roster/backend-registration prerequisite and no stdout probe | A detached unallocated process reaches Ready; a broken listener/config never does; Health reclaims a hung process |
|
||||
| 8.29 `[D:8.26,8.27]` | Separate ENet and Hosted-SDR dynamic/passthrough mappings; supervisor exports local `SDR_LISTEN_PORT` and external `SDR_IP`; validate POP/cert/firewall/NAT | Two isolated matches share a node; Agones-reported public endpoint receives relay traffic on the bound socket; ENet fixture remains independent |
|
||||
| 8.30 `[D:8.18,8.26,8.28,8.29]` | Atomic `GameServerAllocation` from Ready filtered by region/build/protocol/transport, attaching signed roster/non-secret config with bounded race retry | Duplicate commands yield one Allocated server; exhaustion or retry leaves no orphan; no client assignment is exposed merely because process is Ready |
|
||||
| 8.31 `[D:8.9,8.30]` | **Assignment-ready stage:** watch Allocated metadata, verify manifest/bindings, register hosted address, acknowledge backend; only then mint/expose client tickets | Modified/wrong manifest never reaches assignment-ready; clients cannot connect early; secrets never appear in metadata/args/logs |
|
||||
| 8.32 `[D:8.2,8.26,8.30]` | FleetAutoscaler with >=2 Ready processes across >=2 on-demand nodes/failure domains per queue-enabled region; pre-pull current/rollback; scale **Allocated** count to zero, never the Ready floor | Warm allocation meets p95 5 s/p99 10 s; disabled regions alone scale fully to zero; one-node loss retains certified Ready/headroom |
|
||||
| 8.33 `[D:8.26,8.32]` | On-demand-only live capacity and measured N+1: loss of largest node leaves two Ready slots plus headroom for surviving Allocated matches | Interruptible nodes cannot receive live matches; forced node loss neither overloads survivors nor prevents the next allocation |
|
||||
| 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog |
|
||||
| 8.35 `[D:8.17,8.19,8.20,8.30,8.31]` | Initial-connect no-show: 30 s after assignment-ready; ranked cancels/no-show cooldown, casual bot policy, empty allocation exits | No allocation idles indefinitely; innocent players regain original precedence; no pre-live failure changes rating |
|
||||
| 8.36 `[D:8.10,8.25,8.28,8.30]` | Go PID-1 supervisor traps TERM and authenticates localhost drain; 300 s grace/285 s infrastructure abort; PDB + Agones-aware Fleet drain; planned releases never TERM Allocated pods | Rollout/rollback waits Allocated=0; TERM path is exercised; forced timeout is classified/refunded; unexpected node loss is not claimed graceful |
|
||||
| 8.37 `[D:8.5,8.10,8.25,8.26,8.31]` | Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO <=5 m/RTO <=30 m; signed result annotation/reconciliation preserves delivery outages | Restore/failover meet targets; live simulation continues; valid delayed result commits exactly once after recovery |
|
||||
| 8.38 `[D:7.7,8.26,8.36,8.37]` | Provider migration after Valve approves both providers' EU/NA POPs/certs and public UDP: restore, validate coordinator trust, switch allocations, drain old | Both geographies complete Hosted-SDR matches on new provider and no old-provider live match is terminated |
|
||||
|
||||
> **The cold-start tension is real and is not solved by fast boot.** "Only pay
|
||||
> during a match" and "a player never waits" pull against each other: a server
|
||||
> must be listening *before* the matched players connect. 870 ms makes the gap
|
||||
> small, but the risk is not the container — image pull on a cold node,
|
||||
> scheduler placement and network/port programming can each dwarf it.
|
||||
> Recommendation: **match-level scale-to-zero over a small warm node pool**,
|
||||
> not node-level scale-to-zero. The per-match process genuinely exists only for
|
||||
> the match; the pool absorbs cold-start variance. Revisit only when measured
|
||||
> allocation latency on real infrastructure says the pool is unnecessary.
|
||||
#### 8E — Client experience and recovery
|
||||
|
||||
> **Server cost re-enters the design.** Community servers are paid for by
|
||||
> whoever hosts them; allocated servers are paid for by the project, per match.
|
||||
> `README.md`'s original note about a subscription to fund servers is suddenly
|
||||
> load-bearing. 8.12 needs to produce a number before launch, not after.
|
||||
| # | Task | Acceptance |
|
||||
|---|---|---|
|
||||
| 8.39 `[D:8.3,8.14,8.17]` | Queue UI: playlist/quality, elapsed and estimated wait, proposal countdown, allocation/connect state, cancel and latency/capacity explanations | Every backend state and terminal failure has a non-stuck visible state; cancel/decline is acknowledged authoritatively |
|
||||
| 8.40 `[D:8.3,8.14]` | One authenticated revisioned WebSocket plus REST resync; resume valid queue/assignment after client restart | Missed/duplicate/out-of-order events converge and restart never creates a second ticket |
|
||||
| 8.41 `[D:7.8,8.9,8.31,8.40]` | After assignment-ready, install SDR relay ticket before connect and send match-scoped join authorisation in `hello`; retain ENet assignments locally | Production connects/reconnects/fences old generation through SDR, never before assignment-ready; allocated/direct ENet and community flows remain compatible |
|
||||
| 8.42 `[D:8.22,8.23,8.24,8.40]` | Backend-authoritative provisional/rank/tier/delta, abandon status and season countdown UI | Client performs no rating math and displays the committed revision after reconnect |
|
||||
| 8.43 `[D:8.39,8.40,8.41]` | Recovery paths for decline, expiry, startup failure, version mismatch, auth expiry, regional outage and failed reconnect | Automated UI/state tests prove every case returns to a usable queue/menu or resumes the match without a duplicate action |
|
||||
|
||||
#### 8F — Observability, verification, cost and rollout
|
||||
|
||||
| # | Task | Acceptance |
|
||||
|---|---|---|
|
||||
| 8.44 `[D:8.3,8.4,8.28,8.31]` | Propagate queue/proposal/match/server IDs and process-ready/assignment-ready through logs, metrics, traces and replay metadata; redact credentials | One ID traces queue→result across components and automated secret-canary tests find no auth/relay ticket |
|
||||
| 8.45 `[D:8.2,8.44]` | Dashboards/alerts for wait/MMR/RTT, proposals, allocation/Ready/image pull, connect/no-show, tick/crash/flood, result conflict/lag, abandons and cost | Each SLO and security/cost signal has an exercised alert and runbook |
|
||||
| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | Go unit, race, fuzz, property, migration and concurrency suites | CI covers auth/join replay-reclaim, stale revisions, durable matcher fencing with lost Redis ack, rollover, result conflict/delivery retry and PostgreSQL retry |
|
||||
| 8.47 `[D:8.7,8.30]` | Fake Steam verifier and fake allocator for deterministic CI | Normal CI needs no Steam/cloud secret or internet access and can force every success/failure deterministically |
|
||||
| 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | Second Compose flow: fake backend → queue/proposal → process-ready/allocation/assignment-ready → ENet roster → result ack → shutdown; do not edit Phase 6 fixture | Both server models have independent green gates; existing Make invocations remain unchanged |
|
||||
| 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | Disposable `kind` + Agones integration gate | CI covers dynamic ports, both readiness stages, roster/no-show, races, multi-match node, result-pending reconciliation, drain and rollback |
|
||||
| 8.50 `[D:8.25,8.37,8.43,8.49]` | Network/chaos suite: 100 ms RTT, jitter/loss, client/API/matcher restart, game-pod death, node drain, Redis failover and control-plane loss | System recovers to a defined state; infrastructure-caused cases cannot penalise affected players |
|
||||
| 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | Load test >=10,000 queued clients, >=100 proposals/s and forecast launch concurrency x2 | API p95 <=250 ms, durable matcher fence holds, both readiness/allocation SLOs are met and replicas scale without duplicate claims |
|
||||
| 8.52 `[D:8.32,8.34,8.45,8.51]` | Per-region cost model from measured density, warm capacity, bandwidth, DB/Redis and telemetry; add budgets and allocation quotas | Cost per completed match and forecast monthly bands are recorded; a denial-of-wallet test triggers limits/alerts before budget breach |
|
||||
| 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | Progressive release: development → internal → casual canary → casual → provisional ranked → ranked | Each promotion requires SLO/security/cost gates, rollback rehearsal, EU+NA playtests and unchanged legacy gates; rollback criteria and owner are explicit |
|
||||
|
||||
Implementation invariants for every task above:
|
||||
|
||||
- Matchmade mode is opt-in; every new `ServerConfig` default preserves the
|
||||
existing community-server path.
|
||||
- `compose.phase6-smoke.yml`, `make verify-phase6`, and
|
||||
`make verify-enet-integration` are not repurposed or weakened.
|
||||
- Production uses ticketed Hosted Dedicated Server SDR; ENet remains the
|
||||
deterministic local/CI and direct-IP path.
|
||||
- One process serves one match. Warm processes/nodes absorb startup variance;
|
||||
capacity and cost are determined from 8.34 measurements, not old estimates.
|
||||
- Implementation evidence is appended under the completed task as in earlier
|
||||
phases; design changes first update `docs/MATCHMAKING.md` and dependencies.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user