CosmicClash/docs/MATCHMAKING.md

Matchmaking — casual and ranked queues

Architecture and locked product policy for Phase 8. This is a 1.0 launch blocker. The numbered, independently implementable tasks, their acceptance criteria, and current progress all live in multiplayer-next.md.

Nothing in Phase 8 is implemented yet. This document records the decisions those tasks assume so an implementer does not have to redesign the system while building one part of it.

1. Model and non-negotiable constraints

The existing multiplayer is a community-server model: a dedicated server runs continuously, waits for players, rotates arenas, and can be reached by direct IP or the Phase 7 Steam browser. Matchmaking adds a second model: players queue, every selected human accepts a proposal, one server process is allocated for that match, and the process is destroyed after its result is durably recorded. Ranked selects six humans; relaxed casual selects two to six and discloses its bot-filled team composition before acceptance. Both models ship.

Locked constraints:

• One authoritative Godot process hosts exactly one match.
• Solo 3v3 casual and ranked queues launch first. Party-shaped fields are reserved in contracts, but party formation is deferred.
• Production matchmade traffic uses ticketed Steam Hosted Dedicated Server SDR. Direct ENet remains first-class for local development, CI, LAN, self-hosting, and community servers.
• The control plane is Go, PostgreSQL, and Redis, deployed on Kubernetes. Agones owns game-server allocation and lifecycle. Go is chosen for the Agones/Kubernetes-native client ecosystem and its concurrency model, not for raw speed — the control plane never touches a simulation packet. See "Matchmaking control plane" in TECH_STACK.md for the full rationale and the alternatives weighed.
• Infrastructure is provider-portable. Provider-specific cluster, network, DNS, and secret-store configuration lives behind isolated deployment overlays; application code never calls a provider allocation API.
• Launch game fleets run in Europe and North America. Placement is latency-first and never silently violates the ping ceiling to shorten a queue.
• Phase 8 features are opt-in. With allocated mode disabled, ServerConfig, ENet, Docker Compose, and the existing community-server behavior remain unchanged.

2. Steam and trust boundaries

Phase 7 verified Steam identity is a hard prerequisite. The current slot reservation is keyed by display name, so no public queue or rating may rely on it.

Two Steam credentials have different purposes and must not be conflated:

1. A client obtains a single-use Web API ticket for backend login. Only the secure backend calls AuthenticateUserTicket, checks the expected App ID and identity string, and turns the result into a revocable session. A client-supplied SteamID is never identity.
2. For a formed match, the game coordinator creates a short-lived SDR relay ticket authorising one player to one hosted server. The client installs it before connecting. The server separately verifies the signed match join authorisation before admitting the player to the assigned roster slot.

Steam authentication/session tickets are single-use and their lifecycle must include the appropriate cancel/end calls; identity is not valid until Steam's asynchronous validation succeeds. Hosted SDR relay tickets are different: they are short-lived, match/server/identity-scoped and deliberately cached for reconnect. See Steam authentication and the ISteamUserAuth Web API.

Ticketed Hosted Dedicated Server SDR is the production transport because it hides player/server IP addresses and authenticates, encrypts, and rate-limits traffic. It also supplies relay routing that may improve the path. It requires a real App ID, coordinator SDK/signing approval, certificates, and hosted data-centre coordination with Valve; those are explicit release dependencies, not assumptions. See Steam Datagram Relay.

Trust table

Input	Trusted only after	May affect
Steam Web API ticket	Backend validation for the expected App ID/identity and replay check	Backend session identity
Steam ping location + active-probe evidence	Backend verifies nonce/freshness and computes estimates; later compares with observed RTT	Placement only, never results
Queue/accept request	Auth, schema/rate-limit, revision and idempotency validation	That player's queue state
Join authorisation	Server signature, expiry, match/server/SteamID/slot and connection-generation validation	Initial admission or idempotent reclaim of that same slot
Gameplay input	Existing server framing, sequence, byte and rate validation	Authoritative simulation input only
Match result	Assigned server workload identity plus match/server binding	Transactional result/rating commit

Clients never submit ratings, outcome, penalty exemptions, server health, or allocation state. A dedicated server never holds the Steam publisher key or the coordinator root signing key. Keep the offline SDR CA separate from the online leaf ticket key. The online key is exposed only through a narrowly authorised signer backed by KMS/HSM-equivalent non-exportable storage; API, matcher, allocator and game-server pods cannot read it. The signer accepts only allocator-recorded assignments, audits every signature, and supports overlapping-key rotation.

3. Control-plane architecture

Use one repository and shared domain packages, with independently runnable roles rather than independently designed microservices:

Role	Responsibility
API	HTTPS/WebSocket auth, profile, queue commands, status resync, transactional-outbox fan-out
Matcher	Atomic proposal formation from queue state
Allocator	Agones allocation, server registration, assignment delivery
Maintenance worker	Season rollover, initial-connect/no-show expiry, live reconnect-abandonment reconciliation, and other durable lifecycle recovery

API replicas are stateless. Redis sorted sets provide the fast candidate index, but Redis is never the durable allocation fence: asynchronous failover can lose an acknowledged write. A matcher claims a proposal in a PostgreSQL SERIALIZABLE transaction using a partial unique constraint that permits only one active proposal/match participation per player. The transaction records the proposal and participants before Redis cleanup; a stale Redis claim then loses at PostgreSQL and is repaired from the durable record. PostgreSQL is the source of truth for queue ownership, identities, sessions/revocations, seasons, ratings, matches, participants, penalties, result receipts, audits and the transactional outbox. Redis holds expiring presence, candidate indexes, latency evidence, session/revocation caches and transient fan-out. Losing the last acknowledged Redis write may delay/rematerialise a ticket or force a session cache miss, but can neither resurrect a revoked session, split a proposal nor corrupt a result/rating.

For launch, run the horizontally scaled control plane in one primary Kubernetes region with a warm standby and tested restore path. Game fleets remain regional in EU and North America. This avoids a premature multi-writer database while keeping new-match control latency small relative to queue time. An outage may pause new queues/allocations, but live matches must continue. Targets are PostgreSQL RPO <= 5 minutes and control-plane RTO <= 30 minutes.

Stable identifiers and state

Contracts define opaque player_id, queue_ticket_id, proposal_id, match_id, server_id, and season_id. Every mutating request has an idempotency key, expected revision, and versioned schema.

The durable/transient state path is:

QUEUED -> PROPOSED -> ACCEPTED -> ALLOCATING -> PROCESS_READY
       -> ASSIGNMENT_READY -> ASSIGNED -> CONNECTING -> LIVE
       -> RESULT_PENDING -> COMPLETED
       -> CANCELLED / EXPIRED / FAILED from the explicitly legal stages

The API publishes revisioned changes over one authenticated WebSocket. REST GET endpoints are the recovery source after a disconnect or missed revision. Restarting the client resumes an unexpired ticket/assignment instead of creating another one.

Assignments include match/server IDs, protocol and client build, server image digest, playlist version, transport, region, expiry, a match-scoped join authorisation, and either the SDR hosted-server material or an ENet endpoint. The authorisation may be replayed only by the same Steam identity to reclaim the same match/server/slot before expiry. Each successful connection advances a server-owned generation and fences the prior connection; another identity, server or slot is always rejected. This deliberately supports reconnect when Steam or the control plane is temporarily unavailable. Incompatible protocol/build/playlist versions never enter one proposal.

4. Queue and placement policy

Each verified player may own at most one active queue ticket. Heartbeats are sent every 10 seconds and queue presence expires after 30 seconds. Create, cancel, resume, accept, decline, and expiry are atomic and retry-safe.

The client submits its recent opaque Steam ping location plus nonce-bound active-probe responses from each regional endpoint; it does not submit the RTT used for placement. The backend validates a 30-second freshness window and nonce, then uses the Steam coordinator SDK and probe timings to compute the regional matrix. A predicted/observed discrepancy over 25 ms or 30% (whichever is larger) in three matches within 24 hours quarantines the account's samples: it may queue only in regions whose active probe independently remains under the ceiling until five clean matches clear the quarantine. The matchmaker:

1. Finds regions in which every proposed player has predicted RTT <= 100 ms.
2. Minimises the worst player's predicted RTT.
3. Breaks ties by total predicted RTT, then ready server capacity.
4. Widens rating tolerance with wait time, but never automatically widens the 100 ms latency ceiling.

The target is regional observed p95 RTT <= 80 ms. Candidate formation is deterministic:

• Anchor on the oldest compatible ticket (enqueued_at, then ticket ID).
• A ticket's rating tolerance is min(400, 100 + 25 * floor(wait_seconds/30)) rating points. A pair is compatible only when its absolute rating difference is within both tickets' tolerance. Provisional players use the same stored rating; their high RD changes ratings faster, not eligibility arithmetic.
• Every candidate set contains the anchor. Choose the lexicographically smallest tuple (worst_RTT, total_RTT, rating_range, -sum(wait_seconds), sorted_ticket_IDs), so latency dominates once the oldest player anchors fairness and every candidate satisfies the widening rule.
• Partition humans exhaustively into legal teams and minimise absolute team mean-rating difference, then maximum opposing-player difference, then use lexical player IDs. Bots fill remaining casual slots after humans are assigned.

Every selected human receives a 10-second proposal before allocation. Ranked always selects six. Casual requires six before the anchor reaches 60 seconds; afterward it selects the largest compatible human count from six down to two, with at least one human per team, and displays teams/bot slots. Unanimous acceptance by the selected humans advances.

An explicit proposal decline cancels that player's ticket and applies a 30-second casual or 2-minute ranked cooldown. A timeout applies 60 seconds in casual or 5 minutes in ranked. Three ranked proposal declines/timeouts within 30 minutes apply 15 minutes. Players who accepted return with their original enqueued_at and precedence when another player declines, times out, or the allocation fails. Ordering ties use ticket ID. A ranked initial-connect no-show after accepting uses the ranked abandon cooldown ladder but never a rating loss because no rated match began.

The initial-connect clock begins only after the server's durable ASSIGNMENT_READY transition. Player assignments are not exposed before that gate. Each allocated server reports a successfully verified signed-roster admission through the workload-authenticated, idempotent POST /servers/{serverId}/connect boundary; PostgreSQL connected_at values, not client claims, drive no-show reconciliation. The supervisor arms the game process's matching local timeout through an authenticated loopback control call only after the same transition commits.

Casual

• Target six humans in 3v3.
• After 60 seconds, a match may start with at least two humans, one on each team, and fill the other slots with server bots.
• Human backfill may replace a bot/disconnected slot only at a kickoff boundary.
• Backfill is a separate 10-second opt-in proposal showing score, time remaining, team and slot. Declining/timing out carries no cooldown. A backfilled player receives no hidden-rating update or abandon penalty for that match; acceptance removes their queue ticket only when assignment is ready. Choose the oldest ordinary casual ticket that meets the same build, region <=100 ms and current anchor-tolerance rules for the vacated human slot; ties use ticket ID.
• An original casual participant gets 30 seconds to reconnect; leaving after that applies a 60-second queue cooldown. The match's ordinary hidden-rating result still applies, with no extra rating penalty.
• An accepted casual initial-connect no-show gets the same 60-second cooldown. The match proceeds with a bot only if at least one human connected on each team; otherwise it cancels and restores every innocent ticket with original precedence.
• Casual has a separate hidden Glicko-2 rating used only for matching.

Ranked

• Exactly six verified humans; never bots and never backfill.
• Solo queue only at launch.
• The matcher validates ranked admission against its server-owned allowlist of the three floor-goal ArenaRegistry entries. Elevated goals remain excluded until the trained-policy restriction is lifted. It chooses from that list deterministically from the proposal ID, so retrying a proposal cannot change its arena. Every durable proposal, match, and allocation boundary rechecks the same allowlist rather than accepting an arbitrary non-empty path; the PostgreSQL constraints enforce it for new direct SQL writes as well. The selected scene is persisted with the proposal/match plan, included in the allocation identity, and passed through the Agones GameServer annotation into the allocated server's validated --arena-path flag.
• A reconnecting player has 60 seconds to return using the existing assignment. After that, that player receives a loss regardless of the final team result and a rolling seven-day cooldown: 5 minutes, 15 minutes, 1 hour, then 24 hours.
• Delivery failure is not match-integrity failure. A healthy completed match remains rated while its result waits for the control plane. Rating is suppressed only when the authoritative roster, simulation or result is unavailable/corrupt, or a measured regional incident prevented fair play.

5. Rating and seasons

Use canonical Glicko-2 independently per playlist with initial rating 1500, RD 350, volatility 0.06, scale constant 173.7178 and tau 0.5. Updates are immediate per committed match rather than globally batched. Before an update, advance inactivity by whole 24-hour rating periods since the player's last rated match using phi = min(350/173.7178, sqrt(phi^2 + sigma^2 * periods)).

For each player i, transform every opposing human's pre-match rating/RD to mu_j/phi_j and use the canonical equations g(phi)=1/sqrt(1+3*phi^2/pi^2) and E=1/(1+exp(-g(phi_j)*(mu_i-mu_j))). Ranked's three opponent contributions use w=1/3; casual uses w=1/N for the N opposing humans, ignoring bots. Thus human contributions total exactly one match in both sums:

v^-1 = sum(w * g(phi_j)^2 * E_j * (1-E_j))
Delta = v * sum(w * g(phi_j) * (s-E_j))

Then run the canonical Glicko-2 volatility iteration and rating/RD update. If a player has no opposing human, the match is unrated for that player. s is 1/0 for the authoritative winner/loser and 0.5 only for a completed draw. Overtime is an ordinary win/loss. A ranked abandoner gets s=0 regardless of the final team result; non-abandoning players use the authoritative final result. Cancelled or integrity-failed matches do not update anyone.

Lock all six participant rows in lexical player-ID order and compute every new value from the same immutable pre-match snapshot inside one serializable transaction. This prevents update-order bias and concurrent double updates. Golden vectors include canonical one-player examples plus symmetric/asymmetric 3v3, draw, overtime, abandon, inactivity and concurrent-result fixtures.

The first ten ranked matches are provisional. Ranked exposes backend-derived tiers; casual rating remains hidden. Stored Glicko values, not tier labels or client calculations, are authoritative. Ranked seasons last 12 weeks. Ranked rollover sets rating = 1500 + 0.75 * (rating - 1500), raises RD to at least 200 (capped at 350), preserves volatility/history, and is an exactly-once idempotent transaction. Casual rating is continuous and never season-reset.

Rating, penalty, match completion, participant records and outbox events are committed in that transaction. Duplicate identical server results succeed idempotently. A conflicting result changes nothing and pages an operator.

6. Game-server allocation and lifecycle

Use provider-portable Kubernetes manifests and Agones. Create one versioned Fleet per compatible build and region; labels identify region, protocol, transport, and image/build. Allocate atomically with GameServerAllocation. Agones, rather than bespoke allocator code, owns selection and lifecycle. See GameServerAllocation.

The Godot process talks to the Agones REST sidecar through a small adapter that is a no-op when AGONES_SDK_HTTP_PORT is absent. This preserves native, Compose, and CI operation and works with the Agones local SDK emulator. See Agones client SDKs.

Agones has two distinct readiness points; conflating them is a deadlock because GameServerAllocation selects a Ready server and only then attaches the match metadata:

1. The PID-1 supervisor queries the local Agones SDK for the assigned dynamic port/address, exports SDR_LISTEN_PORT plus SDR_IP=public-address:port (or the ENet equivalent), and launches Godot. Godot validates static config, binds the socket and starts Health calls.
2. Godot calls Agones Ready() after the process is genuinely listening. This is process-ready only; never infer it from detached stdout.
3. GameServerAllocation atomically changes that Ready server to Allocated and supplies the signed roster/non-secret match configuration as metadata.
4. The server watches the GameServer, observes Allocated metadata, verifies manifest signature/build/protocol/server binding, registers its hosted SDR address, and calls the backend assignment_ready endpoint.
5. Only after assignment_ready does the allocator mint relay/join tickets and expose the assignment to clients. The server accepts only assigned identities/slots; each selected human has 30 seconds to connect.
6. Run one authoritative match with Health calls independent of the simulation loop. Submit the canonical result using the bound workload identity.
7. Write the signed result hash/payload to the backend and to a non-secret Agones annotation, remain Allocated, and retry until the backend durably acknowledges it. The maintenance worker reconciles the annotation after an API outage. RESULT_PENDING pages at 5 minutes and requires operator review at 30 minutes; it never silently becomes unrated.
8. After acknowledgement call Shutdown() and exit. Invalid/empty allocations shut down immediately. An allocated GameServer is not recycled to Ready.

Agones supplies dynamic host ports so multiple isolated matches can share a node; HTTP ingress is not involved in gameplay routing. Hosted SDR additionally requires Valve approval for every provider/location, a valid SDR_POPID, public-IP/unsolicited-UDP reachability, provider firewall/NAT validation, per-location certificates and coordinator trust. Use an Agones dynamic or passthrough mapping whose externally reported port is the SDR_IP port while the process binds SDR_LISTEN_PORT; test SDR and ENet mappings separately. The control plane's HMAC signing secret arrives through a runtime Secret mount; it never reaches the game pod, command line, logs, or image. For each allocation, the allocator signs a short-lived bearer token containing only the allocation ID and requests Agones to attach it to the selected GameServer's metadata. The allocated pod's local SDK sidecar is the delivery boundary: the supervisor reads that annotation and supplies it only as a child-process environment variable. The backend verifies the HMAC and expiry, then resolves the allocation ID to the durable allocation/match/server tuple in PostgreSQL; the game server cannot choose that binding. A future projected-service-account attestation may replace this delivery mechanism, but it is not a current security claim.

Warm capacity and density

Both launch regions are active whenever their queues are enabled. Each active region maintains at least two Ready processes distributed across at least two on-demand nodes/failure domains; the minimum node floor is therefore two, not one. Pre-pull current and rollback images. A queue/proposal-aware FleetAutoscaler adjusts capacity above that Ready floor. Allocated process count may fall to zero; Ready processes do not. An administratively disabled region may scale both nodes and Fleet to zero and is excluded from placement. See Agones FleetAutoscaler.

Do not run live matches on interruptible nodes. N+1 means loss of the largest single node still leaves two Ready slots plus sufficient headroom for already Allocated matches; certify it in the node-loss test. Set pod requests, limits and node caps only after native x86_64 benchmarks of boot time, p99 CPU/RSS/network and 60 Hz tick behavior, retaining 30% headroom. The current 6–10 processes/core and 150–250 MB estimates are not sizing data.

Godot/GDScript cannot intercept SIGTERM, so Phase 8 adds a small Go PID-1 supervisor. It starts Godot, traps TERM, and sends a per-pod-token-authenticated localhost drain request to an allocated-mode-only Godot control socket. Godot refuses new admissions and reports completion to the supervisor. Kubernetes uses a 300-second termination grace period; at 285 seconds the supervisor forces an infrastructure-failure abort so the pod cannot hang forever in overtime. A PodDisruptionBudget and Agones-aware drain prevent voluntary eviction of Allocated servers. Planned releases create a new Fleet, route new allocations to it and wait for old Allocated count zero without sending TERM. Unexpected node failure cannot be made graceful and follows the integrity failure/refund path.

Live matches continue through an API/control-plane delivery outage. A valid result is retried/reconciled as above; only loss of authoritative match integrity suppresses rating.

7. Security and operational baseline

The threat model must cover forged clients, ticket replay, duplicate queueing, result forgery, queue manipulation, packet floods, botting, server compromise, insider access, DDoS, vulnerable dependencies, and denial-of-wallet attacks.

Minimum controls:

• Non-root containers, read-only root filesystems, dropped Linux capabilities, RuntimeDefault seccomp, resource limits, restricted Pod Security admission, and least-privilege service accounts/RBAC.

• Private PostgreSQL/Redis; default-deny ingress and egress with explicit NetworkPolicy allowances. See the Kubernetes application security checklist and NetworkPolicy.

• Encrypted databases/backups, externally populated Kubernetes Secrets, documented rotation, and no credentials in Git, images, command lines, or telemetry.

• Per-account/IP API limits, request/body/schema limits, replay and duplicate detection, generic public errors, allocation quotas, and budget alerts.

• Put HTTPS/WebSocket traffic behind a provider-portable edge contract implemented by each infrastructure overlay: managed volumetric DDoS absorption, WAF/rate rules, TLS termination, origin-only ingress and health checks. Enforce authenticated queue admission, per-account connection caps, WebSocket handshake/message/idle limits, bounded fan-out and overload shedding. In degraded mode reject new login/queue/allocation work while preserving result ingestion and all live matches.

  The control-plane implements the admission portion of this policy with --degraded at startup, SIGUSR1 to enable it, and SIGUSR2 to disable it. The gate rejects new login, queue, and proposal mutations with 503 service_degraded; assignment reads, events, server registration/results, health, metrics, and other live-match paths remain available.

• Images pinned by digest, SBOM generation, dependency/image scanning, signed releases, admission-time signature verification, and a critical-patch SLA.

• Structured audit events for auth, queue transitions, allocation, roster rejection, result conflict, penalty, season rollover, and operator action.

8. SLOs, observability, and release gates

The operational definitions, owners, windows and alert thresholds for these targets are maintained in MATCHMAKING-SLOs.md.

Launch SLOs:

Measure	Target
Eligible predicted RTT	<= 100 ms for every player
Regional observed RTT	p95 <= 80 ms
Unanimous acceptance to assignment_ready	p95 <= 5 s, p99 <= 10 s with warm capacity
Published assignment to successful connection	p95 <= 5 s
Successful allocation and durable result	>= 99.9%
Certified server density	No tick backlog, with 30% resource headroom
Control-plane API under load	p95 <= 250 ms

Dashboards and alerts cover queue depth/wait, rating spread, predicted versus observed RTT, proposals/declines, allocation latency/failure, Ready capacity, image pulls, connection/no-show, physics overruns, crashes, abnormal packet rates, result lag/conflicts, abandons, and cost per completed match. Correlate all components with queue/proposal/match/server IDs, but never log auth or relay tickets.

Testing layers:

• Go unit, race, fuzz, property, migration, and concurrency tests.
• Fake Steam verifier and fake allocator for deterministic CI.
• A second allocated-server Compose flow; never mutate compose.phase6-smoke.yml.
• Disposable kind + Agones integration for readiness, dynamic ports, health/no-show shutdown, allocation races, multiple matches per node, draining, and rollback.
• Network/chaos cases for 100 ms RTT, jitter/loss, client/backend/matcher restart, game-pod death, node drain, Redis failover, and control-plane loss.
• Load test at least 10,000 queued clients, 100 proposals/second, and forecast launch concurrency x2 while preserving correctness and API latency.
• Provider migration rehearsal: restore data and deploy both regional fleets on a second provider whose EU/NA locations already have Valve approval, POP/certificates, public UDP/firewall verification and coordinator trust; switch new allocations, drain the old provider, and terminate no live match.

Release order is development -> internal -> casual canary -> full casual -> provisional ranked -> full ranked. Each promotion requires its SLO/security gates, rollback rehearsal, EU and North America playtests, a measured cost model, and unchanged make verify-phase6 and make verify-enet-integration.

9. Explicitly out of scope

Parties/premades, tournaments, ranked spectators, public global regions, non-Steam identity providers, and a global leaderboard beyond personal rank display are not launch scope. The contracts reserve party identity and avoid Steam-specific database primary keys so those additions do not require a destructive redesign.