CosmicClash/docs/MATCHMAKING.md

# Matchmaking — casual and ranked queues

Architecture and locked product policy for Phase 8. This is a **1.0 launch
blocker**. The numbered, independently implementable tasks, their acceptance
criteria, and current progress all live in
[`multiplayer-next.md`](../multiplayer-next.md).

Nothing in Phase 8 is implemented yet. This document records the decisions
those tasks assume so an implementer does not have to redesign the system
while building one part of it.

## 1. Model and non-negotiable constraints

The existing multiplayer is a community-server model: a dedicated server
runs continuously, waits for players, rotates arenas, and can be reached by
direct IP or the Phase 7 Steam browser. Matchmaking adds a second model:
players queue, every selected human accepts a proposal, one server process is
allocated for that match, and the process is destroyed after its result is
durably recorded. Ranked selects six humans; relaxed casual selects two to six
and discloses its bot-filled team composition before acceptance. Both models
ship.

Locked constraints:

- One authoritative Godot process hosts exactly one match.
- Solo 3v3 casual and ranked queues launch first. Party-shaped fields are
  reserved in contracts, but party formation is deferred.
- Production matchmade traffic uses ticketed Steam Hosted Dedicated Server
  SDR. Direct ENet remains first-class for local development, CI, LAN,
  self-hosting, and community servers.
- The control plane is Go, PostgreSQL, and Redis, deployed on Kubernetes.
  Agones owns game-server allocation and lifecycle.
- Infrastructure is provider-portable. Provider-specific cluster, network,
  DNS, and secret-store configuration lives behind isolated deployment
  overlays; application code never calls a provider allocation API.
- Launch game fleets run in Europe and North America. Placement is
  latency-first and never silently violates the ping ceiling to shorten a
  queue.
- Phase 8 features are opt-in. With allocated mode disabled, `ServerConfig`,
  ENet, Docker Compose, and the existing community-server behavior remain
  unchanged.

## 2. Steam and trust boundaries

Phase 7 verified Steam identity is a hard prerequisite. The current slot
reservation is keyed by display name, so no public queue or rating may rely on
it.

Two Steam credentials have different purposes and must not be conflated:

1. A client obtains a single-use Web API ticket for backend login. Only the
   secure backend calls `AuthenticateUserTicket`, checks the expected App ID
   and identity string, and turns the result into a revocable session. A
   client-supplied SteamID is never identity.
2. For a formed match, the game coordinator creates a short-lived SDR relay
   ticket authorising one player to one hosted server. The client installs it
   before connecting. The server separately verifies the signed match join
   authorisation before admitting the player to the assigned roster slot.

Steam authentication/session tickets are single-use and their lifecycle must
include the appropriate cancel/end calls; identity is not valid until Steam's
asynchronous validation succeeds. Hosted SDR relay tickets are different:
they are short-lived, match/server/identity-scoped and deliberately cached for
reconnect. See [Steam authentication](https://partner.steamgames.com/doc/features/auth)
and the [`ISteamUserAuth` Web API](https://partner.steamgames.com/doc/webapi/isteamuserauth).

Ticketed Hosted Dedicated Server SDR is the production transport because it
hides player/server IP addresses and authenticates, encrypts, and rate-limits
traffic. It also supplies relay routing that may improve the path. It requires
a real App ID, coordinator SDK/signing approval, certificates, and hosted
data-centre coordination with Valve; those are explicit release dependencies,
not assumptions. See [Steam Datagram Relay](https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay).

### Trust table

| Input | Trusted only after | May affect |
| --- | --- | --- |
| Steam Web API ticket | Backend validation for the expected App ID/identity and replay check | Backend session identity |
| Steam ping location + active-probe evidence | Backend verifies nonce/freshness and computes estimates; later compares with observed RTT | Placement only, never results |
| Queue/accept request | Auth, schema/rate-limit, revision and idempotency validation | That player's queue state |
| Join authorisation | Server signature, expiry, match/server/SteamID/slot and connection-generation validation | Initial admission or idempotent reclaim of that same slot |
| Gameplay input | Existing server framing, sequence, byte and rate validation | Authoritative simulation input only |
| Match result | Assigned server workload identity plus match/server binding | Transactional result/rating commit |

Clients never submit ratings, outcome, penalty exemptions, server health, or
allocation state. A dedicated server never holds the Steam publisher key or
the coordinator root signing key. Keep the offline SDR CA separate from the
online leaf ticket key. The online key is exposed only through a narrowly
authorised signer backed by KMS/HSM-equivalent non-exportable storage; API,
matcher, allocator and game-server pods cannot read it. The signer accepts
only allocator-recorded assignments, audits every signature, and supports
overlapping-key rotation.

## 3. Control-plane architecture

Use one repository and shared domain packages, with independently runnable
roles rather than independently designed microservices:

| Role | Responsibility |
| --- | --- |
| API | HTTPS/WebSocket auth, profile, queue commands, status resync |
| Matcher | Atomic proposal formation from queue state |
| Allocator | Agones allocation, server registration, assignment delivery |
| Maintenance worker | Outbox delivery, season rollover, expiry, reconciliation |

API replicas are stateless. Redis sorted sets provide the fast candidate
index, but Redis is never the durable allocation fence: asynchronous failover
can lose an acknowledged write. A matcher claims a proposal in a PostgreSQL
`SERIALIZABLE` transaction using a partial unique constraint that permits only
one active proposal/match participation per player. The transaction records
the proposal and participants before Redis cleanup; a stale Redis claim then
loses at PostgreSQL and is repaired from the durable record. PostgreSQL is the
source of truth for queue ownership, identities, sessions/revocations,
seasons, ratings, matches, participants, penalties, result receipts, audits
and the transactional outbox. Redis holds expiring presence, candidate
indexes, latency evidence, session/revocation caches and transient fan-out.
Losing the last acknowledged Redis write may delay/rematerialise a ticket or
force a session cache miss, but can neither resurrect a revoked session, split
a proposal nor corrupt a result/rating.

For launch, run the horizontally scaled control plane in one primary
Kubernetes region with a warm standby and tested restore path. Game fleets
remain regional in EU and North America. This avoids a premature multi-writer
database while keeping new-match control latency small relative to queue time.
An outage may pause new queues/allocations, but live matches must continue.
Targets are PostgreSQL RPO <= 5 minutes and control-plane RTO <= 30 minutes.

### Stable identifiers and state

Contracts define opaque `player_id`, `queue_ticket_id`, `proposal_id`,
`match_id`, `server_id`, and `season_id`. Every mutating request has an
idempotency key, expected revision, and versioned schema.

The durable/transient state path is:

```
QUEUED -> PROPOSED -> ACCEPTED -> ALLOCATING -> PROCESS_READY
       -> ASSIGNMENT_READY -> ASSIGNED -> CONNECTING -> LIVE
       -> RESULT_PENDING -> COMPLETED
       -> CANCELLED / EXPIRED / FAILED from the explicitly legal stages
```

The API publishes revisioned changes over one authenticated WebSocket. REST
GET endpoints are the recovery source after a disconnect or missed revision.
Restarting the client resumes an unexpired ticket/assignment instead of
creating another one.

Assignments include match/server IDs, protocol and client build, server image
digest, playlist version, transport, region, expiry, a match-scoped join
authorisation, and either the SDR hosted-server material or an ENet endpoint.
The authorisation may be replayed only by the same Steam identity to reclaim
the same match/server/slot before expiry. Each successful connection advances
a server-owned generation and fences the prior connection; another identity,
server or slot is always rejected. This deliberately supports reconnect when
Steam or the control plane is temporarily unavailable. Incompatible
protocol/build/playlist versions never enter one proposal.

## 4. Queue and placement policy

Each verified player may own at most one active queue ticket. Heartbeats are
sent every 10 seconds and queue presence expires after 30 seconds. Create,
cancel, resume, accept, decline, and expiry are atomic and retry-safe.

The client submits its recent opaque Steam ping location plus nonce-bound
active-probe responses from each regional endpoint; it does not submit the RTT
used for placement. The backend validates a 30-second freshness window and
nonce, then uses the Steam coordinator SDK and probe timings to compute the
regional matrix. A predicted/observed discrepancy over 25 ms or 30% (whichever
is larger) in three matches within 24 hours quarantines the account's samples:
it may queue only in regions whose active probe independently remains under
the ceiling until five clean matches clear the quarantine. The matchmaker:

1. Finds regions in which every proposed player has predicted RTT <= 100 ms.
2. Minimises the worst player's predicted RTT.
3. Breaks ties by total predicted RTT, then ready server capacity.
4. Widens rating tolerance with wait time, but never automatically widens the
   100 ms latency ceiling.

The target is regional observed p95 RTT <= 80 ms. Candidate formation is
deterministic:

- Anchor on the oldest compatible ticket (`enqueued_at`, then ticket ID).
- A ticket's rating tolerance is `min(400, 100 + 25 * floor(wait_seconds/30))`
  rating points. A pair is compatible only when its absolute rating difference
  is within both tickets' tolerance. Provisional players use the same stored
  rating; their high RD changes ratings faster, not eligibility arithmetic.
- Every candidate set contains the anchor. Choose the lexicographically
  smallest tuple `(worst_RTT, total_RTT, rating_range, -sum(wait_seconds),
  sorted_ticket_IDs)`, so latency dominates once the oldest player anchors
  fairness and every candidate satisfies the widening rule.
- Partition humans exhaustively into legal teams and minimise absolute team
  mean-rating difference, then maximum opposing-player difference, then use
  lexical player IDs. Bots fill remaining casual slots after humans are
  assigned.

Every selected human receives a 10-second proposal before allocation. Ranked
always selects six. Casual requires six before the anchor reaches 60 seconds;
afterward it selects the largest compatible human count from six down to two,
with at least one human per team, and displays teams/bot slots. Unanimous
acceptance by the selected humans advances.

An explicit proposal decline cancels that player's ticket and applies a
30-second casual or 2-minute ranked cooldown. A timeout applies 60 seconds in
casual or 5 minutes in ranked. Three ranked proposal declines/timeouts within
30 minutes apply 15 minutes. Players who accepted return with their original
`enqueued_at` and precedence when another player declines, times out, or the
allocation fails. Ordering ties use ticket ID. A ranked initial-connect
no-show after accepting uses the ranked abandon cooldown ladder but never a
rating loss because no rated match began.

### Casual

- Target six humans in 3v3.
- After 60 seconds, a match may start with at least two humans, one on each
  team, and fill the other slots with server bots.
- Human backfill may replace a bot/disconnected slot only at a kickoff
  boundary.
- Backfill is a separate 10-second opt-in proposal showing score, time
  remaining, team and slot. Declining/timing out carries no cooldown. A
  backfilled player receives no hidden-rating update or abandon penalty for
  that match; acceptance removes their queue ticket only when assignment is
  ready. Choose the oldest ordinary casual ticket that meets the same build,
  region <=100 ms and current anchor-tolerance rules for the vacated human
  slot; ties use ticket ID.
- An original casual participant gets 30 seconds to reconnect; leaving after
  that applies a 60-second queue cooldown. The match's ordinary hidden-rating
  result still applies, with no extra rating penalty.
- An accepted casual initial-connect no-show gets the same 60-second cooldown.
  The match proceeds with a bot only if at least one human connected on each
  team; otherwise it cancels and restores every innocent ticket with original
  precedence.
- Casual has a separate hidden Glicko-2 rating used only for matching.

### Ranked

- Exactly six verified humans; never bots and never backfill.
- Solo queue only at launch.
- The matcher validates ranked admission against its server-owned allowlist of
  the three floor-goal `ArenaRegistry` entries. Elevated goals remain excluded
  until the trained-policy restriction is lifted. It chooses from that list
  deterministically from the proposal ID, so retrying a proposal cannot change
  its arena. Every durable proposal, match, and allocation boundary rechecks
  the same allowlist rather than accepting an arbitrary non-empty path; the
  PostgreSQL constraints enforce it for new direct SQL writes as well. The selected scene is
  persisted with the proposal/match plan, included in the allocation identity,
  and passed through the Agones GameServer annotation into the allocated
  server's validated `--arena-path` flag.
- A reconnecting player has 60 seconds to return using the existing
  assignment. After that, that player receives a loss regardless of the final
  team result and a rolling seven-day cooldown: 5 minutes, 15 minutes, 1
  hour, then 24 hours.
- Delivery failure is not match-integrity failure. A healthy completed match
  remains rated while its result waits for the control plane. Rating is
  suppressed only when the authoritative roster, simulation or result is
  unavailable/corrupt, or a measured regional incident prevented fair play.

## 5. Rating and seasons

Use canonical Glicko-2 independently per playlist with initial rating 1500,
RD 350, volatility 0.06, scale constant 173.7178 and tau 0.5. Updates are
immediate per committed match rather than globally batched. Before an update,
advance inactivity by whole 24-hour rating periods since the player's last
rated match using `phi = min(350/173.7178, sqrt(phi^2 + sigma^2 * periods))`.

For each player `i`, transform every opposing human's pre-match rating/RD to
`mu_j`/`phi_j` and use the canonical equations
`g(phi)=1/sqrt(1+3*phi^2/pi^2)` and
`E=1/(1+exp(-g(phi_j)*(mu_i-mu_j)))`. Ranked's three opponent contributions
use `w=1/3`; casual uses `w=1/N` for the `N` opposing humans, ignoring bots.
Thus human contributions total exactly one match in both sums:

```
v^-1 = sum(w * g(phi_j)^2 * E_j * (1-E_j))
Delta = v * sum(w * g(phi_j) * (s-E_j))
```

Then run the canonical Glicko-2 volatility iteration and rating/RD update. If
a player has no opposing human, the match is unrated for that player. `s` is 1/0 for
the authoritative winner/loser and 0.5 only for a completed draw. Overtime is
an ordinary win/loss. A ranked abandoner gets `s=0` regardless of the final
team result; non-abandoning players use the authoritative final result.
Cancelled or integrity-failed matches do not update anyone.

Lock all six participant rows in lexical player-ID order and compute every
new value from the same immutable pre-match snapshot inside one serializable
transaction. This prevents update-order bias and concurrent double updates.
Golden vectors include canonical one-player examples plus symmetric/asymmetric
3v3, draw, overtime, abandon, inactivity and concurrent-result fixtures.

The first ten ranked matches are provisional. Ranked exposes
backend-derived tiers; casual rating remains hidden. Stored Glicko values, not
tier labels or client calculations, are authoritative. Ranked seasons last 12
weeks. Ranked rollover sets
`rating = 1500 + 0.75 * (rating - 1500)`, raises RD to at least 200 (capped at
350), preserves volatility/history, and is an exactly-once idempotent
transaction. Casual rating is continuous and never season-reset.

Rating, penalty, match completion, participant records and outbox events are
committed in that transaction. Duplicate identical server results succeed
idempotently. A conflicting result changes nothing and pages an operator.

## 6. Game-server allocation and lifecycle

Use provider-portable Kubernetes manifests and Agones. Create one versioned
Fleet per compatible build and region; labels identify region, protocol,
transport, and image/build. Allocate atomically with `GameServerAllocation`.
Agones, rather than bespoke allocator code, owns selection and lifecycle. See
[GameServerAllocation](https://agones.dev/site/docs/reference/gameserverallocation/).

The Godot process talks to the Agones REST sidecar through a small adapter
that is a no-op when `AGONES_SDK_HTTP_PORT` is absent. This preserves native,
Compose, and CI operation and works with the Agones local SDK emulator. See
[Agones client SDKs](https://agones.dev/site/docs/guides/client-sdks/).

Agones has two distinct readiness points; conflating them is a deadlock because
`GameServerAllocation` selects a Ready server and only then attaches the match
metadata:

1. The PID-1 supervisor queries the local Agones SDK for the assigned dynamic
   port/address, exports `SDR_LISTEN_PORT` plus `SDR_IP=public-address:port`
   (or the ENet equivalent), and launches Godot. Godot validates static config,
   binds the socket and starts Health calls.
2. Godot calls Agones `Ready()` after the process is genuinely listening.
   This is **process-ready** only; never infer it from detached stdout.
3. `GameServerAllocation` atomically changes that Ready server to Allocated
   and supplies the signed roster/non-secret match configuration as metadata.
4. The server watches the GameServer, observes Allocated metadata, verifies
   manifest signature/build/protocol/server binding, registers its hosted SDR
   address, and calls the backend `assignment_ready` endpoint.
5. Only after `assignment_ready` does the allocator mint relay/join tickets and
   expose the assignment to clients. The server accepts only assigned
   identities/slots; each selected human has 30 seconds to connect.
6. Run one authoritative match with Health calls independent of the simulation
   loop. Submit the canonical result using the bound workload identity.
7. Write the signed result hash/payload to the backend and to a non-secret
   Agones annotation, remain Allocated, and retry until the backend durably
   acknowledges it. The maintenance worker reconciles the annotation after an
   API outage. `RESULT_PENDING` pages at 5 minutes and requires operator review
   at 30 minutes; it never silently becomes unrated.
8. After acknowledgement call `Shutdown()` and exit. Invalid/empty allocations
   shut down immediately. An allocated GameServer is not recycled to Ready.

Agones supplies dynamic host ports so multiple isolated matches can share a
node; HTTP ingress is not involved in gameplay routing. Hosted SDR additionally
requires Valve approval for every provider/location, a valid `SDR_POPID`,
public-IP/unsolicited-UDP reachability, provider firewall/NAT validation,
per-location certificates and coordinator trust. Use an Agones dynamic or
passthrough mapping whose externally reported port is the `SDR_IP` port while
the process binds `SDR_LISTEN_PORT`; test SDR and ENet mappings separately.
Credentials arrive through runtime secret mounts, never allocation metadata,
arguments, logs or images.

Result authentication uses a projected, pod-bound service-account token with
a dedicated audience and one service account per workload class. The backend
validates the configured cluster issuer/JWKS, audience, expiry, namespace,
service account, bound pod UID and allocator-recorded GameServer UID, then
checks that GameServer/match binding in PostgreSQL. Issuers and trust roots are
allowlisted and rotated explicitly for every cluster/provider. A one-match
server credential issued after this attestation is an acceptable equivalent.

### Warm capacity and density

Both launch regions are active whenever their queues are enabled. Each active
region maintains at least two Ready processes distributed across at least two
on-demand nodes/failure domains; the minimum node floor is therefore two, not
one. Pre-pull current and rollback images. A queue/proposal-aware
FleetAutoscaler adjusts capacity above that Ready floor. **Allocated** process
count may fall to zero; Ready processes do not. An administratively disabled
region may scale both nodes and Fleet to zero and is excluded from placement.
See [Agones FleetAutoscaler](https://agones.dev/site/docs/reference/fleetautoscaler/).

Do not run live matches on interruptible nodes. N+1 means loss of the largest
single node still leaves two Ready slots plus sufficient headroom for already
Allocated matches; certify it in the node-loss test. Set pod requests, limits
and node caps only after native x86_64 benchmarks of boot time, p99
CPU/RSS/network and 60 Hz tick behavior, retaining 30% headroom. The current
6–10 processes/core and 150–250 MB estimates are not sizing data.

Godot/GDScript cannot intercept SIGTERM, so Phase 8 adds a small Go PID-1
supervisor. It starts Godot, traps TERM, and sends a per-pod-token-authenticated
localhost drain request to an allocated-mode-only Godot control socket. Godot
refuses new admissions and reports completion to the supervisor. Kubernetes
uses a 300-second termination grace period; at 285 seconds the supervisor
forces an infrastructure-failure abort so the pod cannot hang forever in
overtime. A PodDisruptionBudget and Agones-aware drain prevent voluntary
eviction of Allocated servers. Planned releases create a new Fleet, route new
allocations to it and wait for old Allocated count zero without sending TERM.
Unexpected node failure cannot be made graceful and follows the integrity
failure/refund path.

Live matches continue through an API/control-plane delivery outage. A valid
result is retried/reconciled as above; only loss of authoritative match
integrity suppresses rating.

## 7. Security and operational baseline

The threat model must cover forged clients, ticket replay, duplicate queueing,
result forgery, queue manipulation, packet floods, botting, server compromise,
insider access, DDoS, vulnerable dependencies, and denial-of-wallet attacks.

Minimum controls:

- Non-root containers, read-only root filesystems, dropped Linux
  capabilities, RuntimeDefault seccomp, resource limits, restricted Pod
  Security admission, and least-privilege service accounts/RBAC.
- Private PostgreSQL/Redis; default-deny ingress and egress with explicit
  NetworkPolicy allowances. See the [Kubernetes application security
  checklist](https://kubernetes.io/docs/concepts/security/application-security-checklist/)
  and [NetworkPolicy](https://kubernetes.io/docs/reference/kubernetes-api/networking/network-policy-v1/).
- Encrypted databases/backups, externally populated Kubernetes Secrets,
  documented rotation, and no credentials in Git, images, command lines, or
  telemetry.
- Per-account/IP API limits, request/body/schema limits, replay and duplicate
  detection, generic public errors, allocation quotas, and budget alerts.
- Put HTTPS/WebSocket traffic behind a provider-portable edge contract
  implemented by each infrastructure overlay: managed volumetric DDoS
  absorption, WAF/rate rules, TLS termination, origin-only ingress and health
  checks. Enforce authenticated queue admission, per-account connection caps,
  WebSocket handshake/message/idle limits, bounded fan-out and overload
  shedding. In degraded mode reject new login/queue/allocation work while
  preserving result ingestion and all live matches.

  The control-plane implements the admission portion of this policy with
  `--degraded` at startup, `SIGUSR1` to enable it, and `SIGUSR2` to disable it.
  The gate rejects new login, queue, and proposal mutations with `503
  service_degraded`; assignment reads, events, server registration/results,
  health, metrics, and other live-match paths remain available.
- Images pinned by digest, SBOM generation, dependency/image scanning, signed
  releases, admission-time signature verification, and a critical-patch SLA.
- Structured audit events for auth, queue transitions, allocation, roster
  rejection, result conflict, penalty, season rollover, and operator action.

## 8. SLOs, observability, and release gates

The operational definitions, owners, windows and alert thresholds for these
targets are maintained in [`MATCHMAKING-SLOs.md`](MATCHMAKING-SLOs.md).

Launch SLOs:

| Measure | Target |
| --- | --- |
| Eligible predicted RTT | <= 100 ms for every player |
| Regional observed RTT | p95 <= 80 ms |
| Unanimous acceptance to `assignment_ready` | p95 <= 5 s, p99 <= 10 s with warm capacity |
| Published assignment to successful connection | p95 <= 5 s |
| Successful allocation and durable result | >= 99.9% |
| Certified server density | No tick backlog, with 30% resource headroom |
| Control-plane API under load | p95 <= 250 ms |

Dashboards and alerts cover queue depth/wait, rating spread, predicted versus
observed RTT, proposals/declines, allocation latency/failure, Ready capacity,
image pulls, connection/no-show, physics overruns, crashes, abnormal packet
rates, result lag/conflicts, abandons, and cost per completed match. Correlate
all components with queue/proposal/match/server IDs, but never log auth or
relay tickets.

Testing layers:

- Go unit, race, fuzz, property, migration, and concurrency tests.
- Fake Steam verifier and fake allocator for deterministic CI.
- A second allocated-server Compose flow; never mutate
  `compose.phase6-smoke.yml`.
- Disposable `kind` + Agones integration for readiness, dynamic ports,
  health/no-show shutdown, allocation races, multiple matches per node,
  draining, and rollback.
- Network/chaos cases for 100 ms RTT, jitter/loss, client/backend/matcher
  restart, game-pod death, node drain, Redis failover, and control-plane loss.
- Load test at least 10,000 queued clients, 100 proposals/second, and forecast
  launch concurrency x2 while preserving correctness and API latency.
- Provider migration rehearsal: restore data and deploy both regional fleets
  on a second provider whose EU/NA locations already have Valve approval,
  POP/certificates, public UDP/firewall verification and coordinator trust;
  switch new allocations, drain the old provider, and terminate no live match.

Release order is development -> internal -> casual canary -> full casual ->
provisional ranked -> full ranked. Each promotion requires its SLO/security
gates, rollback rehearsal, EU and North America playtests, a measured cost
model, and unchanged `make verify-phase6` and
`make verify-enet-integration`.

## 9. Explicitly out of scope

Parties/premades, tournaments, ranked spectators, public global regions,
non-Steam identity providers, and a global leaderboard beyond personal rank
display are not launch scope. The contracts reserve party identity and avoid
Steam-specific database primary keys so those additions do not require a
destructive redesign.