From af8592082efcf7b44980ca44c01ce0bf50367ee4 Mon Sep 17 00:00:00 2001 From: Josh Creek <8179928+jcreek@users.noreply.github.com> Date: Mon, 31 Aug 2026 20:07:41 +0100 Subject: [PATCH] docs: define matchmaking launch SLOs --- docs/MATCHMAKING-SLOs.md | 39 +++++++++++++++++++++++++++++++++++++++ docs/MATCHMAKING.md | 3 +++ multiplayer-next.md | 3 ++- multiplayer-todo.md | 2 +- 4 files changed, 45 insertions(+), 2 deletions(-) create mode 100644 docs/MATCHMAKING-SLOs.md diff --git a/docs/MATCHMAKING-SLOs.md b/docs/MATCHMAKING-SLOs.md new file mode 100644 index 00000000..fd478b83 --- /dev/null +++ b/docs/MATCHMAKING-SLOs.md @@ -0,0 +1,39 @@ +# Matchmaking launch SLOs + +These are the measurable release gates for the Phase 8 control plane. All +latency measurements use server-side monotonic timestamps and are labelled by +region, playlist, build, transport and warm/cold capacity. A request or match +is counted only after the corresponding terminal event is durably recorded. + +| SLO | Metric and denominator | Window / target | Owner | Alert threshold | +| --- | --- | --- | --- | --- | +| Placement eligibility | `max(predicted_rtt_ms)` across all accepted players in a candidate | Every candidate; `<=100 ms` | Matcher | Any eligible candidate over 100 ms pages immediately and is rejected | +| Regional observed RTT | p95 of server-observed handshake/game RTT for connected assigned players | Rolling 1 h, per region; `<=80 ms` | Game-server + networking | 15 min above 80 ms, or any region p95 above 100 ms for 5 min | +| Acceptance → assignment-ready | `assignment_ready - unanimous_accept` for accepted proposals with warm capacity | Rolling 1 h; p95 `<=5 s`, p99 `<=10 s` | Allocator | p95 >5 s for 10 min or p99 >10 s for 5 min | +| Assignment → successful connection | `connected - assignment_published` for assignments not cancelled by policy | Rolling 1 h; p95 `<=5 s` | Game-server lifecycle | p95 >5 s for 10 min or connection success <99% for 5 min | +| Allocation + durable result | completed matches with both successful allocation and durable result / matches requiring allocation | Rolling 24 h; `>=99.9%` | Allocator + maintenance | <99.95% warning; <99.9% pages and blocks release | +| Server tick health | Physics ticks completed without backlog / expected physics ticks; resource headroom is measured independently | Every live match; zero backlog and `>=30%` CPU/RSS headroom | Game-server | Any sustained backlog, or headroom <30% for 5 min | +| Control-plane API | p95 request latency for non-streaming authenticated API requests, excluding client cancellation | Rolling 5 min, by route; `<=250 ms` | API | p95 >250 ms for 5 min or 5xx >1% | + +## Measurement rules + +- Do not combine EU and NA into one percentile; a healthy region must not hide + an unhealthy one. +- Exclude explicitly rejected requests from success denominators, but count + accepted work that later expires, fails allocation, or loses result delivery. +- Preserve queue, proposal, match, server and request IDs on every metric and + trace. Never attach Steam auth tickets, SDR relay tickets, publisher keys or + other credentials to labels, logs or traces. +- Warm-capacity SLOs are evaluated only when the region has the declared Ready + floor. Cold-start and capacity-exhaustion outcomes are separate dashboards, + not silently removed from availability accounting. +- Alert thresholds page the owning role; the release gate is the stricter + target in the table, not the warning threshold. + +## Release evidence + +A release candidate must provide one complete 24-hour report, route-level API +histograms, regional RTT histograms, allocation/connection cohort counts, +tick-health samples, and an incident review for every SLO breach. Load and +chaos tests must retain the same event IDs so the report can distinguish +retryable control-plane delay, player no-show, and match-integrity failure. diff --git a/docs/MATCHMAKING.md b/docs/MATCHMAKING.md index 5a1c8aac..581e4090 100644 --- a/docs/MATCHMAKING.md +++ b/docs/MATCHMAKING.md @@ -418,6 +418,9 @@ Minimum controls: ## 8. SLOs, observability, and release gates +The operational definitions, owners, windows and alert thresholds for these +targets are maintained in [`MATCHMAKING-SLOs.md`](MATCHMAKING-SLOs.md). + Launch SLOs: | Measure | Target | diff --git a/multiplayer-next.md b/multiplayer-next.md index ccafaa6e..87485c84 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -31,7 +31,8 @@ product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). - [x] Lock the Go/PostgreSQL/Redis, Kubernetes/Agones, SDR, EU/NA and provider-portability ADR ([ADR-001](docs/ADR-001-matchmaking-platform.md)); - measurable launch SLOs remain task 8.2. + measurable launch SLOs are defined in + [MATCHMAKING-SLOs.md](docs/MATCHMAKING-SLOs.md). - [ ] Publish versioned OpenAPI/WebSocket contracts, stable IDs, legal state transitions, revisions and idempotency semantics. - [ ] Add PostgreSQL queue ownership/active-participation fences, durable diff --git a/multiplayer-todo.md b/multiplayer-todo.md index fba34825..abc366de 100644 --- a/multiplayer-todo.md +++ b/multiplayer-todo.md @@ -1169,7 +1169,7 @@ the local/CI/community transport, not a silent production fallback. | # | Task | Acceptance | |---|---|---| | 8.1 | Add an ADR locking **Go + PostgreSQL + Redis**, provider-portable Kubernetes, Agones, ticketed Hosted Dedicated Server SDR, EU/NA fleets and independently runnable API, matcher, allocator and maintenance roles; keep `README.md`/`docs/TECH_STACK.md` consistent | The ADR names boundaries/rejected alternatives; current docs name the locked stack and replaceable provider; no application code calls a provider allocation API | -| 8.2 `[D:8.1]` | Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | Each SLO has a metric, numerator/denominator, percentile window, owner and alert threshold before implementation is judged against it | +| 8.2 `[D:8.1]` | **DONE.** Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | [`docs/MATCHMAKING-SLOs.md`](docs/MATCHMAKING-SLOs.md) defines each metric, denominator, percentile/window, owner, alert threshold and release evidence | | 8.3 `[D:8.1]` | Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | Generated contract tests cover every request, response, event and external error; clients can REST-resync after a missed WebSocket revision | | 8.4 `[D:8.3]` | Define opaque `player_id`, `queue_ticket_id`, `proposal_id`, `match_id`, `server_id`, `season_id`, legal queue/match state transitions, revisions and idempotency keys | Duplicate/out-of-order commands converge; invalid transitions are rejected without partial state | | 8.5 `[D:8.4]` | Add PostgreSQL migrations for durable queue ownership, active-participation fencing, identities, sessions/revocations, seasons, ratings/events, matches/participants, penalties, results, audits and outbox; document Redis caches/TTLs | A blank DB migrates up; lost Redis writes cannot resurrect revocation, split a proposal or corrupt durable state; rollback/forward compatibility is tested |