feat(multiplayer): export queryable API latency histograms

This commit is contained in:
Josh Creek
2026-09-01 17:21:13 +01:00
parent 20f376f713
commit 0bf33e7fe8
3 changed files with 52 additions and 8 deletions
+2 -2
View File
@@ -1243,8 +1243,8 @@ the local/CI/community transport, not a silent production fallback.
| # | Task | Acceptance |
|---|---|---|
| 8.44 `[D:8.3,8.4,8.28,8.31]` | **IN PROGRESS.** Go observability package encodes queue/proposal/match/server IDs and lifecycle stage in structured events while recursively redacting auth/relay tokens and credentials. It now actually emits: `Service.Log` is a nil-safe optional hook, wired into the two workload-authenticated server routes (register, result) at every outcome plus queue create/heartbeat/cancel and proposal accept/decline (state on success, `rejected` on a domain error, never the error text), and `cmd/control-plane` writes those events as JSON lines to stderr | `server/observability/` covers correlation fields, nested secret redaction and unnamed-event rejection; `server/api/service.go`/`service_test.go` cover the wiring plus a secret-canary test that drives the server routes with real-looking bearer-token/nonce values and asserts neither appears anywhere in what `Log` actually received (stronger than the unit test, which only proves a synthetic value under a denylisted key is stripped), and a lifecycle test asserting the exact event/id/stage sequence across a real create→heartbeat→cancel and an accept→stale-revision-reject. `redact()` is still key-name-based, not content-based — a field logged under an unlisted key would leak and neither test would catch it, only the discipline of never putting raw secret bytes into `Fields`; read-only routes (queue/proposal GET, assignment fetch), early availability/not-found rejections, and a real metrics/traces backend (this is stderr only) remain |
| 8.45 `[D:8.2,8.44]` | **IN PROGRESS.** Go observability package turns the documented RTT, allocation/connect latency, result-success, API-latency and tick/headroom thresholds into executable window checks | `server/observability/slo.go` covers healthy/violating/empty windows; production metrics export, dashboards, alert routing, wait/MMR/proposal/flood/cost series and runbooks remain |
| 8.44 `[D:8.3,8.4,8.28,8.31]` | **IN PROGRESS.** Go observability package encodes queue/proposal/match/server IDs and lifecycle stage in structured events while recursively redacting auth/relay tokens and credentials. It now actually emits: `Service.Log` is a nil-safe optional hook, wired to mutation and read routes at every outcome, and `cmd/control-plane` writes those events as JSON lines to stderr | `server/observability/` covers correlation fields, nested secret redaction, content-aware credential canaries and unnamed-event rejection; API tests cover lifecycle event wiring without logging error text. Remaining work is a real metrics/traces backend and production dashboard/alert routing; the local logger is intentionally stderr-only |
| 8.45 `[D:8.2,8.44]` | **IN PROGRESS.** Go observability package turns the documented RTT, allocation/connect latency, result-success, API-latency and tick/headroom thresholds into executable window checks; the API exporter now emits a bounded cumulative latency histogram suitable for querying the documented p95 API SLO | `server/observability/slo.go`, `metrics.go` and adversarial tests cover healthy/violating/empty windows, fixed operation/status labels, cumulative bucket boundaries and arbitrary-path cardinality safety; production scrape configuration, dashboards, alert routing, wait/MMR/proposal/flood/cost series and runbooks remain |
| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | **IN PROGRESS.** Go unit/race coverage spans the current domain/store/supervisor policies, and fuzz targets now exercise queue input, result payload hashing and revision events | `server/domain/*_test.go`, `server/store/*_test.go`, `server/supervisor/*_test.go`, `server/migrations/*_test.go` and `server/domain/fuzz_test.go` pass normal/race suites; `go test -race ./...` passes across API, domain, migrations, observability, store, supervisor and testkit; `go vet ./...` passes; each of the three declared domain fuzz targets passes a bounded 4-second run; PostgreSQL live migration execution now runs clean (§8.5), and four real-concurrency cases are covered against a live database with `-race`: §8.14's queue-heartbeat revision race, §8.18's two-matcher contested-ticket race, §8.30's cross-allocator-replica capacity race, and §8.21/§8.25's concurrent identical-result-submission race; the "lost Redis" fixture is covered live against a real server (§8.14: real TTL expiry, repair-after-`FLUSHALL`; fake Steam/allocator fixtures are §8.47's testkit, already done). Further transaction fixtures (e.g. concurrent proposal-recovery expiry races, live Redis failover mid-write under load) remain |
| 8.47 `[D:8.7,8.30]` | **IN PROGRESS.** Offline testkit provides deterministic fake Steam verification and fake allocation with forced failure injection | `server/testkit/` covers verified identity/replay, unknown identity, wrong App ID, expiry, no capacity, compatibility-key conflict, idempotent allocation replay and cloud-free forced allocation failure in `TestOfflineFakesCoverVerificationAndAllocationFailureMatrix`; API/Compose integration and live exhaustive matrix remain |
| 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | **IN PROGRESS.** Offline testkit exercises verified queue projection → ranked six-player proposal → ENet allocation → assignment-ready manifest → certified durable result receipt | `server/testkit/pipeline_test.go` covers the cross-domain success path without Steam/cloud secrets; independent Compose fixture, process shutdown, result ack over HTTP and legacy fixture non-regression remain |