ops(multiplayer): add control-plane alert rules

deploy/observability/prometheus-rules.yaml

@@ -0,0 +1,54 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: cosmic-clash-control-plane
+  namespace: cosmic-clash
+  labels:
+    app.kubernetes.io/name: cosmic-clash
+    app.kubernetes.io/component: observability
+spec:
+  groups:
+    - name: cosmic-clash.control-plane
+      rules:
+        - alert: CosmicClashControlPlaneAPIP95High
+          expr: |
+            histogram_quantile(
+              0.95,
+              sum by (le, operation) (
+                rate(cosmic_clash_api_latency_seconds_bucket[5m])
+              )
+            ) > 0.25
+          for: 5m
+          labels:
+            severity: page
+            owner: api
+          annotations:
+            summary: Cosmic Clash control-plane API p95 latency is high
+            description: >-
+              The 5-minute p95 latency for operation {{ $labels.operation }}
+              has exceeded the 250 ms API SLO for 5 minutes.
+            runbook_url: https://example.invalid/cosmic-clash/runbooks/control-plane-api
+        - alert: CosmicClashControlPlaneAPI5xxHigh
+          expr: |
+            (
+              sum by (operation) (
+                rate(cosmic_clash_api_requests_total{status="5xx"}[5m])
+              )
+              /
+              clamp_min(
+                sum by (operation) (
+                  rate(cosmic_clash_api_requests_total[5m])
+                ),
+                0.001
+              )
+            ) > 0.01
+          for: 5m
+          labels:
+            severity: page
+            owner: api
+          annotations:
+            summary: Cosmic Clash control-plane API 5xx rate is high
+            description: >-
+              The 5-minute 5xx ratio for operation {{ $labels.operation }}
+              has exceeded 1 percent for 5 minutes.
+            runbook_url: https://example.invalid/cosmic-clash/runbooks/control-plane-api

docs/OBSERVABILITY.md

@@ -0,0 +1,17 @@
+# Multiplayer observability
+
+The control plane exposes `/metrics` with bounded operation and status labels.
+The API latency metric is a cumulative histogram, so Prometheus can evaluate
+the documented 250 ms p95 SLO with `histogram_quantile`. The optional
+`deploy/observability/prometheus-rules.yaml` resource provides the API p95 and
+5xx alerts for clusters running the Prometheus Operator.
+
+Install the rule only after confirming that the `PrometheusRule` CRD and the
+`cosmic-clash` namespace exist. The example `runbook_url` values are
+placeholders and must be replaced with the operator's incident documentation
+before production use.
+
+This artifact intentionally does not claim coverage for regional RTT,
+allocation/connect latency, tick headroom, durable-result success, or cost.
+Those SLOs need additional server, allocator, and game-server series before
+they can be alerted on safely; the current exporter cannot manufacture them.

multiplayer-next.md

@@ -1453,7 +1453,7 @@ Observability redaction now adds content-aware protection on top of denylisted f
 
 ### Current local completion index (2026-09-01)
 
-The following Phase 8 slices have local implementation and verification evidence in this document: 8.29 dynamic allocated launch flags and endpoint handling; 8.30 allocator claim/reconciliation; 8.31 signed assignment/roster validation; 8.35 initial-connect no-show and casual bot policy; 8.36 controlled drain and shutdown acknowledgment; 8.39–8.43 client state, assignment, profile, recovery, and idempotent action retry; 8.44 structured observability and content-aware redaction; 8.45 bounded API metrics export; 8.46 normal/race/vet/fuzz coverage; and 8.47–8.48 offline testkit coverage. Their remaining acceptance text is infrastructure or production dependent where explicitly noted below the corresponding row.
+The following Phase 8 slices have local implementation and verification evidence in this document: 8.29 dynamic allocated launch flags and endpoint handling; 8.30 allocator claim/reconciliation; 8.31 signed assignment/roster validation; 8.35 initial-connect no-show and casual bot policy; 8.36 controlled drain and shutdown acknowledgment; 8.39–8.43 client state, assignment, profile, recovery, and idempotent action retry; 8.44 structured observability and content-aware redaction; 8.45 bounded API metrics export and optional Prometheus alert rules; 8.46 normal/race/vet/fuzz coverage; and 8.47–8.48 offline testkit coverage. Their remaining acceptance text is infrastructure or production dependent where explicitly noted below the corresponding row.
 
 The following are not locally certifiable from this workspace and remain open prerequisites rather than silently “done”: Valve/GodotSteam credentials and hosted SDR (7.1–7.8), live PostgreSQL/Redis execution where Docker is unavailable, live Agones/kind lifecycle (8.30–8.38, 8.49), public-network chaos/load/cost/release gates (8.50–8.53), and real-hardware graphics profiling (0.15b onward). `TODO.md`’s AI-training and presentation tasks remain separate from multiplayer and are not marked by this index.