feat(multiplayer): harden allocator rollout

This commit is contained in:
Josh Creek
2026-09-01 18:50:09 +01:00
parent 2e1c010fa2
commit 398d6ade61
3 changed files with 33 additions and 1 deletions
+22
View File
@@ -8,6 +8,11 @@ metadata:
app.kubernetes.io/component: allocator
spec:
replicas: 2
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
selector:
matchLabels:
app.kubernetes.io/name: allocator
@@ -17,6 +22,7 @@ spec:
app.kubernetes.io/name: allocator
app.kubernetes.io/component: allocator
spec:
terminationGracePeriodSeconds: 10
serviceAccountName: allocator
automountServiceAccountToken: false
securityContext:
@@ -36,6 +42,22 @@ spec:
ports:
- name: metrics
containerPort: 9091
readinessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 2
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 3
livenessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 10
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
+1 -1
View File
@@ -1251,7 +1251,7 @@ the local/CI/community transport, not a silent production fallback.
| 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | **IN PROGRESS.** `scripts/verify_kind_agones.sh` creates a disposable kind cluster, installs pinned Agones, loads the real `game-server` image, applies the Fleet in an explicitly separate Agones-only supervisor/UDP readiness mode, and verifies readiness plus allocation of a dynamic UDP endpoint; `.github/workflows/agones-integration.yml` runs it for infrastructure changes and on demand | The cloud-free runner is committed and fails clearly when Docker/kind/Helm are unavailable. CI/live evidence for production control-plane registration, roster/no-show, both readiness stages, races, multi-match node, result-pending reconciliation, drain, and rollback remains open |
| 8.50 `[D:8.25,8.37,8.43,8.49]` | **IN PROGRESS.** `make verify-chaos-recovery` provides a disposable PostgreSQL + real testkit API + real maintenance flow: it restarts the API, injects a stale allocation, and verifies no-penalty requeue plus a durable participant-targeted lifecycle event | The API-restart/stalled-allocation slice is implemented and documented; 100 ms RTT/jitter/loss, matcher/client restart, game-pod death, node drain, Redis failover, control-plane loss, and live chaos evidence remain |
| 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | **IN PROGRESS.** The opt-in `make verify-multiplayer-load` gate drives 10,000 real HTTP queue-create requests through the service with 256 in flight and records p95/p99, plus 100 concurrent proposal formations through the real matcher/domain path; the handler and in-process ownership boundary are exercised without weakening normal tests | Local API load passes at p95 <250 ms in normal and race runs, and the matcher forms 100 unique proposals; PostgreSQL saturation, durable matcher fencing under load, forecast launch concurrency x2, and replica scaling remain live infrastructure gates |
| 8.52 `[D:8.32,8.34,8.45,8.51]` | **IN PROGRESS.** Allocator supports both an opt-in per-replica fixed-window quota (`--allocation-quota` / `--allocation-quota-window`) and an optional PostgreSQL-backed EU/NA quota table consumed inside the serializable allocation transaction before any provider call; idempotent replays do not double-count. The allocator now exposes bounded EU/NA Prometheus counters at an explicit `/metrics` listener (`--metrics-addr`), including quota denials; the checked-in rule warns on regional denial activity, and hardened Kubernetes Deployment/Service/ServiceMonitor resources provide the provisioning and discovery contract | Normal/race/vet tests cover local quota exhaustion, window reset, region isolation, invalid input, atomic concurrent consumption, bounded metrics labels, read-only endpoint behavior, and hardened deployment/network policy invariants; migration/SQL and manifest coverage define the shared quota/metrics boundaries; real image digest/secrets, measured regional cost model, threshold tuning, and denial-of-wallet rehearsal remain |
| 8.52 `[D:8.32,8.34,8.45,8.51]` | **IN PROGRESS.** Allocator supports both an opt-in per-replica fixed-window quota (`--allocation-quota` / `--allocation-quota-window`) and an optional PostgreSQL-backed EU/NA quota table consumed inside the serializable allocation transaction before any provider call; idempotent replays do not double-count. The allocator now exposes bounded EU/NA Prometheus counters at an explicit `/metrics` listener (`--metrics-addr`), including quota denials; the checked-in rule warns on regional denial activity, and hardened Kubernetes Deployment/Service/ServiceMonitor resources provide the provisioning, health, rollout, and discovery contract | Normal/race/vet tests cover local quota exhaustion, window reset, region isolation, invalid input, atomic concurrent consumption, bounded metrics labels, read-only endpoint behavior, and hardened deployment/network policy/lifecycle invariants; migration/SQL and manifest coverage define the shared quota/metrics boundaries; real image digest/secrets, measured regional cost model, threshold tuning, and denial-of-wallet rehearsal remain |
| 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | **IN PROGRESS.** `scripts/verify_release_gate.py` provides a fail-closed promotion check for the ordered development → internal → casual canary → casual → provisional ranked → ranked stages, requiring an evidence report for SLO, security, cost, rollback, EU+NA playtests, and both legacy gates | Validator and adversarial tests cover skipped stages, unknown stages, missing gates, non-boolean gate values, and blank release IDs; the actual reports, production rollback rehearsal, regional playtests, and live promotion remain open |
Implementation invariants for every task above:
@@ -38,6 +38,16 @@ class KubernetesPolicyTest(unittest.TestCase):
self.assertIn(required, deployment)
self.assertNotRegex(deployment, r"(?im)^\s*(password|token|private.?key):\s*[^\n]+$")
def test_allocator_rollout_keeps_capacity_and_has_health_probes(self):
deployment = self.read("allocator-deployment.yaml")
for required in (
"type: RollingUpdate", "maxUnavailable: 0", "maxSurge: 1",
"terminationGracePeriodSeconds: 10",
"readinessProbe:", "livenessProbe:",
"path: /metrics", "port: metrics",
):
self.assertIn(required, deployment)
def test_allocator_network_policy_has_only_metrics_data_agones_and_dns_flows(self):
policies = self.read("network-policies.yaml")
allocator = policies.split("name: allocator-allowed-flows", 1)[-1]