feat: protect game fleet during voluntary disruption

This commit is contained in:
Josh Creek
2026-08-31 22:01:18 +01:00
parent 72d4a604c2
commit 52a96de8d6
5 changed files with 28 additions and 3 deletions
+15
View File
@@ -0,0 +1,15 @@
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: cosmic-clash-game
namespace: cosmic-clash
labels:
app.kubernetes.io/name: game-server-pdb
spec:
# Voluntary node drains must preserve the Fleet's two-Ready floor. Agones
# remains responsible for replacing an evicted process before more capacity
# is voluntarily removed.
minAvailable: 2
selector:
matchLabels:
app.kubernetes.io/name: game-server
+1
View File
@@ -8,3 +8,4 @@ resources:
- control-plane-deployment.yaml
- fleet.yaml
- fleet-autoscaler.yaml
- game-server-pdb.yaml
+2 -2
View File
@@ -141,8 +141,8 @@ product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md).
requests/limits and node density from measurements plus 30% headroom.
- [ ] Add 30 s no-show handling, Go PID-1 TERM/drain supervision, PDB/Fleet
drain, signed result annotation/retry, RPO <=5 m and RTO <=30 m. The Go
drain boundary is now authenticated and loopback-only; lifecycle/PDB/Fleet
integration remains.
drain boundary is now authenticated and loopback-only, and the base PDB
protects the two-Ready floor; lifecycle/PDB/Fleet integration remains.
- [ ] Rehearse migration only after the second provider's EU/NA locations have
Valve approval, POP/certs, public UDP/firewall and coordinator trust.
+1 -1
View File
@@ -1218,7 +1218,7 @@ the local/CI/community transport, not a silent production fallback.
| 8.33 `[D:8.26,8.32]` | **IN PROGRESS.** Fleet scheduling now requires on-demand capacity and spreads Ready processes across zones with skew 1; the autoscaler preserves the two-process Ready floor | `deploy/k8s/base/fleet.yaml` and manifest tests reject interruptible placement and single-zone concentration structurally; regional node pools, forced node-loss testing and measured N+1 headroom remain |
| 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog |
| 8.35 `[D:8.17,8.19,8.20,8.30,8.31]` | **IN PROGRESS.** Pure Go initial-connect policy decides ranked 30 s no-show cancellation with abandon ladder and casual 60 s bot start only when each team has a human; empty-team casual allocations cancel | `server/domain/noshow.go` covers wait/deadline boundaries, deterministic no-show/innocent ordering, ranked cooldown history and no pre-live rating action; persistent ticket restoration, allocation shutdown, bot spawn and live integration remain |
| 8.36 `[D:8.10,8.25,8.28,8.30]` | **IN PROGRESS.** Supervisor exposes an authenticated loopback-only drain request boundary that never places the token in command arguments/logs and rejects remote/partial/query-bearing configurations | `server/supervisor/` covers bearer-token enforcement, loopback URL validation, secret-safe configuration and rejection of missing drain credentials; TERM signal handling, 300 s/285 s lifecycle, PDB/Fleet drain and infrastructure-abort classification remain |
| 8.36 `[D:8.10,8.25,8.28,8.30]` | **IN PROGRESS.** Supervisor exposes an authenticated loopback-only drain request boundary that never places the token in command arguments/logs and rejects remote/partial/query-bearing configurations; the base now includes a two-Ready PodDisruptionBudget | `server/supervisor/` and `deploy/k8s/base/game-server-pdb.yaml` cover bearer-token enforcement, loopback URL validation, secret-safe configuration, missing drain credentials and Ready-floor disruption protection; TERM signal handling, 300 s/285 s lifecycle, live PDB/Fleet drain and infrastructure-abort classification remain |
| 8.37 `[D:8.5,8.10,8.25,8.26,8.31]` | Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO <=5 m/RTO <=30 m; signed result annotation/reconciliation preserves delivery outages | Restore/failover meet targets; live simulation continues; valid delayed result commits exactly once after recovery |
| 8.38 `[D:7.7,8.26,8.36,8.37]` | Provider migration after Valve approves both providers' EU/NA POPs/certs and public UDP: restore, validate coordinator trust, switch allocations, drain old | Both geographies complete Hosted-SDR matches on new provider and no old-provider live match is terminated |
+9
View File
@@ -36,6 +36,15 @@ class FleetManifestTest(unittest.TestCase):
):
self.assertIn(field, autoscaler)
def test_pdb_protects_the_ready_floor_and_matches_game_servers(self):
pdb = self.read("base/game-server-pdb.yaml")
for field in (
"kind: PodDisruptionBudget", "apiVersion: policy/v1",
"namespace: cosmic-clash", "minAvailable: 2",
"app.kubernetes.io/name: game-server",
):
self.assertIn(field, pdb)
def test_eu_and_na_overlays_are_distinct_and_namespaced(self):
eu = self.read("overlays/eu/region.yaml")
na = self.read("overlays/na/region.yaml")