diff --git a/SERVER.md b/SERVER.md index f52cc75b..452c9989 100644 --- a/SERVER.md +++ b/SERVER.md @@ -86,6 +86,11 @@ roughly 6–10 simultaneous match processes per modern core, 150–250 MB RSS pe process, and about 630 kbit/s upstream for a full six-player match; use those as a starting point and monitor actual CPU, RSS, and egress. +Per-match autoscaling — allocating a server for one match and shutting it +down afterwards — is designed in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md) +and not yet implemented. The sizing numbers above predate that work and +should be re-measured under real concurrency before they size a bill. + This build must not be exposed to strangers yet. Slot reclaim is still keyed by display name, so a player who knows a disconnected player's name can claim their reserved slot. Phase 7 Steam-auth identity is the required fix. Local, diff --git a/docs/MATCHMAKING.md b/docs/MATCHMAKING.md index 5c1fbe96..2f5a7a9d 100644 --- a/docs/MATCHMAKING.md +++ b/docs/MATCHMAKING.md @@ -94,6 +94,108 @@ The server side needs less new work than it looks: slot, replacing the current first-come model. - Client-side queue UI and the accept/decline flow. +## Server orchestration and autoscaling + +Requirement: game servers scale horizontally and automatically, spin up fast +on demand, serve exactly one match, and shut down — so cost is incurred only +while a match is being played. Docker and the existing CI gates must keep +working unchanged. + +### Why this is achievable: the server is already shaped for it + +Two properties of the current build make per-match allocation practical +rather than aspirational: + +- **The container is small.** The `server` image target is a slim + `ubuntu:24.04` runtime with three shared libraries and the exported + binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image. + Pulling it onto a fresh node is cheap. +- **Boot to listening is sub-second.** Measured on this repo's + `cosmicclash-server:latest`: **~870 ms** from container start to the + `server_started` log line, averaged over three runs, read from the + container's own clock. That measurement was taken under **x86_64 emulation + on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux + should be faster. Re-measure on the real target before setting timeouts. + +Combined with `--max-matches=1`, which already drains and `exit(0)`s after a +single match, the lifecycle the allocator needs mostly exists: start +container → serve one match → process exits → orchestrator reclaims. + +### The cold-start tension, stated honestly + +"Only pay during a match" and "a player never waits" are in tension. A server +must be listening *before* the matched players connect, so some cost always +precedes the match. Sub-second boot makes the gap small enough that a pure +scale-to-zero design is plausible — but the risk is not the container, it is +everything around it: image pull on a cold node, scheduler placement, and +network/port programming can each dwarf 870 ms. + +Recommendation: **scale to zero at the node level is the wrong target; scale +to zero at the match level is the right one.** Keep a small warm pool of +nodes sized to the current queue depth, and start a per-match container on +demand within it. The per-match process genuinely exists only for the match; +the node pool absorbs the cold-start variance. Revisit only if measured +allocation latency on real infrastructure shows the warm pool is unnecessary. + +### Findings that block a naive implementation + +**Readiness cannot be detected from the log line.** Godot's stdout is +block-buffered when it is not attached to a TTY. Run the server image +detached without `-t` and `docker logs` shows **nothing at all** — the +`server_started` line does not appear even after 35 seconds, because the +buffer never flushes. An orchestrator readiness probe that greps for that +line will hang forever, and this was reproduced directly while measuring the +boot time above. Either probe the UDP socket instead, or make the server +flush explicitly. This also means container logs are not a reliable +observability channel for a short-lived match server; treat log shipping as +a separate problem. + +**One fixed port per container does not scale on a shared host.** `--port` +defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing +several matches onto one node needs either a port range allocated per +container, or one address per container. This is a UDP service, so the usual +HTTP ingress/L7 routing answers do not apply — the allocator must hand the +client a concrete `host:port`. + +**The match cannot start on a schedule the players do not control.** Today +the loop waits for `--min-players` then counts down. An allocated server is +told *which* identities to expect, and needs a **no-show timeout**: if a +matched player never connects, the server must abandon and exit rather than +sit idle burning the cost this design is trying to avoid. + +### Keeping Docker and CI green + +The existing gates must not regress. `make verify-phase6` builds the export, +runs it in Compose, joins two headless clients and asserts both saw both +goals and that the arena rotated between matches; `make verify-enet-integration` +runs the source-build ENet matrix. Both depend on current behaviour: +`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot +assignment, and uses `--max-matches=2` to prove rotation. + +The rule that keeps them passing: **every allocation feature is opt-in via a +new `ServerConfig` flag whose default reproduces today's behaviour.** An +assigned roster, a no-show timeout and result reporting must each be inert +unless explicitly enabled. `ServerConfig` is built for exactly this — a flag +declared once is parsed, validated, type-checked, config-file-backed and +documented — and `tests/cases/` can cover the new parsing without a live +server. A second Compose file should cover the allocated-match path rather +than mutating the Phase 6 one, so the community-server model stays tested +alongside the matchmade one. + +### Open questions + +- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this + game-server lifecycle), Nomad, or direct cloud-API container starts. Not + chosen. Agones is the strongest default because it models allocation, + readiness and per-match lifetime natively. +- **Port strategy** — port range per node versus one IP per match. +- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 6–10 match + processes per modern core and 150–250 MB RSS each. That estimate predates + any allocation work and should be re-measured under real concurrency + before it sizes a bill. +- **Draining and deploys.** How a server version rolls out without killing + matches in flight. + ## Casual vs ranked They are different playlists, not a difficulty toggle, and their rules diff --git a/multiplayer-next.md b/multiplayer-next.md index e9c459d4..87d20f5a 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -51,6 +51,28 @@ spoofable identity is worse than no rating. - [ ] Casual and ranked playlist rulesets (backfill, bots, abandon penalties, arena restriction — see the comparison table in the design doc). +Server orchestration (same phase — the servers must autoscale and bill only +for the duration of a match): + +- [ ] Choose the orchestrator (Agones on Kubernetes is the recommended + default; it models allocation, readiness and per-match lifetime natively). +- [ ] Fix readiness detection. Godot's stdout is block-buffered off a TTY, so + `server_started` never appears in `docker logs` for a detached container — + a log-grep readiness probe hangs forever. Probe the UDP socket, or flush. +- [ ] Support more than one match per host: a per-container port from a range, + or one address per match. `--port` defaults to 7777 and the Dockerfile + hardcodes `EXPOSE 7777/udp`. +- [ ] Add a no-show timeout so an allocated server that never fills abandons + and exits instead of idling at cost. +- [ ] Re-measure boot-to-listening on native x86_64 Linux. The repo's current + figure is ~870 ms, measured under emulation on arm64 — a pessimistic bound. +- [ ] Re-measure the SERVER.md sizing estimate (6–10 processes/core, + 150–250 MB RSS) under real concurrency before it sizes a bill. +- [ ] Keep `make verify-phase6` and `make verify-enet-integration` green: + every allocation feature is opt-in via a `ServerConfig` flag defaulting to + today's behaviour, with a second Compose file for the allocated path rather + than mutating `compose.phase6-smoke.yml`. + ## Known issues to resolve before public hosting - [ ] Slot reclaim is currently keyed by display name, so someone can take a