docs: design per-match server autoscaling, with measured boot time

Casual/ranked queues need servers allocated per match and shut down
afterwards, so cost is incurred only while a match runs - while the
existing Docker and CI gates keep passing unchanged.

Measured against the repo's own cosmicclash-server image rather than
estimated: the runtime image is ~148 MB of content, and boot to the
server_started line is ~870 ms on the container's own clock. That was
taken under x86_64 emulation on an arm64 host, so it is a pessimistic
bound and is recorded as one - it needs re-measuring on native Linux
before it sets any timeout.

Two findings that would each break a naive implementation, both hit
while taking that measurement:

- Godot's stdout is block-buffered off a TTY. A detached container logs
  nothing at all - server_started does not appear even after 35s - so an
  orchestrator readiness probe that greps the log hangs forever. Probe
  the UDP socket or flush explicitly.
- --port defaults to 7777 and the Dockerfile hardcodes EXPOSE 7777/udp,
  so several matches cannot share a host without a port range or an
  address per match. Being UDP, L7 ingress routing does not apply.

Also records the honest tension in 'only pay during a match': a server
must listen before players connect, and image pull plus scheduling can
dwarf 870 ms, so the recommendation is match-level scale-to-zero over a
small warm node pool rather than node-level scale-to-zero.

The rule for keeping verify-phase6 and verify-enet-integration green:
every allocation feature is opt-in via a ServerConfig flag defaulting to
current behaviour, with a second Compose file rather than mutating
compose.phase6-smoke.yml.
This commit is contained in:
Josh Creek
2026-08-31 18:45:05 +01:00
parent 3aa0f5b9c2
commit 4ffa1543cc
3 changed files with 129 additions and 0 deletions
+5
View File
@@ -86,6 +86,11 @@ roughly 610 simultaneous match processes per modern core, 150250 MB RSS pe
process, and about 630 kbit/s upstream for a full six-player match; use those
as a starting point and monitor actual CPU, RSS, and egress.
Per-match autoscaling — allocating a server for one match and shutting it
down afterwards — is designed in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md)
and not yet implemented. The sizing numbers above predate that work and
should be re-measured under real concurrency before they size a bill.
This build must not be exposed to strangers yet. Slot reclaim is still keyed
by display name, so a player who knows a disconnected player's name can claim
their reserved slot. Phase 7 Steam-auth identity is the required fix. Local,
+102
View File
@@ -94,6 +94,108 @@ The server side needs less new work than it looks:
slot, replacing the current first-come model.
- Client-side queue UI and the accept/decline flow.
## Server orchestration and autoscaling
Requirement: game servers scale horizontally and automatically, spin up fast
on demand, serve exactly one match, and shut down — so cost is incurred only
while a match is being played. Docker and the existing CI gates must keep
working unchanged.
### Why this is achievable: the server is already shaped for it
Two properties of the current build make per-match allocation practical
rather than aspirational:
- **The container is small.** The `server` image target is a slim
`ubuntu:24.04` runtime with three shared libraries and the exported
binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image.
Pulling it onto a fresh node is cheap.
- **Boot to listening is sub-second.** Measured on this repo's
`cosmicclash-server:latest`: **~870 ms** from container start to the
`server_started` log line, averaged over three runs, read from the
container's own clock. That measurement was taken under **x86_64 emulation
on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux
should be faster. Re-measure on the real target before setting timeouts.
Combined with `--max-matches=1`, which already drains and `exit(0)`s after a
single match, the lifecycle the allocator needs mostly exists: start
container → serve one match → process exits → orchestrator reclaims.
### The cold-start tension, stated honestly
"Only pay during a match" and "a player never waits" are in tension. A server
must be listening *before* the matched players connect, so some cost always
precedes the match. Sub-second boot makes the gap small enough that a pure
scale-to-zero design is plausible — but the risk is not the container, it is
everything around it: image pull on a cold node, scheduler placement, and
network/port programming can each dwarf 870 ms.
Recommendation: **scale to zero at the node level is the wrong target; scale
to zero at the match level is the right one.** Keep a small warm pool of
nodes sized to the current queue depth, and start a per-match container on
demand within it. The per-match process genuinely exists only for the match;
the node pool absorbs the cold-start variance. Revisit only if measured
allocation latency on real infrastructure shows the warm pool is unnecessary.
### Findings that block a naive implementation
**Readiness cannot be detected from the log line.** Godot's stdout is
block-buffered when it is not attached to a TTY. Run the server image
detached without `-t` and `docker logs` shows **nothing at all** — the
`server_started` line does not appear even after 35 seconds, because the
buffer never flushes. An orchestrator readiness probe that greps for that
line will hang forever, and this was reproduced directly while measuring the
boot time above. Either probe the UDP socket instead, or make the server
flush explicitly. This also means container logs are not a reliable
observability channel for a short-lived match server; treat log shipping as
a separate problem.
**One fixed port per container does not scale on a shared host.** `--port`
defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing
several matches onto one node needs either a port range allocated per
container, or one address per container. This is a UDP service, so the usual
HTTP ingress/L7 routing answers do not apply — the allocator must hand the
client a concrete `host:port`.
**The match cannot start on a schedule the players do not control.** Today
the loop waits for `--min-players` then counts down. An allocated server is
told *which* identities to expect, and needs a **no-show timeout**: if a
matched player never connects, the server must abandon and exit rather than
sit idle burning the cost this design is trying to avoid.
### Keeping Docker and CI green
The existing gates must not regress. `make verify-phase6` builds the export,
runs it in Compose, joins two headless clients and asserts both saw both
goals and that the arena rotated between matches; `make verify-enet-integration`
runs the source-build ENet matrix. Both depend on current behaviour:
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
assignment, and uses `--max-matches=2` to prove rotation.
The rule that keeps them passing: **every allocation feature is opt-in via a
new `ServerConfig` flag whose default reproduces today's behaviour.** An
assigned roster, a no-show timeout and result reporting must each be inert
unless explicitly enabled. `ServerConfig` is built for exactly this — a flag
declared once is parsed, validated, type-checked, config-file-backed and
documented — and `tests/cases/` can cover the new parsing without a live
server. A second Compose file should cover the allocated-match path rather
than mutating the Phase 6 one, so the community-server model stays tested
alongside the matchmade one.
### Open questions
- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this
game-server lifecycle), Nomad, or direct cloud-API container starts. Not
chosen. Agones is the strongest default because it models allocation,
readiness and per-match lifetime natively.
- **Port strategy** — port range per node versus one IP per match.
- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 610 match
processes per modern core and 150250 MB RSS each. That estimate predates
any allocation work and should be re-measured under real concurrency
before it sizes a bill.
- **Draining and deploys.** How a server version rolls out without killing
matches in flight.
## Casual vs ranked
They are different playlists, not a difficulty toggle, and their rules
+22
View File
@@ -51,6 +51,28 @@ spoofable identity is worse than no rating.
- [ ] Casual and ranked playlist rulesets (backfill, bots, abandon penalties,
arena restriction — see the comparison table in the design doc).
Server orchestration (same phase — the servers must autoscale and bill only
for the duration of a match):
- [ ] Choose the orchestrator (Agones on Kubernetes is the recommended
default; it models allocation, readiness and per-match lifetime natively).
- [ ] Fix readiness detection. Godot's stdout is block-buffered off a TTY, so
`server_started` never appears in `docker logs` for a detached container —
a log-grep readiness probe hangs forever. Probe the UDP socket, or flush.
- [ ] Support more than one match per host: a per-container port from a range,
or one address per match. `--port` defaults to 7777 and the Dockerfile
hardcodes `EXPOSE 7777/udp`.
- [ ] Add a no-show timeout so an allocated server that never fills abandons
and exits instead of idling at cost.
- [ ] Re-measure boot-to-listening on native x86_64 Linux. The repo's current
figure is ~870 ms, measured under emulation on arm64 — a pessimistic bound.
- [ ] Re-measure the SERVER.md sizing estimate (610 processes/core,
150250 MB RSS) under real concurrency before it sizes a bill.
- [ ] Keep `make verify-phase6` and `make verify-enet-integration` green:
every allocation feature is opt-in via a `ServerConfig` flag defaulting to
today's behaviour, with a second Compose file for the allocated path rather
than mutating `compose.phase6-smoke.yml`.
## Known issues to resolve before public hosting
- [ ] Slot reclaim is currently keyed by display name, so someone can take a