mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
docs: design per-match server autoscaling, with measured boot time
Casual/ranked queues need servers allocated per match and shut down afterwards, so cost is incurred only while a match runs - while the existing Docker and CI gates keep passing unchanged. Measured against the repo's own cosmicclash-server image rather than estimated: the runtime image is ~148 MB of content, and boot to the server_started line is ~870 ms on the container's own clock. That was taken under x86_64 emulation on an arm64 host, so it is a pessimistic bound and is recorded as one - it needs re-measuring on native Linux before it sets any timeout. Two findings that would each break a naive implementation, both hit while taking that measurement: - Godot's stdout is block-buffered off a TTY. A detached container logs nothing at all - server_started does not appear even after 35s - so an orchestrator readiness probe that greps the log hangs forever. Probe the UDP socket or flush explicitly. - --port defaults to 7777 and the Dockerfile hardcodes EXPOSE 7777/udp, so several matches cannot share a host without a port range or an address per match. Being UDP, L7 ingress routing does not apply. Also records the honest tension in 'only pay during a match': a server must listen before players connect, and image pull plus scheduling can dwarf 870 ms, so the recommendation is match-level scale-to-zero over a small warm node pool rather than node-level scale-to-zero. The rule for keeping verify-phase6 and verify-enet-integration green: every allocation feature is opt-in via a ServerConfig flag defaulting to current behaviour, with a second Compose file rather than mutating compose.phase6-smoke.yml.
This commit is contained in:
@@ -94,6 +94,108 @@ The server side needs less new work than it looks:
|
||||
slot, replacing the current first-come model.
|
||||
- Client-side queue UI and the accept/decline flow.
|
||||
|
||||
## Server orchestration and autoscaling
|
||||
|
||||
Requirement: game servers scale horizontally and automatically, spin up fast
|
||||
on demand, serve exactly one match, and shut down — so cost is incurred only
|
||||
while a match is being played. Docker and the existing CI gates must keep
|
||||
working unchanged.
|
||||
|
||||
### Why this is achievable: the server is already shaped for it
|
||||
|
||||
Two properties of the current build make per-match allocation practical
|
||||
rather than aspirational:
|
||||
|
||||
- **The container is small.** The `server` image target is a slim
|
||||
`ubuntu:24.04` runtime with three shared libraries and the exported
|
||||
binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image.
|
||||
Pulling it onto a fresh node is cheap.
|
||||
- **Boot to listening is sub-second.** Measured on this repo's
|
||||
`cosmicclash-server:latest`: **~870 ms** from container start to the
|
||||
`server_started` log line, averaged over three runs, read from the
|
||||
container's own clock. That measurement was taken under **x86_64 emulation
|
||||
on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux
|
||||
should be faster. Re-measure on the real target before setting timeouts.
|
||||
|
||||
Combined with `--max-matches=1`, which already drains and `exit(0)`s after a
|
||||
single match, the lifecycle the allocator needs mostly exists: start
|
||||
container → serve one match → process exits → orchestrator reclaims.
|
||||
|
||||
### The cold-start tension, stated honestly
|
||||
|
||||
"Only pay during a match" and "a player never waits" are in tension. A server
|
||||
must be listening *before* the matched players connect, so some cost always
|
||||
precedes the match. Sub-second boot makes the gap small enough that a pure
|
||||
scale-to-zero design is plausible — but the risk is not the container, it is
|
||||
everything around it: image pull on a cold node, scheduler placement, and
|
||||
network/port programming can each dwarf 870 ms.
|
||||
|
||||
Recommendation: **scale to zero at the node level is the wrong target; scale
|
||||
to zero at the match level is the right one.** Keep a small warm pool of
|
||||
nodes sized to the current queue depth, and start a per-match container on
|
||||
demand within it. The per-match process genuinely exists only for the match;
|
||||
the node pool absorbs the cold-start variance. Revisit only if measured
|
||||
allocation latency on real infrastructure shows the warm pool is unnecessary.
|
||||
|
||||
### Findings that block a naive implementation
|
||||
|
||||
**Readiness cannot be detected from the log line.** Godot's stdout is
|
||||
block-buffered when it is not attached to a TTY. Run the server image
|
||||
detached without `-t` and `docker logs` shows **nothing at all** — the
|
||||
`server_started` line does not appear even after 35 seconds, because the
|
||||
buffer never flushes. An orchestrator readiness probe that greps for that
|
||||
line will hang forever, and this was reproduced directly while measuring the
|
||||
boot time above. Either probe the UDP socket instead, or make the server
|
||||
flush explicitly. This also means container logs are not a reliable
|
||||
observability channel for a short-lived match server; treat log shipping as
|
||||
a separate problem.
|
||||
|
||||
**One fixed port per container does not scale on a shared host.** `--port`
|
||||
defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing
|
||||
several matches onto one node needs either a port range allocated per
|
||||
container, or one address per container. This is a UDP service, so the usual
|
||||
HTTP ingress/L7 routing answers do not apply — the allocator must hand the
|
||||
client a concrete `host:port`.
|
||||
|
||||
**The match cannot start on a schedule the players do not control.** Today
|
||||
the loop waits for `--min-players` then counts down. An allocated server is
|
||||
told *which* identities to expect, and needs a **no-show timeout**: if a
|
||||
matched player never connects, the server must abandon and exit rather than
|
||||
sit idle burning the cost this design is trying to avoid.
|
||||
|
||||
### Keeping Docker and CI green
|
||||
|
||||
The existing gates must not regress. `make verify-phase6` builds the export,
|
||||
runs it in Compose, joins two headless clients and asserts both saw both
|
||||
goals and that the arena rotated between matches; `make verify-enet-integration`
|
||||
runs the source-build ENet matrix. Both depend on current behaviour:
|
||||
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
|
||||
assignment, and uses `--max-matches=2` to prove rotation.
|
||||
|
||||
The rule that keeps them passing: **every allocation feature is opt-in via a
|
||||
new `ServerConfig` flag whose default reproduces today's behaviour.** An
|
||||
assigned roster, a no-show timeout and result reporting must each be inert
|
||||
unless explicitly enabled. `ServerConfig` is built for exactly this — a flag
|
||||
declared once is parsed, validated, type-checked, config-file-backed and
|
||||
documented — and `tests/cases/` can cover the new parsing without a live
|
||||
server. A second Compose file should cover the allocated-match path rather
|
||||
than mutating the Phase 6 one, so the community-server model stays tested
|
||||
alongside the matchmade one.
|
||||
|
||||
### Open questions
|
||||
|
||||
- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this
|
||||
game-server lifecycle), Nomad, or direct cloud-API container starts. Not
|
||||
chosen. Agones is the strongest default because it models allocation,
|
||||
readiness and per-match lifetime natively.
|
||||
- **Port strategy** — port range per node versus one IP per match.
|
||||
- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 6–10 match
|
||||
processes per modern core and 150–250 MB RSS each. That estimate predates
|
||||
any allocation work and should be re-measured under real concurrency
|
||||
before it sizes a bill.
|
||||
- **Draining and deploys.** How a server version rolls out without killing
|
||||
matches in flight.
|
||||
|
||||
## Casual vs ranked
|
||||
|
||||
They are different playlists, not a difficulty toggle, and their rules
|
||||
|
||||
Reference in New Issue
Block a user