mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
docs: design per-match server autoscaling, with measured boot time
Casual/ranked queues need servers allocated per match and shut down afterwards, so cost is incurred only while a match runs - while the existing Docker and CI gates keep passing unchanged. Measured against the repo's own cosmicclash-server image rather than estimated: the runtime image is ~148 MB of content, and boot to the server_started line is ~870 ms on the container's own clock. That was taken under x86_64 emulation on an arm64 host, so it is a pessimistic bound and is recorded as one - it needs re-measuring on native Linux before it sets any timeout. Two findings that would each break a naive implementation, both hit while taking that measurement: - Godot's stdout is block-buffered off a TTY. A detached container logs nothing at all - server_started does not appear even after 35s - so an orchestrator readiness probe that greps the log hangs forever. Probe the UDP socket or flush explicitly. - --port defaults to 7777 and the Dockerfile hardcodes EXPOSE 7777/udp, so several matches cannot share a host without a port range or an address per match. Being UDP, L7 ingress routing does not apply. Also records the honest tension in 'only pay during a match': a server must listen before players connect, and image pull plus scheduling can dwarf 870 ms, so the recommendation is match-level scale-to-zero over a small warm node pool rather than node-level scale-to-zero. The rule for keeping verify-phase6 and verify-enet-integration green: every allocation feature is opt-in via a ServerConfig flag defaulting to current behaviour, with a second Compose file rather than mutating compose.phase6-smoke.yml.
This commit is contained in:
@@ -86,6 +86,11 @@ roughly 6–10 simultaneous match processes per modern core, 150–250 MB RSS pe
|
|||||||
process, and about 630 kbit/s upstream for a full six-player match; use those
|
process, and about 630 kbit/s upstream for a full six-player match; use those
|
||||||
as a starting point and monitor actual CPU, RSS, and egress.
|
as a starting point and monitor actual CPU, RSS, and egress.
|
||||||
|
|
||||||
|
Per-match autoscaling — allocating a server for one match and shutting it
|
||||||
|
down afterwards — is designed in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md)
|
||||||
|
and not yet implemented. The sizing numbers above predate that work and
|
||||||
|
should be re-measured under real concurrency before they size a bill.
|
||||||
|
|
||||||
This build must not be exposed to strangers yet. Slot reclaim is still keyed
|
This build must not be exposed to strangers yet. Slot reclaim is still keyed
|
||||||
by display name, so a player who knows a disconnected player's name can claim
|
by display name, so a player who knows a disconnected player's name can claim
|
||||||
their reserved slot. Phase 7 Steam-auth identity is the required fix. Local,
|
their reserved slot. Phase 7 Steam-auth identity is the required fix. Local,
|
||||||
|
|||||||
@@ -94,6 +94,108 @@ The server side needs less new work than it looks:
|
|||||||
slot, replacing the current first-come model.
|
slot, replacing the current first-come model.
|
||||||
- Client-side queue UI and the accept/decline flow.
|
- Client-side queue UI and the accept/decline flow.
|
||||||
|
|
||||||
|
## Server orchestration and autoscaling
|
||||||
|
|
||||||
|
Requirement: game servers scale horizontally and automatically, spin up fast
|
||||||
|
on demand, serve exactly one match, and shut down — so cost is incurred only
|
||||||
|
while a match is being played. Docker and the existing CI gates must keep
|
||||||
|
working unchanged.
|
||||||
|
|
||||||
|
### Why this is achievable: the server is already shaped for it
|
||||||
|
|
||||||
|
Two properties of the current build make per-match allocation practical
|
||||||
|
rather than aspirational:
|
||||||
|
|
||||||
|
- **The container is small.** The `server` image target is a slim
|
||||||
|
`ubuntu:24.04` runtime with three shared libraries and the exported
|
||||||
|
binary — about 148 MB of content, not the ~2.6 GB `godot-ci` build image.
|
||||||
|
Pulling it onto a fresh node is cheap.
|
||||||
|
- **Boot to listening is sub-second.** Measured on this repo's
|
||||||
|
`cosmicclash-server:latest`: **~870 ms** from container start to the
|
||||||
|
`server_started` log line, averaged over three runs, read from the
|
||||||
|
container's own clock. That measurement was taken under **x86_64 emulation
|
||||||
|
on an arm64 host**, so it is a pessimistic bound — native x86_64 Linux
|
||||||
|
should be faster. Re-measure on the real target before setting timeouts.
|
||||||
|
|
||||||
|
Combined with `--max-matches=1`, which already drains and `exit(0)`s after a
|
||||||
|
single match, the lifecycle the allocator needs mostly exists: start
|
||||||
|
container → serve one match → process exits → orchestrator reclaims.
|
||||||
|
|
||||||
|
### The cold-start tension, stated honestly
|
||||||
|
|
||||||
|
"Only pay during a match" and "a player never waits" are in tension. A server
|
||||||
|
must be listening *before* the matched players connect, so some cost always
|
||||||
|
precedes the match. Sub-second boot makes the gap small enough that a pure
|
||||||
|
scale-to-zero design is plausible — but the risk is not the container, it is
|
||||||
|
everything around it: image pull on a cold node, scheduler placement, and
|
||||||
|
network/port programming can each dwarf 870 ms.
|
||||||
|
|
||||||
|
Recommendation: **scale to zero at the node level is the wrong target; scale
|
||||||
|
to zero at the match level is the right one.** Keep a small warm pool of
|
||||||
|
nodes sized to the current queue depth, and start a per-match container on
|
||||||
|
demand within it. The per-match process genuinely exists only for the match;
|
||||||
|
the node pool absorbs the cold-start variance. Revisit only if measured
|
||||||
|
allocation latency on real infrastructure shows the warm pool is unnecessary.
|
||||||
|
|
||||||
|
### Findings that block a naive implementation
|
||||||
|
|
||||||
|
**Readiness cannot be detected from the log line.** Godot's stdout is
|
||||||
|
block-buffered when it is not attached to a TTY. Run the server image
|
||||||
|
detached without `-t` and `docker logs` shows **nothing at all** — the
|
||||||
|
`server_started` line does not appear even after 35 seconds, because the
|
||||||
|
buffer never flushes. An orchestrator readiness probe that greps for that
|
||||||
|
line will hang forever, and this was reproduced directly while measuring the
|
||||||
|
boot time above. Either probe the UDP socket instead, or make the server
|
||||||
|
flush explicitly. This also means container logs are not a reliable
|
||||||
|
observability channel for a short-lived match server; treat log shipping as
|
||||||
|
a separate problem.
|
||||||
|
|
||||||
|
**One fixed port per container does not scale on a shared host.** `--port`
|
||||||
|
defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`. Packing
|
||||||
|
several matches onto one node needs either a port range allocated per
|
||||||
|
container, or one address per container. This is a UDP service, so the usual
|
||||||
|
HTTP ingress/L7 routing answers do not apply — the allocator must hand the
|
||||||
|
client a concrete `host:port`.
|
||||||
|
|
||||||
|
**The match cannot start on a schedule the players do not control.** Today
|
||||||
|
the loop waits for `--min-players` then counts down. An allocated server is
|
||||||
|
told *which* identities to expect, and needs a **no-show timeout**: if a
|
||||||
|
matched player never connects, the server must abandon and exit rather than
|
||||||
|
sit idle burning the cost this design is trying to avoid.
|
||||||
|
|
||||||
|
### Keeping Docker and CI green
|
||||||
|
|
||||||
|
The existing gates must not regress. `make verify-phase6` builds the export,
|
||||||
|
runs it in Compose, joins two headless clients and asserts both saw both
|
||||||
|
goals and that the arena rotated between matches; `make verify-enet-integration`
|
||||||
|
runs the source-build ENet matrix. Both depend on current behaviour:
|
||||||
|
`compose.phase6-smoke.yml` hardcodes `--port=7777`, relies on first-come slot
|
||||||
|
assignment, and uses `--max-matches=2` to prove rotation.
|
||||||
|
|
||||||
|
The rule that keeps them passing: **every allocation feature is opt-in via a
|
||||||
|
new `ServerConfig` flag whose default reproduces today's behaviour.** An
|
||||||
|
assigned roster, a no-show timeout and result reporting must each be inert
|
||||||
|
unless explicitly enabled. `ServerConfig` is built for exactly this — a flag
|
||||||
|
declared once is parsed, validated, type-checked, config-file-backed and
|
||||||
|
documented — and `tests/cases/` can cover the new parsing without a live
|
||||||
|
server. A second Compose file should cover the allocated-match path rather
|
||||||
|
than mutating the Phase 6 one, so the community-server model stays tested
|
||||||
|
alongside the matchmade one.
|
||||||
|
|
||||||
|
### Open questions
|
||||||
|
|
||||||
|
- **Orchestrator.** Kubernetes (with Agones, which exists for precisely this
|
||||||
|
game-server lifecycle), Nomad, or direct cloud-API container starts. Not
|
||||||
|
chosen. Agones is the strongest default because it models allocation,
|
||||||
|
readiness and per-match lifetime natively.
|
||||||
|
- **Port strategy** — port range per node versus one IP per match.
|
||||||
|
- **Bin-packing.** SERVER.md's Phase 1 sizing estimate is 6–10 match
|
||||||
|
processes per modern core and 150–250 MB RSS each. That estimate predates
|
||||||
|
any allocation work and should be re-measured under real concurrency
|
||||||
|
before it sizes a bill.
|
||||||
|
- **Draining and deploys.** How a server version rolls out without killing
|
||||||
|
matches in flight.
|
||||||
|
|
||||||
## Casual vs ranked
|
## Casual vs ranked
|
||||||
|
|
||||||
They are different playlists, not a difficulty toggle, and their rules
|
They are different playlists, not a difficulty toggle, and their rules
|
||||||
|
|||||||
@@ -51,6 +51,28 @@ spoofable identity is worse than no rating.
|
|||||||
- [ ] Casual and ranked playlist rulesets (backfill, bots, abandon penalties,
|
- [ ] Casual and ranked playlist rulesets (backfill, bots, abandon penalties,
|
||||||
arena restriction — see the comparison table in the design doc).
|
arena restriction — see the comparison table in the design doc).
|
||||||
|
|
||||||
|
Server orchestration (same phase — the servers must autoscale and bill only
|
||||||
|
for the duration of a match):
|
||||||
|
|
||||||
|
- [ ] Choose the orchestrator (Agones on Kubernetes is the recommended
|
||||||
|
default; it models allocation, readiness and per-match lifetime natively).
|
||||||
|
- [ ] Fix readiness detection. Godot's stdout is block-buffered off a TTY, so
|
||||||
|
`server_started` never appears in `docker logs` for a detached container —
|
||||||
|
a log-grep readiness probe hangs forever. Probe the UDP socket, or flush.
|
||||||
|
- [ ] Support more than one match per host: a per-container port from a range,
|
||||||
|
or one address per match. `--port` defaults to 7777 and the Dockerfile
|
||||||
|
hardcodes `EXPOSE 7777/udp`.
|
||||||
|
- [ ] Add a no-show timeout so an allocated server that never fills abandons
|
||||||
|
and exits instead of idling at cost.
|
||||||
|
- [ ] Re-measure boot-to-listening on native x86_64 Linux. The repo's current
|
||||||
|
figure is ~870 ms, measured under emulation on arm64 — a pessimistic bound.
|
||||||
|
- [ ] Re-measure the SERVER.md sizing estimate (6–10 processes/core,
|
||||||
|
150–250 MB RSS) under real concurrency before it sizes a bill.
|
||||||
|
- [ ] Keep `make verify-phase6` and `make verify-enet-integration` green:
|
||||||
|
every allocation feature is opt-in via a `ServerConfig` flag defaulting to
|
||||||
|
today's behaviour, with a second Compose file for the allocated path rather
|
||||||
|
than mutating `compose.phase6-smoke.yml`.
|
||||||
|
|
||||||
## Known issues to resolve before public hosting
|
## Known issues to resolve before public hosting
|
||||||
|
|
||||||
- [ ] Slot reclaim is currently keyed by display name, so someone can take a
|
- [ ] Slot reclaim is currently keyed by display name, so someone can take a
|
||||||
|
|||||||
Reference in New Issue
Block a user