The root blocker (issue #14). The worker bound the provider allocation
and stopped. Service.PublishRoster and store.SaveVerifiedAssignmentRoster
both existed, fully tested, with zero non-test callers, and the
production allocator configured neither a roster store nor a signing
key. Nothing ever wrote the assignments table.
The allocated supervisor fetches a non-empty roster before it launches
the game child, so every real allocation failed at that fetch: no match
could reach ASSIGNMENT_READY or accept a player. Existing tests seeded
assignments directly, which is exactly why the missing hand-off went
unnoticed.
The worker now builds one join authorisation per durable participant,
signs each with the active key, and publishes them. Participants are
read through the same query SaveVerifiedAssignmentRoster re-validates
against, so the allocator cannot construct a roster the persistence
boundary would reject. The manifest commits to a digest over the whole
roster, so a server cannot be handed a truncated roster whose surviving
entries are each individually valid.
Persist the provider endpoint on the allocation: it arrived on the
provider response and was never stored, so a worker crashing between
allocating and publishing had no endpoint to recover and would have
stranded the match permanently. Republishing is idempotent, so that
crash now simply retries.
cmd/allocator refuses to start without key material rather than running
an allocator that binds allocations and silently strands every match.
The k8s allocator Deployment mounts the same key set the Fleet does, and
both now take the JSON key map so a rotation can publish several.
New integration test drives the real worker through to the supervisor's
own roster read path without seeding the assignments table. Verified it
fails with "assignments = 0, want 2" when the publish step is removed.
Prerequisite for wiring the allocator to publish rosters. The signing
key is a shared HMAC secret mounted into both the allocator and the
allocated game server; without a key ID, rotating it would invalidate
every authorisation already issued for an in-flight match, because a
server holding only the new key cannot verify a token signed with the
old one.
Add KeyID to JoinAuthorisation and append it to the canonical claim
bytes, so it is covered by the signature and cannot be repointed at a
different key than the one that actually signed. Allocated servers now
hold a set of currently-valid keys and select by ID: a rotation
publishes the new key alongside the old, and the old is dropped once no
live match can still reference it.
The key file becomes a JSON map of key ID to base64 key. A file of raw
key bytes is still accepted as a single key under the empty ID, which is
what an unrotated deployment and the kind fixture use.
Game/scripts/match_net.gd builds the canonical bytes independently, so
it changes in lockstep; the cross-language golden token in
test_match_net.gd is regenerated from the Go implementation and now
carries a key ID. Added tests cover accepting either key mid-rotation,
rejecting a retired key ID, and rejecting a token whose key ID was
swapped to name a key the server does hold.
Go suite and 223 Godot tests pass.
test_contracts.py required operation ID `recordPlayerConnected`, but
openapi.json names that endpoint `claimPlayerConnection` — the accurate
name, since POST /servers/{id}/connect claims a connection lease and
returns a generation. Align the test on the document and assert the set
difference, so a future mismatch names the missing operation instead of
reporting "False is not true".
test_observability_manifests.py copied only two of the four files the
checker reads, so it died on a missing kustomization.yaml before ever
reaching the mutated namespace. Copy the full fixture, split the
namespace and scrape-path mutations into separate cases so either
defect produces its own diagnostic, and add an unmutated-copy case so a
broken fixture can't make the mutation cases pass vacuously.
Neither suite was invoked by any Make target or workflow, which is why
both could sit red. Add them, plus test_threat_model.py, to
verify_multiplayer_local.sh.
Every existing test of the client/control-plane boundary is either a
Go unit test with a mocked HTTP layer or a GDScript unit test with no
network at all (multiplayer-next.md 8.40's own evidence names "live
multi-process control-plane/game verification" as remaining). Nothing
before this actually ran the real compiled Go binary, a real
PostgreSQL instance, and a real headless Godot process talking real
HTTP to each other -- and it immediately found a real bug (previous
commit).
server/cmd/testkit-api is a new, deliberately separate, clearly-marked
test-only binary wired identically to cmd/control-plane except for
SteamLogin: cmd/control-plane has no way to authenticate against a
real Steam Web API from this sandbox (task 8.7's own documented
blocker), so testkit-api accepts any non-empty ticket string and
derives a deterministic identity instead. This bypass is confined to
its own binary -- never a flag on cmd/control-plane, never referenced
by any Dockerfile stage or Kubernetes manifest -- specifically so it
can't become a footgun on the real one.
Game/tests/control_plane_smoke.gd drives the real ControlPlaneClient
autoload through login -> queue_create -> heartbeat against a real
server and prints SMOKE PASS/FAIL, matching the existing net_smoke.gd
convention. scripts/verify_control_plane_integration.sh orchestrates
both sides (real postgres:17-alpine, the built testkit-api binary, the
Godot client) end to end.
Two real bugs surfaced building this, both fixed and re-verified, not
just the target bug: the smoke script's own use of `go run` left a
zombie process that survived cleanup and squatting on its port
corrupted the NEXT run with a misleading "http=401 unauthorized" (now
builds and runs a real binary directly, plus a belt-and-suspenders
port-kill in cleanup); and calling heartbeat() synchronously from
within a request_succeeded handler produced a spurious "Busy" because
ControlPlaneClient's own internal resync (see previous commit) was
still in flight -- the test now waits for ControlPlaneClient to go
idle via a real Timer (call_deferred alone floods the message queue
without ever yielding a frame for the in-flight request to complete).
Verified stable across 3 consecutive full runs: real PostgreSQL
container up, migrations applied, testkit-api built and started, real
headless Godot client round-tripping login/queue/heartbeat, clean
teardown with no leftover processes, containers, or bound ports each
time.
Every existing RedisCandidateIndex test runs against miniredis -- a
from-scratch Go reimplementation of the Redis command set, not real
Redis's own float64 score encoding, TTL/expiry, or RESP behavior.
Add an opt-in integration suite (mirroring postgres_integration_test.go's
pattern: //go:build integration, COSMIC_CLASH_REDIS_ADDR-gated) plus
scripts/run_redis_integration.sh against a disposable redis:7-alpine
container, covering:
- upsert/snapshot/remove against a real server
- a real TTL actually waited out (not miniredis's manual
FastForward), proving expiry really happens on the wire
- the documented 'Redis restart or lost keyspace' repair path
exercised against an actual FLUSHALL, not a simulated empty map,
including that the repair actually persists back to Redis (a
second snapshot reads it without a second durable-source call)
Verified stable across 3 runs with -race against a real container.
Josh Creek