The Steam adapter took --steam-publisher-key/--steam-app-id and the
matching env vars, but no manifest supplied them, so a deployed control
plane would have kept sign-in returning 503 even once the App ID from
#15 arrived -- that issue would have unblocked nothing on landing.
Mount them from a new cosmic-clash-steam Secret, into the control-plane
Deployment alone: the publisher key is issued to us, never to a client,
and no other workload (least of all a game server) has any use for it. A
manifest test asserts both the wiring and that the Secret appears in no
other manifest; verified it fails in both directions.
Both keys are optional, so the Deployment still rolls out before the App
ID exists and sign-in simply stays 503.
Also correct task rows this branch made stale: 7.4 (durable ban storage
landed), 8.7 (adapter, bans and secret store landed), 8.39 (cross-replica
fan-out landed), and 8.5's migration range, which stopped at 0013.
Move the branch review into docs/ with a header marking it a point-in-time
artefact -- all thirteen findings are addressed, and its present tense
would otherwise read as current behaviour.
Record gotcha 52: the integration scripts use `docker run --rm`, which
reclaims the container but not its anonymous volume. Sixty-four of them,
~4 GB, accumulated during this session until PostgreSQL stopped starting
-- surfacing only as the script's own readiness timeout, not as a disk
error. That is the real cause behind the "Docker storage exhausted
locally" notes those rows carried.
The Kubernetes base deployed a control-plane image the Dockerfile never
built -- cmd/control-plane was absent from the Go build stage and no
target existed -- while the Dockerfile built a matcher image no manifest
ever deployed. Applying the checked-in base therefore could not produce
the advertised topology: one required workload had no repository-defined
artifact, and nothing consumed queued tickets. Tickets could be created
but never became proposals.
Add the production control-plane build and image target, explicitly not
the testkit-api target, which injects a fake login accepting any ticket.
Add casual and ranked matcher Deployments as separate workloads: they
have different match sizes, and separating them keeps a ranked backlog
from delaying casual formation. One replica each -- CreateProposal's
SKIP LOCKED fences make more replicas safe, but they would halve the
candidate pool each worker sees per poll and worsen formation for no
throughput gain at this scale. Their PDB uses maxUnavailable, since
minAvailable against a single replica blocks node drains outright.
Also fix both blocked traffic directions. No ingress policy admitted
UDP/7777 to game-server pods, so an allocated server was unreachable
from the internet under the namespace-wide default deny. And
control-plane ingress admitted only edge-gateway pods, so roster fetch,
registration, connection receipts, shutdown and result submission from
game servers were dropped even inside the cluster, despite their egress
being permitted. The default deny stays.
Manifest tests now assert every required role is deployed, both
playlists are scheduled, every referenced image maps to a real
Dockerfile target, and both traffic directions are permitted. Each was
verified to fail against the defect it covers. The control-plane image
was built and run to confirm the target works.
The root blocker (issue #14). The worker bound the provider allocation
and stopped. Service.PublishRoster and store.SaveVerifiedAssignmentRoster
both existed, fully tested, with zero non-test callers, and the
production allocator configured neither a roster store nor a signing
key. Nothing ever wrote the assignments table.
The allocated supervisor fetches a non-empty roster before it launches
the game child, so every real allocation failed at that fetch: no match
could reach ASSIGNMENT_READY or accept a player. Existing tests seeded
assignments directly, which is exactly why the missing hand-off went
unnoticed.
The worker now builds one join authorisation per durable participant,
signs each with the active key, and publishes them. Participants are
read through the same query SaveVerifiedAssignmentRoster re-validates
against, so the allocator cannot construct a roster the persistence
boundary would reject. The manifest commits to a digest over the whole
roster, so a server cannot be handed a truncated roster whose surviving
entries are each individually valid.
Persist the provider endpoint on the allocation: it arrived on the
provider response and was never stored, so a worker crashing between
allocating and publishing had no endpoint to recover and would have
stranded the match permanently. Republishing is idempotent, so that
crash now simply retries.
cmd/allocator refuses to start without key material rather than running
an allocator that binds allocations and silently strands every match.
The k8s allocator Deployment mounts the same key set the Fleet does, and
both now take the JSON key map so a rotation can publish several.
New integration test drives the real worker through to the supervisor's
own roster read path without seeding the assignments table. Verified it
fails with "assignments = 0, want 2" when the publish step is removed.
Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.
observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.
Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.
Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
Godot's stdout is fully (block) buffered whenever it isn't attached to
a TTY -- true of every real deployment path this repo documents:
'docker run -d' (Docker's log driver presents a pipe), a plain
'docker run' without -d, and systemd's journal capture (also a pipe).
Confirmed directly, not from the existing gotcha note alone: a real
'docker run -d' container sat for 20+ seconds with 'docker logs'
showing nothing at all -- not even the startup line -- while the
process was confirmed alive and running (ps aux inside the container).
'docker stop' then killed it via SIGTERM (Godot has no SIGTERM hook)
without ever flushing that buffered output, losing it permanently
rather than merely delaying it.
This affects the already-shipped community server path today, not
just the not-yet-built Agones fleet path multiplayer-next.md's task
8.28 gotcha originally flagged this for -- SERVER.md's Docker AND
native-systemd instructions both route through this exact launcher
script, and journald's capture has the same non-TTY-pipe buffering
problem docker logs does.
Wrap the exec in 'stdbuf -oL -eL' (LD_PRELOAD-based line buffering,
touches no binary) when available, falling back to the unwrapped exec
otherwise so a minimal image without GNU coreutils still starts.
Re-verified the same failing scenario against the actual launcher
script in a real image: the startup line now appears within 3s of a
genuinely detached 'docker run -d'. Re-ran the full make verify-phase6
gate end to end afterward to confirm no regression: both arenas
rotated, both clients observed both goals, clean teardown.
Josh Creek