The root blocker (issue #14). The worker bound the provider allocation
and stopped. Service.PublishRoster and store.SaveVerifiedAssignmentRoster
both existed, fully tested, with zero non-test callers, and the
production allocator configured neither a roster store nor a signing
key. Nothing ever wrote the assignments table.
The allocated supervisor fetches a non-empty roster before it launches
the game child, so every real allocation failed at that fetch: no match
could reach ASSIGNMENT_READY or accept a player. Existing tests seeded
assignments directly, which is exactly why the missing hand-off went
unnoticed.
The worker now builds one join authorisation per durable participant,
signs each with the active key, and publishes them. Participants are
read through the same query SaveVerifiedAssignmentRoster re-validates
against, so the allocator cannot construct a roster the persistence
boundary would reject. The manifest commits to a digest over the whole
roster, so a server cannot be handed a truncated roster whose surviving
entries are each individually valid.
Persist the provider endpoint on the allocation: it arrived on the
provider response and was never stored, so a worker crashing between
allocating and publishing had no endpoint to recover and would have
stranded the match permanently. Republishing is idempotent, so that
crash now simply retries.
cmd/allocator refuses to start without key material rather than running
an allocator that binds allocations and silently strands every match.
The k8s allocator Deployment mounts the same key set the Fleet does, and
both now take the JSON key map so a rotation can publish several.
New integration test drives the real worker through to the supervisor's
own roster read path without seeding the assignments table. Verified it
fails with "assignments = 0, want 2" when the publish step is removed.
Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.
observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.
Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.
Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
Godot's stdout is fully (block) buffered whenever it isn't attached to
a TTY -- true of every real deployment path this repo documents:
'docker run -d' (Docker's log driver presents a pipe), a plain
'docker run' without -d, and systemd's journal capture (also a pipe).
Confirmed directly, not from the existing gotcha note alone: a real
'docker run -d' container sat for 20+ seconds with 'docker logs'
showing nothing at all -- not even the startup line -- while the
process was confirmed alive and running (ps aux inside the container).
'docker stop' then killed it via SIGTERM (Godot has no SIGTERM hook)
without ever flushing that buffered output, losing it permanently
rather than merely delaying it.
This affects the already-shipped community server path today, not
just the not-yet-built Agones fleet path multiplayer-next.md's task
8.28 gotcha originally flagged this for -- SERVER.md's Docker AND
native-systemd instructions both route through this exact launcher
script, and journald's capture has the same non-TTY-pipe buffering
problem docker logs does.
Wrap the exec in 'stdbuf -oL -eL' (LD_PRELOAD-based line buffering,
touches no binary) when available, falling back to the unwrapped exec
otherwise so a minimal image without GNU coreutils still starts.
Re-verified the same failing scenario against the actual launcher
script in a real image: the startup line now appears within 3s of a
genuinely detached 'docker run -d'. Re-ran the full make verify-phase6
gate end to end afterward to confirm no regression: both arenas
rotated, both clients observed both goals, clean teardown.
Josh Creek