Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.
observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.
Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.
Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
Godot's stdout is fully (block) buffered whenever it isn't attached to
a TTY -- true of every real deployment path this repo documents:
'docker run -d' (Docker's log driver presents a pipe), a plain
'docker run' without -d, and systemd's journal capture (also a pipe).
Confirmed directly, not from the existing gotcha note alone: a real
'docker run -d' container sat for 20+ seconds with 'docker logs'
showing nothing at all -- not even the startup line -- while the
process was confirmed alive and running (ps aux inside the container).
'docker stop' then killed it via SIGTERM (Godot has no SIGTERM hook)
without ever flushing that buffered output, losing it permanently
rather than merely delaying it.
This affects the already-shipped community server path today, not
just the not-yet-built Agones fleet path multiplayer-next.md's task
8.28 gotcha originally flagged this for -- SERVER.md's Docker AND
native-systemd instructions both route through this exact launcher
script, and journald's capture has the same non-TTY-pipe buffering
problem docker logs does.
Wrap the exec in 'stdbuf -oL -eL' (LD_PRELOAD-based line buffering,
touches no binary) when available, falling back to the unwrapped exec
otherwise so a minimal image without GNU coreutils still starts.
Re-verified the same failing scenario against the actual launcher
script in a real image: the startup line now appears within 3s of a
genuinely detached 'docker run -d'. Re-ran the full make verify-phase6
gate end to end afterward to confirm no regression: both arenas
rotated, both clients observed both goals, clean teardown.
Josh Creek