Both playlists shared one hash and sorted set, causing two independent
failures.
Starvation: Snapshot performed an unbounded ZRANGEBYSCORE and HMGET,
decoded the whole queue, and the matcher then truncated to its candidate
limit *before* filtering by playlist. A large casual prefix could
therefore leave the ranked worker with zero candidates indefinitely even
while ranked tickets were queued further down the set.
Mutual erasure: each matcher captured only its own playlist as the
durable source, but Rebuild replaced the shared keys, so a casual repair
wiped ranked projections and vice versa.
Namespace the keys per playlist, push the limit into Redis (LIMIT 0 N)
so reads no longer scale with total queue depth, and scope Rebuild to
one namespace. Rebuild now rejects a candidate whose playlist does not
match the namespace, which would reintroduce the starvation. Upsert
derives the namespace from the candidate; Remove takes the playlist,
since a ticket ID alone no longer identifies its namespace.
Add tests for a 300-deep casual backlog not starving ranked, for neither
playlist's rebuild erasing the other, and for the limit being applied
without losing enqueue ordering.
Closes §8.43's 'version-mismatch-specific client messaging' gap. Per
the user's explicit go-ahead to design new server behavior for this
(rather than only wiring up something that already existed, the
pattern every other fix this session followed): before this, there
was no server-side protocol rejection at all. queue_create accepted
any protocol_version >= 1 unconditionally, so an outdated client could
only ever discover a mismatch by waiting in the queue forever
unmatched -- the matcher's own compatibility check requires every
formed player to share an identical protocol_version -- with no error
and no explanation given to the player.
Server: Service.MinProtocolVersion (opt-in, zero by default so every
existing caller keeps accepting protocol_version 1 unconditionally)
rejects a below-floor queue_create with 426 Upgrade Required /
client_outdated before the request ever reaches the candidate
provider. Wired via cmd/control-plane's new --min-protocol-version
flag (validated non-negative at startup).
Client: ControlPlaneClient recognises HTTPClient.RESPONSE_UPGRADE_REQUIRED
on queue_create specifically and sets a distinct 'Your client is out
of date -- please update to continue searching' message instead of
the server's raw generic error string, and clears _last_queue_create
so can_retry_queue_create() never offers 'Retry Search' for a failure
that retrying with the same build can never fix.
Verified: go build/vet/test -race clean across every server package.
TestQueueCreateEnforcesMinProtocolVersion covers below-floor rejection
(candidate provider never reached), exactly-at-floor acceptance, and
the error body naming client_outdated; TestQueueCreateMinProtocolVersionZeroIsDisabled
proves the opt-in default doesn't change behavior for every existing
caller. Godot: test_outdated_client_receives_a_distinct_message_and_no_retry_offer
proves the distinct message and suppressed retry offer. Full Godot
suite (217/217, 0 failed, no crash), full make verify-multiplayer-local
gate, zero new crash reports.
Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.
observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.
Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.
Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
Same discovery pattern as ResultSubmitter/SessionIssuer, one level
deeper: api.Service.rankedProfile and .profile both only ever read
from an in-memory RankedProfiles map with no durable-store equivalent
at all -- not "adapter exists but unwired" this time, there was no
adapter. Every real request to GET /v1/profile/ranked or /api/v1/profile
always 404'd regardless of a player's actual rating.
Add RankedProfileProvider (an interface, not a struct-literal adapter
this time) and store.PostgresRankedProfiles reading the ratings table;
Service.rankedProfileFor prefers it when set and falls back to the
map otherwise, so every existing test/direct Service literal keeps
compiling and passing unchanged. A missing ratings row maps to the
exact same (zero value, false, nil) the map lookup already produced,
preserving existing not-found semantics rather than reinterpreting
them. LastSeasonID/SeasonHistory are deliberately left unset -- the
ratings table has no season pointer, and reconstructing history needs
its own query and display semantics, not bundled in here speculatively.
Wired into both cmd/control-plane and cmd/testkit-api. Verified
against real PostgreSQL via curl: a fresh identity's ranked profile
correctly 404s through the real adapter (same behavior as before,
now for a real reason instead of an empty map).
Wire the same Service.Log hook added for the server register/result
routes into queue create/heartbeat/cancel and proposal accept/decline:
log the resulting state on success (queue_create, queue_heartbeat,
queue_cancel, proposal_response) or 'rejected' on a domain error,
using only the ticket/proposal ID and outcome -- never the domain
error text itself, which isn't documented as credential-free.
Read-only routes (queue GET, proposal GET, assignment fetch) and the
early availability/not-found rejections that return before reaching
the domain call are deliberately not logged in this pass.
Covered by a new end-to-end test driving real create/heartbeat/cancel
and an accept followed by a stale-revision accept (fenced for real by
the domain layer behind proposalBackendSpy, unlike the dumb queue
spy), asserting the exact sequence of events logged.
server/observability existed fully unit-tested but was imported by
nothing outside its own package -- no HTTP handler ever called it, so
its credential redaction protected zero real log output. Wire it into
Service via an optional Log field (nil-safe, so every existing Service
literal keeps compiling unchanged) and call it from the two
workload-authenticated server routes -- register and result -- at
every outcome: unauthorized, rejected, conflict and success. Wire
cmd/control-plane to actually emit those events as JSON lines on
stderr.
Add a secret canary test that drives both routes end to end with
realistic bearer-token and result-nonce values and asserts neither
literal secret appears anywhere in what Service.Log actually received
-- a stronger claim than the existing observability unit test, which
only proves redact() strips a synthetic value under a denylisted key
name. redact() is still key-name-based, not content-based: a future
call site that logs a secret under an unlisted key name would not be
caught by this test or by redact() itself, only by the same discipline
applied here of never putting raw request/token bytes into Fields.
Queue, proposal and assignment mutation routes are not wired yet.
Add POST /v1/servers/{id}/register (and its /api/v1 contract alias),
authenticated by the same workload binding as the result route. A
game server reports its protocol version and image digest and asks
to advance ALLOCATING -> PROCESS_READY -> ASSIGNMENT_READY; the store
boundary (AdvanceServerRegistration) does this as one idempotent
SERIALIZABLE transaction that also advances every participant's queue
ticket, and gates the final transition on every participant having a
live, unexpired assignment.
Adversarial review of the surrounding routing turned up a pre-existing
bug: contractServerMutation rejected any path containing '/', so the
already-documented /api/v1/servers/{id}/result route (and this new
/register route) 404'd for every real caller despite being declared
in the OpenAPI contract. Fix it to delegate shape validation to
serverMutation, matching how contractQueueMutation handles its own
two-segment paths, and add a regression test covering both contract
routes end to end.
Josh Creek