Closes part of the 'live Redis failover' gap in §8.46, found by
reproducing a genuine Redis outage (not just an empty/partial cache)
against CandidateProjection.Snapshot with a killed miniredis instance.
CandidateProjection.Snapshot funnelled two different situations into
the same code path: the index erroring outright (Redis unreachable)
and the index coming back empty (ambiguous — a genuinely empty queue,
or a lost keyspace). Both went through Repair, which itself calls
Index.Rebuild — a second Redis round-trip that fails for exactly the
same reason the first one did. The result: a real Redis outage, or
the window during a failover, made Snapshot fail outright even though
PostgreSQL — the documented authoritative source everywhere
(RedisCandidateIndex's own comment, cmd/matcher, cmd/control-plane's
--redis-addr help text all call it a rebuildable/optional
acceleration layer) — was completely healthy. Matchmaking would stop
entirely on a Redis outage despite the architecture explicitly not
requiring that.
Snapshot now falls back to serving Source (PostgreSQL) directly
whenever the index errors OR comes back empty, and only best-effort
attempts to repopulate Redis afterward — that attempt's outcome is
deliberately ignored, since a caller must never be denied service
just because the opportunistic rebuild also hit the same down Redis.
Snapshot still fails when Source itself is unavailable; the fallback
is not unconditional.
Verified: reproduced the bug first (killed-miniredis Snapshot call
failed even though Source was healthy), then fixed it. go build/vet
clean; all pre-existing store-package tests pass unmodified,
including the two live-redis:7-alpine-container tests
(TestRealRedisCandidateIndexUpsertSnapshotRemove,
TestRealRedisCandidateProjectionRepairsAfterFlush, run against a real
container and torn down after). Two new tests cover the fallback
directly (killed miniredis, Source still served, exactly one Source
call) and that the fallback is not unconditional (both Redis and
Source down still fails). Full go test ./... -race clean across
every server package.
Remaining: live matcher-worker-under-load-during-failover integration,
i.e. running the actual matcher process against a real Redis that
goes down mid-run under concurrent load, not just this unit-level
reproduction.
Closes the 'innocent-ticket restoration' gap noted in §8.20 and found
by re-examining §8.16's matcher worker. domain.FormFromQueue's anchor
is always the single oldest candidate, deterministically. If
domain.PrepareProposal then rejected that exact formation for a
reason specific to those particular players — mismatched protocol,
incomplete ranked identity metadata, a duplicate-SteamID pair, ranked
admission generally — RunOnce returned immediately and the next
matcher interval reproduced the identical formation and failed again.
Forever: nothing in the queue ever changes, so the same doomed anchor
group would be retried every single pass, permanently head-of-line-
blocking every other waiting player behind it too, not just the
players actually at fault. This is worse than the already-fixed
no-common-region crash-loop (§8.16) — that one killed the process;
this one fails silently and just never matches anyone again.
Two changes, both required together:
1. RunOnce now excludes a failed formation's players and retries
with the remaining candidate pool, bounded to 8 attempts per pass.
A batch with no viable formation at all (the pre-existing
no-common-region case) still returns immediately, since retrying
that can't help.
2. That fix was inert without a second one: RunOnce was asking
Source for exactly w.Size candidates, so after excluding one
failed formation's players there was nothing left to retry
against. domain.SelectCandidates was always designed to search a
larger pool (anchor plus an arbitrary remainder, widening through
it) — the call site just never gave it one. RunOnce now requests
up to 10x w.Size, capped at 200.
Verified: go build/vet/test -race clean across every server package.
Three new matcher tests cover the exclusion retry (an 8-candidate
batch whose permanently-doomed oldest 4 still lets the remaining 4
form and claim, correctly excluding the doomed players from the
claimed ticket set), that exhausting every attempt surfaces the last
real error rather than a silent false/nil, and that Source is
actually asked for more than w.Size candidates — a regression guard
for exactly the companion bug above. All six pre-existing worker
tests still pass unmodified, confirming the fix preserves every prior
guarantee (mixed-playlist/duplicate-identity rejection, incomplete
batch handling, durable claim failure propagation, Run's existing
per-pass-error survival).
Closes the 'concurrent proposal-recovery expiry races' gap noted in
§8.46. GetProposal (read-side recovery) and RespondToProposal both
run the identical expiry-advance SQL in their own transaction, so any
number of them can observe the same past-expiry proposal at once —
this had never been exercised concurrently, only sequentially (the
existing late-response test drives one call at a time).
TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
races 8 concurrent GetProposal/RespondToProposal calls, each with a
distinct 'now' past the proposal window, against one proposal and
asserts: EXPIRED lands on the proposal and both tickets exactly once,
a PROPOSAL_TIMEOUT penalty lands exactly once per offending player
(not once per racing transaction), and no idempotency row survives a
closed-proposal response. The design already defends against this —
ProposalParticipantExpireSQL only ever flips a still-PENDING row
once, so a losing racer's 'now' can't match
recordProposalTimeoutCooldowns' responded_at filter — this test is
what actually proves that holds under real concurrent load rather
than by inspection.
Verified: real postgres:17-alpine container, go test -tags
integration ./store/... -run
TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
-race -count=3 clean; full -tags integration ./store/... -race run
clean; full non-integration go build/vet/test -race clean across
every server package; container removed after the run.
Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.
observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.
Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.
Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
Josh Creek