Both playlists shared one hash and sorted set, causing two independent
failures.
Starvation: Snapshot performed an unbounded ZRANGEBYSCORE and HMGET,
decoded the whole queue, and the matcher then truncated to its candidate
limit *before* filtering by playlist. A large casual prefix could
therefore leave the ranked worker with zero candidates indefinitely even
while ranked tickets were queued further down the set.
Mutual erasure: each matcher captured only its own playlist as the
durable source, but Rebuild replaced the shared keys, so a casual repair
wiped ranked projections and vice versa.
Namespace the keys per playlist, push the limit into Redis (LIMIT 0 N)
so reads no longer scale with total queue depth, and scope Rebuild to
one namespace. Rebuild now rejects a candidate whose playlist does not
match the namespace, which would reintroduce the starvation. Upsert
derives the namespace from the candidate; Remove takes the playlist,
since a ticket ID alone no longer identifies its namespace.
Add tests for a 300-deep casual backlog not starving ranked, for neither
playlist's rebuild erasing the other, and for the limit being applied
without losing enqueue ordering.
The candidate projection selected only from queue_tickets and its scan
never set Candidate.Rating, so every PostgreSQL-sourced ranked candidate
arrived with Go's zero value. Rating tolerance, selection scoring and
team partitioning all read that field, so ranked matchmaking treated a
900-rated player as identical to a 2100-rated one. Unit tests missed it
because they construct candidates with ratings already populated.
Join the ratings table, defaulting to domain.GlickoInitialRating for a
player with no ratings row yet -- a genuinely new profile, matching the
column default.
Fix the same defect on the Redis path too, which is reached differently:
the projection is seeded from the candidate CreateQueueTicket builds,
not from the candidate query, and that candidate also left Rating unset.
Resolve the rating inside the enqueue transaction so both projections
agree on one authoritative value. The rating is never client-supplied.
Add a store-backed test with deliberately distant ratings (900 vs 2100)
plus an unrated player, asserting both projections and that the spread
survives. Verified it fails without the fix.
ApplyInitialConnectPlan wrote a payload of {match_id,state,action},
omitting event, revision, resource_id, occurred_at and player_ids --
every field deliverStateOutboxEvent requires. Delivery rejected the row,
dispatch returned on the first error so it was never acknowledged, and
because reads are ordered oldest-first it was retried ahead of every
later state_changed event on every 100ms poll. One initial-connect
transition therefore blocked lifecycle delivery for all matches, not
just its own.
Two independent fixes, since either alone leaves the system fragile:
Build envelopes through one validating helper (MarshalOutboxEnvelope)
and convert all five writers to it. A writer that omits a required
field now fails its own transaction instead of committing a row that
can only ever poison the queue. The helper takes revision as int64 so
the -1 "nothing matched" sentinel some CTEs return surfaces as an error
rather than wrapping to a huge uint64.
Make dispatch resilient regardless: a delivery failure is now counted
against that row and the batch continues, with the row dead-lettered
after MaxOutboxDeliveryAttempts so a poison event degrades to one lost
notification instead of a stalled queue. Ordering within an aggregate
is still honoured -- later events of a failed match are deferred, so no
client observes that match's newer state before its older state. An ack
failure still stops the batch, being a database rather than a payload
problem.
Initial-connect events now address every participant, not just the
connected ones: a no-show needs to learn their ticket was failed and a
penalty applied.
Closes part of the 'live Redis failover' gap in §8.46, found by
reproducing a genuine Redis outage (not just an empty/partial cache)
against CandidateProjection.Snapshot with a killed miniredis instance.
CandidateProjection.Snapshot funnelled two different situations into
the same code path: the index erroring outright (Redis unreachable)
and the index coming back empty (ambiguous — a genuinely empty queue,
or a lost keyspace). Both went through Repair, which itself calls
Index.Rebuild — a second Redis round-trip that fails for exactly the
same reason the first one did. The result: a real Redis outage, or
the window during a failover, made Snapshot fail outright even though
PostgreSQL — the documented authoritative source everywhere
(RedisCandidateIndex's own comment, cmd/matcher, cmd/control-plane's
--redis-addr help text all call it a rebuildable/optional
acceleration layer) — was completely healthy. Matchmaking would stop
entirely on a Redis outage despite the architecture explicitly not
requiring that.
Snapshot now falls back to serving Source (PostgreSQL) directly
whenever the index errors OR comes back empty, and only best-effort
attempts to repopulate Redis afterward — that attempt's outcome is
deliberately ignored, since a caller must never be denied service
just because the opportunistic rebuild also hit the same down Redis.
Snapshot still fails when Source itself is unavailable; the fallback
is not unconditional.
Verified: reproduced the bug first (killed-miniredis Snapshot call
failed even though Source was healthy), then fixed it. go build/vet
clean; all pre-existing store-package tests pass unmodified,
including the two live-redis:7-alpine-container tests
(TestRealRedisCandidateIndexUpsertSnapshotRemove,
TestRealRedisCandidateProjectionRepairsAfterFlush, run against a real
container and torn down after). Two new tests cover the fallback
directly (killed miniredis, Source still served, exactly one Source
call) and that the fallback is not unconditional (both Redis and
Source down still fails). Full go test ./... -race clean across
every server package.
Remaining: live matcher-worker-under-load-during-failover integration,
i.e. running the actual matcher process against a real Redis that
goes down mid-run under concurrent load, not just this unit-level
reproduction.
Closes the 'concurrent proposal-recovery expiry races' gap noted in
§8.46. GetProposal (read-side recovery) and RespondToProposal both
run the identical expiry-advance SQL in their own transaction, so any
number of them can observe the same past-expiry proposal at once —
this had never been exercised concurrently, only sequentially (the
existing late-response test drives one call at a time).
TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
races 8 concurrent GetProposal/RespondToProposal calls, each with a
distinct 'now' past the proposal window, against one proposal and
asserts: EXPIRED lands on the proposal and both tickets exactly once,
a PROPOSAL_TIMEOUT penalty lands exactly once per offending player
(not once per racing transaction), and no idempotency row survives a
closed-proposal response. The design already defends against this —
ProposalParticipantExpireSQL only ever flips a still-PENDING row
once, so a losing racer's 'now' can't match
recordProposalTimeoutCooldowns' responded_at filter — this test is
what actually proves that holds under real concurrent load rather
than by inspection.
Verified: real postgres:17-alpine container, go test -tags
integration ./store/... -run
TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
-race -count=3 clean; full -tags integration ./store/... -race run
clean; full non-integration go build/vet/test -race clean across
every server package; container removed after the run.
The just-landed signed workload token embedded (allocation_id, match_id,
server_id) as claims. That doesn't actually work for its intended delivery
channel: the token is meant to be requested as a GameServerAllocation
annotation in the SAME request that asks Agones to pick a server, so at mint
time the allocator knows allocation_id (it generates it) but not yet which
server_id Agones will return -- server_id only exists in Agones's response,
after the annotation request has already been sent. Embedding it was simply
not possible for the real caller this was built for; only the (allocator ->
signed_token) unit tests and hand-constructed integration tests happened to
supply it directly, masking the gap.
Fixes it by having the token bind only allocation_id (the one identifier
actually known at mint time) plus expiry. match_id/server_id are resolved at
verify time from the durable allocations table via the new
store.AllocationBindingByAllocationID, keyed by allocation_id -- which the
allocator already records immediately after Agones responds. This is
strictly stronger, not just a workaround: a caller can no longer claim any
match/server pairing at all, even one that happens to be internally
consistent -- the binding returned is entirely durable-record-derived.
Verified: server/workload's unit tests updated for the new two-field claim
shape; server/api's Postgres integration suite gains
TestWorkloadVerifierFromSignedTokenNeverTrustsCallerSuppliedBinding (two
distinct real allocations each resolve to their own, and only their own,
match/server pairing) replacing the now-inapplicable mismatched-triple test.
Full `go build ./... && go vet ./... && gofmt -l . && go test ./... -race`
and `go test -tags integration ./... -race` both clean; the api integration
suite re-run 3x clean against a live postgres:17-alpine container.
Josh Creek