Both idempotency paths returned a bare fmt.Errorf, and writeDomainError
maps anything it does not recognise to its 422 "invalid_request"
default. So reusing a key with a different payload answered 422 where
openapi.json declares 409 and state-transitions.json requires
"reject_conflict_without_state_change".
That is the difference between "your request was malformed" and "that
key is taken". A client acting on 422 would rewrite a request that was
never wrong, and the 409 branch of every generated client was
unreachable.
Wrap domain.ErrConflict on both the create and mutate paths, and add an
integration test covering identical replay and conflicting reuse.
Pre-existing: both bare errors are unchanged from 089c127c, which is why
verify-allocated-compose failed in CI before this branch's work as well.
Found only after adding the diagnostics in 432e5a11 and fc2f5c86 --
until then the assertion aborted silently and three CI runs reported
nothing but "make: *** Error 1".
Two things that made the integration gate untrustworthy.
The retry budget was too small for expected contention.
TestPostgreSQLConcurrentIdenticalResultSubmission fires five identical
concurrent submissions and requires all five to succeed; it failed 4 runs
in 20. The error was retryable and retries did fire -- three attempts
simply was not enough. Contention here is normal rather than
exceptional: several game servers can submit results, and several
matchers can claim candidates, against the same rows at once. Raised to
five attempts, which is 0 failures in 40 runs.
Also jittered the backoff, but measured rather than assumed: my first
theory was a thundering herd, since the delay was exactly
RetryBackoff*(attempt+1) and every loser of a race woke at the same
instant. Isolating the two changes showed jitter alone moved 4/20 to
3/20, while the budget alone reached 0/20. The budget was the real
constraint. Jitter is kept because it costs nothing and its benefit
grows with the number of contending writers -- production is not capped
at five -- but the comment now says plainly that it is the smaller half,
so nobody inherits my wrong explanation.
Second, the integration scripts leaked one throwaway database volume per
run. --rm does reclaim anonymous volumes on a normal exit, but these
scripts force-remove the container from a trap, and `docker rm -f`
without -v keeps the volume. Sixty-four accumulated during this branch
until PostgreSQL stopped starting, surfacing only as the scripts' own
readiness timeout rather than as a disk error -- which is what the
"Docker storage exhausted locally" notes were really describing.
Measured at one volume per run before, zero after, across all five
scripts.
The audit established which rows understated what was built by locating
implementations and their call sites. That proves code exists, not that
it works, so each corrected claim is now tied to an executable test.
Seven of the nine were already covered and just needed naming: the
allocated ServerConfig fields, signed-authorisation admission, endpoint
wiring, casual lineup being reached through formation, three of the four
penalty kinds, signed roster metadata, the season countdown, and the
matcher deployments.
Two had no proof at all:
- INITIAL_CONNECT_NO_SHOW was the one penalty kind with no integration
coverage, so "all four penalty kinds are written durably" rested
entirely on reading the code.
- Cross-replica revocation is a behavioural property. An in-memory cache
in front of the session read would break it while leaving every call
site looking correct, so no amount of reading establishes it.
Writing the first one found my own error rather than a defect: casual
deliberately waits past InitialConnectWindow to CasualBotStartAfter
before deciding a no-show, giving a slow-loading player longer than the
ranked deadline. Reconciling at the earlier window only yields WAIT.
Both new tests were mutation-checked -- removing the penalty insert and
removing the revoked_at check each make them fail -- so they assert
something real rather than passing incidentally.
Task 8.22. Tier bands lived in domain.DefaultTierPolicy(), compiled into
every API binary, so retuning one meant building and rolling a new image
-- least attractive exactly when it is most needed, as the rating
distribution settles after launch.
Bands now live in a tier_bands table, seeded by the migration with the
exact policy the binaries hardcode, so this changes durable state without
changing behaviour. Retuning is a rolling restart rather than a rebuild.
Three properties the loader deliberately holds:
- A malformed durable policy stops startup. Falling back on error would
silently mis-tier every player, which is worse than not starting.
- An empty table is supported and falls back to the compiled default, so
an operator can truncate back to known-good without a deploy, and a
fresh database works before the seed is reviewed.
- PROVISIONAL is rejected as a band. It is derived from ranked game
count, not rating, so a band claiming it would be unreachable at best
and would shadow a real tier at worst.
Bands stay backend-owned; clients still receive only the resulting label,
per docs/MATCHMAKING.md. UNIQUE(min_rating) rejects two bands sharing a
threshold, catching an ambiguous policy before NewTierPolicy does.
testkit-api loads it too, so the control-plane integration scripts
exercise the durable path rather than the compiled default.
Integration tests cover the seeded policy matching the compiled one,
retuning taking effect from the database alone, truncation falling back,
and each invalid-policy shape being rejected. Verified they fail against
a loader that ignores durable bands.
The other two parts of 8.22 needed no work: the client UI already renders
tier, provisional status, ranked games and the season countdown, and
reconnect transport is 8.42's, dependent on live backend events.
newAPIService never supplied SteamLogin, so POST /v1/session/steam
always returned 503 auth_unavailable in production. The only
implementation was cmd/testkit-api's fake, which derives an identity
from the ticket string itself and accepts anything -- so the passing
integration path was neither deployable nor secure. On the client side
the game started with an empty token and a loopback base URL, and no
production code called configure() or login_steam(); the menu entered
matchmaking directly, so every request failed ERR_UNAUTHORIZED before
reaching the network.
Add a real ISteamUserAuth/AuthenticateUserTicket adapter behind an
interface, so the production login path is testable with only the Valve
call stubbed. It rejects family-shared copies (the account playing does
not own the app) and, by default, VAC- or publisher-banned accounts, and
refuses malformed tickets locally rather than forwarding them.
Crucially it separates our faults from the player's: a Valve outage or a
revoked publisher key returns 503, not 401. Answering 401 would tell a
legitimate player their login failed and send them to fix an account
that is fine while the real fault went unnoticed. A banned identity now
returns 403 rather than a misleading 503.
Sign-in is configuration-gated on the publisher key and App ID: without
them the endpoint keeps returning 503, since silently accepting an
unverified ticket would be worse than refusing to authenticate. A
returning player keeps the player ID they already had, so ratings,
penalties and bans follow the account rather than the session.
Client side: acquire a web-API ticket through GodotSteam's async
signal -- requesting one returns a handle, not a ticket -- using the
existing dynamic-call pattern so stock Godot still parses the project.
The endpoint is configurable for release builds, and matchmaking
completes sign-in before it will queue.
Verified against real PostgreSQL; 232 Godot tests pass.
domain.validCandidate hard-requires a non-empty PredictedRTT map, but
CreateQueueTicket persisted an empty one and the only endpoint that
could fill it returned 503 in every real binary, because Service.Probe
was assigned nowhere outside api tests. No client-created ticket could
ever be selected by the matcher. The Godot client had no probe method at
all, so even a wired backend was unreachable from the game.
Four distinct defects had to be fixed for this path to work:
Nothing issued the nonce ProbeProvider was meant to compare against, so
the contract could not be satisfied even in principle. Add
POST /v1/probes/{region}/challenge, backed by a durable single-use
challenge -- durable because any replica may serve the answer for a
challenge another replica issued. RTT is the interval between issuing
and receiving, so no client-reported latency reaches placement.
CreateQueueTicket marshalled a nil map to JSON `null`, a JSONB scalar
rather than an object, and jsonb_set rejects that with "cannot set path
in scalar". RecordProbe would have failed at runtime even once wired.
Persist an object, and normalise non-object values in the update for
rows already written.
A nil ProbeRecorder made the handler report success while persisting
nothing, which silently leaves the ticket unmatchable. That is a
misconfiguration, not a successful probe; it now returns 503.
A successful probe updated PostgreSQL only. The candidate inserted at
enqueue time carries an empty RTT map, and the Redis keyspace has its
TTL continually refreshed, so the stale entry need never repair itself.
Refresh that player's projection after the probe commits.
Client side: add the challenge/answer round trip and have the
matchmaking screen collect evidence before creating a ticket, since
queueing first produces a search that can never match. Probing every
region fully is not required -- placement uses whichever regions
answered -- but queueing with none is refused rather than silently
stalling.
New integration test drives the real enqueue and probe paths and then
asks the actual matcher predicate, rather than hand-building a candidate
the way the unit tests do -- which is exactly why they missed this.
Also make the integration schema reset drop the whole public schema: the
enumerated table list silently broke with each new migration.
The root blocker (issue #14). The worker bound the provider allocation
and stopped. Service.PublishRoster and store.SaveVerifiedAssignmentRoster
both existed, fully tested, with zero non-test callers, and the
production allocator configured neither a roster store nor a signing
key. Nothing ever wrote the assignments table.
The allocated supervisor fetches a non-empty roster before it launches
the game child, so every real allocation failed at that fetch: no match
could reach ASSIGNMENT_READY or accept a player. Existing tests seeded
assignments directly, which is exactly why the missing hand-off went
unnoticed.
The worker now builds one join authorisation per durable participant,
signs each with the active key, and publishes them. Participants are
read through the same query SaveVerifiedAssignmentRoster re-validates
against, so the allocator cannot construct a roster the persistence
boundary would reject. The manifest commits to a digest over the whole
roster, so a server cannot be handed a truncated roster whose surviving
entries are each individually valid.
Persist the provider endpoint on the allocation: it arrived on the
provider response and was never stored, so a worker crashing between
allocating and publishing had no endpoint to recover and would have
stranded the match permanently. Republishing is idempotent, so that
crash now simply retries.
cmd/allocator refuses to start without key material rather than running
an allocator that binds allocations and silently strands every match.
The k8s allocator Deployment mounts the same key set the Fleet does, and
both now take the JSON key map so a rotation can publish several.
New integration test drives the real worker through to the supervisor's
own roster read path without seeding the assignments table. Verified it
fails with "assignments = 0, want 2" when the publish step is removed.
Each ten-second queue heartbeat mints a fresh idempotency key and
permanently inserts a row. Published outbox rows and expired/revoked
sessions were never purged either -- the maintenance role performed
lifecycle reconciliation only. At 10,000 queued players heartbeats alone
add roughly 60,000 durable rows per minute, so table and index growth,
vacuum pressure, backup size and recovery time were all unbounded on a
service intended to scale horizontally.
Add retention windows chosen to exceed every retry and recovery horizon
that could still consult the row -- deleting an idempotency key early
would turn a client replay into a second real mutation, so this is a
correctness bound, not just a housekeeping one. Dead-lettered outbox
rows are kept longest, being the record of events never delivered.
Deletes run in bounded SKIP LOCKED batches so a purge never blocks live
traffic, never holds a long transaction, and concurrent maintenance
replicas do not contend. Indexes back each predicate so a pass cannot
degrade into a sequential scan of the table it is bounding. The
maintenance role reports rows purged, the backlog past its window
(deletion lag), and any dead-lettered events.
Also make the migration-rollback test derive its step counts instead of
hardcoding them: adding a migration silently shifted the fixed counts so
the failure surfaced as an unrelated "0006 rollback did not drop
matches.allocation_id".
The Deployment runs two replicas, but WebSocket subscribers live only in
each process's in-memory hub. Every replica races to read the same
global unpublished outbox rows, and publishing succeeded even when the
winning replica held no matching local subscriber -- that replica then
set the single global published_at. A client connected to the other
replica never received the event, and delivery degraded further with
each replica added. REST recovery eventually converged, but short-lived
proposal transitions could be observed late or not at all.
Publish committed events through PostgreSQL LISTEN/NOTIFY so the replica
that owns the subscriber's connection delivers it, regardless of which
replica drained the row. The listener holds its own pgx connection --
LISTEN is session state, so a pooled database/sql connection cannot
carry it -- and reconnects with backoff, since losing it would silently
downgrade that replica's subscribers to REST-only recovery.
The fan-out is optional: without EventFanout configured, behaviour is
unchanged local-hub publication, which stays correct for a single
replica and for tests. Only outbox-sourced events are routed through it;
the in-request-path publishes remain local, as those are a latency
optimisation for the caller's own connection.
Fan-out needs a wire shape of its own because ControlPlaneEvent hides
PlayerID from clients, and the recipient is exactly what a peer replica
needs to route on.
banned_until and ban_reason have been in the schema since 0001, but no
production query ever read them -- grepping the tree found no reference
outside the migration itself. The only ban check was an in-memory map on
domain.TicketVerifier used by domain tests. Once real Steam login is
wired, a banned identity would keep full access through every existing
session until expiry and could obtain new ones.
Make the ban part of the durable authentication transaction rather than
a policy each login adapter must remember to re-implement:
- Session issuance inserts only when the identity exists and has no
active ban, so a banned player cannot mint a session.
- Authentication joins the identity and rejects an active ban on every
request, so a ban takes effect immediately on every replica rather
than at session expiry.
- ApplyIdentityBan sets the ban and revokes that identity's sessions in
one serializable transaction, closing the window where the ban is
durable but another replica still accepts an issued session.
Bans are time-bounded and clearing one does not resurrect sessions the
ban revoked.
Tests cover enforcement across two independently constructed stores
standing in for two replicas, expiry/unban semantics, and -- separately,
because revocation would otherwise mask it -- that a ban applied without
revoking anything still blocks the next request.
Both playlists shared one hash and sorted set, causing two independent
failures.
Starvation: Snapshot performed an unbounded ZRANGEBYSCORE and HMGET,
decoded the whole queue, and the matcher then truncated to its candidate
limit *before* filtering by playlist. A large casual prefix could
therefore leave the ranked worker with zero candidates indefinitely even
while ranked tickets were queued further down the set.
Mutual erasure: each matcher captured only its own playlist as the
durable source, but Rebuild replaced the shared keys, so a casual repair
wiped ranked projections and vice versa.
Namespace the keys per playlist, push the limit into Redis (LIMIT 0 N)
so reads no longer scale with total queue depth, and scope Rebuild to
one namespace. Rebuild now rejects a candidate whose playlist does not
match the namespace, which would reintroduce the starvation. Upsert
derives the namespace from the candidate; Remove takes the playlist,
since a ticket ID alone no longer identifies its namespace.
Add tests for a 300-deep casual backlog not starving ranked, for neither
playlist's rebuild erasing the other, and for the limit being applied
without losing enqueue ordering.
The candidate projection selected only from queue_tickets and its scan
never set Candidate.Rating, so every PostgreSQL-sourced ranked candidate
arrived with Go's zero value. Rating tolerance, selection scoring and
team partitioning all read that field, so ranked matchmaking treated a
900-rated player as identical to a 2100-rated one. Unit tests missed it
because they construct candidates with ratings already populated.
Join the ratings table, defaulting to domain.GlickoInitialRating for a
player with no ratings row yet -- a genuinely new profile, matching the
column default.
Fix the same defect on the Redis path too, which is reached differently:
the projection is seeded from the candidate CreateQueueTicket builds,
not from the candidate query, and that candidate also left Rating unset.
Resolve the rating inside the enqueue transaction so both projections
agree on one authoritative value. The rating is never client-supplied.
Add a store-backed test with deliberately distant ratings (900 vs 2100)
plus an unrated player, asserting both projections and that the spread
survives. Verified it fails without the fix.
ApplyInitialConnectPlan wrote a payload of {match_id,state,action},
omitting event, revision, resource_id, occurred_at and player_ids --
every field deliverStateOutboxEvent requires. Delivery rejected the row,
dispatch returned on the first error so it was never acknowledged, and
because reads are ordered oldest-first it was retried ahead of every
later state_changed event on every 100ms poll. One initial-connect
transition therefore blocked lifecycle delivery for all matches, not
just its own.
Two independent fixes, since either alone leaves the system fragile:
Build envelopes through one validating helper (MarshalOutboxEnvelope)
and convert all five writers to it. A writer that omits a required
field now fails its own transaction instead of committing a row that
can only ever poison the queue. The helper takes revision as int64 so
the -1 "nothing matched" sentinel some CTEs return surfaces as an error
rather than wrapping to a huge uint64.
Make dispatch resilient regardless: a delivery failure is now counted
against that row and the batch continues, with the row dead-lettered
after MaxOutboxDeliveryAttempts so a poison event degrades to one lost
notification instead of a stalled queue. Ordering within an aggregate
is still honoured -- later events of a failed match are deferred, so no
client observes that match's newer state before its older state. An ack
failure still stops the batch, being a database rather than a payload
problem.
Initial-connect events now address every participant, not just the
connected ones: a no-show needs to learn their ticket was failed and a
penalty applied.
Closes part of the 'live Redis failover' gap in §8.46, found by
reproducing a genuine Redis outage (not just an empty/partial cache)
against CandidateProjection.Snapshot with a killed miniredis instance.
CandidateProjection.Snapshot funnelled two different situations into
the same code path: the index erroring outright (Redis unreachable)
and the index coming back empty (ambiguous — a genuinely empty queue,
or a lost keyspace). Both went through Repair, which itself calls
Index.Rebuild — a second Redis round-trip that fails for exactly the
same reason the first one did. The result: a real Redis outage, or
the window during a failover, made Snapshot fail outright even though
PostgreSQL — the documented authoritative source everywhere
(RedisCandidateIndex's own comment, cmd/matcher, cmd/control-plane's
--redis-addr help text all call it a rebuildable/optional
acceleration layer) — was completely healthy. Matchmaking would stop
entirely on a Redis outage despite the architecture explicitly not
requiring that.
Snapshot now falls back to serving Source (PostgreSQL) directly
whenever the index errors OR comes back empty, and only best-effort
attempts to repopulate Redis afterward — that attempt's outcome is
deliberately ignored, since a caller must never be denied service
just because the opportunistic rebuild also hit the same down Redis.
Snapshot still fails when Source itself is unavailable; the fallback
is not unconditional.
Verified: reproduced the bug first (killed-miniredis Snapshot call
failed even though Source was healthy), then fixed it. go build/vet
clean; all pre-existing store-package tests pass unmodified,
including the two live-redis:7-alpine-container tests
(TestRealRedisCandidateIndexUpsertSnapshotRemove,
TestRealRedisCandidateProjectionRepairsAfterFlush, run against a real
container and torn down after). Two new tests cover the fallback
directly (killed miniredis, Source still served, exactly one Source
call) and that the fallback is not unconditional (both Redis and
Source down still fails). Full go test ./... -race clean across
every server package.
Remaining: live matcher-worker-under-load-during-failover integration,
i.e. running the actual matcher process against a real Redis that
goes down mid-run under concurrent load, not just this unit-level
reproduction.
Closes the 'concurrent proposal-recovery expiry races' gap noted in
§8.46. GetProposal (read-side recovery) and RespondToProposal both
run the identical expiry-advance SQL in their own transaction, so any
number of them can observe the same past-expiry proposal at once —
this had never been exercised concurrently, only sequentially (the
existing late-response test drives one call at a time).
TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
races 8 concurrent GetProposal/RespondToProposal calls, each with a
distinct 'now' past the proposal window, against one proposal and
asserts: EXPIRED lands on the proposal and both tickets exactly once,
a PROPOSAL_TIMEOUT penalty lands exactly once per offending player
(not once per racing transaction), and no idempotency row survives a
closed-proposal response. The design already defends against this —
ProposalParticipantExpireSQL only ever flips a still-PENDING row
once, so a losing racer's 'now' can't match
recordProposalTimeoutCooldowns' responded_at filter — this test is
what actually proves that holds under real concurrent load rather
than by inspection.
Verified: real postgres:17-alpine container, go test -tags
integration ./store/... -run
TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
-race -count=3 clean; full -tags integration ./store/... -race run
clean; full non-integration go build/vet/test -race clean across
every server package; container removed after the run.
Josh Creek