Each ten-second queue heartbeat mints a fresh idempotency key and
permanently inserts a row. Published outbox rows and expired/revoked
sessions were never purged either -- the maintenance role performed
lifecycle reconciliation only. At 10,000 queued players heartbeats alone
add roughly 60,000 durable rows per minute, so table and index growth,
vacuum pressure, backup size and recovery time were all unbounded on a
service intended to scale horizontally.
Add retention windows chosen to exceed every retry and recovery horizon
that could still consult the row -- deleting an idempotency key early
would turn a client replay into a second real mutation, so this is a
correctness bound, not just a housekeeping one. Dead-lettered outbox
rows are kept longest, being the record of events never delivered.
Deletes run in bounded SKIP LOCKED batches so a purge never blocks live
traffic, never holds a long transaction, and concurrent maintenance
replicas do not contend. Indexes back each predicate so a pass cannot
degrade into a sequential scan of the table it is bounding. The
maintenance role reports rows purged, the backlog past its window
(deletion lag), and any dead-lettered events.
Also make the migration-rollback test derive its step counts instead of
hardcoding them: adding a migration silently shifted the fixed counts so
the failure surfaced as an unrelated "0006 rollback did not drop
matches.allocation_id".
Closes the last named item on task 8.28 (Health-reclaim): nothing
currently detects or cleans up a match stuck in
ALLOCATING/PROCESS_READY/ASSIGNMENT_READY forever because its server
crashed or was reclaimed by Agones as unhealthy before ever
registering -- players would wait indefinitely for a match that was
never coming.
The design question this was blocked on -- does an abandoned match
auto-requeue its players, or fail and make them re-queue -- isn't
actually open: task 8.50's own stated acceptance criterion already
answers it ("infrastructure-caused cases cannot penalise affected
players"). A server-side crash/reclaim is exactly that, not player
behaviour, so store.ExpireStalledAllocations fails the match but
requeues every participant's ticket to QUEUED with a fresh expiry
(matching the ordinary 30s queue window), releases their
match_participants row (participation_active = false, so they're
matchable again immediately), all inside one FOR UPDATE SKIP LOCKED
pass so a second maintenance replica continues past whatever a
concurrent one is already reclaiming.
Wired into cmd/maintenance alongside the existing season-rollover
sweep: --stalled-allocation-deadline (default 2m) and
--stalled-allocation-batch (default 100).
Covered by a SQL-fragment test and a real PostgreSQL integration test:
two matches (one genuinely stalled, one recent), confirming the
deadline boundary is respected (recent match untouched), both
stranded participants' tickets requeue with a refreshed expiry, the
match_participants row releases, and a second pass doesn't reprocess
an already-FAILED match. Verified clean across 5 runs, plus the full
integration and unit suites.
Josh Creek