jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-11 08:23:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(server): add retention for idempotency, outbox and session records

Browse Source 
Each ten-second queue heartbeat mints a fresh idempotency key and
permanently inserts a row. Published outbox rows and expired/revoked
sessions were never purged either -- the maintenance role performed
lifecycle reconciliation only. At 10,000 queued players heartbeats alone
add roughly 60,000 durable rows per minute, so table and index growth,
vacuum pressure, backup size and recovery time were all unbounded on a
service intended to scale horizontally.

Add retention windows chosen to exceed every retry and recovery horizon
that could still consult the row -- deleting an idempotency key early
would turn a client replay into a second real mutation, so this is a
correctness bound, not just a housekeeping one. Dead-lettered outbox
rows are kept longest, being the record of events never delivered.

Deletes run in bounded SKIP LOCKED batches so a purge never blocks live
traffic, never holds a long transaction, and concurrent maintenance
replicas do not contend. Indexes back each predicate so a pass cannot
degrade into a sequential scan of the table it is bounding. The
maintenance role reports rows purged, the backlog past its window
(deletion lag), and any dead-lettered events.

Also make the migration-rollback test derive its step counts instead of
hardcoding them: adding a migration silently shifted the fixed counts so
the failure surfaced as an unrelated "0006 rollback did not drop
matches.allocation_id".
This commit is contained in:
Josh Creek
2026-09-05 10:32:08 +01:00
parent 129b0c7ef0
commit 5453e19761
5 changed files with 302 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
server/cmd/maintenance/main.go
+27
View File
@@ -26,6 +26,7 @@ func main() {
stalledAllocationBatch := flag.Int("stalled-allocation-batch", 100, "maximum stalled matches reclaimed per pass")
initialConnectBatch := flag.Int("initial-connect-batch", 100, "maximum pre-live matches evaluated per pass")
liveAbandonmentBatch := flag.Int("live-abandonment-batch", 100, "maximum live ranked matches evaluated for expired reconnect leases per pass")
retentionBatch := flag.Int("retention-batch", 500, "maximum rows deleted per table per retention pass")
flag.Parse()
if *dsn == "" {
fatalf("--dsn or COSMIC_CLASH_POSTGRES_DSN is required")
@@ -69,6 +70,32 @@ func main() {
if reclaimed > 0 {
log.Printf("reclaimed %d stalled allocations, requeuing their participants", reclaimed)
}
// Retention. Without this, idempotency keys alone grow by roughly one
// row per queued player per heartbeat interval, forever.
purged, err := store.PurgeExpiredRecords(ctx, db, now, *retentionBatch)
if err != nil {
fatalf("retention maintenance: %v", err)
}
if purged.Total() > 0 {
log.Printf("purged %d expired records (idempotency=%d outbox=%d dead-lettered=%d sessions=%d)",
purged.Total(), purged.IdempotencyKeys, purged.PublishedOutbox, purged.DeadLetteredOutbox, purged.ExpiredSessions)
}
// Deletion lag: a backlog that keeps climbing means the interval or
// batch size is too small for current volume.
backlog, err := store.RetentionBacklog(ctx, db, now)
if err != nil {
fatalf("retention backlog: %v", err)
}
if backlog > 0 {
log.Printf("retention backlog is %d rows past their window", backlog)
}
deadLettered, err := store.CountDeadLetteredOutboxEvents(ctx, db)
if err != nil {
fatalf("dead-letter count: %v", err)
}
if deadLettered > 0 {
log.Printf("WARNING: %d outbox events were never delivered and are dead-lettered", deadLettered)
}
}
runInitialConnect := func(now time.Time) {
reconciled, err := store.ReconcileInitialConnect(ctx, db, now, *initialConnectBatch)