fix(multiplayer): make match promotion replay lifecycle-safe

multiplayer-next.md

@@ -1206,7 +1206,7 @@ production fallback.
 | 8.15 `[D:7.8,8.3]` | **IN PROGRESS.** Pure Go probe validation treats Steam location as opaque, requires nonce/freshness/region and server-computed RTT, and implements discrepancy quarantine/release; authenticated HTTP now accepts only opaque location/nonce input through a server-owned probe provider and records validated RTT into the active player's durable queue ticket; durable queue projections have a server-derived RTT JSON field for matcher reads | `server/domain/probes.go`, `server/migrations/0003_queue_probe_metadata.sql`, adversarial fixtures and `server/api/service.go`/`store/queue_sql.go` cover stale/wrong/forged evidence, the 25 ms/30% threshold, three-sample quarantine, five-clean release, authenticated provider arguments, rejection of client RTT fields, player/ticket/expiry binding, persistence failure, playlist-scoped candidate reads and bounded metadata decoding; Steam coordinator, regional probe adapters and multi-region probe population remain |
 | 8.16 `[D:8.14,8.15]` | **IN PROGRESS.** Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction; the runnable matcher now supports explicitly enabled ranked six-player polling with durable verified-Steam identity lookup. **Fixed a real crash-loop**: `Worker.Run` treated every `RunOnce` error as fatal to the whole loop, including "no compatible candidates" (`FormFromQueue`'s completely routine answer when currently-queued players share no verified region) — found building a live two-player integration attempt (see below): two real players with no common region crashed the entire matcher process, taking matchmaking down for every other player in the playlist, and would crash-loop again on restart since the same incompatible candidates stay queued. Now only genuine static misconfiguration (`ErrWorkerNotConfigured`/`ErrUnsupportedPlaylist`/`ErrInvalidMatcherSize`) stops the loop; everything else retries next interval | `server/domain/matcher.go`, `teams.go`, `server/matcher/worker.go`, `server/store/queue_sql.go` and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure, queue-backed oldest-anchor formation and incomplete ranked identity metadata; two new tests cover `Run` (not just `RunOnce`) surviving a per-pass error via a real concurrent goroutine, and still stopping immediately on a real configuration error, both clean across repeated `-race` runs. **A live two-player Godot proposal integration attempt is on disk but not committed**: `Game/tests/control_plane_proposal_smoke.gd`/`.tscn` and `scripts/verify_control_plane_proposal_integration.sh` exist and found the bug above, but the session paused running further concurrent headless Godot processes after discovering they'd been causing native engine crashes (macOS crash reporter, `EXC_BAD_ACCESS`/`SIGBUS`) intermittently all session, confirmed by the user; the two-player script was never itself verified to a clean pass. Arena selection and long-running worker integration remain |
 | 8.17 `[D:8.14,8.16]` | **IN PROGRESS.** Pure Go proposal policy sends a 10-second response window to every selected human, requires unanimous acceptance, applies exact casual/ranked decline and timeout cooldowns with ranked escalation, and exposes revisioned idempotent responses through the authenticated API. Proposal closure now atomically separates offenders from innocents: a decliner's ticket is `CANCELLED`; a timed-out player's ticket is `EXPIRED`; accepted or otherwise innocent participants return to `QUEUED` with their original `enqueued_at` and refreshed expiry. Direct queue cancellation closes the open proposal and requeues remaining participants immediately. Late API responses commit expiry, timeout penalties, and ticket release before returning `ErrProposalClosed`; recovery of an old declined proposal cannot misclassify its pending innocents as timeouts. Cooldown history rejects future, foreign-playlist, and invalid-kind events, and database rows are closed before penalty writes | Domain/store/API fixtures cover partial/unanimous response, expiry, replay/conflict, stale revision, exact cooldown windows/escalation, corrupt history filtering, offender ticket termination, innocent precedence preservation, direct-cancel cascade, and the former late-response rollback. PostgreSQL-tagged regressions compile and assert the durable split and penalty rows; the full local Go suite passes. Live PostgreSQL execution and allocation integration remain |
-| 8.18 `[D:8.5,8.14,8.17]` | **IN PROGRESS.** Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using `FOR UPDATE SKIP LOCKED` plus durable uniqueness/revision fences; proposal creation now persists its matcher-selected region/protocol/team/slot plan alongside proposal/participant claims, queue creation has a durable idempotency/owner-read adapter, participant-scoped proposal recovery now expires OPEN proposals and pending participants transactionally at read time, and proposal accept/decline now uses participant/proposal locks, revision fencing and durable idempotency; final unanimous API acceptance invokes replay-safe promotion of the exact stored `ALLOCATING` match/team/slot topology and claimed tickets to `ACCEPTED`; runnable matcher polling supports an optional Redis candidate projection that repairs empty/lost cache state from authoritative PostgreSQL before the durable final claim | `server/store/serializable.go`, `queue_sql.go`, `proposal_sql.go`, `proposal_recovery_sql.go`, `match_sql.go`, `redis_candidates.go`, `server/matcher/worker.go`, `server/api/service.go` and tests cover retry classification, claim-boundary invariants, player/ticket/playlist mapping, durable queue replay/conflict, owner-scoped queue/proposal recovery, expiry at read and mutation boundaries, final-response promotion retry, exact match-promotion replay/conflict, fixed team/slot persistence, zero-row claim aborts, atomic statement ordering, incomplete matcher batches, source failures and empty-index Redis repair; opt-in PostgreSQL execution now covers queue create/replay/fencing, assignment persistence, proposal claim/promotion, accepted-proposal match promotion, participant recovery, unanimous response and rollback of partial claims — actually running this suite live (it had not been before) found `ProposalParticipantExpireSQL` had no expiry-time condition at all, so every call timed out every pending participant on the spot; the very first accept on any proposal then failed with a false conflict. Fixed with the same `expires_at <=` gate `ProposalExpireSQL` already used, re-verified live. A real concurrent-goroutine test now covers the two-matcher race this was missing: two proposals sharing one contested ticket, racing two real Postgres connections under `-race`, exactly-one-wins/loser-fully-rolls-back including the loser's own uncontested ticket, stable across 8 runs; allocation runtime integration tests remain |
+| 8.18 `[D:8.5,8.14,8.17]` | **IN PROGRESS.** Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using `FOR UPDATE SKIP LOCKED` plus durable uniqueness/revision fences; proposal creation persists matcher-selected region/protocol/team/slot topology before acceptance, and final unanimous acceptance promotes the exact roster into one `ALLOCATING` match. Promotion replay validates immutable playlist/region/protocol/arena, participant, ticket, team, and slot identity but deliberately ignores mutable match state/server ownership, so a retry after a lost response still succeeds after allocation has advanced. Result sets are closed before crossing into promotion writes, avoiding one-connection pool stalls. Redis remains a rebuildable candidate projection over PostgreSQL authority | Store/API tests cover retries, claims, owner/revision fencing, expiry, exact promotion replay/conflict, progressed-match replay, rollback of partial claims, concurrent contested-ticket formation, and lost-cache repair. PostgreSQL-tagged regressions compile; prior live runs covered queue/proposal promotion and races, while this progressed-replay change awaits a live database rerun. Allocation runtime integration remains |
 | 8.19 `[D:8.18]` | **IN PROGRESS.** Pure Go casual lineup requires 2–6 humans with at least one per team, fills missing slots with explicit bots, permits kickoff-only bot-slot backfill and assigns no backfill penalty/rating update; proposal preparation now derives the lineup from formed teams | `server/domain/casual.go`, `formation.go` cover both-team minimum, bot shape, live-play rejection, zero-penalty backfill and casual proposal composition; queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties and live integration remain |
 | 8.20 `[D:8.18]` | **IN PROGRESS.** Pure Go ranked admission requires six unique verified solo humans, rejects bots/backfill/parties, and allows only random-enabled non-elevated arenas; proposal preparation requires matching metadata for every formed player | `server/domain/ranked.go`, `formation.go` cover count, identity, party, bot/backfill, arena eligibility and formed-player metadata rejection; `ArenaRegistry` integration, allocation wiring and innocent-ticket restoration remain |
 | 8.21 `[D:8.5,8.20]` | **IN PROGRESS.** Pure Go rating core implements canonical Glicko-2, daily inactivity, ranked 1/3 and casual 1/N human-opponent weights, deterministic opponent ordering, and authoritative draw/overtime/abandon scoring; certified result completion now applies per-player updates inside the durable transaction with lexical row locks and revision increments | `server/domain/rating.go`, `server/store/result_sql.go` and tests cover canonical/inactivity/weight/invalid-input, draw/OT/abandon, ordered participant snapshots, lock/value re-read and rating update SQL; live PostgreSQL rating and seasons execution now covered (§8.23), and concurrent result transaction cases are covered: `TestPostgreSQLConcurrentIdenticalResultSubmissionAppliesRatingsExactlyOnce` races 5 identical submissions and confirms one rating application, while `TestPostgreSQLConcurrentConflictingResultSubmissionsKeepOneReceipt` races different payloads and confirms exactly one winner, one conflict, one receipt and one completion event; live maintenance/DB execution remains |

server/store/match_sql.go

@@ -45,7 +45,7 @@ const AcceptedMatchInsertSQL = `INSERT INTO matches
 VALUES ($1, $2, 'ALLOCATING', $3, $4, NULLIF($5, ''))
 ON CONFLICT (match_id) DO NOTHING`
 
-const AcceptedMatchSelectSQL = `SELECT playlist, state, region, protocol_version, arena_path, server_id
+const AcceptedMatchSelectSQL = `SELECT playlist, region, protocol_version, arena_path
 FROM matches
 WHERE match_id = $1
 FOR UPDATE`
@@ -99,6 +99,9 @@ func PromoteStoredAcceptedProposal(ctx context.Context, db *sql.DB, proposalID s
 	if err := rows.Err(); err != nil {
 		return err
 	}
+	if err := rows.Close(); err != nil {
+		return err
+	}
 	return CreateMatchFromAcceptedProposal(ctx, db, plan, now)
 }
 
@@ -206,6 +209,9 @@ func acceptedProposalParticipants(ctx context.Context, tx *sql.Tx, plan Accepted
 	if err := rows.Err(); err != nil {
 		return nil, err
 	}
+	if err := rows.Close(); err != nil {
+		return nil, err
+	}
 	if len(participants) != len(plan.Players) {
 		return nil, fmt.Errorf("proposal participants do not match accepted plan")
 	}
@@ -218,14 +224,16 @@ func acceptedProposalParticipants(ctx context.Context, tx *sql.Tx, plan Accepted
 }
 
 func verifyAcceptedMatchReplay(ctx context.Context, tx *sql.Tx, plan AcceptedMatchPlan, playlist domain.Playlist, tickets map[string]string) error {
-	var existingPlaylist, state, region string
+	var existingPlaylist, region string
 	var protocol int
 	var arenaPath sql.NullString
-	var serverID sql.NullString
-	if err := tx.QueryRowContext(ctx, AcceptedMatchSelectSQL, plan.MatchID).Scan(&existingPlaylist, &state, &region, &protocol, &arenaPath, &serverID); err != nil {
+	if err := tx.QueryRowContext(ctx, AcceptedMatchSelectSQL, plan.MatchID).Scan(&existingPlaylist, &region, &protocol, &arenaPath); err != nil {
 		return err
 	}
-	if existingPlaylist != string(playlist) || state != string(domain.Allocating) || region != plan.Region || protocol != plan.Protocol || arenaPath.String != plan.ArenaPath || arenaPath.Valid != (plan.ArenaPath != "") || serverID.Valid {
+	// Match state and server ownership are intentionally absent: allocation may
+	// advance immediately after the first promotion commits. A retry after a
+	// lost API response is valid whenever the immutable topology still matches.
+	if existingPlaylist != string(playlist) || region != plan.Region || protocol != plan.Protocol || arenaPath.String != plan.ArenaPath || arenaPath.Valid != (plan.ArenaPath != "") {
 		return domain.ErrConflict
 	}
 	rows, err := tx.QueryContext(ctx, AcceptedMatchParticipantsSQL, plan.MatchID)

server/store/match_sql_test.go

@@ -12,6 +12,7 @@ func TestAcceptedMatchSQLPreservesAtomicProposalToMatchBoundary(t *testing.T) {
 		AcceptedProposalLockSQL:           {"FOR UPDATE", "proposal_id = $1"},
 		AcceptedProposalParticipantsSQL:   {"response", "ORDER BY player_id", "FOR UPDATE"},
 		AcceptedMatchInsertSQL:            {"'ALLOCATING'", "ON CONFLICT (match_id) DO NOTHING"},
+		AcceptedMatchSelectSQL:            {"playlist", "region", "protocol_version", "arena_path", "FOR UPDATE"},
 		AcceptedTicketSQL:                 {"state = 'ACCEPTED'", "state = 'PROPOSED'", "revision = revision + 1"},
 		AcceptedMatchParticipantInsertSQL: {"match_participants", "slot", "team"},
 	}
@@ -24,6 +25,12 @@ func TestAcceptedMatchSQLPreservesAtomicProposalToMatchBoundary(t *testing.T) {
 	}
 }
 
+func TestAcceptedMatchReplayDoesNotDependOnMutableLifecycleFields(t *testing.T) {
+	if contains(AcceptedMatchSelectSQL, "state") || contains(AcceptedMatchSelectSQL, "server_id") {
+		t.Fatalf("accepted promotion replay is coupled to mutable lifecycle fields: %s", AcceptedMatchSelectSQL)
+	}
+}
+
 func TestAcceptedMatchPlanRejectsInvalidPlansBeforeDatabaseUse(t *testing.T) {
 	valid := AcceptedMatchPlan{
 		MatchID: "match-1", ProposalID: "proposal-1", Region: "EU", Protocol: 1,

server/store/postgres_integration_test.go

@@ -252,10 +252,16 @@ func TestPostgreSQLAcceptedProposalPromotesOneAtomicAllocatingMatch(t *testing.T
 	if err := CreateMatchFromAcceptedProposal(ctx, db, plan, now.Add(time.Second)); err != nil {
 		t.Fatalf("identical match promotion replay: %v", err)
 	}
+	if _, err := db.ExecContext(ctx, `UPDATE matches SET state = 'LIVE' WHERE match_id = 'promote-match'`); err != nil {
+		t.Fatal(err)
+	}
+	if err := CreateMatchFromAcceptedProposal(ctx, db, plan, now.Add(2*time.Second)); err != nil {
+		t.Fatalf("promotion replay after match lifecycle advanced: %v", err)
+	}
 	conflict := plan
 	conflict.Players = append([]MatchPlayer(nil), plan.Players...)
 	conflict.Players[1].Slot = 4
-	if err := CreateMatchFromAcceptedProposal(ctx, db, conflict, now.Add(2*time.Second)); err == nil {
+	if err := CreateMatchFromAcceptedProposal(ctx, db, conflict, now.Add(3*time.Second)); err == nil {
 		t.Fatal("conflicting match promotion replay was accepted")
 	}
 }