test(multiplayer): cover concurrent proposal-expiry recovery race

Closes the 'concurrent proposal-recovery expiry races' gap noted in
§8.46. GetProposal (read-side recovery) and RespondToProposal both
run the identical expiry-advance SQL in their own transaction, so any
number of them can observe the same past-expiry proposal at once —
this had never been exercised concurrently, only sequentially (the
existing late-response test drives one call at a time).

TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
races 8 concurrent GetProposal/RespondToProposal calls, each with a
distinct 'now' past the proposal window, against one proposal and
asserts: EXPIRED lands on the proposal and both tickets exactly once,
a PROPOSAL_TIMEOUT penalty lands exactly once per offending player
(not once per racing transaction), and no idempotency row survives a
closed-proposal response. The design already defends against this —
ProposalParticipantExpireSQL only ever flips a still-PENDING row
once, so a losing racer's 'now' can't match
recordProposalTimeoutCooldowns' responded_at filter — this test is
what actually proves that holds under real concurrent load rather
than by inspection.

Verified: real postgres:17-alpine container, go test -tags
integration ./store/... -run
TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
-race -count=3 clean; full -tags integration ./store/... -race run
clean; full non-integration go build/vet/test -race clean across
every server package; container removed after the run.

multiplayer-next.md

@@ -1251,7 +1251,7 @@ The allocated-runtime result reporter now keeps a completed match in `RESULTS` u
 |---|---|---|
 | 8.44 `[D:8.3,8.4,8.28,8.31]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Go observability package encodes queue/proposal/match/server IDs and lifecycle stage while recursively redacting auth/relay tokens and credentials. `Service.Log` is wired to mutation and read routes at every outcome, and `cmd/control-plane` writes those events as JSON lines to stderr | `server/observability/` covers correlation fields, nested secret redaction, content-aware credential canaries and unnamed-event rejection; API tests cover lifecycle event wiring without logging error text. A production metrics/traces backend and dashboard/alert routing remain open; the local logger is intentionally stderr-only |
 | 8.45 `[D:8.2,8.44]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Go observability package turns the documented RTT, allocation/connect latency, result-success, API-latency and tick/headroom thresholds into executable window checks; the API exporter emits a bounded cumulative latency histogram suitable for querying the documented p95 API SLO | `server/observability/slo.go`, `metrics.go`, `deploy/observability/prometheus-rules.yaml` and adversarial tests cover healthy/violating/empty windows, fixed operation/status labels, cumulative bucket boundaries, arbitrary-path cardinality safety, and optional API p95/5xx alerts. Production scrape configuration, alert routing, wait/MMR/proposal/flood/cost series and runbooks remain |
-| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | **IN PROGRESS.** Go unit/race coverage spans the current domain/store/supervisor policies, and fuzz targets now exercise queue input, result payload hashing and revision events. The local gate uses the pinned headless Godot container when the native executable is unavailable or crashes by signal, while preserving ordinary nonzero test failures, so its full cross-language suite remains runnable without an image export | `scripts/verify_multiplayer_local.sh` passed end to end on the current tree: Go normal/race/vet, all three bounded fuzz targets, 212 Godot tests, contracts, migrations, and manifests. `server/domain/*_test.go`, `server/store/*_test.go`, `server/supervisor/*_test.go`, `server/migrations/*_test.go` and `server/domain/fuzz_test.go` provide the underlying coverage; PostgreSQL live migration execution now runs clean (§8.5), and four real-concurrency cases are covered against a live database with `-race`: §8.14's queue-heartbeat revision race, §8.18's two-matcher contested-ticket race, §8.30's cross-allocator-replica capacity race, and §8.21/§8.25's concurrent identical-result-submission race; the "lost Redis" fixture is covered live against a real server (§8.14: real TTL expiry, repair-after-`FLUSHALL`; fake Steam/allocator fixtures are §8.47's testkit, already done). Further transaction fixtures (e.g. concurrent proposal-recovery expiry races, live Redis failover mid-write under load) remain |
+| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | **IN PROGRESS.** Go unit/race coverage spans the current domain/store/supervisor policies, and fuzz targets now exercise queue input, result payload hashing and revision events. The local gate uses the pinned headless Godot container when the native executable is unavailable or crashes by signal, while preserving ordinary nonzero test failures, so its full cross-language suite remains runnable without an image export | `scripts/verify_multiplayer_local.sh` passed end to end on the current tree: Go normal/race/vet, all three bounded fuzz targets, 212 Godot tests, contracts, migrations, and manifests. `server/domain/*_test.go`, `server/store/*_test.go`, `server/supervisor/*_test.go`, `server/migrations/*_test.go` and `server/domain/fuzz_test.go` provide the underlying coverage; PostgreSQL live migration execution now runs clean (§8.5), and five real-concurrency cases are covered against a live database with `-race`: §8.14's queue-heartbeat revision race, §8.18's two-matcher contested-ticket race, §8.30's cross-allocator-replica capacity race, §8.21/§8.25's concurrent identical-result-submission race, and now `TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce`, which races 8 concurrent `GetProposal`/`RespondToProposal` calls (mixed read-recovery and a late accept) against one already-expired proposal and proves the design's own defense holds: `ProposalParticipantExpireSQL` only ever flips a still-PENDING row once, so a losing racer's `now` never matches `recordProposalTimeoutCooldowns`' `responded_at = $2` filter and cannot double-apply a `PROPOSAL_TIMEOUT` penalty -- verified against a real PostgreSQL container, `-race`, 3 repeated runs plus a full store-package integration run, all clean; the "lost Redis" fixture is covered live against a real server (§8.14: real TTL expiry, repair-after-`FLUSHALL`; fake Steam/allocator fixtures are §8.47's testkit, already done). Live Redis failover mid-write under load remains |
 | 8.47 `[D:8.7,8.30]` | **IN PROGRESS.** Offline testkit provides deterministic fake Steam verification and fake allocation with forced failure injection; the independent Compose runner drives fake-Steam session issuance, real HTTP queue create/heartbeat/cancel, matcher-backed six-player proposal formation/acceptance, and idempotency-conflict checks | `server/testkit/` covers verified identity/replay, unknown identity, wrong App ID, expiry, no capacity, compatibility-key conflict, idempotent allocation replay and cloud-free forced allocation failure; the Compose API/matcher slice is wired into CI, while live exhaustive matrix and production Steam remain |
 | 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Offline testkit exercises verified queue projection → ranked six-player proposal → ENet allocation → assignment-ready manifest → certified durable result receipt; `compose.allocated-smoke.yml` independently runs the real testkit API, matcher, allocator, maintenance, HTTPS Agones-shaped provider, PostgreSQL, and game-server supervisor with generated TLS, roster, and signed workload credentials. It verifies an expired ranked reconnect becomes one durable abandonment/cooldown, queue/proposal/allocation binding, authenticated result, idempotent retry, shutdown acknowledgment, durable receipt/audit rows, and supervised game-process stop without repurposing the Phase 6 fixture | `scripts/verify_allocated_compose.sh` passed on 2026-09-04 in this workspace; `.github/workflows/allocated-compose.yml` runs `make verify-allocated-compose`. Real Agones/kind and production evidence remain open |
 | 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | **IN PROGRESS.** `scripts/verify_kind_agones.sh` creates a disposable kind cluster, installs pinned Agones, loads the real `game-server` image, applies the Fleet in an explicitly separate Agones-only supervisor/UDP readiness mode, and verifies readiness plus allocation of a dynamic UDP endpoint; `.github/workflows/agones-integration.yml` runs it for infrastructure changes and on demand | The cloud-free runner is committed and fails clearly when Docker/kind/Helm are unavailable. CI/live evidence for production control-plane registration, roster/no-show, both readiness stages, races, multi-match node, result-pending reconciliation, drain, and rollback remains open |

server/store/postgres_integration_test.go

@@ -1020,6 +1020,113 @@ func TestPostgreSQLLateProposalResponseCommitsExpiryRecovery(t *testing.T) {
 	}
 }
 
+// TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce
+// covers the race §8.46 flagged as still open: multiple concurrent recovery
+// paths (a read-side GetProposal from each participant polling for an
+// update, and a RespondToProposal arriving right at the same boundary) can
+// all observe the same past-expiry proposal simultaneously. Every one of
+// them runs the identical expiry-advance SQL in its own transaction, so this
+// proves that racing recovery does not multiply the durable side effects: a
+// PROPOSAL_TIMEOUT cooldown must land exactly once per offending player, not
+// once per racing transaction that happened to perform the PENDING ->
+// TIMED_OUT flip. The design's own defense is that ProposalParticipantExpireSQL
+// only ever flips a still-PENDING row once, and recordProposalTimeoutCooldowns
+// only cooldowns participants whose responded_at equals this transaction's
+// own `now` -- so a loser transaction's `now` simply matches nothing.
+func TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce(t *testing.T) {
+	db := openIntegrationPostgres(t)
+	applyIntegrationMigrations(t, db)
+
+	now := time.Now().UTC().Truncate(time.Microsecond)
+	ctx := context.Background()
+	for _, player := range []string{"race-expiry-a", "race-expiry-b"} {
+		if _, err := db.ExecContext(ctx, `INSERT INTO identities (player_id, steam_id) VALUES ($1, $1)`, player); err != nil {
+			t.Fatal(err)
+		}
+	}
+	for i, player := range []string{"race-expiry-a", "race-expiry-b"} {
+		if _, err := db.ExecContext(ctx, `INSERT INTO queue_tickets (ticket_id, player_id, playlist, state, client_build, protocol_version, enqueued_at, expires_at) VALUES ($1, $2, 'casual', 'QUEUED', 'integration-build', 1, $3, $4)`, fmt.Sprintf("race-expiry-ticket-%d", i), player, now, now.Add(time.Minute)); err != nil {
+			t.Fatal(err)
+		}
+	}
+	proposal, err := domain.NewProposal("race-expiry-proposal", domain.Casual, []string{"race-expiry-a", "race-expiry-b"}, now)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := CreateProposal(ctx, db, proposal, map[string]string{"race-expiry-a": "race-expiry-ticket-0", "race-expiry-b": "race-expiry-ticket-1"}, now); err != nil {
+		t.Fatalf("create proposal: %v", err)
+	}
+
+	late := now.Add(domain.ProposalWindow + time.Second)
+	const racers = 8
+	var wg sync.WaitGroup
+	errs := make([]error, racers)
+	wg.Add(racers)
+	for i := 0; i < racers; i++ {
+		go func(i int) {
+			defer wg.Done()
+			// Each racer's `now` is distinct (and every one is past expiry), so a
+			// real implementation bug would show up as several of them believing
+			// they were the one that performed the PENDING -> TIMED_OUT flip.
+			racerNow := late.Add(time.Duration(i) * time.Millisecond)
+			switch i % 3 {
+			case 0:
+				_, errs[i] = GetProposal(ctx, db, "race-expiry-a", proposal.ProposalID, racerNow)
+			case 1:
+				_, errs[i] = GetProposal(ctx, db, "race-expiry-b", proposal.ProposalID, racerNow)
+			default:
+				_, errs[i] = RespondToProposal(ctx, db, "race-expiry-a", proposal.ProposalID, fmt.Sprintf("race-expiry-key-%04d", i), true, 0, racerNow)
+			}
+		}(i)
+	}
+	wg.Wait()
+	for i, err := range errs {
+		// GetProposal never errors on an already-expired proposal (it's a pure
+		// read-with-recovery); RespondToProposal on an already-closed proposal
+		// must report exactly ErrProposalClosed, nothing else.
+		if err != nil && !errors.Is(err, domain.ErrProposalClosed) {
+			t.Fatalf("racer %d: unexpected error %v", i, err)
+		}
+	}
+
+	var proposalState string
+	if err := db.QueryRow(`SELECT state FROM proposals WHERE proposal_id = 'race-expiry-proposal'`).Scan(&proposalState); err != nil {
+		t.Fatal(err)
+	}
+	if proposalState != "EXPIRED" {
+		t.Fatalf("proposal state = %s, want EXPIRED", proposalState)
+	}
+	var ticketA, ticketB string
+	if err := db.QueryRow(`SELECT state FROM queue_tickets WHERE ticket_id = 'race-expiry-ticket-0'`).Scan(&ticketA); err != nil {
+		t.Fatal(err)
+	}
+	if err := db.QueryRow(`SELECT state FROM queue_tickets WHERE ticket_id = 'race-expiry-ticket-1'`).Scan(&ticketB); err != nil {
+		t.Fatal(err)
+	}
+	if ticketA != "EXPIRED" || ticketB != "EXPIRED" {
+		t.Fatalf("tickets not expired exactly once: a=%s b=%s", ticketA, ticketB)
+	}
+	// The crux of the race: exactly one PROPOSAL_TIMEOUT penalty per player,
+	// however many transactions raced to observe the expiry.
+	var penaltiesA, penaltiesB int
+	if err := db.QueryRow(`SELECT count(*) FROM penalties WHERE kind = 'PROPOSAL_TIMEOUT' AND player_id = 'race-expiry-a'`).Scan(&penaltiesA); err != nil {
+		t.Fatal(err)
+	}
+	if err := db.QueryRow(`SELECT count(*) FROM penalties WHERE kind = 'PROPOSAL_TIMEOUT' AND player_id = 'race-expiry-b'`).Scan(&penaltiesB); err != nil {
+		t.Fatal(err)
+	}
+	if penaltiesA != 1 || penaltiesB != 1 {
+		t.Fatalf("cooldown was not applied exactly once per player: a=%d b=%d", penaltiesA, penaltiesB)
+	}
+	var idempotencyRows int
+	if err := db.QueryRow(`SELECT count(*) FROM idempotency_keys WHERE scope = $1`, ProposalResponseIdempotencyScope).Scan(&idempotencyRows); err != nil {
+		t.Fatal(err)
+	}
+	if idempotencyRows != 0 {
+		t.Fatalf("closed-proposal responses left stray idempotency rows: %d", idempotencyRows)
+	}
+}
+
 // TestPostgreSQLCancellingAProposedTicketImmediatelyRequeuesTheOtherParticipant
 // covers the responsiveness gap the decline/timeout fixes above left bounded
 // but not closed: cancelling a ticket that's part of an OPEN proposal used