diff --git a/multiplayer-next.md b/multiplayer-next.md index 1ce64496..5996ee4b 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1204,7 +1204,7 @@ production fallback. |---|---|---| | 8.14 `[D:8.4,8.5,8.8]` | **IN PROGRESS.** Pure Go queue policy and PostgreSQL enforce one active ticket per verified player, 10 s heartbeat/30 s expiry, retry-safe owner/revision-scoped create/heartbeat/cancel, and deterministic candidate projection. Client cancellation is limited to `QUEUED`/`PROPOSED`; it cannot overwrite match-owned `ACCEPTED` through `LIVE` lifecycle states. A locked rejection classifier maps missing ticket, wrong owner, expiry, stale revision, and invalid state to distinct domain/API outcomes without weakening the atomic mutation predicate. Queue admission also honors both pre-live and live ranked abandonment penalties, so an expired reconnect cannot immediately requeue after result completion. Redis is an optional rebuildable projection over authoritative PostgreSQL | Domain/store/API tests cover ownership, expiry, idempotency, candidate binding, exact mutation-state fences, live-ticket cancellation rejection, stale revision classification, abandonment cooldown selection, concurrent create/heartbeat races, durable-source cache repair, Redis TTL/lost-keyspace behavior, and playlist/build/protocol compatibility. PostgreSQL-tagged lifecycle regressions compile and prior live runs cover the queue races; live database reruns remain blocked by Docker storage. Live Redis failover-under-load and worker integration remain | | 8.15 `[D:7.8,8.3]` | **IN PROGRESS.** Pure Go probe validation treats Steam location as opaque, requires nonce/freshness/region and server-computed RTT, and implements discrepancy quarantine/release; authenticated HTTP now accepts only opaque location/nonce input through a server-owned probe provider and records validated RTT into the active player's durable queue ticket; durable queue projections have a server-derived RTT JSON field for matcher reads | `server/domain/probes.go`, `server/migrations/0003_queue_probe_metadata.sql`, adversarial fixtures and `server/api/service.go`/`store/queue_sql.go` cover stale/wrong/forged evidence, the 25 ms/30% threshold, three-sample quarantine, five-clean release, authenticated provider arguments, rejection of client RTT fields, player/ticket/expiry binding, persistence failure, playlist-scoped candidate reads and bounded metadata decoding; Steam coordinator, regional probe adapters and multi-region probe population remain | -| 8.16 `[D:8.14,8.15]` | **IN PROGRESS.** Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction; the runnable matcher now supports explicitly enabled ranked six-player polling with durable verified-Steam identity lookup. **Fixed a real crash-loop**: `Worker.Run` treated every `RunOnce` error as fatal to the whole loop, including "no compatible candidates" (`FormFromQueue`'s completely routine answer when currently-queued players share no verified region) — found building a live two-player integration attempt (see below): two real players with no common region crashed the entire matcher process, taking matchmaking down for every other player in the playlist, and would crash-loop again on restart since the same incompatible candidates stay queued. Now only genuine static misconfiguration (`ErrWorkerNotConfigured`/`ErrUnsupportedPlaylist`/`ErrInvalidMatcherSize`) stops the loop; everything else retries next interval. **Fixed a second, quieter wedge in the same area**: `FormFromQueue`'s anchor is always the single oldest candidate, deterministically, so when `domain.PrepareProposal` rejected that exact formation for a reason specific to those particular players (mismatched protocol, incomplete ranked identity metadata, a duplicate-SteamID pair) rather than "no compatible batch exists", `RunOnce` returned immediately and the next interval reproduced the identical formation and failed again — forever, permanently head-of-line-blocking every other waiting player behind that anchor too, not just the players actually at fault (this is the "innocent-ticket restoration" gap task 8.20 named: the innocents were never stuck in the database, since no claim had happened yet, but they were durably starved of ever being tried). `RunOnce` now excludes a failed formation's players and retries with the remaining pool, bounded to 8 attempts per pass; a batch that has no viable formation at all (the pre-existing no-common-region case) still returns immediately rather than looping pointlessly. **This fix was inert without a companion one**: `RunOnce` was asking `Source` for exactly `w.Size` candidates -- `SelectCandidates` was always designed to search a larger pool (it takes an anchor plus an arbitrary remainder and widens through it), but the call site never gave it one, so there was never a "remainder" for the exclusion retry to fall back to in production. `RunOnce` now requests up to 10x `w.Size` (capped at 200) instead | `server/domain/matcher.go`, `teams.go`, `server/matcher/worker.go`, `server/store/queue_sql.go` and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure, queue-backed oldest-anchor formation and incomplete ranked identity metadata; two new tests cover `Run` (not just `RunOnce`) surviving a per-pass error via a real concurrent goroutine, and still stopping immediately on a real configuration error, both clean across repeated `-race` runs; three further tests cover the formation-exclusion retry (an 8-candidate batch whose oldest 4 are permanently doomed still forms and claims the remaining 4, excluding the doomed players from the claimed ticket set), that exhausting every attempt still surfaces the last real error rather than a silent `false,nil`, and that `Source` is actually asked for more than `w.Size` candidates. **A live two-player Godot proposal integration attempt is on disk but not committed**: `Game/tests/control_plane_proposal_smoke.gd`/`.tscn` and `scripts/verify_control_plane_proposal_integration.sh` exist and found the original crash-loop bug above, but the session paused running further concurrent headless Godot processes after discovering they'd been causing native engine crashes (macOS crash reporter, `EXC_BAD_ACCESS`/`SIGBUS`) intermittently all session, confirmed by the user; the two-player script was never itself verified to a clean pass. Arena selection and long-running worker integration remain | +| 8.16 `[D:8.14,8.15]` | **IN PROGRESS.** Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction; the runnable matcher now supports explicitly enabled ranked six-player polling with durable verified-Steam identity lookup. **Fixed a real crash-loop**: `Worker.Run` treated every `RunOnce` error as fatal to the whole loop, including "no compatible candidates" (`FormFromQueue`'s completely routine answer when currently-queued players share no verified region) — found building a live two-player integration attempt (see below): two real players with no common region crashed the entire matcher process, taking matchmaking down for every other player in the playlist, and would crash-loop again on restart since the same incompatible candidates stay queued. Now only genuine static misconfiguration (`ErrWorkerNotConfigured`/`ErrUnsupportedPlaylist`/`ErrInvalidMatcherSize`) stops the loop; everything else retries next interval. **Fixed a second, quieter wedge in the same area**: `FormFromQueue`'s anchor is always the single oldest candidate, deterministically, so when `domain.PrepareProposal` rejected that exact formation for a reason specific to those particular players (mismatched protocol, incomplete ranked identity metadata, a duplicate-SteamID pair) rather than "no compatible batch exists", `RunOnce` returned immediately and the next interval reproduced the identical formation and failed again — forever, permanently head-of-line-blocking every other waiting player behind that anchor too, not just the players actually at fault (this is the "innocent-ticket restoration" gap task 8.20 named: the innocents were never stuck in the database, since no claim had happened yet, but they were durably starved of ever being tried). `RunOnce` now excludes a failed formation's players and retries with the remaining pool, bounded to 8 attempts per pass; a batch that has no viable formation at all (the pre-existing no-common-region case) still returns immediately rather than looping pointlessly. **This fix was inert without a companion one**: `RunOnce` was asking `Source` for exactly `w.Size` candidates -- `SelectCandidates` was always designed to search a larger pool (it takes an anchor plus an arbitrary remainder and widens through it), but the call site never gave it one, so there was never a "remainder" for the exclusion retry to fall back to in production. `RunOnce` now requests up to 10x `w.Size` (capped at 200) instead | `server/domain/matcher.go`, `teams.go`, `server/matcher/worker.go`, `server/store/queue_sql.go` and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure, queue-backed oldest-anchor formation and incomplete ranked identity metadata; two new tests cover `Run` (not just `RunOnce`) surviving a per-pass error via a real concurrent goroutine, and still stopping immediately on a real configuration error, both clean across repeated `-race` runs; three further tests cover the formation-exclusion retry (an 8-candidate batch whose oldest 4 are permanently doomed still forms and claims the remaining 4, excluding the doomed players from the claimed ticket set), that exhausting every attempt still surfaces the last real error rather than a silent `false,nil`, and that `Source` is actually asked for more than `w.Size` candidates. **The two-player Godot proposal integration now passes**: `Game/tests/control_plane_proposal_smoke.gd`/`.tscn` and `scripts/verify_control_plane_proposal_integration.sh` found the original crash-loop bug above; headless Godot testing was then paused for several sessions after a run of native engine crashes (macOS crash reporter, `EXC_BAD_ACCESS`/`SIGBUS`) that day. Reading the actual `~/Library/Logs/DiagnosticReports/Godot-*.ips` crash reports (rather than relying on temporal correlation) found every one of the 25 reports on the machine named `ChatGPT`/`codex` (17), an already-exited process under that same tree (6), or a manual `iTerm2` session (1) as the responsible/parent process — none named Claude Code. Godot testing was resumed on that evidence (with the user's explicit go-ahead) and re-verified clean: `test_runner.tscn` (212/212), the full `make verify-enet-integration` suite (all five cases including the 3-process match), `verify_control_plane_proposal_integration.sh` (passed twice, real matcher forms the proposal and both clients accept), and the complete `make verify-multiplayer-local` gate -- zero new crash reports across all of it. Arena selection and long-running worker integration remain | | 8.17 `[D:8.14,8.16]` | **IN PROGRESS.** Pure Go proposal policy sends a 10-second response window to every selected human, requires unanimous acceptance, applies exact casual/ranked decline and timeout cooldowns with ranked escalation, and exposes revisioned idempotent responses through the authenticated API. Proposal closure now atomically separates offenders from innocents: a decliner's ticket is `CANCELLED`; a timed-out player's ticket is `EXPIRED`; accepted or otherwise innocent participants return to `QUEUED` with their original `enqueued_at` and refreshed expiry. Direct queue cancellation closes the open proposal and requeues remaining participants immediately. Late API responses commit expiry, timeout penalties, and ticket release before returning `ErrProposalClosed`; recovery of an old declined proposal cannot misclassify its pending innocents as timeouts. Cooldown history rejects future, foreign-playlist, and invalid-kind events, and database rows are closed before penalty writes | Domain/store/API fixtures cover partial/unanimous response, expiry, replay/conflict, stale revision, exact cooldown windows/escalation, corrupt history filtering, offender ticket termination, innocent precedence preservation, direct-cancel cascade, and the former late-response rollback. PostgreSQL-tagged regressions compile and assert the durable split and penalty rows; the full local Go suite passes. Live PostgreSQL execution and allocation integration remain | | 8.18 `[D:8.5,8.14,8.17]` | **IN PROGRESS.** Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using `FOR UPDATE SKIP LOCKED` plus durable uniqueness/revision fences; proposal creation persists matcher-selected region/protocol/team/slot topology before acceptance, and the same serializable final-acceptance transaction now promotes the exact roster into one `ALLOCATING` match, closing the process-crash gap that could otherwise strand an accepted proposal before the former second promotion transaction. The API promoter remains a replay check. Promotion replay validates immutable playlist/region/protocol/arena, participant, ticket, team, and slot identity but deliberately ignores mutable match state/server ownership, so a retry after a lost response still succeeds after allocation has advanced. Result sets are closed before crossing into promotion writes, avoiding one-connection pool stalls. Redis remains a rebuildable candidate projection over PostgreSQL authority | Store/API tests cover retries, claims, owner/revision fencing, expiry, exact promotion replay/conflict, progressed-match replay, rollback of partial claims, concurrent contested-ticket formation, and lost-cache repair. PostgreSQL-tagged regressions compile and assert acceptance, ticket transitions, match creation, and roster insertion are one durable outcome; prior live runs covered queue/proposal promotion and races, while this atomic-promotion change awaits a live database rerun. Allocation runtime integration remains | | 8.19 `[D:8.18]` | **IN PROGRESS.** Pure Go casual lineup requires 2–6 humans with at least one per team, fills missing slots with explicit bots, permits kickoff-only bot-slot backfill and assigns no backfill penalty/rating update; proposal preparation now derives the lineup from formed teams | `server/domain/casual.go`, `formation.go` cover both-team minimum, bot shape, live-play rejection, zero-penalty backfill and casual proposal composition; queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties and live integration remain |