test(multiplayer): cover generic mutation retry recovery

Closes most of §8.43's 'decline, regional outage retry UI, failed
reconnect, duplicate-action recovery beyond proposals' remaining
list -- turned out to be mostly stale doc, not missing code.

matchmaking.gd's decline button/handler already existed
(%DeclineButton, _on_decline_pressed, visibility toggled by
MatchmakingState.PROPOSED alongside accept). ControlPlaneClient's
can_retry_last_mutation()/retry_last_mutation() -- the generic
'duplicate-action recovery beyond proposals' and 'regional outage
retry' mechanism -- also already existed: any mutation (not just a
proposal response) becomes retryable on a transport failure or a
408/429/503 response, and matchmaking.gd's queue button already fell
back to it ('Retry Request'). Neither had any test coverage proving
the mechanism actually works for a non-proposal mutation --
is_retryable_mutation_response's pure classification was the only
thing tested.

Two new tests: test_generic_mutation_retry_recovers_after_a_transient_failure
proves can_retry_last_mutation() transitions from false (mutation
in flight) to true after a transport-level failure on an ordinary
queue_heartbeat, exactly the 'regional outage' case; test_generic_mutation_retry_is_not_offered_for_unsafe_failures
proves a 409 (revision conflict) is never offered as a blind retry
and that retry_last_mutation() fails closed with ERR_INVALID_DATA
rather than resending a stale mutation. retry_last_mutation's literal
network dispatch (HTTPRequest.request()) is not exercised -- it needs
a live SceneTree that test_runner.tscn's synchronous single-_ready()
execution model cannot provide mid-suite; the two tests cover the
can_retry_last_mutation() decision boundary and the fail-closed path
instead, which is what's actually new here.

Verified against the real Godot 4.7.1 binary now that headless
testing has resumed: test_runner.tscn 214/214 clean (no crash, no
engine-level error), full make verify-multiplayer-local re-run clean,
zero new crash reports.

Remaining in §8.43: version-mismatch-specific messaging (a protocol
rejection currently surfaces only as the server's generic error
string), failed-reconnect UX, and §8.16's arena selection/long-running
worker integration.

Game/tests/cases/test_control_plane_client.gd

@@ -181,6 +181,56 @@ func test_retryable_mutation_policy_only_retries_safe_failures() -> void:
 	assert_true(not ControlPlaneClient.is_retryable_mutation_response(409), "revision/idempotency conflict is not blindly replayed")
 
 
+# multiplayer-next.md 8.43 named "duplicate-action recovery beyond proposals"
+# and "regional outage retry UI" as remaining. Both mechanisms (can_retry_last_mutation /
+# retry_last_mutation, and matchmaking.gd's queue button falling back to them)
+# already existed in the client, but had no test coverage proving the
+# generic (non-proposal) mutation path actually recovers end to end -- only
+# is_retryable_mutation_response's pure classification was covered above.
+func test_generic_mutation_retry_recovers_after_a_transient_failure() -> void:
+	var client := ControlPlaneClient.new()
+	client._ready()
+	assert_true(client.configure("https://match.example", "session-id:opaque-token"), "client configures")
+	assert_true(client.state.begin_queue("ticket-retry-generic", "casual"), "queue setup succeeds")
+	# Simulate what _start_request itself would already have recorded before
+	# a real network call was in flight, the same way the pre-existing
+	# conflict-handler tests above set _operation directly.
+	client._operation = "queue_heartbeat"
+	client._last_mutation = {"operation": "queue_heartbeat", "method": HTTPClient.METHOD_POST, "path": "/v1/queue/ticket-retry-generic/heartbeat", "payload": {"revision": 0}, "key": "heartbeat-retry-key-123456", "expected_revision": 0}
+	assert_true(not client.can_retry_last_mutation(), "a mutation still in flight is never offered as retryable")
+
+	# A regional outage: the transport itself failed rather than returning a
+	# decoded HTTP status -- exactly the "regional outage retry" case. This
+	# transition is the actual previously-uncovered boundary: nothing tested
+	# that a generic (non-proposal) mutation ever becomes retryable at all,
+	# only is_retryable_mutation_response's pure classification above.
+	# retry_last_mutation's own dispatch is not exercised here: it reaches
+	# HTTPRequest.request(), which needs the node inside a live SceneTree,
+	# and test_runner.tscn runs every test method from within its own
+	# _ready() while the tree is still being built, so that is out of reach
+	# for this harness -- the "not offered at all" boundary below covers the
+	# part of retry_last_mutation this environment can exercise safely.
+	client._on_request_completed(HTTPRequest.RESULT_CANT_CONNECT, 0, PackedStringArray(), PackedByteArray())
+	assert_true(client.can_retry_last_mutation(), "a transport failure on a non-proposal mutation is offered as retryable")
+	client.free()
+
+
+func test_generic_mutation_retry_is_not_offered_for_unsafe_failures() -> void:
+	var client := ControlPlaneClient.new()
+	client._ready()
+	assert_true(client.configure("https://match.example", "session-id:opaque-token"), "client configures")
+	assert_true(client.state.begin_queue("ticket-retry-unsafe", "casual"), "queue setup succeeds")
+	client._operation = "queue_cancel"
+	client._last_mutation = {"operation": "queue_cancel", "method": HTTPClient.METHOD_POST, "path": "/v1/queue/ticket-retry-unsafe/cancel", "payload": {}, "key": "cancel-retry-key-123456", "expected_revision": 0}
+	# A 409 is a revision/idempotency conflict, not a transient failure --
+	# should_recover_queue_after_conflict owns recovering it instead, and a
+	# blind resend would replay a mutation whose precondition already failed.
+	client._on_request_completed(HTTPRequest.RESULT_SUCCESS, 409, PackedStringArray(), JSON.stringify({"error": "revision conflict"}).to_utf8_buffer())
+	assert_true(not client.can_retry_last_mutation(), "a conflict response is never offered as a blind retry")
+	assert_eq(client.retry_last_mutation(), ERR_INVALID_DATA, "retrying when not offered fails closed rather than resending a stale mutation")
+	client.free()
+
+
 func test_rest_resource_identifiers_use_the_opaque_contract_shape() -> void:
 	assert_true(ControlPlaneClient.is_valid_resource_id("ticket_1234567890"), "contract-sized resource id is accepted")
 	assert_true(not ControlPlaneClient.is_valid_resource_id("ticket-1"), "short resource id is rejected")

multiplayer-next.md

@@ -1243,7 +1243,7 @@ The allocated-runtime result reporter now keeps a completed match in `RESULTS` u
 | 8.40 `[D:8.3,8.14]` | **IN PROGRESS.** Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated `/v1/events` WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot `ControlPlaneClient` can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return, including a deferred recovery when an event arrives during an in-flight HTTP mutation; `OutboxDispatcher` now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match; proposal creation writes one durable targeted `proposal_changed` outbox event per proposal aggregate revision, while result completion writes `match_completed` and production `cmd/control-plane` plus the test-only API harness dispatch both event types through separate filtered consumers | `server/domain/sync.go`, `server/api/events.go`, `server/api/outbox.go`, `server/api/service.go`, `server/store/outbox.go`, `server/store/proposal_sql.go`, `server/cmd/control-plane`, `server/cmd/testkit-api`, `service_test.go`, `outbox_test.go`, `matchmaking_state.gd`, `control_plane_client.gd`, `local_prediction_history.gd`, `net_ship_predictor.gd` and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation, prediction warm-up/hard-resync separation, deferred proposal recovery and proposal/result outbox filtering and delivery failures; Godot 4.7.1 headless project parse and 150-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; `cmd/control-plane` provides a signal-bound API role, `cmd/matcher` now supports casual and explicitly enabled ranked roles with durable identity lookup, and `cmd/maintenance` provides bounded season maintenance. **Live multi-process control-plane/game verification now exists**: `scripts/verify_control_plane_integration.sh` runs a real `postgres:17-alpine`, the real `api.Service` (via the new test-only `server/cmd/testkit-api`, wired identically to `cmd/control-plane` except for a fake Steam login — see §8.7), and a real headless Godot client (`control_plane_smoke.gd`) round-tripping login → `fetch_ranked_profile` (expect 404, §8.22) → `queue_create` → heartbeat → `cancel_queue` over an actual network connection — the first time this boundary was exercised end to end rather than against a mock on either side. **The two-player proposal extension now also passes**: `scripts/verify_control_plane_proposal_integration.sh` starts isolated PostgreSQL, the real matcher and testkit API, launches two headless Godot clients, seeds deterministic RTT projections, observes the real WebSocket `OPEN` proposal, accepts from both clients, and verifies both durable tickets plus the proposal reach `ACCEPTED` (including authoritative recovery when the concurrent revision advances). `scripts/run_result_fanout_integration.sh` additionally verifies a real PostgreSQL-backed authenticated WebSocket receives a completed-match event; allocator and Redis fan-out live verification remain |
 | 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows; a workload-authenticated `GET /v1/servers/{serverId}/roster` now returns the complete signed envelope set only for the bound allocation, and the allocated supervisor atomically materializes it before launching Godot; live authenticated Godot/PostgreSQL assignment verification now exists via `scripts/verify_assignment_integration.sh`, which seeds a durable player-scoped `ASSIGNMENT_READY` row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary); `scripts/run_supervisor_integration.sh` additionally verifies the allocated-server process-ready → roster fetch/materialization → assignment-ready registration path against real PostgreSQL and the real API, with the workload token recovered from an Agones-shaped GameServer annotation | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql`, `server/allocator/allocator_integration_test.go`, `server/supervisor/supervisor_integration_test.go`, `server/supervisor/supervisor.go` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior, workload-authenticated roster delivery, atomic file installation, real allocator reconciliation, real supervisor registration and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; dynamic per-match launch flags, SDR relay-ticket installation and live Agones cluster integration remain |
 | 8.42 `[D:8.22,8.23,8.24,8.40]` | **IN PROGRESS.** `RankedProfileState` and `ControlPlaneClient.fetch_ranked_profile()` expose the backend-authoritative rating/RD/volatility/games/tier/provisional/season view; matchmaking UI displays provisional/tier status without client-side rating math; runnable API binaries now wire the backend-owned default tier policy; live authenticated profile verification now exists via `scripts/verify_ranked_profile_integration.sh`, which seeds a durable rating and verifies the populated profile over real PostgreSQL, Go HTTP, and Godot JSON | `test_control_plane_client.gd` validates profile shape, numeric safety and provisional display; the real populated profile path now covers durable rating/games/tier/provisional decoding and caught the missing runtime tier-policy wiring; committed revision after reconnect, abandon status and season countdown remain dependent on live auth/backend events and Godot runtime verification |
-| 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path; decline, version mismatch, regional outage retry UI, failed reconnect, duplicate-action recovery beyond proposals and broader live Godot verification remain |
+| 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path. **This row's own "remain" list was stale**: `matchmaking.gd`'s decline button/handler already existed (`%DeclineButton`, `_on_decline_pressed`, visibility toggled by `MatchmakingState.PROPOSED` alongside accept) with no test gap; `ControlPlaneClient.can_retry_last_mutation()`/`retry_last_mutation()` -- the generic "duplicate-action recovery beyond proposals" and "regional outage retry" mechanism -- also already existed (any mutation, not just a proposal response, becomes retryable on a transport failure or 503/408/429, and the matchmaking queue button already fell back to it), it simply had zero test coverage proving the transition actually happens for a non-proposal mutation; two new tests close that (`test_generic_mutation_retry_recovers_after_a_transient_failure`, `test_generic_mutation_retry_is_not_offered_for_unsafe_failures`). `MatchmakingClient`'s real dispatch (`HTTPRequest.request()`) needs a live SceneTree that `test_runner.tscn`'s synchronous single-`_ready()` execution model cannot provide mid-suite, so the two new tests exercise the `can_retry_last_mutation()` decision boundary and the `ERR_INVALID_DATA` fail-closed path rather than the literal network call; verified against the real Godot 4.7.1 binary (214/214 tests, no crash, no engine-level error), plus a full `make verify-multiplayer-local` re-run. Version-mismatch-specific messaging (a protocol rejection currently surfaces only as the server's generic error string, not a distinguished "update your client" affordance), failed-reconnect UX, and arena selection/long-running worker integration (§8.16) remain |
 
 #### 8F — Observability, verification, cost and rollout