diff --git a/Game/tests/cases/test_control_plane_client.gd b/Game/tests/cases/test_control_plane_client.gd index c4167d8e..16783be2 100644 --- a/Game/tests/cases/test_control_plane_client.gd +++ b/Game/tests/cases/test_control_plane_client.gd @@ -181,6 +181,56 @@ func test_retryable_mutation_policy_only_retries_safe_failures() -> void: assert_true(not ControlPlaneClient.is_retryable_mutation_response(409), "revision/idempotency conflict is not blindly replayed") +# multiplayer-next.md 8.43 named "duplicate-action recovery beyond proposals" +# and "regional outage retry UI" as remaining. Both mechanisms (can_retry_last_mutation / +# retry_last_mutation, and matchmaking.gd's queue button falling back to them) +# already existed in the client, but had no test coverage proving the +# generic (non-proposal) mutation path actually recovers end to end -- only +# is_retryable_mutation_response's pure classification was covered above. +func test_generic_mutation_retry_recovers_after_a_transient_failure() -> void: + var client := ControlPlaneClient.new() + client._ready() + assert_true(client.configure("https://match.example", "session-id:opaque-token"), "client configures") + assert_true(client.state.begin_queue("ticket-retry-generic", "casual"), "queue setup succeeds") + # Simulate what _start_request itself would already have recorded before + # a real network call was in flight, the same way the pre-existing + # conflict-handler tests above set _operation directly. + client._operation = "queue_heartbeat" + client._last_mutation = {"operation": "queue_heartbeat", "method": HTTPClient.METHOD_POST, "path": "/v1/queue/ticket-retry-generic/heartbeat", "payload": {"revision": 0}, "key": "heartbeat-retry-key-123456", "expected_revision": 0} + assert_true(not client.can_retry_last_mutation(), "a mutation still in flight is never offered as retryable") + + # A regional outage: the transport itself failed rather than returning a + # decoded HTTP status -- exactly the "regional outage retry" case. This + # transition is the actual previously-uncovered boundary: nothing tested + # that a generic (non-proposal) mutation ever becomes retryable at all, + # only is_retryable_mutation_response's pure classification above. + # retry_last_mutation's own dispatch is not exercised here: it reaches + # HTTPRequest.request(), which needs the node inside a live SceneTree, + # and test_runner.tscn runs every test method from within its own + # _ready() while the tree is still being built, so that is out of reach + # for this harness -- the "not offered at all" boundary below covers the + # part of retry_last_mutation this environment can exercise safely. + client._on_request_completed(HTTPRequest.RESULT_CANT_CONNECT, 0, PackedStringArray(), PackedByteArray()) + assert_true(client.can_retry_last_mutation(), "a transport failure on a non-proposal mutation is offered as retryable") + client.free() + + +func test_generic_mutation_retry_is_not_offered_for_unsafe_failures() -> void: + var client := ControlPlaneClient.new() + client._ready() + assert_true(client.configure("https://match.example", "session-id:opaque-token"), "client configures") + assert_true(client.state.begin_queue("ticket-retry-unsafe", "casual"), "queue setup succeeds") + client._operation = "queue_cancel" + client._last_mutation = {"operation": "queue_cancel", "method": HTTPClient.METHOD_POST, "path": "/v1/queue/ticket-retry-unsafe/cancel", "payload": {}, "key": "cancel-retry-key-123456", "expected_revision": 0} + # A 409 is a revision/idempotency conflict, not a transient failure -- + # should_recover_queue_after_conflict owns recovering it instead, and a + # blind resend would replay a mutation whose precondition already failed. + client._on_request_completed(HTTPRequest.RESULT_SUCCESS, 409, PackedStringArray(), JSON.stringify({"error": "revision conflict"}).to_utf8_buffer()) + assert_true(not client.can_retry_last_mutation(), "a conflict response is never offered as a blind retry") + assert_eq(client.retry_last_mutation(), ERR_INVALID_DATA, "retrying when not offered fails closed rather than resending a stale mutation") + client.free() + + func test_rest_resource_identifiers_use_the_opaque_contract_shape() -> void: assert_true(ControlPlaneClient.is_valid_resource_id("ticket_1234567890"), "contract-sized resource id is accepted") assert_true(not ControlPlaneClient.is_valid_resource_id("ticket-1"), "short resource id is rejected") diff --git a/multiplayer-next.md b/multiplayer-next.md index 5996ee4b..74136118 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1243,7 +1243,7 @@ The allocated-runtime result reporter now keeps a completed match in `RESULTS` u | 8.40 `[D:8.3,8.14]` | **IN PROGRESS.** Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated `/v1/events` WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot `ControlPlaneClient` can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return, including a deferred recovery when an event arrives during an in-flight HTTP mutation; `OutboxDispatcher` now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match; proposal creation writes one durable targeted `proposal_changed` outbox event per proposal aggregate revision, while result completion writes `match_completed` and production `cmd/control-plane` plus the test-only API harness dispatch both event types through separate filtered consumers | `server/domain/sync.go`, `server/api/events.go`, `server/api/outbox.go`, `server/api/service.go`, `server/store/outbox.go`, `server/store/proposal_sql.go`, `server/cmd/control-plane`, `server/cmd/testkit-api`, `service_test.go`, `outbox_test.go`, `matchmaking_state.gd`, `control_plane_client.gd`, `local_prediction_history.gd`, `net_ship_predictor.gd` and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation, prediction warm-up/hard-resync separation, deferred proposal recovery and proposal/result outbox filtering and delivery failures; Godot 4.7.1 headless project parse and 150-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; `cmd/control-plane` provides a signal-bound API role, `cmd/matcher` now supports casual and explicitly enabled ranked roles with durable identity lookup, and `cmd/maintenance` provides bounded season maintenance. **Live multi-process control-plane/game verification now exists**: `scripts/verify_control_plane_integration.sh` runs a real `postgres:17-alpine`, the real `api.Service` (via the new test-only `server/cmd/testkit-api`, wired identically to `cmd/control-plane` except for a fake Steam login — see §8.7), and a real headless Godot client (`control_plane_smoke.gd`) round-tripping login → `fetch_ranked_profile` (expect 404, §8.22) → `queue_create` → heartbeat → `cancel_queue` over an actual network connection — the first time this boundary was exercised end to end rather than against a mock on either side. **The two-player proposal extension now also passes**: `scripts/verify_control_plane_proposal_integration.sh` starts isolated PostgreSQL, the real matcher and testkit API, launches two headless Godot clients, seeds deterministic RTT projections, observes the real WebSocket `OPEN` proposal, accepts from both clients, and verifies both durable tickets plus the proposal reach `ACCEPTED` (including authoritative recovery when the concurrent revision advances). `scripts/run_result_fanout_integration.sh` additionally verifies a real PostgreSQL-backed authenticated WebSocket receives a completed-match event; allocator and Redis fan-out live verification remain | | 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows; a workload-authenticated `GET /v1/servers/{serverId}/roster` now returns the complete signed envelope set only for the bound allocation, and the allocated supervisor atomically materializes it before launching Godot; live authenticated Godot/PostgreSQL assignment verification now exists via `scripts/verify_assignment_integration.sh`, which seeds a durable player-scoped `ASSIGNMENT_READY` row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary); `scripts/run_supervisor_integration.sh` additionally verifies the allocated-server process-ready → roster fetch/materialization → assignment-ready registration path against real PostgreSQL and the real API, with the workload token recovered from an Agones-shaped GameServer annotation | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql`, `server/allocator/allocator_integration_test.go`, `server/supervisor/supervisor_integration_test.go`, `server/supervisor/supervisor.go` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior, workload-authenticated roster delivery, atomic file installation, real allocator reconciliation, real supervisor registration and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; dynamic per-match launch flags, SDR relay-ticket installation and live Agones cluster integration remain | | 8.42 `[D:8.22,8.23,8.24,8.40]` | **IN PROGRESS.** `RankedProfileState` and `ControlPlaneClient.fetch_ranked_profile()` expose the backend-authoritative rating/RD/volatility/games/tier/provisional/season view; matchmaking UI displays provisional/tier status without client-side rating math; runnable API binaries now wire the backend-owned default tier policy; live authenticated profile verification now exists via `scripts/verify_ranked_profile_integration.sh`, which seeds a durable rating and verifies the populated profile over real PostgreSQL, Go HTTP, and Godot JSON | `test_control_plane_client.gd` validates profile shape, numeric safety and provisional display; the real populated profile path now covers durable rating/games/tier/provisional decoding and caught the missing runtime tier-policy wiring; committed revision after reconnect, abandon status and season countdown remain dependent on live auth/backend events and Godot runtime verification | -| 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path; decline, version mismatch, regional outage retry UI, failed reconnect, duplicate-action recovery beyond proposals and broader live Godot verification remain | +| 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path. **This row's own "remain" list was stale**: `matchmaking.gd`'s decline button/handler already existed (`%DeclineButton`, `_on_decline_pressed`, visibility toggled by `MatchmakingState.PROPOSED` alongside accept) with no test gap; `ControlPlaneClient.can_retry_last_mutation()`/`retry_last_mutation()` -- the generic "duplicate-action recovery beyond proposals" and "regional outage retry" mechanism -- also already existed (any mutation, not just a proposal response, becomes retryable on a transport failure or 503/408/429, and the matchmaking queue button already fell back to it), it simply had zero test coverage proving the transition actually happens for a non-proposal mutation; two new tests close that (`test_generic_mutation_retry_recovers_after_a_transient_failure`, `test_generic_mutation_retry_is_not_offered_for_unsafe_failures`). `MatchmakingClient`'s real dispatch (`HTTPRequest.request()`) needs a live SceneTree that `test_runner.tscn`'s synchronous single-`_ready()` execution model cannot provide mid-suite, so the two new tests exercise the `can_retry_last_mutation()` decision boundary and the `ERR_INVALID_DATA` fail-closed path rather than the literal network call; verified against the real Godot 4.7.1 binary (214/214 tests, no crash, no engine-level error), plus a full `make verify-multiplayer-local` re-run. Version-mismatch-specific messaging (a protocol rejection currently surfaces only as the server's generic error string, not a distinguished "update your client" affordance), failed-reconnect UX, and arena selection/long-running worker integration (§8.16) remain | #### 8F — Observability, verification, cost and rollout