diff --git a/multiplayer-next.md b/multiplayer-next.md index baa73e40..b38a2a16 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1235,7 +1235,7 @@ the local/CI/community transport, not a silent production fallback. |---|---|---| | 8.39 `[D:8.3,8.14,8.17]` | **IN PROGRESS.** `MatchmakingState` now projects queue → proposal → allocation/process-ready/assignment-ready/connect/live plus terminal failure states; autoload `ControlPlaneClient` provides authenticated queue create/recovery/heartbeat/cancel and proposal response requests with idempotency/revision headers; `matchmaking.tscn`/`matchmaking.gd` expose the state and authoritative actions from the main menu; authenticated proposal recovery now reconciles missed proposal events and expires them at read time; the Go API publishes targeted authenticated revisioned queue/proposal/assignment events and the Godot client consumes state/proposal/assignment events, fetching the authoritative assignment after assignment readiness; the API can now use the durable participant-scoped proposal provider for both recovery and accept/decline mutations | `test_matchmaking_state.gd`, `test_control_plane_client.gd`, `test_matchmaking_ui.gd`, `TestProposalRecoveryIsParticipantScopedAndExpiresAtReadBoundary`, `TestAuthenticatedWebSocketDeliversOnlyTargetedRevisionedEvents`, `TestStateChangingAPIActionsPublishTargetedEvents`, `TestProposalRecoveryUsesDurableBackendAndRemainsParticipantScoped` and store proposal SQL tests reject stale/gapped/conflicting updates, validate endpoint/token/payload normalization, preserve idempotent duplicates, and guarantee visible phase/terminal copy; wiring durable allocation events into the outbox, wait/latency explanations and Godot runtime verification remain | | 8.40 `[D:8.3,8.14]` | **IN PROGRESS.** Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated `/v1/events` WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot `ControlPlaneClient` can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return, including a deferred recovery when an event arrives during an in-flight HTTP mutation; `OutboxDispatcher` now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match; proposal creation now writes one durable targeted `proposal_changed` outbox event per proposal aggregate revision, and both production `cmd/control-plane` and the test-only API harness dispatch only that event type to authenticated participants, leaving result events for their separate consumer | `server/domain/sync.go`, `server/api/events.go`, `server/api/outbox.go`, `server/api/service.go`, `server/store/outbox.go`, `server/store/proposal_sql.go`, `server/cmd/control-plane`, `server/cmd/testkit-api`, `service_test.go`, `outbox_test.go`, `matchmaking_state.gd`, `control_plane_client.gd`, `local_prediction_history.gd`, `net_ship_predictor.gd` and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation, prediction warm-up/hard-resync separation, deferred proposal recovery and proposal outbox filtering/delivery failures; Godot 4.7.1 headless project parse and 150-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; `cmd/control-plane` provides a signal-bound API role, `cmd/matcher` now supports casual and explicitly enabled ranked roles with durable identity lookup, and `cmd/maintenance` provides bounded season maintenance. **Live multi-process control-plane/game verification now exists**: `scripts/verify_control_plane_integration.sh` runs a real `postgres:17-alpine`, the real `api.Service` (via the new test-only `server/cmd/testkit-api`, wired identically to `cmd/control-plane` except for a fake Steam login — see §8.7), and a real headless Godot client (`control_plane_smoke.gd`) round-tripping login → `fetch_ranked_profile` (expect 404, §8.22) → `queue_create` → heartbeat → `cancel_queue` over an actual network connection — the first time this boundary was exercised end to end rather than against a mock on either side. **The two-player proposal extension now also passes**: `scripts/verify_control_plane_proposal_integration.sh` starts isolated PostgreSQL, the real matcher and testkit API, launches two headless Godot clients, seeds deterministic RTT projections, observes the real WebSocket `OPEN` proposal, accepts from both clients, and verifies both durable tickets plus the proposal reach `ACCEPTED` (including authoritative recovery when the concurrent revision advances). Allocator and Redis fan-out live verification remain | -| 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows; live authenticated Godot/PostgreSQL assignment verification now exists via `scripts/verify_assignment_integration.sh`, which seeds a durable player-scoped `ASSIGNMENT_READY` row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary) | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; SDR relay-ticket installation and live allocated-token process integration remain | +| 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows; live authenticated Godot/PostgreSQL assignment verification now exists via `scripts/verify_assignment_integration.sh`, which seeds a durable player-scoped `ASSIGNMENT_READY` row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary); `scripts/run_supervisor_integration.sh` additionally verifies the allocated-server process-ready → assignment-ready registration path against real PostgreSQL and the real API, with the workload token recovered from an Agones-shaped GameServer annotation | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql`, `server/allocator/allocator_integration_test.go`, `server/supervisor/supervisor_integration_test.go` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior, real allocator reconciliation, real supervisor registration and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; SDR relay-ticket installation and live Agones cluster integration remain | | 8.42 `[D:8.22,8.23,8.24,8.40]` | **IN PROGRESS.** `RankedProfileState` and `ControlPlaneClient.fetch_ranked_profile()` expose the backend-authoritative rating/RD/volatility/games/tier/provisional/season view; matchmaking UI displays provisional/tier status without client-side rating math; runnable API binaries now wire the backend-owned default tier policy; live authenticated profile verification now exists via `scripts/verify_ranked_profile_integration.sh`, which seeds a durable rating and verifies the populated profile over real PostgreSQL, Go HTTP, and Godot JSON | `test_control_plane_client.gd` validates profile shape, numeric safety and provisional display; the real populated profile path now covers durable rating/games/tier/provisional decoding and caught the missing runtime tier-policy wiring; committed revision after reconnect, abandon status and season countdown remain dependent on live auth/backend events and Godot runtime verification | | 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path; decline, version mismatch, regional outage retry UI, failed reconnect, duplicate-action recovery beyond proposals and broader live Godot verification remain | diff --git a/scripts/run_supervisor_integration.sh b/scripts/run_supervisor_integration.sh new file mode 100755 index 00000000..cd3c7b3a --- /dev/null +++ b/scripts/run_supervisor_integration.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root="$(cd "$(dirname "$0")/.." && pwd)" +container_name="cosmic-clash-supervisor-integration" +database="cosmic_clash_test" +user="cosmic_clash_test" +password="cosmic_clash_test" + +cleanup() { docker rm -f "$container_name" >/dev/null 2>&1 || true; } +trap cleanup EXIT +cleanup +docker run --rm -d --name "$container_name" \ + -e POSTGRES_DB="$database" -e POSTGRES_USER="$user" -e POSTGRES_PASSWORD="$password" \ + -p 55437:5432 postgres:17-alpine >/dev/null +for attempt in $(seq 1 30); do + if docker exec "$container_name" pg_isready -U "$user" -d "$database" >/dev/null 2>&1; then + break + fi + if [ "$attempt" = 30 ]; then + echo "PostgreSQL did not become ready" >&2 + exit 1 + fi + sleep 1 +done +cd "$repo_root/server" +COSMIC_CLASH_POSTGRES_DSN="postgres://${user}:${password}@127.0.0.1:55437/${database}?sslmode=disable" \ + go test -tags integration ./supervisor -count=1 diff --git a/server/supervisor/supervisor_integration_test.go b/server/supervisor/supervisor_integration_test.go new file mode 100644 index 00000000..9a83317a --- /dev/null +++ b/server/supervisor/supervisor_integration_test.go @@ -0,0 +1,133 @@ +//go:build integration + +package supervisor + +import ( + "context" + "database/sql" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + "time" + + "github.com/cosmic-clash/cosmic-clash/server/api" + "github.com/cosmic-clash/cosmic-clash/server/domain" + "github.com/cosmic-clash/cosmic-clash/server/migrations" + "github.com/cosmic-clash/cosmic-clash/server/store" + "github.com/cosmic-clash/cosmic-clash/server/workload" + _ "github.com/jackc/pgx/v5/stdlib" +) + +func TestRealSupervisorRegistersAllocatedServerThroughControlPlane(t *testing.T) { + dsn := os.Getenv("COSMIC_CLASH_POSTGRES_DSN") + if dsn == "" { + t.Skip("COSMIC_CLASH_POSTGRES_DSN is not set") + } + db, err := sql.Open("pgx", dsn) + if err != nil { + t.Fatal(err) + } + defer db.Close() + ctx := context.Background() + if err := db.PingContext(ctx); err != nil { + t.Fatal(err) + } + if _, err := db.ExecContext(ctx, `DROP TABLE IF EXISTS schema_migrations, assignments, audit_events, outbox, result_receipts, ranked_season_rollovers, penalties, seasons, ratings, match_participants, matches, proposal_participants, proposals, allocations, game_servers, queue_tickets, idempotency_keys, sessions, identities CASCADE`); err != nil { + t.Fatal(err) + } + if err := migrations.Apply(ctx, db, filepath.Join("..", "migrations")); err != nil { + t.Fatal(err) + } + now := time.Now().UTC().Truncate(time.Microsecond) + players := []string{"supervisor-live-a", "supervisor-live-b"} + for index, player := range players { + if _, err := db.ExecContext(ctx, `INSERT INTO identities (player_id, steam_id) VALUES ($1, $1)`, player); err != nil { + t.Fatal(err) + } + if _, err := db.ExecContext(ctx, `INSERT INTO queue_tickets (ticket_id, player_id, playlist, state, client_build, protocol_version, enqueued_at, expires_at) VALUES ($1, $2, 'casual', 'ACCEPTED', 'build-1', 1, $3, $4)`, fmt.Sprintf("supervisor-live-ticket-%d", index), player, now, now.Add(time.Hour)); err != nil { + t.Fatal(err) + } + } + if _, err := db.ExecContext(ctx, `INSERT INTO matches (match_id, playlist, state, region, protocol_version) VALUES ('supervisor-live-match', 'casual', 'ALLOCATING', 'EU', 1)`); err != nil { + t.Fatal(err) + } + for index, player := range players { + if _, err := db.ExecContext(ctx, `INSERT INTO match_participants (match_id, player_id, ticket_id, slot, team) VALUES ('supervisor-live-match', $1, $2, $3, $4)`, player, fmt.Sprintf("supervisor-live-ticket-%d", index), index, index); err != nil { + t.Fatal(err) + } + } + if err := store.RegisterReadyServer(ctx, db, domain.ReadyServer{ServerID: "supervisor-live-server", Region: "EU", Build: "build-1", Protocol: 1, Transport: "enet", State: domain.ServerReady}, now); err != nil { + t.Fatal(err) + } + claim, found, err := store.ClaimAllocatingMatch(ctx, db, "enet", now) + if err != nil || !found { + t.Fatalf("claim allocating match found=%t err=%v", found, err) + } + request := claim.Request + allocation, err := store.ClaimAllocation(ctx, db, request, now) + if err != nil { + t.Fatal(err) + } + if err := store.BindAllocatedMatch(ctx, db, allocation); err != nil { + t.Fatal(err) + } + for index, player := range players { + if err := store.SaveAssignment(ctx, db, store.DurableAssignment{ + MatchID: "supervisor-live-match", PlayerID: player, AllocationID: request.AllocationID, ServerID: "supervisor-live-server", Slot: index, + Region: "EU", ClientBuild: "build-1", ProtocolVersion: 1, Transport: "enet", Endpoint: "127.0.0.1:7777", + JoinAuthorisation: "join-" + player, ManifestDigest: []byte{0, 1, 2, 3}, ExpiresAt: now.Add(time.Hour), Revision: 1, + }); err != nil { + t.Fatal(err) + } + } + secret := []byte("supervisor-live-workload-secret") + token, err := workload.IssueSignedWorkloadToken(secret, request.AllocationID, now, time.Hour) + if err != nil { + t.Fatal(err) + } + sdk := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + switch r.URL.Path { + case "/gameserver": + _, _ = fmt.Fprintf(w, `{"object_meta":{"annotations":{"cosmic-clash.io/match-id":"supervisor-live-match","cosmic-clash.io/workload-token":%q}},"status":{"address":"127.0.0.1","ports":[{"name":"game","port":7777}]}}`, token) + case "/ready-probe", "/ready": + w.WriteHeader(http.StatusOK) + default: + w.WriteHeader(http.StatusNotFound) + } + })) + defer sdk.Close() + service := &api.Service{ServerRegistrar: api.ServerRegistrarFromStore(db), WorkloadVerify: api.WorkloadVerifierFromSignedToken(secret, db), Now: func() time.Time { return now }} + control := httptest.NewServer(service.Handler()) + defer control.Close() + supervisor, err := New(Config{ + Command: []string{"/bin/sh", "-c", "true"}, SDKBaseURL: sdk.URL, ReadyURL: sdk.URL + "/ready-probe", ControlPlaneURL: control.URL, + ServerID: "supervisor-live-server", ProtocolVersion: 1, ImageDigest: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + ReadyTimeout: time.Second, PollInterval: time.Millisecond, AssignmentReadyAttempts: 1, + }) + if err != nil { + t.Fatal(err) + } + if err := supervisor.Start(ctx); err != nil { + t.Fatal(err) + } + if err := supervisor.Wait(); err != nil { + t.Fatal(err) + } + var matchState, ticketState string + if err := db.QueryRowContext(ctx, `SELECT state FROM matches WHERE match_id = 'supervisor-live-match'`).Scan(&matchState); err != nil { + t.Fatal(err) + } + if err := db.QueryRowContext(ctx, `SELECT state FROM queue_tickets WHERE ticket_id = 'supervisor-live-ticket-0'`).Scan(&ticketState); err != nil { + t.Fatal(err) + } + if matchState != "ASSIGNMENT_READY" || ticketState != "ASSIGNMENT_READY" { + t.Fatalf("registration lifecycle match=%q ticket=%q", matchState, ticketState) + } + var registrationCount int + if err := db.QueryRowContext(ctx, `SELECT count(*) FROM idempotency_keys WHERE scope = 'server.register' AND idempotency_key LIKE 'supervisor-register-supervisor-live-server-supervisor-live-match-%'`).Scan(®istrationCount); err != nil || registrationCount != 2 { + t.Fatalf("registration idempotency rows=%d err=%v", registrationCount, err) + } +}