test(multiplayer): verify allocated supervisor registration

multiplayer-next.md

@@ -1235,7 +1235,7 @@ the local/CI/community transport, not a silent production fallback.
 |---|---|---|
 | 8.39 `[D:8.3,8.14,8.17]` | **IN PROGRESS.** `MatchmakingState` now projects queue → proposal → allocation/process-ready/assignment-ready/connect/live plus terminal failure states; autoload `ControlPlaneClient` provides authenticated queue create/recovery/heartbeat/cancel and proposal response requests with idempotency/revision headers; `matchmaking.tscn`/`matchmaking.gd` expose the state and authoritative actions from the main menu; authenticated proposal recovery now reconciles missed proposal events and expires them at read time; the Go API publishes targeted authenticated revisioned queue/proposal/assignment events and the Godot client consumes state/proposal/assignment events, fetching the authoritative assignment after assignment readiness; the API can now use the durable participant-scoped proposal provider for both recovery and accept/decline mutations | `test_matchmaking_state.gd`, `test_control_plane_client.gd`, `test_matchmaking_ui.gd`, `TestProposalRecoveryIsParticipantScopedAndExpiresAtReadBoundary`, `TestAuthenticatedWebSocketDeliversOnlyTargetedRevisionedEvents`, `TestStateChangingAPIActionsPublishTargetedEvents`, `TestProposalRecoveryUsesDurableBackendAndRemainsParticipantScoped` and store proposal SQL tests reject stale/gapped/conflicting updates, validate endpoint/token/payload normalization, preserve idempotent duplicates, and guarantee visible phase/terminal copy; wiring durable allocation events into the outbox, wait/latency explanations and Godot runtime verification remain |
 | 8.40 `[D:8.3,8.14]` | **IN PROGRESS.** Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated `/v1/events` WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot `ControlPlaneClient` can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return, including a deferred recovery when an event arrives during an in-flight HTTP mutation; `OutboxDispatcher` now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match; proposal creation now writes one durable targeted `proposal_changed` outbox event per proposal aggregate revision, and both production `cmd/control-plane` and the test-only API harness dispatch only that event type to authenticated participants, leaving result events for their separate consumer | `server/domain/sync.go`, `server/api/events.go`, `server/api/outbox.go`, `server/api/service.go`, `server/store/outbox.go`, `server/store/proposal_sql.go`, `server/cmd/control-plane`, `server/cmd/testkit-api`, `service_test.go`, `outbox_test.go`, `matchmaking_state.gd`, `control_plane_client.gd`, `local_prediction_history.gd`, `net_ship_predictor.gd` and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation, prediction warm-up/hard-resync separation, deferred proposal recovery and proposal outbox filtering/delivery failures; Godot 4.7.1 headless project parse and 150-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; `cmd/control-plane` provides a signal-bound API role, `cmd/matcher` now supports casual and explicitly enabled ranked roles with durable identity lookup, and `cmd/maintenance` provides bounded season maintenance. **Live multi-process control-plane/game verification now exists**: `scripts/verify_control_plane_integration.sh` runs a real `postgres:17-alpine`, the real `api.Service` (via the new test-only `server/cmd/testkit-api`, wired identically to `cmd/control-plane` except for a fake Steam login — see §8.7), and a real headless Godot client (`control_plane_smoke.gd`) round-tripping login → `fetch_ranked_profile` (expect 404, §8.22) → `queue_create` → heartbeat → `cancel_queue` over an actual network connection — the first time this boundary was exercised end to end rather than against a mock on either side. **The two-player proposal extension now also passes**: `scripts/verify_control_plane_proposal_integration.sh` starts isolated PostgreSQL, the real matcher and testkit API, launches two headless Godot clients, seeds deterministic RTT projections, observes the real WebSocket `OPEN` proposal, accepts from both clients, and verifies both durable tickets plus the proposal reach `ACCEPTED` (including authoritative recovery when the concurrent revision advances). Allocator and Redis fan-out live verification remain |
-| 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows; live authenticated Godot/PostgreSQL assignment verification now exists via `scripts/verify_assignment_integration.sh`, which seeds a durable player-scoped `ASSIGNMENT_READY` row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary) | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; SDR relay-ticket installation and live allocated-token process integration remain |
+| 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows; live authenticated Godot/PostgreSQL assignment verification now exists via `scripts/verify_assignment_integration.sh`, which seeds a durable player-scoped `ASSIGNMENT_READY` row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary); `scripts/run_supervisor_integration.sh` additionally verifies the allocated-server process-ready → assignment-ready registration path against real PostgreSQL and the real API, with the workload token recovered from an Agones-shaped GameServer annotation | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql`, `server/allocator/allocator_integration_test.go`, `server/supervisor/supervisor_integration_test.go` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior, real allocator reconciliation, real supervisor registration and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; SDR relay-ticket installation and live Agones cluster integration remain |
 | 8.42 `[D:8.22,8.23,8.24,8.40]` | **IN PROGRESS.** `RankedProfileState` and `ControlPlaneClient.fetch_ranked_profile()` expose the backend-authoritative rating/RD/volatility/games/tier/provisional/season view; matchmaking UI displays provisional/tier status without client-side rating math; runnable API binaries now wire the backend-owned default tier policy; live authenticated profile verification now exists via `scripts/verify_ranked_profile_integration.sh`, which seeds a durable rating and verifies the populated profile over real PostgreSQL, Go HTTP, and Godot JSON | `test_control_plane_client.gd` validates profile shape, numeric safety and provisional display; the real populated profile path now covers durable rating/games/tier/provisional decoding and caught the missing runtime tier-policy wiring; committed revision after reconnect, abandon status and season countdown remain dependent on live auth/backend events and Godot runtime verification |
 | 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path; decline, version mismatch, regional outage retry UI, failed reconnect, duplicate-action recovery beyond proposals and broader live Godot verification remain |
 

scripts/run_supervisor_integration.sh

@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+container_name="cosmic-clash-supervisor-integration"
+database="cosmic_clash_test"
+user="cosmic_clash_test"
+password="cosmic_clash_test"
+
+cleanup() { docker rm -f "$container_name" >/dev/null 2>&1 || true; }
+trap cleanup EXIT
+cleanup
+docker run --rm -d --name "$container_name" \
+  -e POSTGRES_DB="$database" -e POSTGRES_USER="$user" -e POSTGRES_PASSWORD="$password" \
+  -p 55437:5432 postgres:17-alpine >/dev/null
+for attempt in $(seq 1 30); do
+  if docker exec "$container_name" pg_isready -U "$user" -d "$database" >/dev/null 2>&1; then
+    break
+  fi
+  if [ "$attempt" = 30 ]; then
+    echo "PostgreSQL did not become ready" >&2
+    exit 1
+  fi
+  sleep 1
+done
+cd "$repo_root/server"
+COSMIC_CLASH_POSTGRES_DSN="postgres://${user}:${password}@127.0.0.1:55437/${database}?sslmode=disable" \
+  go test -tags integration ./supervisor -count=1

server/supervisor/supervisor_integration_test.go

@@ -0,0 +1,133 @@
+//go:build integration
+
+package supervisor
+
+import (
+	"context"
+	"database/sql"
+	"fmt"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/cosmic-clash/cosmic-clash/server/api"
+	"github.com/cosmic-clash/cosmic-clash/server/domain"
+	"github.com/cosmic-clash/cosmic-clash/server/migrations"
+	"github.com/cosmic-clash/cosmic-clash/server/store"
+	"github.com/cosmic-clash/cosmic-clash/server/workload"
+	_ "github.com/jackc/pgx/v5/stdlib"
+)
+
+func TestRealSupervisorRegistersAllocatedServerThroughControlPlane(t *testing.T) {
+	dsn := os.Getenv("COSMIC_CLASH_POSTGRES_DSN")
+	if dsn == "" {
+		t.Skip("COSMIC_CLASH_POSTGRES_DSN is not set")
+	}
+	db, err := sql.Open("pgx", dsn)
+	if err != nil {
+		t.Fatal(err)
+	}
+	defer db.Close()
+	ctx := context.Background()
+	if err := db.PingContext(ctx); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := db.ExecContext(ctx, `DROP TABLE IF EXISTS schema_migrations, assignments, audit_events, outbox, result_receipts, ranked_season_rollovers, penalties, seasons, ratings, match_participants, matches, proposal_participants, proposals, allocations, game_servers, queue_tickets, idempotency_keys, sessions, identities CASCADE`); err != nil {
+		t.Fatal(err)
+	}
+	if err := migrations.Apply(ctx, db, filepath.Join("..", "migrations")); err != nil {
+		t.Fatal(err)
+	}
+	now := time.Now().UTC().Truncate(time.Microsecond)
+	players := []string{"supervisor-live-a", "supervisor-live-b"}
+	for index, player := range players {
+		if _, err := db.ExecContext(ctx, `INSERT INTO identities (player_id, steam_id) VALUES ($1, $1)`, player); err != nil {
+			t.Fatal(err)
+		}
+		if _, err := db.ExecContext(ctx, `INSERT INTO queue_tickets (ticket_id, player_id, playlist, state, client_build, protocol_version, enqueued_at, expires_at) VALUES ($1, $2, 'casual', 'ACCEPTED', 'build-1', 1, $3, $4)`, fmt.Sprintf("supervisor-live-ticket-%d", index), player, now, now.Add(time.Hour)); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if _, err := db.ExecContext(ctx, `INSERT INTO matches (match_id, playlist, state, region, protocol_version) VALUES ('supervisor-live-match', 'casual', 'ALLOCATING', 'EU', 1)`); err != nil {
+		t.Fatal(err)
+	}
+	for index, player := range players {
+		if _, err := db.ExecContext(ctx, `INSERT INTO match_participants (match_id, player_id, ticket_id, slot, team) VALUES ('supervisor-live-match', $1, $2, $3, $4)`, player, fmt.Sprintf("supervisor-live-ticket-%d", index), index, index); err != nil {
+			t.Fatal(err)
+		}
+	}
+	if err := store.RegisterReadyServer(ctx, db, domain.ReadyServer{ServerID: "supervisor-live-server", Region: "EU", Build: "build-1", Protocol: 1, Transport: "enet", State: domain.ServerReady}, now); err != nil {
+		t.Fatal(err)
+	}
+	claim, found, err := store.ClaimAllocatingMatch(ctx, db, "enet", now)
+	if err != nil || !found {
+		t.Fatalf("claim allocating match found=%t err=%v", found, err)
+	}
+	request := claim.Request
+	allocation, err := store.ClaimAllocation(ctx, db, request, now)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := store.BindAllocatedMatch(ctx, db, allocation); err != nil {
+		t.Fatal(err)
+	}
+	for index, player := range players {
+		if err := store.SaveAssignment(ctx, db, store.DurableAssignment{
+			MatchID: "supervisor-live-match", PlayerID: player, AllocationID: request.AllocationID, ServerID: "supervisor-live-server", Slot: index,
+			Region: "EU", ClientBuild: "build-1", ProtocolVersion: 1, Transport: "enet", Endpoint: "127.0.0.1:7777",
+			JoinAuthorisation: "join-" + player, ManifestDigest: []byte{0, 1, 2, 3}, ExpiresAt: now.Add(time.Hour), Revision: 1,
+		}); err != nil {
+			t.Fatal(err)
+		}
+	}
+	secret := []byte("supervisor-live-workload-secret")
+	token, err := workload.IssueSignedWorkloadToken(secret, request.AllocationID, now, time.Hour)
+	if err != nil {
+		t.Fatal(err)
+	}
+	sdk := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		switch r.URL.Path {
+		case "/gameserver":
+			_, _ = fmt.Fprintf(w, `{"object_meta":{"annotations":{"cosmic-clash.io/match-id":"supervisor-live-match","cosmic-clash.io/workload-token":%q}},"status":{"address":"127.0.0.1","ports":[{"name":"game","port":7777}]}}`, token)
+		case "/ready-probe", "/ready":
+			w.WriteHeader(http.StatusOK)
+		default:
+			w.WriteHeader(http.StatusNotFound)
+		}
+	}))
+	defer sdk.Close()
+	service := &api.Service{ServerRegistrar: api.ServerRegistrarFromStore(db), WorkloadVerify: api.WorkloadVerifierFromSignedToken(secret, db), Now: func() time.Time { return now }}
+	control := httptest.NewServer(service.Handler())
+	defer control.Close()
+	supervisor, err := New(Config{
+		Command: []string{"/bin/sh", "-c", "true"}, SDKBaseURL: sdk.URL, ReadyURL: sdk.URL + "/ready-probe", ControlPlaneURL: control.URL,
+		ServerID: "supervisor-live-server", ProtocolVersion: 1, ImageDigest: "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
+		ReadyTimeout: time.Second, PollInterval: time.Millisecond, AssignmentReadyAttempts: 1,
+	})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := supervisor.Start(ctx); err != nil {
+		t.Fatal(err)
+	}
+	if err := supervisor.Wait(); err != nil {
+		t.Fatal(err)
+	}
+	var matchState, ticketState string
+	if err := db.QueryRowContext(ctx, `SELECT state FROM matches WHERE match_id = 'supervisor-live-match'`).Scan(&matchState); err != nil {
+		t.Fatal(err)
+	}
+	if err := db.QueryRowContext(ctx, `SELECT state FROM queue_tickets WHERE ticket_id = 'supervisor-live-ticket-0'`).Scan(&ticketState); err != nil {
+		t.Fatal(err)
+	}
+	if matchState != "ASSIGNMENT_READY" || ticketState != "ASSIGNMENT_READY" {
+		t.Fatalf("registration lifecycle match=%q ticket=%q", matchState, ticketState)
+	}
+	var registrationCount int
+	if err := db.QueryRowContext(ctx, `SELECT count(*) FROM idempotency_keys WHERE scope = 'server.register' AND idempotency_key LIKE 'supervisor-register-supervisor-live-server-supervisor-live-match-%'`).Scan(&registrationCount); err != nil || registrationCount != 2 {
+		t.Fatalf("registration idempotency rows=%d err=%v", registrationCount, err)
+	}
+}