diff --git a/multiplayer-next.md b/multiplayer-next.md index 23d3a843..3bd58000 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -61,10 +61,12 @@ product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). `server/allocator` now requires provider allocation reconciliation into the durable registry before returning an endpoint and exposes an accepted-proposal gate that validates unanimous responses and playlist/participant invariants; - allocator-facing roster + matches now also use a leased durable allocation claim, derived from their + persisted compatibility tuple, to fence allocator replicas before provider + calls; allocator-facing roster publication now requires an allocated endpoint and verifies canonical - join-authorisation signatures before exposing player rows; live Agones - integration remains. + join-authorisation signatures before exposing player rows; allocator runtime + wiring and live Agones integration remain. - [ ] **IN PROGRESS:** Run the Go control plane against PostgreSQL/Redis with independently runnable API, matcher, allocator and maintenance roles. The `cmd/control-plane` API role now opens PostgreSQL, applies migrations, wires diff --git a/multiplayer-todo.md b/multiplayer-todo.md index 47deead7..58e9d51a 100644 --- a/multiplayer-todo.md +++ b/multiplayer-todo.md @@ -1173,7 +1173,7 @@ the local/CI/community transport, not a silent production fallback. | 8.2 `[D:8.1]` | **DONE.** Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | [`docs/MATCHMAKING-SLOs.md`](docs/MATCHMAKING-SLOs.md) defines each metric, denominator, percentile/window, owner, alert threshold and release evidence | | 8.3 `[D:8.1]` | **DONE.** Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | [`server/contracts/v1/`](server/contracts/v1/) contains machine-readable REST/events contracts and dependency-free structural tests; REST resync is specified by the contract; `server/api/service.go` also exposes the documented `/api/v1` route names (including server-assigned idempotent queue ticket IDs and DELETE cancellation) alongside the existing client `/v1` routes, covered by `TestDocumentedContractRoutesAdaptToServiceAPI` | | 8.4 `[D:8.3]` | **DONE.** Define opaque IDs, legal queue/match state transitions, revisions and idempotency keys | [`server/contracts/v1/state-transitions.json`](server/contracts/v1/state-transitions.json) locks terminal states, legal edges, stale-revision handling and same-key replay/conflict behavior; contract tests cover the invariants | -| 8.5 `[D:8.4]` | **IN PROGRESS.** Initial PostgreSQL migration now defines durable idempotency keys, queue ownership/active-participation fencing, identities, sessions/revocations, ranked seasons, ratings/events, matches/participants, penalties, results, audits and outbox; follow-up migrations persist server-derived queue probe RTT metadata, the allocator GameServer/allocation registry, and matcher-selected proposal region/protocol/team/slot plans | `server/migrations/0001_initial.sql`, `0003_queue_probe_metadata.sql`, `0004_allocator_registry.sql`, `0005_proposal_match_plans.sql`, `migrations/runner.go`, `cmd/migrate` and static checks cover the durable tables, uniqueness/check constraints, Redis-as-cache boundary and serialized forward migration recording; opt-in `scripts/run_postgres_integration.sh` now runs the runner and real queue/assignment ownership, idempotency, revision and expiry checks through pgx; rollback/down migration, remaining serializable adapters and cache-loss repair remain | +| 8.5 `[D:8.4]` | **IN PROGRESS.** Initial PostgreSQL migration now defines durable idempotency keys, queue ownership/active-participation fencing, identities, sessions/revocations, ranked seasons, ratings/events, matches/participants, penalties, results, audits and outbox; follow-up migrations persist server-derived queue probe RTT metadata, the allocator GameServer/allocation registry, matcher-selected proposal region/protocol/team/slot plans, and leased allocating-match claims | `server/migrations/0001_initial.sql`, `0003_queue_probe_metadata.sql`, `0004_allocator_registry.sql`, `0005_proposal_match_plans.sql`, `0006_match_allocation_claims.sql`, `migrations/runner.go`, `cmd/migrate` and static checks cover the durable tables, uniqueness/check constraints, Redis-as-cache boundary and serialized forward migration recording; opt-in `scripts/run_postgres_integration.sh` now runs the runner and real queue/assignment ownership, idempotency, revision and expiry checks through pgx; rollback/down migration, remaining serializable adapters and cache-loss repair remain | | 8.6 `[D:8.3,8.4]` | **IN PROGRESS.** Add allocated-mode `ServerConfig` compatibility fields as opt-in defaults | `ServerConfig` now validates allocation mode, match/server IDs, playlist version, client build, future assignment expiry, image digest, transport and EU/NA region; `server_boot.gd` fails closed for the not-yet-wired Steam SDR transport, constrains allocated processes to one match, and emits allocation identity/transport in `server_started`; signed-authorisation admission, dynamic endpoint wiring and full manifest/runtime tests remain | #### 8B — Authentication and secure control plane @@ -1213,7 +1213,7 @@ the local/CI/community transport, not a silent production fallback. | 8.27 `[D:8.26]` | **IN PROGRESS.** Go supervisor package provides local-safe Agones REST discovery, validates assigned address/port data, injects dynamic `SDR_LISTEN_PORT`/`SDR_IP`, performs explicit process-ready probing and Ready transition; direct mode bypasses Agones | `server/supervisor/` covers allocated/direct startup, invalid endpoint rejection, dynamic endpoint/Ready ordering and authenticated drain; allocated Godot now supplies a loopback readiness/drain control surface and `agones_sdk.gd` supplies sidecar Health/Ready/Shutdown/annotation REST operations; metadata watch, real Agones annotation/shutdown confirmation and emulator integration remain | | 8.28 `[D:8.6,8.27]` | **IN PROGRESS.** Supervisor separates explicit process-ready from Agones Ready and never scrapes stdout; allocated mode refuses to mark Ready without a configured readiness probe | `server/supervisor/` tests prove Ready follows the probe and direct mode remains functional; `server_control.gd`, `agones_sdk.gd` and process-level smokes prove loopback `/ready`, `/health`, bearer-protected `/drain`, sidecar-shaped Health/Ready calls and drain admission fencing; detached-container and Health-reclaim integration remain | | 8.29 `[D:8.26,8.27]` | **IN PROGRESS.** Supervisor discovers and validates the Agones endpoint, propagates the actual dynamic `--port`, and exports `SDR_LISTEN_PORT`/`SDR_IP` only for Hosted-SDR while preserving an isolated ENet path | `server/supervisor/` tests cover invalid address/port rejection, dynamic port argument/env propagation and SDR-vs-ENet separation; real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT and multi-match fixture remain | -| 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** Pure Go allocator filters Ready servers by region/build/protocol/transport, atomically claims one with idempotent allocation replay, verifies unanimous accepted-proposal/playlist/participant invariants before provider invocation, and now owns the assignment-publication boundary; PostgreSQL adds durable GameServer registration and compatible `SKIP LOCKED` claims with request-digest fencing; `server/agones` submits and validates namespaced `GameServerAllocation` responses and dynamic endpoints; `server/allocator` reconciles provider success into durable state before exposing the endpoint | `server/domain/allocator.go`, `server/store/allocator_sql.go`, `server/agones/allocation.go`, `server/allocator/service.go`, `server/migrations/0004_allocator_registry.sql` and tests cover deterministic compatible selection, exhaustion, conflicting/identical allocation replay, unknown allocations, SQL claim ordering, invalid input, provider error/malformed response/IPv6 endpoint handling, accepted-proposal gating, durable-reconciliation failure isolation and assignment replay/conflict; `TestPostgreSQLAllocatorClaimReplayAndCapacityFence` now covers live registration/selection/replay/conflict/no-capacity when the disposable database gate is run; signed roster metadata, bounded cross-replica retry and live integration remain | +| 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** Pure Go allocator filters Ready servers by region/build/protocol/transport, atomically claims one with idempotent allocation replay, verifies unanimous accepted-proposal/playlist/participant invariants before provider invocation, and now owns the assignment-publication boundary; PostgreSQL adds durable GameServer registration and compatible `SKIP LOCKED` claims with request-digest fencing, plus a leased `ALLOCATING`-match claim that derives the immutable compatibility tuple and fences bind/release by deterministic allocation ID; `server/agones` submits and validates namespaced `GameServerAllocation` responses and dynamic endpoints; `server/allocator` reconciles provider success into durable state before exposing the endpoint | `server/domain/allocator.go`, `server/store/allocator_sql.go`, `server/store/allocation_match_sql.go`, `server/agones/allocation.go`, `server/allocator/service.go`, `server/migrations/0004_allocator_registry.sql`, `0006_match_allocation_claims.sql` and tests cover deterministic compatible selection, exhaustion, conflicting/identical allocation replay, allocation-match lease recovery/bind fencing, unknown allocations, SQL claim ordering, invalid input, provider error/malformed response/IPv6 endpoint handling, accepted-proposal gating, durable-reconciliation failure isolation and assignment replay/conflict; `TestPostgreSQLAllocatorClaimReplayAndCapacityFence` and `TestPostgreSQLAllocationMatchClaimLeaseAndBindFence` cover the live database paths when the disposable database gate is run; allocator runtime wiring, signed roster metadata, bounded cross-replica retry and live Agones integration remain | | 8.31 `[D:8.9,8.30]` | **IN PROGRESS.** Pure Go assignment gate requires Allocated state, exact allocation ID/match/server/region/build/protocol/transport compatibility, non-empty hosted endpoint and verified manifest signature before exposure; allocator publication cannot expose Ready state; durable roster persistence now verifies each canonical join-authorisation signature before publishing player rows; allocator service gates roster publication on allocated state and endpoint presence | `server/domain/assignment.go`, `allocator.go`, `store/assignment_sql.go` plus adversarial fixtures cover early-connect, tampered signature/manifest, wrong compatibility, empty endpoint, unknown allocation, forged roster signature, valid signature, premature publication and post-publication mutation rejection; Agones metadata watch, hosted-address registration, production signer and client-ticket publication remain | | 8.32 `[D:8.2,8.26,8.30]` | **IN PROGRESS.** Provider-neutral FleetAutoscaler baseline preserves a two-process Ready buffer, caps warm capacity, and leaves Allocated scale-down independent of the Ready floor; Fleet image references remain digest-pinned for current/rollback pre-pull | `deploy/k8s/base/fleet-autoscaler.yaml` and manifest tests cover Fleet ownership, Buffer policy and floor/cap invariants; regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99 and N+1 certification remain | | 8.33 `[D:8.26,8.32]` | **IN PROGRESS.** Fleet scheduling now requires on-demand capacity and spreads Ready processes across zones with skew 1; the autoscaler preserves the two-process Ready floor | `deploy/k8s/base/fleet.yaml` and manifest tests reject interruptible placement and single-zone concentration structurally; regional node pools, forced node-loss testing and measured N+1 headroom remain | diff --git a/server/migrations/0006_match_allocation_claims.sql b/server/migrations/0006_match_allocation_claims.sql new file mode 100644 index 00000000..c6b09436 --- /dev/null +++ b/server/migrations/0006_match_allocation_claims.sql @@ -0,0 +1,12 @@ +-- Allocation is an external call, so a durable leased claim fences competing +-- allocator replicas before any provider request. A timed-out claim can be +-- recovered with the same deterministic allocation ID after a worker crash. +ALTER TABLE matches + ADD COLUMN allocation_id TEXT UNIQUE, + ADD COLUMN allocation_claimed_at TIMESTAMPTZ, + ADD CONSTRAINT matches_allocation_claim_pair + CHECK ((allocation_id IS NULL) = (allocation_claimed_at IS NULL)); + +CREATE INDEX matches_allocating_claimable + ON matches (created_at, match_id) + WHERE state = 'ALLOCATING' AND server_id IS NULL; diff --git a/server/store/allocation_match_sql.go b/server/store/allocation_match_sql.go new file mode 100644 index 00000000..6a844fd0 --- /dev/null +++ b/server/store/allocation_match_sql.go @@ -0,0 +1,134 @@ +package store + +import ( + "context" + "database/sql" + "fmt" + "time" + + "github.com/cosmic-clash/cosmic-clash/server/domain" +) + +const AllocationClaimLease = time.Minute + +type PendingAllocation struct { + Request domain.AllocationRequest +} + +const ClaimAllocatingMatchSQL = `WITH candidate AS ( + SELECT match_id FROM matches + WHERE state = 'ALLOCATING' AND server_id IS NULL + AND (allocation_id IS NULL OR allocation_claimed_at <= $1) + ORDER BY created_at, match_id + LIMIT 1 + FOR UPDATE SKIP LOCKED +) +UPDATE matches m +SET allocation_id = 'allocation-' || candidate.match_id, allocation_claimed_at = $2 +FROM candidate +WHERE m.match_id = candidate.match_id +RETURNING m.match_id, m.region, m.protocol_version, m.allocation_id` + +const AllocatingMatchBuildSQL = `SELECT client_build +FROM queue_tickets q +JOIN match_participants mp ON mp.ticket_id = q.ticket_id AND mp.player_id = q.player_id +WHERE mp.match_id = $1 +ORDER BY q.client_build` + +const BindAllocatedMatchSQL = `UPDATE matches +SET server_id = $3 +WHERE match_id = $1 AND state = 'ALLOCATING' AND allocation_id = $2 AND server_id IS NULL + AND EXISTS ( + SELECT 1 FROM allocations + WHERE allocation_id = $2 AND match_id = $1 AND server_id = $3 AND state = 'ALLOCATED' + )` + +const ReleaseAllocatedMatchClaimSQL = `UPDATE matches +SET allocation_id = NULL, allocation_claimed_at = NULL +WHERE match_id = $1 AND state = 'ALLOCATING' AND allocation_id = $2 AND server_id IS NULL` + +// ClaimAllocatingMatch returns one durable provider work item. The fixed +// allocation ID is retained across a lease recovery, allowing every later +// reconciliation step to reject a different server for the same match. +func ClaimAllocatingMatch(ctx context.Context, db *sql.DB, transport string, now time.Time) (PendingAllocation, bool, error) { + if db == nil || (transport != "enet" && transport != "steam_sdr") || now.IsZero() { + return PendingAllocation{}, false, fmt.Errorf("invalid allocation claim arguments") + } + var item PendingAllocation + found := false + err := RunSerializable(ctx, db, DefaultSerializableAttempts, func(ctx context.Context, tx *sql.Tx) error { + var matchID, region string + var protocol int + var claimedID string + err := tx.QueryRowContext(ctx, ClaimAllocatingMatchSQL, now.Add(-AllocationClaimLease), now).Scan(&matchID, ®ion, &protocol, &claimedID) + if err == sql.ErrNoRows { + return nil + } + if err != nil { + return err + } + rows, err := tx.QueryContext(ctx, AllocatingMatchBuildSQL, matchID) + if err != nil { + return err + } + defer rows.Close() + build := "" + for rows.Next() { + var candidate string + if err := rows.Scan(&candidate); err != nil { + return err + } + if build == "" { + build = candidate + } else if build != candidate { + return fmt.Errorf("allocating match has mixed client builds") + } + } + if err := rows.Err(); err != nil { + return err + } + if build == "" { + return fmt.Errorf("allocating match has no participants") + } + item.Request = domain.AllocationRequest{AllocationID: claimedID, MatchID: matchID, Region: region, Build: build, Protocol: protocol, Transport: transport} + found = true + return nil + }) + return item, found, err +} + +func BindAllocatedMatch(ctx context.Context, db *sql.DB, allocation domain.Allocation) error { + if db == nil || allocation.MatchID == "" || allocation.AllocationID == "" || allocation.ServerID == "" || allocation.State != domain.ServerAllocated { + return fmt.Errorf("invalid allocated match binding") + } + result, err := db.ExecContext(ctx, BindAllocatedMatchSQL, allocation.MatchID, allocation.AllocationID, allocation.ServerID) + if err != nil { + return err + } + changed, err := result.RowsAffected() + if err != nil { + return err + } + if changed != 1 { + return domain.ErrConflict + } + return nil +} + +func ReleaseAllocatedMatchClaim(ctx context.Context, db *sql.DB, matchID, allocationID string) error { + if db == nil || matchID == "" || allocationID == "" { + return fmt.Errorf("invalid allocated match claim release") + } + result, err := db.ExecContext(ctx, ReleaseAllocatedMatchClaimSQL, matchID, allocationID) + if err != nil { + return err + } + changed, err := result.RowsAffected() + if err != nil { + return err + } + if changed != 1 { + return domain.ErrConflict + } + return nil +} diff --git a/server/store/allocation_match_sql_test.go b/server/store/allocation_match_sql_test.go new file mode 100644 index 00000000..d758b3a6 --- /dev/null +++ b/server/store/allocation_match_sql_test.go @@ -0,0 +1,44 @@ +package store + +import ( + "testing" + "time" + + "github.com/cosmic-clash/cosmic-clash/server/domain" +) + +func TestAllocationMatchClaimSQLFencesConcurrentWorkers(t *testing.T) { + checks := map[string][]string{ + ClaimAllocatingMatchSQL: {"FOR UPDATE SKIP LOCKED", "allocation_id = 'allocation-' || candidate.match_id", "allocation_claimed_at <= $1", "ORDER BY created_at, match_id"}, + AllocatingMatchBuildSQL: {"match_participants", "queue_tickets", "ORDER BY q.client_build"}, + BindAllocatedMatchSQL: {"allocation_id = $2", "server_id IS NULL", "SET server_id = $3", "FROM allocations"}, + ReleaseAllocatedMatchClaimSQL: {"allocation_id = $2", "allocation_id = NULL", "allocation_claimed_at = NULL"}, + } + for query, fragments := range checks { + for _, fragment := range fragments { + if !contains(query, fragment) { + t.Fatalf("query missing %q", fragment) + } + } + } +} + +func TestAllocationMatchClaimRejectsInvalidArgumentsWithoutDatabase(t *testing.T) { + now := time.Unix(1_000, 0) + if _, _, err := ClaimAllocatingMatch(nil, nil, "enet", now); err == nil { + t.Fatal("nil database accepted") + } + if _, _, err := ClaimAllocatingMatch(nil, nil, "udp", now); err == nil { + t.Fatal("invalid transport accepted") + } + if _, _, err := ClaimAllocatingMatch(nil, nil, "enet", time.Time{}); err == nil { + t.Fatal("zero claim time accepted") + } + allocated := domain.Allocation{AllocationID: "allocation-match-1", MatchID: "match-1", ServerID: "server-1", State: domain.ServerAllocated} + if err := BindAllocatedMatch(nil, nil, allocated); err == nil { + t.Fatal("nil database accepted for bind") + } + if err := ReleaseAllocatedMatchClaim(nil, nil, "match-1", "allocation-match-1"); err == nil { + t.Fatal("nil database accepted for release") + } +} diff --git a/server/store/postgres_integration_test.go b/server/store/postgres_integration_test.go index 10a37949..0f86b70d 100644 --- a/server/store/postgres_integration_test.go +++ b/server/store/postgres_integration_test.go @@ -143,6 +143,57 @@ func TestPostgreSQLAcceptedProposalPromotesOneAtomicAllocatingMatch(t *testing.T } } +func TestPostgreSQLAllocationMatchClaimLeaseAndBindFence(t *testing.T) { + db := openIntegrationPostgres(t) + applyIntegrationMigrations(t, db) + now := time.Now().UTC().Truncate(time.Microsecond) + ctx := context.Background() + for index, player := range []string{"allocation-match-a", "allocation-match-b"} { + if _, err := db.ExecContext(ctx, `INSERT INTO identities (player_id, steam_id) VALUES ($1, $1)`, player); err != nil { + t.Fatal(err) + } + if _, err := db.ExecContext(ctx, `INSERT INTO queue_tickets (ticket_id, player_id, playlist, state, client_build, protocol_version, enqueued_at, expires_at) VALUES ($1, $2, 'casual', 'ACCEPTED', 'build-1', 1, $3, $4)`, fmt.Sprintf("allocation-match-ticket-%d", index), player, now, now.Add(time.Minute)); err != nil { + t.Fatal(err) + } + } + if _, err := db.ExecContext(ctx, `INSERT INTO matches (match_id, playlist, state, region, protocol_version) VALUES ('allocation-match', 'casual', 'ALLOCATING', 'EU', 1)`); err != nil { + t.Fatal(err) + } + for index, player := range []string{"allocation-match-a", "allocation-match-b"} { + if _, err := db.ExecContext(ctx, `INSERT INTO match_participants (match_id, player_id, ticket_id, slot, team) VALUES ('allocation-match', $1, $2, $3, $4)`, player, fmt.Sprintf("allocation-match-ticket-%d", index), index*3, index); err != nil { + t.Fatal(err) + } + } + claim, found, err := ClaimAllocatingMatch(ctx, db, "enet", now) + if err != nil || !found || claim.Request != (domain.AllocationRequest{AllocationID: "allocation-allocation-match", MatchID: "allocation-match", Region: "EU", Build: "build-1", Protocol: 1, Transport: "enet"}) { + t.Fatalf("claim=%+v found=%t err=%v", claim, found, err) + } + if err := ReleaseAllocatedMatchClaim(ctx, db, claim.Request.MatchID, "different-allocation"); err != domain.ErrConflict { + t.Fatalf("wrong-claim release err=%v", err) + } + if err := ReleaseAllocatedMatchClaim(ctx, db, claim.Request.MatchID, claim.Request.AllocationID); err != nil { + t.Fatalf("release claim: %v", err) + } + reclaimed, found, err := ClaimAllocatingMatch(ctx, db, "enet", now.Add(time.Second)) + if err != nil || !found || reclaimed.Request.AllocationID != claim.Request.AllocationID { + t.Fatalf("reclaimed=%+v found=%t err=%v", reclaimed, found, err) + } + server := domain.ReadyServer{ServerID: "allocation-server", Region: "EU", Build: "build-1", Protocol: 1, Transport: "enet", State: domain.ServerReady} + if err := RegisterReadyServer(ctx, db, server, now); err != nil { + t.Fatalf("register allocation server: %v", err) + } + allocation, err := ClaimAllocation(ctx, db, reclaimed.Request, now.Add(time.Second)) + if err != nil { + t.Fatalf("record provider allocation: %v", err) + } + if err := BindAllocatedMatch(ctx, db, allocation); err != nil { + t.Fatalf("bind allocation: %v", err) + } + if _, found, err := ClaimAllocatingMatch(ctx, db, "enet", now.Add(2*time.Second)); err != nil || found { + t.Fatalf("bound match re-claimed found=%t err=%v", found, err) + } +} + func TestPostgreSQLQueueAdapterAgainstRealDatabase(t *testing.T) { db := openIntegrationPostgres(t) applyIntegrationMigrations(t, db)