diff --git a/multiplayer-next.md b/multiplayer-next.md index 0175f5ba..b3a6dff0 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -54,7 +54,9 @@ product policy are in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). authenticated durable queue/proposal/assignment/session adapters, and shuts down gracefully; optional `--redis-addr` publishes queue mutations to a TTL-bound best-effort candidate projection without making Redis authoritative; - worker roles, Redis worker wiring and live service checks remain. + a runnable casual `cmd/matcher` role now polls PostgreSQL and delegates + proposal claims to the durable transaction; allocator/maintenance roles, + ranked provider wiring, Redis worker wiring and live service checks remain. - [ ] **IN PROGRESS:** Define assignment compatibility and opt-in `ServerConfig` flags whose defaults reproduce the community-server path. Allocation manifest validation now covers client build and future expiry; allocated servers now diff --git a/multiplayer-todo.md b/multiplayer-todo.md index 1bde30e7..2f5c0d2c 100644 --- a/multiplayer-todo.md +++ b/multiplayer-todo.md @@ -1196,7 +1196,7 @@ the local/CI/community transport, not a silent production fallback. | 8.15 `[D:7.8,8.3]` | **IN PROGRESS.** Pure Go probe validation treats Steam location as opaque, requires nonce/freshness/region and server-computed RTT, and implements discrepancy quarantine/release; authenticated HTTP now accepts only opaque location/nonce input through a server-owned probe provider | `server/domain/probes.go`, adversarial fixtures and `server/api/service.go` cover stale/wrong/forged evidence, the 25 ms/30% threshold, three-sample quarantine, five-clean release, authenticated provider arguments and rejection of client RTT fields; Steam coordinator and regional probe adapters remain | | 8.16 `[D:8.14,8.15]` | **IN PROGRESS.** Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction | `server/domain/matcher.go`, `teams.go`, `server/matcher/worker.go` and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure and queue-backed oldest-anchor formation; ranked provider and long-running worker integration remain | | 8.17 `[D:8.14,8.16]` | **IN PROGRESS.** Pure Go proposal policy sends a 10-second response window to every selected human, requires unanimous acceptance, applies exact decline/timeout cooldowns and ranked escalation; authenticated API exposes revisioned accept/decline mutations; formed matches now pass through a playlist-aware proposal boundary | `server/domain/proposal.go`, `formation.go` and `server/api/service.go` plus adversarial fixtures cover partial/unanimous response, expiry, replay/conflict, stale API revision, casual lineup preparation and ranked metadata validation; queue precedence and allocation integration remain | -| 8.18 `[D:8.5,8.14,8.17]` | **IN PROGRESS.** Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using `FOR UPDATE SKIP LOCKED` plus durable uniqueness/revision fences; proposal creation now inserts proposal/participants and promotes every ticket in one rollback-safe transaction with player- and playlist-bound claim predicates, queue creation has a durable idempotency/owner-read adapter, participant-scoped proposal recovery now expires OPEN proposals and pending participants transactionally at read time, and proposal accept/decline now uses participant/proposal locks, revision fencing and durable idempotency; response attempts also advance expired proposals and pending participants before returning closed | `server/store/serializable.go`, `queue_sql.go`, `proposal_sql.go`, `proposal_recovery_sql.go` and tests cover retry classification, claim-boundary invariants, player/ticket/playlist mapping, durable queue replay/conflict, owner-scoped queue/proposal recovery, expiry at read and mutation boundaries, response replay/conflict, stale revisions, zero-row claim aborts and atomic statement ordering; opt-in PostgreSQL execution now covers queue create/replay/fencing, assignment persistence, proposal claim/promotion, participant recovery, unanimous response and rollback of partial claims; Redis candidate index/repair, worker-failure and concurrent two-matcher integration tests remain | +| 8.18 `[D:8.5,8.14,8.17]` | **IN PROGRESS.** Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using `FOR UPDATE SKIP LOCKED` plus durable uniqueness/revision fences; proposal creation now inserts proposal/participants and promotes every ticket in one rollback-safe transaction with player- and playlist-bound claim predicates, queue creation has a durable idempotency/owner-read adapter, participant-scoped proposal recovery now expires OPEN proposals and pending participants transactionally at read time, and proposal accept/decline now uses participant/proposal locks, revision fencing and durable idempotency; response attempts also advance expired proposals and pending participants before returning closed; runnable casual matcher polling now reads an authoritative PostgreSQL candidate batch and delegates its final claim to this transaction | `server/store/serializable.go`, `queue_sql.go`, `proposal_sql.go`, `proposal_recovery_sql.go`, `server/matcher/worker.go` and tests cover retry classification, claim-boundary invariants, player/ticket/playlist mapping, durable queue replay/conflict, owner-scoped queue/proposal recovery, expiry at read and mutation boundaries, response replay/conflict, stale revisions, zero-row claim aborts, atomic statement ordering, incomplete matcher batches and source/claim failures; opt-in PostgreSQL execution now covers queue create/replay/fencing, assignment persistence, proposal claim/promotion, participant recovery, unanimous response and rollback of partial claims; ranked provider, Redis-backed worker repair, worker-failure and concurrent two-matcher integration tests remain | | 8.19 `[D:8.18]` | **IN PROGRESS.** Pure Go casual lineup requires 2–6 humans with at least one per team, fills missing slots with explicit bots, permits kickoff-only bot-slot backfill and assigns no backfill penalty/rating update; proposal preparation now derives the lineup from formed teams | `server/domain/casual.go`, `formation.go` cover both-team minimum, bot shape, live-play rejection, zero-penalty backfill and casual proposal composition; queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties and live integration remain | | 8.20 `[D:8.18]` | **IN PROGRESS.** Pure Go ranked admission requires six unique verified solo humans, rejects bots/backfill/parties, and allows only random-enabled non-elevated arenas; proposal preparation requires matching metadata for every formed player | `server/domain/ranked.go`, `formation.go` cover count, identity, party, bot/backfill, arena eligibility and formed-player metadata rejection; `ArenaRegistry` integration, allocation wiring and innocent-ticket restoration remain | | 8.21 `[D:8.5,8.20]` | **IN PROGRESS.** Pure Go rating core implements canonical Glicko-2, daily inactivity, ranked 1/3 and casual 1/N human-opponent weights, deterministic opponent ordering, and authoritative draw/overtime/abandon scoring | `server/domain/rating.go` has canonical/inactivity/weight/invalid-input plus draw/OT/abandon fixtures; PostgreSQL snapshot locking, rating transaction integration, seasons and concurrent result transaction tests remain | @@ -1228,7 +1228,7 @@ the local/CI/community transport, not a silent production fallback. | # | Task | Acceptance | |---|---|---| | 8.39 `[D:8.3,8.14,8.17]` | **IN PROGRESS.** `MatchmakingState` now projects queue → proposal → allocation/process-ready/assignment-ready/connect/live plus terminal failure states; autoload `ControlPlaneClient` provides authenticated queue create/recovery/heartbeat/cancel and proposal response requests with idempotency/revision headers; `matchmaking.tscn`/`matchmaking.gd` expose the state and authoritative actions from the main menu; authenticated proposal recovery now reconciles missed proposal events and expires them at read time; the Go API publishes targeted authenticated revisioned queue/proposal/assignment events and the Godot client consumes state/proposal/assignment events, fetching the authoritative assignment after assignment readiness; the API can now use the durable participant-scoped proposal provider for both recovery and accept/decline mutations | `test_matchmaking_state.gd`, `test_control_plane_client.gd`, `test_matchmaking_ui.gd`, `TestProposalRecoveryIsParticipantScopedAndExpiresAtReadBoundary`, `TestAuthenticatedWebSocketDeliversOnlyTargetedRevisionedEvents`, `TestStateChangingAPIActionsPublishTargetedEvents`, `TestProposalRecoveryUsesDurableBackendAndRemainsParticipantScoped` and store proposal SQL tests reject stale/gapped/conflicting updates, validate endpoint/token/payload normalization, preserve idempotent duplicates, and guarantee visible phase/terminal copy; wiring durable allocation events into the outbox, wait/latency explanations and Godot runtime verification remain | -| 8.40 `[D:8.3,8.14]` | **IN PROGRESS.** Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated `/v1/events` WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot `ControlPlaneClient` can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return; `OutboxDispatcher` now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match | `server/domain/sync.go`, `server/api/events.go`, `server/api/service.go`, `server/store/outbox.go`, `service_test.go`, `outbox_test.go`, `matchmaking_state.gd`, `control_plane_client.gd`, `local_prediction_history.gd`, `net_ship_predictor.gd` and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation and prediction warm-up/hard-resync separation; Godot 4.7.1 headless project parse and 143-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; `cmd/control-plane` now provides a signal-bound API role that opens PostgreSQL, applies migrations, and wires durable session/queue/proposal/assignment adapters; worker roles, Redis fan-out and live multi-process control-plane/game verification remain | +| 8.40 `[D:8.3,8.14]` | **IN PROGRESS.** Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated `/v1/events` WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot `ControlPlaneClient` can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return; `OutboxDispatcher` now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match | `server/domain/sync.go`, `server/api/events.go`, `server/api/service.go`, `server/store/outbox.go`, `service_test.go`, `outbox_test.go`, `matchmaking_state.gd`, `control_plane_client.gd`, `local_prediction_history.gd`, `net_ship_predictor.gd` and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation and prediction warm-up/hard-resync separation; Godot 4.7.1 headless project parse and 143-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; `cmd/control-plane` now provides a signal-bound API role and `cmd/matcher` provides a signal-bound casual matcher role, both applying migrations and using durable PostgreSQL boundaries; allocator/maintenance roles, Redis fan-out and live multi-process control-plane/game verification remain | | 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior and 146-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; SDR relay-ticket installation, live allocated-token process integration and live Godot/PostgreSQL verification remain | | 8.42 `[D:8.22,8.23,8.24,8.40]` | **IN PROGRESS.** `RankedProfileState` and `ControlPlaneClient.fetch_ranked_profile()` expose the backend-authoritative rating/RD/volatility/games/tier/provisional/season view; matchmaking UI displays provisional/tier status without client-side rating math | `test_control_plane_client.gd` validates profile shape, numeric safety and provisional display; ranked profile fetch/display, committed revision after reconnect, abandon status and season countdown remain dependent on live auth/backend events and Godot runtime verification | | 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; decline, version mismatch, regional outage retry UI, failed reconnect, duplicate-action recovery and live Godot verification remain | diff --git a/server/cmd/matcher/main.go b/server/cmd/matcher/main.go new file mode 100644 index 00000000..f8ba5fbe --- /dev/null +++ b/server/cmd/matcher/main.go @@ -0,0 +1,71 @@ +package main + +import ( + "context" + "database/sql" + "flag" + "fmt" + "log" + "os" + "os/signal" + "syscall" + "time" + + "github.com/cosmic-clash/cosmic-clash/server/domain" + "github.com/cosmic-clash/cosmic-clash/server/matcher" + "github.com/cosmic-clash/cosmic-clash/server/migrations" + "github.com/cosmic-clash/cosmic-clash/server/store" + _ "github.com/jackc/pgx/v5/stdlib" +) + +func main() { + dsn := flag.String("dsn", os.Getenv("COSMIC_CLASH_POSTGRES_DSN"), "PostgreSQL connection string") + migrationDir := flag.String("migrations", "migrations", "directory containing numbered SQL migrations") + playlist := flag.String("playlist", string(domain.Casual), "playlist to match; ranked requires a provider-enabled role") + size := flag.Int("size", 4, "players per match") + interval := flag.Duration("interval", time.Second, "poll interval") + flag.Parse() + if *dsn == "" { + fatalf("--dsn or COSMIC_CLASH_POSTGRES_DSN is required") + } + if *playlist != string(domain.Casual) { + fatalf("unsupported playlist %q; only casual is currently enabled", *playlist) + } + db, err := sql.Open("pgx", *dsn) + if err != nil { + fatalf("open PostgreSQL: %v", err) + } + defer db.Close() + startupCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + if err := db.PingContext(startupCtx); err != nil { + fatalf("ping PostgreSQL: %v", err) + } + if err := migrations.Apply(startupCtx, db, *migrationDir); err != nil { + fatalf("apply migrations: %v", err) + } + now := func() time.Time { return time.Now().UTC() } + worker := matcher.Worker{ + Source: func(ctx context.Context, at time.Time, limit int) ([]domain.Candidate, error) { + return store.ListQueuedCandidates(ctx, db, at, limit) + }, + Creator: matcher.ProposalCreatorFunc(func(ctx context.Context, proposal domain.Proposal, ticketIDs map[string]string, at time.Time) error { + return store.CreateProposal(ctx, db, proposal, ticketIDs, at) + }), + Playlist: domain.Casual, Size: *size, Now: now, + NextID: func() string { return fmt.Sprintf("proposal-%d", time.Now().UnixNano()) }, + Prepare: func(id string, playlist domain.Playlist, formation domain.MatchFormation, at time.Time) (domain.PreparedProposal, error) { + return domain.PrepareProposal(id, playlist, formation, nil, domain.RankedArena{}, at) + }, + } + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + if err := worker.Run(ctx, *interval); err != nil && ctx.Err() == nil { + fatalf("matcher stopped: %v", err) + } +} + +func fatalf(format string, args ...any) { + log.Printf("matcher: "+format, args...) + os.Exit(1) +} diff --git a/server/matcher/worker.go b/server/matcher/worker.go index 5831880f..5cf526d3 100644 --- a/server/matcher/worker.go +++ b/server/matcher/worker.go @@ -34,6 +34,26 @@ type Worker struct { Prepare PrepareFunc } +// Run polls until cancellation. A failed attempt is returned so a supervisor +// can restart the role rather than silently dropping durable claim failures. +func (w Worker) Run(ctx context.Context, interval time.Duration) error { + if interval <= 0 { + return fmt.Errorf("matcher interval must be positive") + } + for { + if _, err := w.RunOnce(ctx); err != nil { + return err + } + timer := time.NewTimer(interval) + select { + case <-ctx.Done(): + timer.Stop() + return nil + case <-timer.C: + } + } +} + // RunOnce performs one bounded matchmaking attempt. The source may be Redis // backed, but the creator must be the durable transaction that claims tickets; // a stale cache therefore fails safely and can be retried on the next pass. diff --git a/server/store/queue_sql.go b/server/store/queue_sql.go index 9a59ca8b..19d83cb8 100644 --- a/server/store/queue_sql.go +++ b/server/store/queue_sql.go @@ -37,6 +37,41 @@ WHERE ticket_id = $1 AND player_id = $2 AND revision = $3 RETURNING ticket_id, player_id, playlist, state, client_build, protocol_version, enqueued_at, expires_at, revision` ) +const QueueCandidateProjectionSQL = `SELECT ticket_id, player_id, playlist, client_build, + protocol_version, enqueued_at +FROM queue_tickets +WHERE state = 'QUEUED' AND expires_at > $1 +ORDER BY enqueued_at, ticket_id +LIMIT $2` + +// ListQueuedCandidates is an authoritative, expiry-filtered source for the +// matcher projection. It deliberately does not claim rows; CreateProposal is +// the transaction that performs the competing claim with SKIP LOCKED fences. +func ListQueuedCandidates(ctx context.Context, db *sql.DB, now time.Time, limit int) ([]domain.Candidate, error) { + if db == nil || now.IsZero() || limit < 1 || limit > 1000 { + return nil, fmt.Errorf("invalid queued candidate arguments") + } + rows, err := db.QueryContext(ctx, QueueCandidateProjectionSQL, now, limit) + if err != nil { + return nil, err + } + defer rows.Close() + var candidates []domain.Candidate + for rows.Next() { + var candidate domain.Candidate + var playlist string + if err := rows.Scan(&candidate.TicketID, &candidate.PlayerID, &playlist, &candidate.ClientBuild, &candidate.ProtocolVersion, &candidate.EnqueuedAt); err != nil { + return nil, err + } + candidate.Playlist = domain.Playlist(playlist) + candidates = append(candidates, candidate) + } + if err := rows.Err(); err != nil { + return nil, err + } + return candidates, nil +} + func CreateQueueTicket(ctx context.Context, db *sql.DB, ticketID, playerID, idempotencyKey string, spec domain.QueueSpec, now time.Time) (domain.QueueTicket, error) { if db == nil || ticketID == "" || playerID == "" || len(idempotencyKey) < 16 || len(idempotencyKey) > 128 || (spec.Playlist != domain.Casual && spec.Playlist != domain.Ranked) || spec.ClientBuild == "" || len(spec.ClientBuild) > 128 || spec.ProtocolVersion < 1 || now.IsZero() { return domain.QueueTicket{}, fmt.Errorf("invalid queue transaction arguments")