diff --git a/multiplayer-next.md b/multiplayer-next.md index 34522028..baa73e40 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1219,7 +1219,7 @@ the local/CI/community transport, not a silent production fallback. | 8.27 `[D:8.26]` | **IN PROGRESS.** Go supervisor package provides local-safe Agones REST discovery, validates assigned address/port data, injects dynamic `SDR_LISTEN_PORT`/`SDR_IP`, performs explicit process-ready probing and Ready transition; direct mode bypasses Agones | `server/supervisor/` covers allocated/direct startup, invalid endpoint rejection, dynamic endpoint/Ready ordering and authenticated drain; allocated Godot now supplies a loopback readiness/drain control surface and `agones_sdk.gd` supplies sidecar Health/Ready/Shutdown/annotation REST operations; metadata watch, real Agones annotation/shutdown confirmation and emulator integration remain | | 8.28 `[D:8.6,8.27]` | **IN PROGRESS.** Supervisor separates explicit process-ready from Agones Ready and never scrapes stdout; allocated mode refuses to mark Ready without a configured readiness probe. The durable control-plane counterpart now exists: a workload-authenticated `POST /v1/servers/{id}/register` (and its `/api/v1` contract alias) advances a match's `ALLOCATING -> PROCESS_READY -> ASSIGNMENT_READY` state, and every participant's queue ticket with it, as one idempotent SERIALIZABLE transaction, gated on every participant already holding a live, unexpired assignment. The supervisor now calls it: once Agones Ready succeeds, it POSTs process-ready (`assignment_ready=false`) using a workload token read fresh from disk each call (matching kubelet's in-place rotation of a projected service-account token), and a registration failure kills the child rather than leaving an Agones-Ready-but-control-plane-unregistered process running; `ControlPlaneURL` unset (the default) is a total no-op. It then reports assignment-ready too: `server_boot.gd` already verifies its mounted roster synchronously before `/ready` is ever exposed (so process-ready implies the roster was valid), and the API's `ASSIGNMENT_READY` gate checks only durable `assignments` rows server-side — so no new Godot-side state was needed, correcting an earlier overcautious note here. The supervisor retries assignment-ready (default 5 attempts, 2s apart, configurable) since those durable rows may lag process-ready slightly; a persistent failure there does not kill the child, unlike process-ready. Per-allocation data (currently `match-id`) now has a real channel to an already-Ready pod: `server/agones.Client.Allocate` requests `cosmic-clash.io/match-id`/`cosmic-clash.io/allocation-id` as `GameServerAllocation.spec.metadata.annotations` (Agones applies these to the allocated GameServer's own `object_meta` — the only channel that exists post-allocation, since env vars are fixed at pod creation), and the supervisor reads them back from its existing `/gameserver` SDK call, falling back to them only when `MatchID` isn't explicitly configured. The image now exists: a new `Dockerfile` `game-server` target packages the supervisor as ENTRYPOINT alongside the same dedicated-server export `server` produces | `server/supervisor/` tests prove Ready follows the probe and direct mode remains functional; `server_control.gd`, `agones_sdk.gd` and process-level smokes prove loopback `/ready`, `/health`, bearer-protected `/drain`, sidecar-shaped Health/Ready calls and drain admission fencing; `server/api/service.go`, `server/store/allocation_match_sql.go` and adversarial tests cover the registration route, digest/protocol validation, idempotent replay/conflict and the assignment-count gate; `server/supervisor/supervisor_test.go` covers opt-in registration, the workload-token/body/idempotency-key shape, the kill-on-failure path, the match-ID annotation fallback (both that it's used and that its absence fails closed before any HTTP call), the full process-ready-then-assignment-ready sequence and body shapes, and a retry test that fails assignment-ready twice with 409 before succeeding, asserting `Start()` still succeeds and the child is never killed; `server/agones/allocation_test.go` covers the requested annotations. `docker build --target game-server` verified for real: both binaries present, correct permissions, supervisor prints its usage; `server/store/stalled_allocation_sql.go`/`_test.go` and a live `TestPostgreSQLStalledAllocationsAreReclaimedWithoutPenalisingPlayers` cover the deadline boundary (a recent match must survive untouched), the no-penalty requeue and refreshed expiry, participant release, and idempotence against a second pass. **Fixed in passing**: `Dockerfile`'s `server` stage's `ubuntu` base digest had gone dead on Docker Hub (`docker pull` returned "not found", verified independently) — `make verify-phase6` was silently broken for a clean build before the re-pin; confirmed fixed with a full `make verify-phase6` run (arenas rotated, both goals observed, clean teardown). **Still not wired into `deploy/k8s/base/fleet.yaml`**: the manifest doesn't reference the `game-server` image or invoke any supervisor flags yet — the concrete remaining step is deciding and adding the per-deployment values (`--control-plane-url`, `--workload-token-path` plus the projected token volume, `--server-id-env`/`--image-digest-env` Downward API wiring), deliberately not guessed at here since they're environment-specific. `deploy/cosmic-clash-server` now wraps its exec in `stdbuf -oL -eL` (falling back to unwrapped if unavailable), fixing a real, live-confirmed bug: a genuinely detached (`docker run -d`) container showed zero `docker logs` output — not even the startup line — for 20+ seconds while the process ran normally, and `docker stop`'s SIGTERM lost that buffered output permanently rather than delaying it; re-verified fixed against the real launcher script, then a full `make verify-phase6` re-run confirmed no regression. Health-reclaim now exists: `store.ExpireStalledAllocations` reclaims a match stuck in `ALLOCATING`/`PROCESS_READY`/`ASSIGNMENT_READY` past a deadline (server crashed, or was reclaimed by Agones as unhealthy, before ever registering) by failing the match and requeuing every participant to `QUEUED` with a fresh expiry rather than penalising them — task 8.50's own "infrastructure-caused cases cannot penalise affected players" criterion directly settles the requeue-vs-fail design question this had been blocked on. Wired into `cmd/maintenance` alongside the season-rollover sweep (`--stalled-allocation-deadline` default 2m, `--stalled-allocation-batch`). **Superseding the `fleet.yaml` framing above**: §8.10's `WorkloadVerify` blocker, and its delivery channel, are both now closed — a control-plane-self-issued signed token (not a Kubernetes JWT), minted by `cmd/allocator` into a `cosmic-clash.io/workload-token` annotation and read back by the supervisor, exactly the way `match-id` already worked — see §8.10. `/register` and `/result` no longer 503 unconditionally once every `--workload-secret` (control plane, allocator) is set consistently. What remains for `fleet.yaml` is now purely the manifest itself: it doesn't yet reference the `game-server` image or invoke any supervisor flags (`--control-plane-url`, `--server-id-env`/`--image-digest-env` Downward API wiring — `--workload-token-path` is no longer required, since the annotation fallback covers it) — deliberately not guessed at here since these are environment-specific values, and this whole path has only run against HTTP-level Agones fakes, never a real cluster (see §8.10's "what's still missing") | | 8.29 `[D:8.26,8.27]` | **IN PROGRESS.** Supervisor discovers and validates the Agones endpoint, propagates the actual dynamic `--port`, and exports `SDR_LISTEN_PORT`/`SDR_IP` only for Hosted-SDR while preserving an isolated ENet path | `server/supervisor/` tests cover invalid address/port rejection, dynamic port argument/env propagation and SDR-vs-ENet separation; real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT and multi-match fixture remain | -| 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** Pure Go allocator filters Ready servers by region/build/protocol/transport, atomically claims one with idempotent allocation replay, verifies unanimous accepted-proposal/playlist/participant invariants before provider invocation, and now owns the assignment-publication boundary; PostgreSQL adds durable GameServer registration and compatible `SKIP LOCKED` claims with request-digest fencing, plus a leased `ALLOCATING`-match claim that derives the immutable compatibility tuple and fences bind/release by deterministic allocation ID; bind atomically attaches only a recorded provider allocation and advances every accepted participant ticket to `ALLOCATING`; `server/agones` strictly projects Ready GameServers from Fleet compatibility labels, submits/validates namespaced `GameServerAllocation` responses and dynamic endpoints; `server/allocator` reconciles provider success into durable state before exposing the endpoint; `cmd/allocator` refreshes that Ready projection before driving the lease → provider → durable-record → match-bind sequence and rebinds a recovered durable provider allocation without a second provider call | `server/domain/allocator.go`, `server/store/allocator_sql.go`, `server/store/allocation_match_sql.go`, `server/store/allocation_match_adapter.go`, `server/agones/allocation.go`, `server/allocator/service.go`, `server/allocator/worker.go`, `server/cmd/allocator`, `server/migrations/0004_allocator_registry.sql`, `0006_match_allocation_claims.sql` and tests cover strict Ready-server projection, stale-projection protection for allocated rows, deterministic compatible selection, exhaustion, conflicting/identical allocation replay, allocation-match lease recovery/bind fencing, atomic participant lifecycle transition, recovered-allocation binding without provider recall, immutable Fleet selector labels, provider ambiguity lease retention, unknown allocations, SQL claim ordering, invalid input, provider error/malformed response/IPv6 endpoint handling, accepted-proposal gating, durable-reconciliation failure isolation and assignment replay/conflict; `TestPostgreSQLAllocatorClaimReplayAndCapacityFence` and `TestPostgreSQLAllocationMatchClaimLeaseAndBindFence` cover the live database paths when the disposable database gate is run; `TestPostgreSQLConcurrentAllocationClaimNeverDoubleBooksAReadyServer` now races real concurrent claims against fewer Ready servers than requesters and proves no double-booking and no stray errors, stable across repeated `-race` runs; unknown provider-outcome reconciliation, signed roster metadata and live Agones integration remain | +| 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** Pure Go allocator filters Ready servers by region/build/protocol/transport, atomically claims one with idempotent allocation replay, verifies unanimous accepted-proposal/playlist/participant invariants before provider invocation, and now owns the assignment-publication boundary; PostgreSQL adds durable GameServer registration and compatible `SKIP LOCKED` claims with request-digest fencing, plus a leased `ALLOCATING`-match claim that derives the immutable compatibility tuple and fences bind/release by deterministic allocation ID; bind atomically attaches only a recorded provider allocation and advances every accepted participant ticket to `ALLOCATING`; `server/agones` strictly projects Ready GameServers from Fleet compatibility labels, submits/validates namespaced `GameServerAllocation` responses and dynamic endpoints; `server/allocator` reconciles provider success into durable state before exposing the endpoint; `cmd/allocator` refreshes that Ready projection before driving the lease → provider → durable-record → match-bind sequence and rebinds a recovered durable provider allocation without a second provider call | `server/domain/allocator.go`, `server/store/allocator_sql.go`, `server/store/allocation_match_sql.go`, `server/store/allocation_match_adapter.go`, `server/agones/allocation.go`, `server/allocator/service.go`, `server/allocator/worker.go`, `server/cmd/allocator`, `server/migrations/0004_allocator_registry.sql`, `0006_match_allocation_claims.sql` and tests cover strict Ready-server projection, stale-projection protection for allocated rows, deterministic compatible selection, exhaustion, conflicting/identical allocation replay, allocation-match lease recovery/bind fencing, atomic participant lifecycle transition, recovered-allocation binding without provider recall, immutable Fleet selector labels, provider ambiguity lease retention, unknown allocations, SQL claim ordering, invalid input, provider error/malformed response/IPv6 endpoint handling, accepted-proposal gating, durable-reconciliation failure isolation and assignment replay/conflict; `TestPostgreSQLAllocatorClaimReplayAndCapacityFence` and `TestPostgreSQLAllocationMatchClaimLeaseAndBindFence` cover the live database paths when the disposable database gate is run; `TestPostgreSQLConcurrentAllocationClaimNeverDoubleBooksAReadyServer` now races real concurrent claims against fewer Ready servers than requesters and proves no double-booking and no stray errors, stable across repeated `-race` runs; `scripts/run_allocator_integration.sh` and `TestRealAllocatorWorkerReconcilesAgonesAllocationAndBindsMatch` now add a real PostgreSQL + Agones-shaped HTTP provider integration covering Ready projection → worker lease → provider request → durable reconciliation → match/ticket bind; unknown provider-outcome reconciliation, signed roster metadata and live Agones cluster integration remain | | 8.31 `[D:8.9,8.30]` | **IN PROGRESS.** Pure Go assignment gate requires Allocated state, exact allocation ID/match/server/region/build/protocol/transport compatibility, non-empty hosted endpoint and verified manifest signature before exposure; allocator publication cannot expose Ready state; durable roster persistence now verifies each canonical join-authorisation signature before publishing player rows; allocator service gates roster publication on allocated state and endpoint presence | `server/domain/assignment.go`, `allocator.go`, `store/assignment_sql.go` plus adversarial fixtures cover early-connect, tampered signature/manifest, wrong compatibility, empty endpoint, unknown allocation, forged roster signature, valid signature, premature publication and post-publication mutation rejection; Agones metadata watch, hosted-address registration, production signer and client-ticket publication remain | | 8.32 `[D:8.2,8.26,8.30]` | **IN PROGRESS.** Provider-neutral FleetAutoscaler baseline preserves a two-process Ready buffer, caps warm capacity, and leaves Allocated scale-down independent of the Ready floor; Fleet image references remain digest-pinned for current/rollback pre-pull | `deploy/k8s/base/fleet-autoscaler.yaml` and manifest tests cover Fleet ownership, Buffer policy and floor/cap invariants; regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99 and N+1 certification remain | | 8.33 `[D:8.26,8.32]` | **IN PROGRESS.** Fleet scheduling now requires on-demand capacity and spreads Ready processes across zones with skew 1; the autoscaler preserves the two-process Ready floor | `deploy/k8s/base/fleet.yaml` and manifest tests reject interruptible placement and single-zone concentration structurally; regional node pools, forced node-loss testing and measured N+1 headroom remain | diff --git a/scripts/run_allocator_integration.sh b/scripts/run_allocator_integration.sh new file mode 100755 index 00000000..48371cfc --- /dev/null +++ b/scripts/run_allocator_integration.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +set -euo pipefail + +repo_root="$(cd "$(dirname "$0")/.." && pwd)" +container_name="cosmic-clash-allocator-integration" +database="cosmic_clash_test" +user="cosmic_clash_test" +password="cosmic_clash_test" + +cleanup() { + docker rm -f "$container_name" >/dev/null 2>&1 || true +} +trap cleanup EXIT + +cleanup +docker run --rm -d --name "$container_name" \ + -e POSTGRES_DB="$database" \ + -e POSTGRES_USER="$user" \ + -e POSTGRES_PASSWORD="$password" \ + -p 55436:5432 postgres:17-alpine >/dev/null + +for attempt in $(seq 1 30); do + if docker exec "$container_name" pg_isready -U "$user" -d "$database" >/dev/null 2>&1; then + break + fi + if [ "$attempt" = 30 ]; then + echo "PostgreSQL did not become ready" >&2 + exit 1 + fi + sleep 1 +done + +cd "$repo_root/server" +COSMIC_CLASH_POSTGRES_DSN="postgres://${user}:${password}@127.0.0.1:55436/${database}?sslmode=disable" \ + go test -tags integration ./allocator -count=1 diff --git a/server/allocator/allocator_integration_test.go b/server/allocator/allocator_integration_test.go new file mode 100644 index 00000000..a33acb2f --- /dev/null +++ b/server/allocator/allocator_integration_test.go @@ -0,0 +1,114 @@ +//go:build integration + +package allocator + +import ( + "context" + "database/sql" + "encoding/json" + "fmt" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "testing" + "time" + + "github.com/cosmic-clash/cosmic-clash/server/agones" + "github.com/cosmic-clash/cosmic-clash/server/migrations" + "github.com/cosmic-clash/cosmic-clash/server/store" + _ "github.com/jackc/pgx/v5/stdlib" +) + +func TestRealAllocatorWorkerReconcilesAgonesAllocationAndBindsMatch(t *testing.T) { + dsn := os.Getenv("COSMIC_CLASH_POSTGRES_DSN") + if dsn == "" { + t.Skip("COSMIC_CLASH_POSTGRES_DSN is not set") + } + db, err := sql.Open("pgx", dsn) + if err != nil { + t.Fatal(err) + } + defer db.Close() + ctx := context.Background() + if err := db.PingContext(ctx); err != nil { + t.Fatal(err) + } + if _, err := db.ExecContext(ctx, `DROP TABLE IF EXISTS schema_migrations, assignments, audit_events, outbox, result_receipts, ranked_season_rollovers, penalties, seasons, ratings, match_participants, matches, proposal_participants, proposals, allocations, game_servers, queue_tickets, idempotency_keys, sessions, identities CASCADE`); err != nil { + t.Fatal(err) + } + if err := migrations.Apply(ctx, db, filepath.Join("..", "migrations")); err != nil { + t.Fatal(err) + } + now := time.Now().UTC().Truncate(time.Microsecond) + for index, player := range []string{"allocator-worker-a", "allocator-worker-b"} { + if _, err := db.ExecContext(ctx, `INSERT INTO identities (player_id, steam_id) VALUES ($1, $1)`, player); err != nil { + t.Fatal(err) + } + if _, err := db.ExecContext(ctx, `INSERT INTO queue_tickets (ticket_id, player_id, playlist, state, client_build, protocol_version, enqueued_at, expires_at) VALUES ($1, $2, 'casual', 'ACCEPTED', 'build-1', 1, $3, $4)`, fmt.Sprintf("allocator-worker-ticket-%d", index), player, now, now.Add(time.Minute)); err != nil { + t.Fatal(err) + } + } + if _, err := db.ExecContext(ctx, `INSERT INTO matches (match_id, playlist, state, region, protocol_version) VALUES ('allocator-worker-match', 'casual', 'ALLOCATING', 'EU', 1)`); err != nil { + t.Fatal(err) + } + for index, player := range []string{"allocator-worker-a", "allocator-worker-b"} { + if _, err := db.ExecContext(ctx, `INSERT INTO match_participants (match_id, player_id, ticket_id, slot, team) VALUES ('allocator-worker-match', $1, $2, $3, $4)`, player, fmt.Sprintf("allocator-worker-ticket-%d", index), index, index); err != nil { + t.Fatal(err) + } + } + + var allocationCalls int + provider := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodGet { + _, _ = w.Write([]byte(`{"items":[{"metadata":{"name":"agones-ready-1","labels":{"cosmic-clash.io/region":"EU","cosmic-clash.io/build":"build-1","cosmic-clash.io/protocol":"1","cosmic-clash.io/transport":"enet"}},"status":{"state":"Ready"}}]}`)) + return + } + if r.Method != http.MethodPost { + t.Fatalf("provider method = %s", r.Method) + } + allocationCalls++ + var body map[string]any + if err := json.NewDecoder(r.Body).Decode(&body); err != nil { + t.Fatal(err) + } + if body["kind"] != "GameServerAllocation" { + t.Fatalf("provider body kind = %v", body["kind"]) + } + w.Header().Set("Content-Type", "application/json") + _, _ = w.Write([]byte(`{"status":{"state":"Allocated","gameServerName":"agones-ready-1","address":"127.0.0.1","ports":[{"name":"default","port":7777}]}}`)) + })) + defer provider.Close() + + agonesClient := agones.Client{BaseURL: provider.URL, Namespace: "games", HTTP: provider.Client()} + ready, err := agonesClient.ListReadyServers(ctx) + if err != nil || len(ready) != 1 { + t.Fatalf("ready projection = %+v, err=%v", ready, err) + } + if err := store.RegisterReadyServer(ctx, db, ready[0], now); err != nil { + t.Fatal(err) + } + worker := Worker{ + Claims: store.AllocatingMatchClaims{DB: db, Transport: "enet"}, + Service: Service{Provider: agonesClient, Durable: store.AllocationRegistry{DB: db}, Now: func() time.Time { return now }}, + Now: func() time.Time { return now }, + } + processed, err := worker.RunOnce(ctx) + if err != nil || !processed || allocationCalls != 1 { + t.Fatalf("worker processed=%t err=%v provider calls=%d", processed, err, allocationCalls) + } + var serverID, matchState, ticketState string + if err := db.QueryRowContext(ctx, `SELECT server_id, state FROM matches WHERE match_id = 'allocator-worker-match'`).Scan(&serverID, &matchState); err != nil { + t.Fatal(err) + } + if err := db.QueryRowContext(ctx, `SELECT state FROM queue_tickets WHERE ticket_id = 'allocator-worker-ticket-0'`).Scan(&ticketState); err != nil { + t.Fatal(err) + } + if serverID != "agones-ready-1" || matchState != "ALLOCATING" || ticketState != "ALLOCATING" { + t.Fatalf("durable lifecycle server=%q match=%q ticket=%q", serverID, matchState, ticketState) + } + var recorded int + if err := db.QueryRowContext(ctx, `SELECT count(*) FROM allocations WHERE allocation_id = 'allocation-allocator-worker-match' AND server_id = 'agones-ready-1' AND state = 'ALLOCATED'`).Scan(&recorded); err != nil || recorded != 1 { + t.Fatalf("recorded allocations=%d err=%v", recorded, err) + } +}