From 00d6b1edd326c6c4ed9afc17569e42bf5c06718e Mon Sep 17 00:00:00 2001 From: Josh Creek <8179928+jcreek@users.noreply.github.com> Date: Tue, 1 Sep 2026 17:56:47 +0100 Subject: [PATCH] test(multiplayer): cover compose queue lifecycle --- multiplayer-next.md | 2 +- scripts/verify_allocated_compose.sh | 31 +++++++++++++++++++++++ server/security/test_compose_manifests.py | 3 +++ 3 files changed, 35 insertions(+), 1 deletion(-) diff --git a/multiplayer-next.md b/multiplayer-next.md index 52a2d715..5c00092e 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1246,7 +1246,7 @@ the local/CI/community transport, not a silent production fallback. | 8.44 `[D:8.3,8.4,8.28,8.31]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Go observability package encodes queue/proposal/match/server IDs and lifecycle stage while recursively redacting auth/relay tokens and credentials. `Service.Log` is wired to mutation and read routes at every outcome, and `cmd/control-plane` writes those events as JSON lines to stderr | `server/observability/` covers correlation fields, nested secret redaction, content-aware credential canaries and unnamed-event rejection; API tests cover lifecycle event wiring without logging error text. A production metrics/traces backend and dashboard/alert routing remain open; the local logger is intentionally stderr-only | | 8.45 `[D:8.2,8.44]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Go observability package turns the documented RTT, allocation/connect latency, result-success, API-latency and tick/headroom thresholds into executable window checks; the API exporter emits a bounded cumulative latency histogram suitable for querying the documented p95 API SLO | `server/observability/slo.go`, `metrics.go`, `deploy/observability/prometheus-rules.yaml` and adversarial tests cover healthy/violating/empty windows, fixed operation/status labels, cumulative bucket boundaries, arbitrary-path cardinality safety, and optional API p95/5xx alerts. Production scrape configuration, alert routing, wait/MMR/proposal/flood/cost series and runbooks remain | | 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | **IN PROGRESS.** Go unit/race coverage spans the current domain/store/supervisor policies, and fuzz targets now exercise queue input, result payload hashing and revision events | `server/domain/*_test.go`, `server/store/*_test.go`, `server/supervisor/*_test.go`, `server/migrations/*_test.go` and `server/domain/fuzz_test.go` pass normal/race suites; `go test -race ./...` passes across API, domain, migrations, observability, store, supervisor and testkit; `go vet ./...` passes; each of the three declared domain fuzz targets passes a bounded 4-second run; PostgreSQL live migration execution now runs clean (§8.5), and four real-concurrency cases are covered against a live database with `-race`: §8.14's queue-heartbeat revision race, §8.18's two-matcher contested-ticket race, §8.30's cross-allocator-replica capacity race, and §8.21/§8.25's concurrent identical-result-submission race; the "lost Redis" fixture is covered live against a real server (§8.14: real TTL expiry, repair-after-`FLUSHALL`; fake Steam/allocator fixtures are §8.47's testkit, already done). Further transaction fixtures (e.g. concurrent proposal-recovery expiry races, live Redis failover mid-write under load) remain | -| 8.47 `[D:8.7,8.30]` | **IN PROGRESS.** Offline testkit provides deterministic fake Steam verification and fake allocation with forced failure injection | `server/testkit/` covers verified identity/replay, unknown identity, wrong App ID, expiry, no capacity, compatibility-key conflict, idempotent allocation replay and cloud-free forced allocation failure in `TestOfflineFakesCoverVerificationAndAllocationFailureMatrix`; API/Compose integration and live exhaustive matrix remain | +| 8.47 `[D:8.7,8.30]` | **IN PROGRESS.** Offline testkit provides deterministic fake Steam verification and fake allocation with forced failure injection; the independent Compose runner now also drives fake-Steam session issuance and the real HTTP queue create/heartbeat/cancel boundary with an idempotency-conflict check | `server/testkit/` covers verified identity/replay, unknown identity, wrong App ID, expiry, no capacity, compatibility-key conflict, idempotent allocation replay and cloud-free forced allocation failure; the Compose API slice is wired into CI, while live exhaustive matrix and production Steam remain | | 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | **IN PROGRESS.** Offline testkit exercises verified queue projection → ranked six-player proposal → ENet allocation → assignment-ready manifest → certified durable result receipt; `compose.allocated-smoke.yml` independently runs the real testkit API against PostgreSQL and the real game-server supervisor with a generated signed roster, verifying authenticated result, idempotent retry, shutdown acknowledgment, durable receipt/audit rows, and SIGTERM-driven game-process drain | `.github/workflows/allocated-compose.yml` runs `make verify-allocated-compose`. Full queue/proposal/allocation orchestration over HTTP, live Docker evidence from this workspace, and legacy fixture non-regression remain open | | 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | **IN PROGRESS.** `scripts/verify_kind_agones.sh` creates a disposable kind cluster, installs pinned Agones, loads the real `game-server` image, applies the Fleet in an explicitly separate Agones-only supervisor/UDP readiness mode, and verifies readiness plus allocation of a dynamic UDP endpoint; `.github/workflows/agones-integration.yml` runs it for infrastructure changes and on demand | The cloud-free runner is committed and fails clearly when Docker/kind/Helm are unavailable. CI/live evidence for production control-plane registration, roster/no-show, both readiness stages, races, multi-match node, result-pending reconciliation, drain, and rollback remains open | | 8.50 `[D:8.25,8.37,8.43,8.49]` | Network/chaos suite: 100 ms RTT, jitter/loss, client/API/matcher restart, game-pod death, node drain, Redis failover and control-plane loss | System recovers to a defined state; infrastructure-caused cases cannot penalise affected players | diff --git a/scripts/verify_allocated_compose.sh b/scripts/verify_allocated_compose.sh index a4e676f1..791b61ee 100755 --- a/scripts/verify_allocated_compose.sh +++ b/scripts/verify_allocated_compose.sh @@ -63,6 +63,37 @@ for attempt in $(seq 1 60); do sleep 1 done +session_json="$(curl -fsS -X POST "$api_url/v1/session/steam" \ + -H 'Content-Type: application/json' -d '{"web_api_ticket":"compose-queue-ticket"}')" +access_token="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["access_token"])' <<<"$session_json")" +queue_body='{"ticket_id":"compose-queue-ticket","playlist":"casual","client_build":"build-1","protocol_version":1}' +queue_json="$(curl -fsS -X POST "$api_url/v1/queue" \ + -H "Authorization: Bearer $access_token" \ + -H 'Idempotency-Key: compose-queue-key-123456' \ + -H 'Content-Type: application/json' -d "$queue_body")" +queue_revision="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["revision"])' <<<"$queue_json")" +[[ "$queue_revision" == 0 ]] + +# Reusing a queue idempotency key with different command material must not +# silently turn into a second ticket or a successful replay. +conflict_status="$(curl -sS -o /dev/null -w '%{http_code}' -X POST "$api_url/v1/queue" \ + -H "Authorization: Bearer $access_token" \ + -H 'Idempotency-Key: compose-queue-key-123456' \ + -H 'Content-Type: application/json' \ + -d '{"ticket_id":"compose-other-ticket","playlist":"casual","client_build":"build-1","protocol_version":1}')" +[[ "$conflict_status" == 409 ]] + +heartbeat_json="$(curl -fsS -X POST "$api_url/v1/queue/compose-queue-ticket/heartbeat" \ + -H "Authorization: Bearer $access_token" \ + -H 'Idempotency-Key: compose-heartbeat-key-123456' \ + -H 'If-Match-Revision: 0')" +heartbeat_revision="$(python3 -c 'import json,sys; print(json.load(sys.stdin)["revision"])' <<<"$heartbeat_json")" +[[ "$heartbeat_revision" == 1 ]] +curl -fsS -o /dev/null -X POST "$api_url/v1/queue/compose-queue-ticket/cancel" \ + -H "Authorization: Bearer $access_token" \ + -H 'Idempotency-Key: compose-cancel-key-123456' \ + -H "If-Match-Revision: $heartbeat_revision" + # Model the durable state produced by the allocator, then use the real HTTP # workload authentication and mutation boundaries for every action below. "${compose[@]}" exec -T database psql -v ON_ERROR_STOP=1 -U cosmic_clash_test -d cosmic_clash_test <<'SQL' diff --git a/server/security/test_compose_manifests.py b/server/security/test_compose_manifests.py index 3fade56f..1c204300 100644 --- a/server/security/test_compose_manifests.py +++ b/server/security/test_compose_manifests.py @@ -26,6 +26,9 @@ class ComposeManifestTest(unittest.TestCase): "result_receipts", "/v1/servers/compose-server/shutdown", "SERVER_SHUTDOWN", + "/v1/session/steam", + "compose-queue-key-123456", + "/v1/queue/compose-queue-ticket/heartbeat", "down --volumes --remove-orphans", ): self.assertIn(marker, runner)