diff --git a/multiplayer-next.md b/multiplayer-next.md index 81525e83..d5ff3f35 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1223,7 +1223,7 @@ production fallback. | 8.27 `[D:8.26]` | **IN PROGRESS.** Go supervisor package provides local-safe Agones REST discovery, validates assigned address/port data, injects dynamic `SDR_LISTEN_PORT`/`SDR_IP`, performs explicit process-ready probing and Ready transition; direct mode bypasses Agones | `server/supervisor/` covers allocated/direct startup, invalid endpoint rejection, dynamic endpoint/Ready ordering and authenticated drain; allocated Godot now supplies a loopback readiness/drain control surface and `agones_sdk.gd` supplies sidecar Health/Ready/Shutdown/annotation REST operations; metadata watch, real Agones annotation/shutdown confirmation and emulator integration remain | | 8.28 `[D:8.6,8.27]` | **IN PROGRESS.** Supervisor separates explicit process-ready from Agones Ready and never scrapes stdout; allocated mode refuses to mark Ready without a configured readiness probe. The durable control-plane counterpart now exists: a workload-authenticated `POST /v1/servers/{id}/register` (and its `/api/v1` contract alias) advances a match's `ALLOCATING -> PROCESS_READY -> ASSIGNMENT_READY` state, and every participant's queue ticket with it, as one idempotent SERIALIZABLE transaction, gated on every participant already holding a live, unexpired assignment. The supervisor now calls it: once Agones Ready succeeds, it POSTs process-ready (`assignment_ready=false`) using a workload token read fresh from disk each call (matching kubelet's in-place rotation of a projected service-account token), and a registration failure kills the child rather than leaving an Agones-Ready-but-control-plane-unregistered process running; `ControlPlaneURL` unset (the default) is a total no-op. It then reports assignment-ready too: `server_boot.gd` already verifies its mounted roster synchronously before `/ready` is ever exposed (so process-ready implies the roster was valid), and the API's `ASSIGNMENT_READY` gate checks only durable `assignments` rows server-side — so no new Godot-side state was needed, correcting an earlier overcautious note here. The supervisor retries assignment-ready (default 5 attempts, 2s apart, configurable) since those durable rows may lag process-ready slightly; a persistent failure there does not kill the child, unlike process-ready. Per-allocation data (currently `match-id`) now has a real channel to an already-Ready pod: `server/agones.Client.Allocate` requests `cosmic-clash.io/match-id`/`cosmic-clash.io/allocation-id` as `GameServerAllocation.spec.metadata.annotations` (Agones applies these to the allocated GameServer's own `object_meta` — the only channel that exists post-allocation, since env vars are fixed at pod creation), and the supervisor reads them back from its existing `/gameserver` SDK call, falling back to them only when `MatchID` isn't explicitly configured. The image now exists: a new `Dockerfile` `game-server` target packages the supervisor as ENTRYPOINT alongside the same dedicated-server export `server` produces; the base Fleet now invokes that target with the control-plane URL, server/image Downward API identity, roster/signing/drain material, and exported Godot executable. | `server/supervisor/` tests prove Ready follows the probe and direct mode remains functional; `server_control.gd`, `agones_sdk.gd` and process-level smokes prove loopback `/ready`, `/health`, bearer-protected `/drain`, sidecar-shaped Health/Ready calls and drain admission fencing; `server/api/service.go`, `server/store/allocation_match_sql.go` and adversarial tests cover the registration route, digest/protocol validation, idempotent replay/conflict and the assignment-count gate; `server/supervisor/supervisor_test.go` covers opt-in registration, the workload-token/body/idempotency-key shape, the kill-on-failure path, the match-ID annotation fallback (both that it's used and that its absence fails closed before any HTTP call), the full process-ready-then-assignment-ready sequence and body shapes, and a retry test that fails assignment-ready twice with 409 before succeeding, asserting `Start()` still succeeds and the child is never killed; `server/agones/allocation_test.go` covers the requested annotations. `docker build --target game-server` verified for real: both binaries present, correct permissions, supervisor prints its usage; `server/store/stalled_allocation_sql.go`/`_test.go` and a live `TestPostgreSQLStalledAllocationsAreReclaimedWithoutPenalisingPlayers` cover the deadline boundary (a recent match must survive untouched), the no-penalty requeue and refreshed expiry, participant release, and idempotence against a second pass. **Fixed in passing**: `Dockerfile`'s `server` stage's `ubuntu` base digest had gone dead on Docker Hub (`docker pull` returned "not found", verified independently) — `make verify-phase6` was silently broken for a clean build before the re-pin; confirmed fixed with a full `make verify-phase6` run (arenas rotated, both goals observed, clean teardown). `deploy/cosmic-clash-server` now wraps its exec in `stdbuf -oL -eL` (falling back to unwrapped if unavailable), fixing a real, live-confirmed bug: a genuinely detached (`docker run -d`) container showed zero `docker logs` output — not even the startup line — for 20+ seconds while the process ran normally, and `docker stop`'s SIGTERM lost that buffered output permanently rather than delaying it; re-verified fixed against the real launcher script, then a full `make verify-phase6` re-run confirmed no regression. Health-reclaim now exists: `store.ExpireStalledAllocations` reclaims a match stuck in `ALLOCATING`/`PROCESS_READY`/`ASSIGNMENT_READY` past a deadline (server crashed, or was reclaimed by Agones as unhealthy, before ever registering) by failing the match and requeuing every participant to `QUEUED` with a fresh expiry rather than penalising them — task 8.50's own "infrastructure-caused cases cannot penalise affected players" criterion directly settles the requeue-vs-fail design question this had been blocked on. Wired into `cmd/maintenance` alongside the season-rollover sweep (`--stalled-allocation-deadline` default 2m, `--stalled-allocation-batch`). **Superseding the `fleet.yaml` framing above**: §8.10's `WorkloadVerify` blocker, and its delivery channel, are both now closed — a control-plane-self-issued signed token (not a Kubernetes JWT), minted by `cmd/allocator` into a `cosmic-clash.io/workload-token` annotation and read back by the supervisor, exactly the way `match-id` already worked — see §8.10. `/register` and `/result` no longer 503 unconditionally once every `--workload-secret` (control plane, allocator) is set consistently. The remaining gates are live Agones annotation/shutdown behavior and production cluster readiness; those are covered by §8.49 and remain explicitly open. | | 8.29 `[D:8.26,8.27]` | **IN PROGRESS.** Supervisor discovers and validates the Agones endpoint, propagates the actual dynamic `--port`, and exports `SDR_LISTEN_PORT`/`SDR_IP` only for Hosted-SDR while preserving an isolated ENet path | `server/supervisor/` tests cover invalid address/port rejection, dynamic port argument/env propagation and SDR-vs-ENet separation; real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT and multi-match fixture remain | -| 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** Pure Go allocator filters Ready servers by region/build/protocol/transport, atomically claims one with idempotent allocation replay, verifies unanimous accepted-proposal/playlist/participant invariants before provider invocation, and now owns the assignment-publication boundary; PostgreSQL adds durable GameServer registration and compatible `SKIP LOCKED` claims with request-digest fencing, plus a leased `ALLOCATING`-match claim that derives the immutable compatibility tuple and fences bind/release by deterministic allocation ID; bind atomically attaches only a recorded provider allocation and advances every accepted participant ticket to `ALLOCATING`; `server/agones` strictly projects Ready GameServers from Fleet compatibility labels, submits/validates namespaced `GameServerAllocation` responses and dynamic endpoints, and can recover an already-Allocated GameServer by allocation ID after an ambiguous write; `server/allocator` reconciles provider success into durable state before exposing the endpoint; `cmd/allocator` refreshes that Ready projection before driving the lease → provider → durable-record → match-bind sequence and rebinds a recovered durable provider allocation without a second provider call | `server/domain/allocator.go`, `server/store/allocator_sql.go`, `server/store/allocation_match_sql.go`, `server/store/allocation_match_adapter.go`, `server/agones/allocation.go`, `server/allocator/service.go`, `server/allocator/worker.go`, `server/cmd/allocator`, `server/migrations/0004_allocator_registry.sql`, `0006_match_allocation_claims.sql` and tests cover strict Ready-server projection, stale-projection protection for allocated rows, deterministic compatible selection, exhaustion, conflicting/identical allocation replay, allocation-match lease recovery/bind fencing, atomic participant lifecycle transition, recovered-allocation binding without provider recall, immutable Fleet selector labels, provider ambiguity lease retention, unknown allocations, SQL claim ordering, invalid input, provider error/malformed response/IPv6 endpoint handling, accepted-proposal gating, provider recovery metadata/duplicate detection, durable-reconciliation failure isolation and assignment replay/conflict; `TestPostgreSQLAllocatorClaimReplayAndCapacityFence` and `TestPostgreSQLAllocationMatchClaimLeaseAndBindFence` cover the live database paths when the disposable database gate is run; `TestPostgreSQLConcurrentAllocationClaimNeverDoubleBooksAReadyServer` now races real concurrent claims against fewer Ready servers than requesters and proves no double-booking and no stray errors, stable across repeated `-race` runs; `scripts/run_allocator_integration.sh` and `TestRealAllocatorWorkerReconcilesAgonesAllocationAndBindsMatch` now add a real PostgreSQL + Agones-shaped HTTP provider integration covering Ready projection → worker lease → provider request → durable reconciliation → match/ticket bind; full live unknown-provider-outcome recovery and signed roster metadata/cluster integration remain | +| 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** PostgreSQL leases each `ALLOCATING` match under a deterministic allocation ID, derives immutable compatibility from its accepted roster, and atomically binds only a durably recorded provider allocation while advancing every ticket. Fresh and recovered provider results now share the same fail-closed validation of allocation/match/server identity, region, build, protocol, arena, transport, allocated state, and non-empty endpoint before persistence or binding. Workers bind the canonical allocation returned by durable reconciliation rather than the provider's pre-persistence object, preserving server-owned timestamps and normalization. Ambiguous provider outcomes retain the lease and recover by allocation ID before another external request. Agones request/response parsing and Fleet labels remain provider-portable | Unit/adversarial tests cover every fresh/recovered compatibility mismatch, empty endpoint, canonical durable result propagation, lease recovery, bind/release fencing, quota behavior, accepted-proposal gating, provider ambiguity, malformed responses, and immutable labels. PostgreSQL-tagged allocator/race/integration suites and the Agones-shaped HTTP runner remain committed; this provider-validation change awaits live database/cluster reruns while Docker storage, kind, and Helm are unavailable. Full unknown-outcome cluster recovery and signed roster metadata remain | | 8.31 `[D:8.9,8.30]` | **IN PROGRESS.** Pure Go assignment gate requires Allocated state, exact allocation ID/match/server/region/build/protocol/transport compatibility, non-empty hosted endpoint and verified manifest signature before exposure; allocator publication cannot expose Ready state; durable roster persistence now verifies each canonical join-authorisation signature before publishing player rows; allocator service gates roster publication on allocated state and endpoint presence | `server/domain/assignment.go`, `allocator.go`, `store/assignment_sql.go` plus adversarial fixtures cover early-connect, tampered signature/manifest, wrong compatibility, empty endpoint, unknown allocation, forged roster signature, valid signature, premature publication and post-publication mutation rejection; Agones metadata watch, hosted-address registration, production signer and client-ticket publication remain | | 8.32 `[D:8.2,8.26,8.30]` | **IN PROGRESS.** Provider-neutral FleetAutoscaler baseline preserves a two-process Ready buffer, caps warm capacity, and leaves Allocated scale-down independent of the Ready floor; Fleet image references remain digest-pinned for current/rollback pre-pull | `deploy/k8s/base/fleet-autoscaler.yaml` and manifest tests cover Fleet ownership, Buffer policy and floor/cap invariants; regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99 and N+1 certification remain | | 8.33 `[D:8.26,8.32]` | **IN PROGRESS.** Fleet scheduling now requires on-demand capacity and spreads Ready processes across zones with skew 1; the autoscaler preserves the two-process Ready floor | `deploy/k8s/base/fleet.yaml` and manifest tests reject interruptible placement and single-zone concentration structurally; regional node pools, forced node-loss testing and measured N+1 headroom remain | diff --git a/server/allocator/service.go b/server/allocator/service.go index 6ac88276..aa88ee9a 100644 --- a/server/allocator/service.go +++ b/server/allocator/service.go @@ -122,12 +122,20 @@ func (s Service) Allocate(ctx context.Context, request domain.AllocationRequest, } return agones.AllocatedServer{}, err } - if _, err := s.Durable.RecordProviderAllocation(ctx, result.Allocation, now); err != nil { + if err := validateProviderAllocation(request, result); err != nil { if s.Metrics != nil { s.Metrics.ObserveFailure(request.Region) } return agones.AllocatedServer{}, err } + recorded, err := s.Durable.RecordProviderAllocation(ctx, result.Allocation, now) + if err != nil { + if s.Metrics != nil { + s.Metrics.ObserveFailure(request.Region) + } + return agones.AllocatedServer{}, err + } + result.Allocation = recorded if s.Metrics != nil { s.Metrics.ObserveSuccess(request.Region) } diff --git a/server/allocator/service_test.go b/server/allocator/service_test.go index 60f546b2..8f26c297 100644 --- a/server/allocator/service_test.go +++ b/server/allocator/service_test.go @@ -25,6 +25,7 @@ func (p *providerSpy) Allocate(_ context.Context, _ domain.AllocationRequest, _ type durableSpy struct { calls int allocation domain.Allocation + result domain.Allocation err error } @@ -51,15 +52,19 @@ func (r *rosterSpy) PublishRoster(_ context.Context, _ domain.Assignment, _ []do func (d *durableSpy) RecordProviderAllocation(_ context.Context, allocation domain.Allocation, _ time.Time) (domain.Allocation, error) { d.calls++ d.allocation = allocation + if d.result.AllocationID != "" { + return d.result, d.err + } return allocation, d.err } func TestServiceDurablyRecordsProviderAllocationBeforeReturning(t *testing.T) { - provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", MatchID: "m", ServerID: "gs", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} + request := domain.AllocationRequest{AllocationID: "a", MatchID: "m", Region: "EU", Build: "b", Protocol: 1, Transport: "enet"} + provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", MatchID: "m", ServerID: "gs", Region: "EU", Build: "b", Protocol: 1, Transport: "enet", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} durable := &durableSpy{} metrics := NewMetrics() service := Service{Provider: provider, Durable: durable, Metrics: metrics, Now: func() time.Time { return time.Unix(1000, 0) }} - result, err := service.Allocate(context.Background(), domain.AllocationRequest{AllocationID: "a", MatchID: "m", Region: "EU", Build: "b", Protocol: 1, Transport: "enet"}, map[string]string{"region": "EU"}) + result, err := service.Allocate(context.Background(), request, map[string]string{"region": "EU"}) if err != nil || result.Endpoint == "" || durable.calls != 1 || durable.allocation.ServerID != "gs" { t.Fatalf("result=%+v err=%v durable=%+v", result, err, durable) } @@ -69,8 +74,54 @@ func TestServiceDurablyRecordsProviderAllocationBeforeReturning(t *testing.T) { } } +func TestServiceRejectsMismatchedFreshProviderAllocationBeforePersistence(t *testing.T) { + request := domain.AllocationRequest{AllocationID: "allocation-1", MatchID: "match-1", Playlist: domain.Ranked, Region: "EU", Build: "build-1", Protocol: 1, ArenaPath: "res://scenes/arena_01.tscn", Transport: "enet"} + base := agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: request.AllocationID, MatchID: request.MatchID, ServerID: "server-1", Region: request.Region, Build: request.Build, Protocol: request.Protocol, ArenaPath: request.ArenaPath, Transport: request.Transport, State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"} + for name, mutate := range map[string]func(*agones.AllocatedServer){ + "allocation id": func(r *agones.AllocatedServer) { r.Allocation.AllocationID = "other" }, + "match": func(r *agones.AllocatedServer) { r.Allocation.MatchID = "other" }, + "region": func(r *agones.AllocatedServer) { r.Allocation.Region = "NA" }, + "build": func(r *agones.AllocatedServer) { r.Allocation.Build = "other" }, + "protocol": func(r *agones.AllocatedServer) { r.Allocation.Protocol++ }, + "arena": func(r *agones.AllocatedServer) { r.Allocation.ArenaPath = "res://scenes/arena_02.tscn" }, + "transport": func(r *agones.AllocatedServer) { r.Allocation.Transport = "steam_sdr" }, + "server": func(r *agones.AllocatedServer) { r.Allocation.ServerID = "" }, + "state": func(r *agones.AllocatedServer) { r.Allocation.State = domain.ServerReady }, + "endpoint": func(r *agones.AllocatedServer) { r.Endpoint = "" }, + } { + t.Run(name, func(t *testing.T) { + result := base + mutate(&result) + durable := &durableSpy{} + service := Service{Provider: &providerSpy{result: result}, Durable: durable, Now: func() time.Time { return time.Unix(1000, 0) }} + if _, err := service.Allocate(context.Background(), request, nil); err == nil { + t.Fatal("mismatched provider result accepted") + } + if durable.calls != 0 { + t.Fatalf("mismatched result reached durable store %d times", durable.calls) + } + }) + } +} + +func TestServiceReturnsCanonicalDurableAllocation(t *testing.T) { + now := time.Unix(1000, 0) + request := domain.AllocationRequest{AllocationID: "a", MatchID: "m", Region: "EU", Build: "b", Protocol: 1, Transport: "enet"} + providerAllocation := domain.Allocation{AllocationID: "a", MatchID: "m", ServerID: "gs", Region: "EU", Build: "b", Protocol: 1, Transport: "enet", State: domain.ServerAllocated} + canonical := providerAllocation + canonical.AllocatedAt = now + service := Service{ + Provider: &providerSpy{result: agones.AllocatedServer{Allocation: providerAllocation, Endpoint: "127.0.0.1:7777"}}, + Durable: &durableSpy{result: canonical}, Now: func() time.Time { return now }, + } + result, err := service.Allocate(context.Background(), request, nil) + if err != nil || result.Allocation != canonical { + t.Fatalf("result=%+v err=%v, want canonical %+v", result, err, canonical) + } +} + func TestServiceDoesNotReturnProviderResultAfterDurableFailure(t *testing.T) { - provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} + provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", MatchID: "m", ServerID: "gs", Region: "EU", Build: "b", Protocol: 1, Transport: "enet", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} durable := &durableSpy{err: errors.New("database unavailable")} service := Service{Provider: provider, Durable: durable, Now: func() time.Time { return time.Unix(1000, 0) }} result, err := service.Allocate(context.Background(), domain.AllocationRequest{AllocationID: "a", MatchID: "m", Region: "EU", Build: "b", Protocol: 1, Transport: "enet"}, map[string]string{"region": "EU"}) @@ -113,7 +164,7 @@ func TestServiceDoesNotConsumeSharedQuotaWhenReconcilingProviderResult(t *testin func TestServiceDoesNotDoubleChargeQuotaAfterProviderResultRecovery(t *testing.T) { quota := "aSpy{} durable := &durableSpy{err: errors.New("recording unavailable")} - provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", MatchID: "m", Region: "EU", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} + provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", MatchID: "m", ServerID: "gs", Region: "EU", Build: "b", Protocol: 1, Transport: "enet", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} service := Service{Provider: provider, Durable: durable, Quota: quota, Now: func() time.Time { return time.Unix(1000, 0) }} request := domain.AllocationRequest{AllocationID: "a", MatchID: "m", Region: "EU", Build: "b", Protocol: 1, Transport: "enet"} if _, err := service.Allocate(context.Background(), request, nil); err == nil { @@ -136,7 +187,7 @@ func TestServiceAllocatesOnlyUnanimouslyAcceptedMatchingProposal(t *testing.T) { {PlayerID: "player-b", Response: domain.AcceptedResponse}, }, } - provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", MatchID: "m", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} + provider := &providerSpy{result: agones.AllocatedServer{Allocation: domain.Allocation{AllocationID: "a", MatchID: "m", ServerID: "gs", Region: "EU", Build: "b", Protocol: 1, Transport: "enet", State: domain.ServerAllocated}, Endpoint: "127.0.0.1:7777"}} durable := &durableSpy{} service := Service{Provider: provider, Durable: durable, Now: func() time.Time { return time.Unix(1000, 0) }} request := domain.AllocationRequest{AllocationID: "a", MatchID: "m", Region: "EU", Build: "b", Protocol: 1, Transport: "enet"} diff --git a/server/allocator/worker.go b/server/allocator/worker.go index b6e352a1..ee490448 100644 --- a/server/allocator/worker.go +++ b/server/allocator/worker.go @@ -50,13 +50,14 @@ func (w Worker) RunOnce(ctx context.Context) (bool, error) { return true, fmt.Errorf("recover provider allocation for match %s: %w", request.MatchID, err) } if found { - if err := validateRecoveredAllocation(request, recovered); err != nil { + if err := validateProviderAllocation(request, recovered); err != nil { return true, fmt.Errorf("recovered provider allocation for match %s: %w", request.MatchID, err) } - if _, err := w.Service.RecordProviderAllocation(ctx, recovered, w.Now()); err != nil { + recorded, err := w.Service.RecordProviderAllocation(ctx, recovered, w.Now()) + if err != nil { return true, fmt.Errorf("record recovered allocation for match %s: %w", request.MatchID, err) } - allocation = recovered.Allocation + allocation = recorded } else { result, err := w.Service.Allocate(ctx, request, AllocationLabels(request)) if err != nil { @@ -78,10 +79,10 @@ func (w Worker) RunOnce(ctx context.Context) (bool, error) { return true, nil } -func validateRecoveredAllocation(request domain.AllocationRequest, result agones.AllocatedServer) error { +func validateProviderAllocation(request domain.AllocationRequest, result agones.AllocatedServer) error { allocation := result.Allocation if result.Endpoint == "" || allocation.State != domain.ServerAllocated || allocation.AllocationID != request.AllocationID || allocation.MatchID != request.MatchID || allocation.ServerID == "" || allocation.Region != request.Region || allocation.Build != request.Build || allocation.Protocol != request.Protocol || allocation.Transport != request.Transport || allocation.ArenaPath != request.ArenaPath { - return fmt.Errorf("recovered allocation does not match request") + return fmt.Errorf("provider allocation does not match request") } return nil } diff --git a/server/allocator/worker_test.go b/server/allocator/worker_test.go index e090a9bc..11cc7802 100644 --- a/server/allocator/worker_test.go +++ b/server/allocator/worker_test.go @@ -93,6 +93,21 @@ func TestWorkerRecoversProviderAllocationBeforeIssuingSecondAllocation(t *testin } } +func TestWorkerBindsCanonicalRecordedRecovery(t *testing.T) { + now := time.Unix(1_000, 0) + request := domain.AllocationRequest{AllocationID: "allocation-1", MatchID: "match-1", Region: "EU", Build: "build-1", Protocol: 1, Transport: "enet"} + providerAllocation := domain.Allocation{AllocationID: request.AllocationID, MatchID: request.MatchID, ServerID: "server-recovered", Region: request.Region, Build: request.Build, Protocol: request.Protocol, Transport: request.Transport, State: domain.ServerAllocated} + canonical := providerAllocation + canonical.AllocatedAt = now + claims := &matchClaimSpy{request: request, found: true} + provider := &recoverableProviderSpy{recovered: agones.AllocatedServer{Allocation: providerAllocation, Endpoint: "127.0.0.1:31001"}, found: true} + worker := Worker{Claims: claims, Service: Service{Provider: provider, Durable: &durableSpy{result: canonical}, Now: func() time.Time { return now }}, Now: func() time.Time { return now }} + processed, err := worker.RunOnce(context.Background()) + if err != nil || !processed || claims.bound != canonical { + t.Fatalf("processed=%t err=%v bound=%+v, want %+v", processed, err, claims.bound, canonical) + } +} + func TestWorkerRejectsRecoveredAllocationForDifferentCompatibility(t *testing.T) { request := domain.AllocationRequest{AllocationID: "allocation-1", MatchID: "match-1", Region: "EU", Build: "build-1", Protocol: 1, Transport: "enet"} claims := &matchClaimSpy{request: request, found: true}