Closes the remaining gap the previous two commits left open: WorkloadVerify
itself worked, but nothing minted a real token at allocation time or handed
it to a running pod, so it had no real caller yet.
agones.Client gains WorkloadSecret/WorkloadTokenTTL. When set, Allocate
mints a signed workload token for the allocation (allocation_id is known at
request-construction time, before Agones has picked a server -- see the
previous commit for why that's the only identifier the token can bind) and
requests it as a third cosmic-clash.io/workload-token annotation, alongside
the existing match-id/allocation-id ones. Left unset (the default), Allocate
requests no such annotation, so a deployment not yet using this path is
unaffected. cmd/allocator wires it from a new --workload-secret /
COSMIC_CLASH_WORKLOAD_SECRET flag (must match cmd/control-plane's own), with
a startup warning if left unset.
supervisor.Supervisor.workloadToken() resolves the bearer credential for
control-plane registration: an explicitly configured --workload-token-path
always wins (kept for a future Kubernetes-projected-JWT WorkloadVerify path,
not yet wired server-side), otherwise it falls back to the
cosmic-clash.io/workload-token annotation on the allocated GameServer --
the same annotation-fallback pattern matchID already used for
cosmic-clash.io/match-id. WorkloadTokenPath is accordingly no longer
required at construction time when ControlPlaneURL is set.
Verified: new agones test proves the annotation is requested (and parses/
verifies against the same secret, naming the right allocation) when
WorkloadSecret is configured, and that it's absent when it isn't; new
supervisor tests prove the annotation-sourced token is what's actually sent
as the Authorization bearer, and that Start fails closed with neither a
configured path nor an annotation present. Full
`go build ./... && go vet ./... && gofmt -l . && go test ./... -race` and
`go test -tags integration ./... -race` both clean.
The just-landed signed workload token embedded (allocation_id, match_id,
server_id) as claims. That doesn't actually work for its intended delivery
channel: the token is meant to be requested as a GameServerAllocation
annotation in the SAME request that asks Agones to pick a server, so at mint
time the allocator knows allocation_id (it generates it) but not yet which
server_id Agones will return -- server_id only exists in Agones's response,
after the annotation request has already been sent. Embedding it was simply
not possible for the real caller this was built for; only the (allocator ->
signed_token) unit tests and hand-constructed integration tests happened to
supply it directly, masking the gap.
Fixes it by having the token bind only allocation_id (the one identifier
actually known at mint time) plus expiry. match_id/server_id are resolved at
verify time from the durable allocations table via the new
store.AllocationBindingByAllocationID, keyed by allocation_id -- which the
allocator already records immediately after Agones responds. This is
strictly stronger, not just a workaround: a caller can no longer claim any
match/server pairing at all, even one that happens to be internally
consistent -- the binding returned is entirely durable-record-derived.
Verified: server/workload's unit tests updated for the new two-field claim
shape; server/api's Postgres integration suite gains
TestWorkloadVerifierFromSignedTokenNeverTrustsCallerSuppliedBinding (two
distinct real allocations each resolve to their own, and only their own,
match/server pairing) replacing the now-inapplicable mismatched-triple test.
Full `go build ./... && go vet ./... && gofmt -l . && go test ./... -race`
and `go test -tags integration ./... -race` both clean; the api integration
suite re-run 3x clean against a live postgres:17-alpine container.
WorkloadVerify (api.Service.WorkloadVerify) was permanently unwired: both
/v1/servers/{id}/register and /v1/servers/{id}/result always 503, because
the only design considered so far was verifying a Kubernetes-projected
service-account JWT (server/workload/jwt.go), which needs a live cluster's
TokenReview/JWKS endpoint to validate against safely -- something this
sandbox cannot do without guessing at a trust boundary.
The API layer doesn't actually require that specific mechanism. serverMutation
only compares WorkloadBinding.ServerID and .MatchID (server/api/service.go);
AdvanceServerRegistration only uses .MatchID/.ServerID/.AllocationID. Nothing
downstream needs Namespace/ServiceAcct/PodUID/GameServerUID populated.
This adds a self-contained alternative: a short-lived, HMAC-signed token the
control plane mints and verifies with a secret only it holds (server/workload/
signed_token.go), the same trust model domain.SessionStore already uses for
player sessions elsewhere in this codebase. It needs no cluster to verify --
signature + expiry is fully self-contained and unit-testable.
The design's soundness rests on the delivery channel, not the crypto: the
token is meant to reach the allocated GameServer via the same Agones
GameServerAllocation annotation channel allocation.go already uses for
match-id/allocation-id, readable only by that pod's own local SDK sidecar. A
caller presenting this token has already proven, via that channel, that it is
the pod Agones allocated. (Wiring the actual annotation delivery -- extending
agones.Client.Allocate and the supervisor's token source -- is a separate,
follow-up change; this commit lands the verification core it depends on.)
store.AllocationBindingStillValid adds defense-in-depth on top of signature
and expiry: it cross-checks the token's claims against the durable
allocations table (append-only, never leaves 'ALLOCATED'), so a validly-signed
token naming an allocation that was never recorded -- or a real allocation id
paired with a mismatched match/server -- is still rejected.
api.WorkloadVerifierFromSignedToken wires the two together and is now plugged
into cmd/control-plane (new --workload-secret / COSMIC_CLASH_WORKLOAD_SECRET
flag; a startup warning is logged if it's left unset, since the route then
stays 503 exactly as before) and cmd/testkit-api (fixed test secret, since
that binary is test-only already).
Verified: new unit tests in server/workload (signature tamper, wrong secret,
expiry boundary, malformed input) and a new Postgres integration suite in
server/api (real allocation row, real signed token, acceptance / unknown-
allocation rejection / mismatched-triple rejection / the previously-503
Service.WorkloadVerify field itself) -- both run clean with -race across
multiple passes against a live postgres:17-alpine container. Full
`go build ./... && go vet ./... && gofmt -l . && go test ./... -race` and
`go test -tags integration ./... -race` both clean.
The last two commits fixed the severe stranding bug in decline and
timeout, but left a real responsiveness gap: cancelling a ticket
directly while it's part of an OPEN proposal used to leave the OTHER
participant waiting out the full response window for something the
system already knew couldn't happen -- their proposal partner just
abandoned the queue. ProposalExpireRequeueSQL eventually rescues them,
but only after the full window elapses, not immediately.
CascadeCancelToOpenProposal runs inside the same transaction as the
cancel itself: if the cancelled ticket belonged to a currently-OPEN
proposal, decline that proposal right now and requeue every other
participant immediately via the same ProposalDeclineRequeueSQL the
decline path already uses. The cancelling player's own ticket
correctly stays CANCELLED, not swept back into the requeue meant for
everyone else (ProposalDeclineRequeueSQL only touches tickets still at
PROPOSED).
Covered by a real PostgreSQL integration test: cancelling one
participant's ticket mid-proposal immediately declines the proposal
and requeues the other participant with a refreshed expiry, while the
cancelling player's own ticket stays CANCELLED. First draft used a
stale expected revision (0) for the cancel call -- CreateProposal's
own QueueTicketProposeSQL already bumps a ticket's revision to 1 when
forming the proposal, caught immediately by actually running the test
against real Postgres rather than assuming. Clean across 5 runs after
the fix, plus the full integration and unit suites.
The timeout sibling of the previous commit's decline fix: a proposal
that simply times out (the 10s window elapses with no unanimous
response) hits ProposalExpireSQL/ProposalParticipantExpireSQL, and
neither of those -- same as the decline path -- ever touched
queue_tickets. Same severe consequence: every participant still
holding a PROPOSED ticket, response pending or already accepted, is
left stranded (invisible to the matcher, blocking a fresh
queue_create, renewable forever by heartbeat) with no automatic way
back into matchmaking. This path is reached from both GetProposal
(the recovery/read boundary -- a client that missed the expiry event
entirely) and RespondToProposal (a response arriving after the
window), so both needed the fix.
ProposalExpireRequeueSQL mirrors ProposalDeclineRequeueSQL, guarded on
state = 'EXPIRED' so it's safe to call unconditionally right after
ProposalExpireSQL: a no-op on a proposal that's still OPEN, and a
no-op on a proposal that was already EXPIRED on a prior pass (nothing
left at PROPOSED to requeue a second time).
Covered by a real PostgreSQL integration test via GetProposal (nobody
ever responds; recovering the proposal well after its window expires
it and must requeue both participants), confirming both tickets land
back at QUEUED with a refreshed expiry and are visible again to
ListQueuedCandidates. Clean across 5 runs, plus the full integration
and unit suites.
Found by reading the code, not a failing test: no path anywhere
transitioned a queue ticket from PROPOSED back to QUEUED after a
proposal was declined. A stranded PROPOSED ticket is invisible to the
matcher (ListQueuedCandidates only ever reads state='QUEUED'), still
counts as that player's one active ticket (blocking a fresh
queue_create), and is renewable forever by an ordinary heartbeat --
a player proposed a match with someone who then declines had no way
back into matchmaking without realising, on their own, that they
needed to manually cancel first. This affects every participant, not
just the decliner: an uninvolved player who never even responded was
left stuck by someone else's decision.
ProposalDeclineRequeueSQL requeues every participant's ticket,
including the decliner's own -- nothing yet enforces the decline
cooldown task 8.17 documents as a separate, not-yet-built feature, so
leaving anyone behind at PROPOSED today isn't "cooldown behaviour",
it's just broken. Once that cooldown exists it can exempt the
decliner from this immediate requeue; today nothing does.
Covered by a real PostgreSQL integration test: after one player
declines, both the decliner's and an uninvolved participant's tickets
land back at QUEUED with a refreshed expiry, and -- the actual
end-to-end regression -- both are visible again to
ListQueuedCandidates, the same query the matcher itself uses. Clean
across 5 runs, plus the full integration and unit suites.
Found building the two-player proposal integration test (next commit):
Worker.Run treated ANY RunOnce error as fatal to the whole loop,
including domain.FormFromQueue's "no compatible candidates" -- which
is not a failure, it's the completely routine and expected outcome of
a queue whose currently-waiting players don't share a verified region
yet. Two real players with no common region formed exactly this
shape, and the entire matcher process exited -- taking matchmaking
down for every OTHER player in the same playlist, not just the
incompatible pair, since cmd/matcher runs one process per playlist.
Worse: on a real supervisor restart, the same still-incompatible
candidates are still queued, so it would crash again immediately --
an actual crash loop, not a one-off.
RunOnce's own per-call contract (return an error for source failure,
bad formation, mixed playlist, an incomplete batch, a lost durable
claim) is deliberately tested and unchanged. The fix is entirely in
Run's loop: only the three genuinely static misconfiguration errors
(nil dependencies, unsupported playlist, invalid size -- true on every
future pass just as much as this one, so retrying can never help) now
stop it, via new exported sentinels (ErrWorkerNotConfigured,
ErrUnsupportedPlaylist, ErrInvalidMatcherSize) and errors.Is. Every
other RunOnce error is a single pass's worth of "no match formed this
time" and Run keeps polling.
Covered by two new tests: Run recovering from a first-pass error and
still forming a proposal once the pool becomes viable (a real
concurrent goroutine driving Run, not just calling RunOnce directly --
Run's own loop had no test coverage at all before this), and Run still
stopping immediately on a genuine configuration error. Both clean
across 3 runs with -race.
Same discovery pattern as ResultSubmitter/SessionIssuer, one level
deeper: api.Service.rankedProfile and .profile both only ever read
from an in-memory RankedProfiles map with no durable-store equivalent
at all -- not "adapter exists but unwired" this time, there was no
adapter. Every real request to GET /v1/profile/ranked or /api/v1/profile
always 404'd regardless of a player's actual rating.
Add RankedProfileProvider (an interface, not a struct-literal adapter
this time) and store.PostgresRankedProfiles reading the ratings table;
Service.rankedProfileFor prefers it when set and falls back to the
map otherwise, so every existing test/direct Service literal keeps
compiling and passing unchanged. A missing ratings row maps to the
exact same (zero value, false, nil) the map lookup already produced,
preserving existing not-found semantics rather than reinterpreting
them. LastSeasonID/SeasonHistory are deliberately left unset -- the
ratings table has no season pointer, and reconstructing history needs
its own query and display semantics, not bundled in here speculatively.
Wired into both cmd/control-plane and cmd/testkit-api. Verified
against real PostgreSQL via curl: a fresh identity's ranked profile
correctly 404s through the real adapter (same behavior as before,
now for a real reason instead of an empty map).
Every existing test of the client/control-plane boundary is either a
Go unit test with a mocked HTTP layer or a GDScript unit test with no
network at all (multiplayer-next.md 8.40's own evidence names "live
multi-process control-plane/game verification" as remaining). Nothing
before this actually ran the real compiled Go binary, a real
PostgreSQL instance, and a real headless Godot process talking real
HTTP to each other -- and it immediately found a real bug (previous
commit).
server/cmd/testkit-api is a new, deliberately separate, clearly-marked
test-only binary wired identically to cmd/control-plane except for
SteamLogin: cmd/control-plane has no way to authenticate against a
real Steam Web API from this sandbox (task 8.7's own documented
blocker), so testkit-api accepts any non-empty ticket string and
derives a deterministic identity instead. This bypass is confined to
its own binary -- never a flag on cmd/control-plane, never referenced
by any Dockerfile stage or Kubernetes manifest -- specifically so it
can't become a footgun on the real one.
Game/tests/control_plane_smoke.gd drives the real ControlPlaneClient
autoload through login -> queue_create -> heartbeat against a real
server and prints SMOKE PASS/FAIL, matching the existing net_smoke.gd
convention. scripts/verify_control_plane_integration.sh orchestrates
both sides (real postgres:17-alpine, the built testkit-api binary, the
Godot client) end to end.
Two real bugs surfaced building this, both fixed and re-verified, not
just the target bug: the smoke script's own use of `go run` left a
zombie process that survived cleanup and squatting on its port
corrupted the NEXT run with a misleading "http=401 unauthorized" (now
builds and runs a real binary directly, plus a belt-and-suspenders
port-kill in cleanup); and calling heartbeat() synchronously from
within a request_succeeded handler produced a spurious "Busy" because
ControlPlaneClient's own internal resync (see previous commit) was
still in flight -- the test now waits for ControlPlaneClient to go
idle via a real Timer (call_deferred alone floods the message queue
without ever yielding a frame for the in-flight request to complete).
Verified stable across 3 consecutive full runs: real PostgreSQL
container up, migrations applied, testkit-api built and started, real
headless Godot client round-tripping login/queue/heartbeat, clean
teardown with no leftover processes, containers, or bound ports each
time.
Same pattern and same discovery method as the ResultSubmitter fix:
store.PostgresSessions already implements SessionIssuer.Issue (used by
steamSession() to mint sessions) as well as SessionBackend.Authenticate
(used to verify them), and was already wired for the latter -- but not
the former, so /v1/session/steam always 503'd with auth_unavailable
even before considering whether SteamLogin (the real, still-correctly-
unwired Steam blocker) was available. Wire it: same struct value,
second field.
Not independently visible via a black-box HTTP test yet -- SteamLogin
still nil means the handler's first guard clause still 503s before
ever reaching SessionIssuer, so the observable symptom is unchanged
until Steam access exists. Verified by reading the handler's actual
branch order, not by a test that would currently pass for the wrong
reason.
Investigating the fleet.yaml wiring task found something more
fundamental than a manifest problem: cmd/control-plane/main.go never
wires WorkloadVerify, and store.PostgresResults (a ready-made,
already-correct ResultSubmitter adapter matching the interface
exactly) was referenced from nowhere outside its own file -- not even
a test. Both server-authenticated routes this session built
(/v1/servers/{id}/register and the pre-existing /result) are
completely unreachable in the actual running control-plane binary
today: Service.serverMutation treats a nil WorkloadVerify as fatal
for both routes regardless of ServerRegistrar/ResultSubmitter being
present, so every real request 503s.
Wire the safe, obviously-correct half: ResultSubmitter now uses
store.PostgresResults{DB: db}, same pattern as ServerRegistrar.
Deliberately NOT attempting a WorkloadVerify implementation here.
server/workload/jwt.go's ParseAndValidate needs a pre-known "expected"
WorkloadBinding to construct its policy against (itself needing a
durable per-allocation lookup that doesn't exist yet) plus a real
cryptographic SignatureVerifier -- which for a Kubernetes projected
service account token means either fetching/caching the cluster's own
JWKS or delegating to the API server's TokenReview endpoint, a
different verification model that doesn't fit ParseAndValidate's
signature-callback shape at all and would need its own domain-level
adapter. This is authentication-critical code with no existing
wiring example anywhere in the codebase to follow, and the actual
trust boundary (a live cluster's key material) can't be validated
from this sandbox regardless of how carefully the client code is
written. Building it fast under this session's already-heavy pace
risked a subtle, dangerous mistake far more costly than leaving the
gap named precisely, which is what this commit does instead.
Added TestServerRoutesRequireWorkloadVerifyToBeWired: pins the
current 503-on-every-request behavior as an explicit, visible
regression trip-wire rather than a silent gap -- it's designed to
start failing (and be updated, not deleted) the day WorkloadVerify is
actually wired.
Closes the last named item on task 8.28 (Health-reclaim): nothing
currently detects or cleans up a match stuck in
ALLOCATING/PROCESS_READY/ASSIGNMENT_READY forever because its server
crashed or was reclaimed by Agones as unhealthy before ever
registering -- players would wait indefinitely for a match that was
never coming.
The design question this was blocked on -- does an abandoned match
auto-requeue its players, or fail and make them re-queue -- isn't
actually open: task 8.50's own stated acceptance criterion already
answers it ("infrastructure-caused cases cannot penalise affected
players"). A server-side crash/reclaim is exactly that, not player
behaviour, so store.ExpireStalledAllocations fails the match but
requeues every participant's ticket to QUEUED with a fresh expiry
(matching the ordinary 30s queue window), releases their
match_participants row (participation_active = false, so they're
matchable again immediately), all inside one FOR UPDATE SKIP LOCKED
pass so a second maintenance replica continues past whatever a
concurrent one is already reclaiming.
Wired into cmd/maintenance alongside the existing season-rollover
sweep: --stalled-allocation-deadline (default 2m) and
--stalled-allocation-batch (default 100).
Covered by a SQL-fragment test and a real PostgreSQL integration test:
two matches (one genuinely stalled, one recent), confirming the
deadline boundary is respected (recent match untouched), both
stranded participants' tickets requeue with a refreshed expiry, the
match_participants row releases, and a second pass doesn't reprocess
an already-FAILED match. Verified clean across 5 runs, plus the full
integration and unit suites.
Josh Creek