Every allocated GameServer reached Ready and was recycled by Agones ~20s
later. Health pings are the game process's job by design -- the
supervisor has no health implementation at all -- so a server that stops
pinging is exactly what Agones is built to reclaim.
start_health() armed a Timer on a node that might not be inside the
SceneTree. A Timer only ticks inside the tree, so the node reported
itself configured, sent nothing, and said nothing about it. It now
returns a bool, refuses loudly when unconfigured, and defers to _ready()
when called before parenting, so the SDK arms its own timer and no
caller has to get the ordering right. server_boot.gd defers the add like
every sibling does (§9 gotcha 27) and logs when AGONES_SDK_HTTP_PORT is
missing, which previously read identically to a healthy start.
Also bounded the in-flight latch: it is set across an await, so a request
that never completes would silence health permanently. Defence in depth
rather than an observed fault.
Tests target the contract rather than the mechanism: a test that parents
the SDK correctly and asserts pings passes with the bug present, because
the defect was in the wiring. The unit tests assert start_health()
cannot claim success out of tree, and were confirmed to fail against the
previous code. The smoke gains a counting sidecar and asserts a
*repeating* ping -- it reports "health pings in 3.0s = 1, want at least
2" when the loop is broken, which is the production symptom exactly. It
is also now actually run: nothing referenced it before.
Two diagnostic fixes, both of which changed conclusions during this work:
The kind gate only built the game-server image when the tag was absent,
so a local rerun silently verified whatever was built last. That is why
local runs and CI disagreed about the same commit. It now builds by
default, with KIND_REUSE_GAME_SERVER_IMAGE=1 as the opt-in fast path.
The failure dump logged only not-ready pods, and used --all-containers
with a shared tail. A GameServer recycled after reaching Ready leaves no
unready pod behind, and the Agones sidecar out-logs the game server, so
the relevant output was never captured. It now dumps every pod, per
container, current and previous, plus the GameServer and Fleet resources
-- Agones' own state machine is what rejects these.
newAPIService never supplied SteamLogin, so POST /v1/session/steam
always returned 503 auth_unavailable in production. The only
implementation was cmd/testkit-api's fake, which derives an identity
from the ticket string itself and accepts anything -- so the passing
integration path was neither deployable nor secure. On the client side
the game started with an empty token and a loopback base URL, and no
production code called configure() or login_steam(); the menu entered
matchmaking directly, so every request failed ERR_UNAUTHORIZED before
reaching the network.
Add a real ISteamUserAuth/AuthenticateUserTicket adapter behind an
interface, so the production login path is testable with only the Valve
call stubbed. It rejects family-shared copies (the account playing does
not own the app) and, by default, VAC- or publisher-banned accounts, and
refuses malformed tickets locally rather than forwarding them.
Crucially it separates our faults from the player's: a Valve outage or a
revoked publisher key returns 503, not 401. Answering 401 would tell a
legitimate player their login failed and send them to fix an account
that is fine while the real fault went unnoticed. A banned identity now
returns 403 rather than a misleading 503.
Sign-in is configuration-gated on the publisher key and App ID: without
them the endpoint keeps returning 503, since silently accepting an
unverified ticket would be worse than refusing to authenticate. A
returning player keeps the player ID they already had, so ratings,
penalties and bans follow the account rather than the session.
Client side: acquire a web-API ticket through GodotSteam's async
signal -- requesting one returns a handle, not a ticket -- using the
existing dynamic-call pattern so stock Godot still parses the project.
The endpoint is configurable for release builds, and matchmaking
completes sign-in before it will queue.
Verified against real PostgreSQL; 232 Godot tests pass.
domain.validCandidate hard-requires a non-empty PredictedRTT map, but
CreateQueueTicket persisted an empty one and the only endpoint that
could fill it returned 503 in every real binary, because Service.Probe
was assigned nowhere outside api tests. No client-created ticket could
ever be selected by the matcher. The Godot client had no probe method at
all, so even a wired backend was unreachable from the game.
Four distinct defects had to be fixed for this path to work:
Nothing issued the nonce ProbeProvider was meant to compare against, so
the contract could not be satisfied even in principle. Add
POST /v1/probes/{region}/challenge, backed by a durable single-use
challenge -- durable because any replica may serve the answer for a
challenge another replica issued. RTT is the interval between issuing
and receiving, so no client-reported latency reaches placement.
CreateQueueTicket marshalled a nil map to JSON `null`, a JSONB scalar
rather than an object, and jsonb_set rejects that with "cannot set path
in scalar". RecordProbe would have failed at runtime even once wired.
Persist an object, and normalise non-object values in the update for
rows already written.
A nil ProbeRecorder made the handler report success while persisting
nothing, which silently leaves the ticket unmatchable. That is a
misconfiguration, not a successful probe; it now returns 503.
A successful probe updated PostgreSQL only. The candidate inserted at
enqueue time carries an empty RTT map, and the Redis keyspace has its
TTL continually refreshed, so the stale entry need never repair itself.
Refresh that player's projection after the probe commits.
Client side: add the challenge/answer round trip and have the
matchmaking screen collect evidence before creating a ticket, since
queueing first produces a search that can never match. Probing every
region fully is not required -- placement uses whichever regions
answered -- but queueing with none is refused rather than silently
stalling.
New integration test drives the real enqueue and probe paths and then
asks the actual matcher predicate, rather than hand-building a candidate
the way the unit tests do -- which is exactly why they missed this.
Also make the integration schema reset drop the whole public schema: the
enumerated table list silently broke with each new migration.
Prerequisite for wiring the allocator to publish rosters. The signing
key is a shared HMAC secret mounted into both the allocator and the
allocated game server; without a key ID, rotating it would invalidate
every authorisation already issued for an in-flight match, because a
server holding only the new key cannot verify a token signed with the
old one.
Add KeyID to JoinAuthorisation and append it to the canonical claim
bytes, so it is covered by the signature and cannot be repointed at a
different key than the one that actually signed. Allocated servers now
hold a set of currently-valid keys and select by ID: a rotation
publishes the new key alongside the old, and the old is dropped once no
live match can still reference it.
The key file becomes a JSON map of key ID to base64 key. A file of raw
key bytes is still accepted as a single key under the empty ID, which is
what an unrotated deployment and the kind fixture use.
Game/scripts/match_net.gd builds the canonical bytes independently, so
it changes in lockstep; the cross-language golden token in
test_match_net.gd is regenerated from the Go implementation and now
carries a key ID. Added tests cover accepting either key mid-rotation,
rejecting a retired key ID, and rejecting a token whose key ID was
swapped to name a key the server does hold.
Go suite and 223 Godot tests pass.
multiplayer-next.md was a 1662-line mix of standing architecture spec
and task-completion tracking, most of which was dense per-task DONE
evidence for finished Phases 0-6. Split it:
- MULTIPLAYER_SPEC.md (new): the locked architecture decisions, wire
format, server-side input handling, prediction/reconciliation,
latency/frame-rate budget, and match lifecycle state machine -
standing design reference, not task-tracked.
- multiplayer-next.md (trimmed 1662 -> ~370 lines): only outstanding
work remains - §0 status, §7 Phase 7/8 task tables condensed to
"what's left" per task, §8-11 reference material (refactoring notes,
gotchas, testing, flagged items). Phases 0-6 collapsed to a pointer
at git history instead of ~500 lines of DONE evidence.
Also:
- Repointed every `multiplayer-next.md §N` code comment (N 1-6) across
Game/scripts, Game/tools and Game/tests to MULTIPLAYER_SPEC.md, since
those sections moved. Task-number references (`task N.N`, §7-11)
correctly still point at multiplayer-next.md.
- Updated CLAUDE.md's doc index and docs/TECH_STACK.md's spec-section
citations to match.
- TODO.md: added a "what's left to actually finish multiplayer
(human-actionable)" checklist pulled from multiplayer-next.md §0 and
docs/MATCHMAKING.md - things that need a person (hardware, a design
decision, a Steam App ID, hands on a controller), not more agent code.
Closes §8.43's 'failed-reconnect UX' gap, found immediately after
wiring connect_to_assignment() itself: even with that fix in place, a
connection failure had nowhere to go. connect_to_assignment()'s
synchronous failures (assignment missing/expired, invalid endpoint,
NetworkManager.join() erroring immediately) only ever emitted
assignment_connection_failed -- a signal nothing in the client
listened to. state.phase would stay stuck at ASSIGNED, the UI would
keep showing "Your match server is ready" forever, with no way back
to a fresh search.
Worse, the likelier real-world failure mode had no handler at all:
NetworkManager.join() returns OK immediately once the attempt starts,
but the actual ENet handshake can still fail asynchronously afterward
(unreachable server, refused connection, ENet's own ~5s connect
timeout). This is exactly the gap main_menu.gd's own
_on_connection_failed exists to cover for the direct-join flow (see
its header comment) -- nothing covered the equivalent for a
matchmaking-driven connect.
ControlPlaneClient now connects both assignment_connection_failed and
NetworkManager.connection_failed (guarded to state.phase == CONNECTING,
so it never misattributes an unrelated direct-join failure to a
matchmaking search) to state.fail(...), so either failure mode now
surfaces as a failed search the player can actually retry from.
Verified: three new test_control_plane_client.gd tests cover the
synchronous failure path, the async NetworkManager.connection_failed
path (via a real end-to-end ASSIGNED -> CONNECTING flow), and that an
unrelated connection_failed outside CONNECTING is correctly ignored.
220/220 tests pass, stable across 3 repeated runs, no crash, no
engine-level error; full make verify-multiplayer-local and the
complete make verify-enet-integration suite (all five cases) both
clean; zero new crash reports throughout.
Also fixes a markdown table-integrity mistake introduced while
documenting this in the same edit pass: an earlier Edit call
accidentally duplicated a sentence and dropped the row's closing
'remains' clause in §8.43 -- caught and corrected before commit via
the usual pipe-count check.
Closes §8.43's 'version-mismatch-specific client messaging' gap. Per
the user's explicit go-ahead to design new server behavior for this
(rather than only wiring up something that already existed, the
pattern every other fix this session followed): before this, there
was no server-side protocol rejection at all. queue_create accepted
any protocol_version >= 1 unconditionally, so an outdated client could
only ever discover a mismatch by waiting in the queue forever
unmatched -- the matcher's own compatibility check requires every
formed player to share an identical protocol_version -- with no error
and no explanation given to the player.
Server: Service.MinProtocolVersion (opt-in, zero by default so every
existing caller keeps accepting protocol_version 1 unconditionally)
rejects a below-floor queue_create with 426 Upgrade Required /
client_outdated before the request ever reaches the candidate
provider. Wired via cmd/control-plane's new --min-protocol-version
flag (validated non-negative at startup).
Client: ControlPlaneClient recognises HTTPClient.RESPONSE_UPGRADE_REQUIRED
on queue_create specifically and sets a distinct 'Your client is out
of date -- please update to continue searching' message instead of
the server's raw generic error string, and clears _last_queue_create
so can_retry_queue_create() never offers 'Retry Search' for a failure
that retrying with the same build can never fix.
Verified: go build/vet/test -race clean across every server package.
TestQueueCreateEnforcesMinProtocolVersion covers below-floor rejection
(candidate provider never reached), exactly-at-floor acceptance, and
the error body naming client_outdated; TestQueueCreateMinProtocolVersionZeroIsDisabled
proves the opt-in default doesn't change behavior for every existing
caller. Godot: test_outdated_client_receives_a_distinct_message_and_no_retry_offer
proves the distinct message and suppressed retry offer. Full Godot
suite (217/217, 0 failed, no crash), full make verify-multiplayer-local
gate, zero new crash reports.
A player who completed the entire queue -> proposal -> allocate ->
assign pipeline would reach ASSIGNED, see "Your match server is
ready", and then simply sit there forever. connect_to_assignment()
existed in control_plane_client.gd, fully validated (checks the
assignment is available and fresh, splits and validates the
endpoint, carries the join authorisation via MatchNet's hello payload
rather than the URL) with its own assignment_connection_started/
assignment_connection_failed signals, and multiplayer-next.md's own
§8.41 row already described it as wired -- but grepping the whole
client found zero callers. Nothing anywhere in matchmaking.gd or
control_plane_client.gd itself ever invoked it.
ControlPlaneClient._connect_when_assigned() now calls it automatically
the moment state.phase reaches ASSIGNED. Wired into the single call
site every queue-shaped HTTP response already shares (heartbeat,
recover, and resync-triggered recover alike, since the WebSocket
match-lifecycle path always funnels into a REST resync first), so
both the ordinary poll path and the WebSocket-push path are covered
without a second call site to keep in sync. Two orderings are handled:
if the assignment fetch triggered earlier by ASSIGNMENT_READY has
already completed, it connects immediately; if not, it defers via
_pending_connect_match_id and resolves once the assignment becomes
available. _connect_attempted_match_id guards against a duplicate or
replayed ASSIGNED event reattempting the connection.
Verified against the real Godot 4.7.1 binary now that headless
testing has resumed: two new test_control_plane_client.gd tests cover
both orderings and the duplicate-attempt guard directly (216/216
total, 0 failed, no crash, no engine-level error, stable across
repeated runs); full make verify-multiplayer-local and the complete
make verify-enet-integration suite (all five cases, including the
3-process match) both pass clean; zero new crash reports throughout.
multiplayer-next.md's §8.41 row is corrected to describe what was
actually true (connect_to_assignment existed but was never called)
rather than repeating the prior, inaccurate 'already wired' claim.
Closes most of §8.43's 'decline, regional outage retry UI, failed
reconnect, duplicate-action recovery beyond proposals' remaining
list -- turned out to be mostly stale doc, not missing code.
matchmaking.gd's decline button/handler already existed
(%DeclineButton, _on_decline_pressed, visibility toggled by
MatchmakingState.PROPOSED alongside accept). ControlPlaneClient's
can_retry_last_mutation()/retry_last_mutation() -- the generic
'duplicate-action recovery beyond proposals' and 'regional outage
retry' mechanism -- also already existed: any mutation (not just a
proposal response) becomes retryable on a transport failure or a
408/429/503 response, and matchmaking.gd's queue button already fell
back to it ('Retry Request'). Neither had any test coverage proving
the mechanism actually works for a non-proposal mutation --
is_retryable_mutation_response's pure classification was the only
thing tested.
Two new tests: test_generic_mutation_retry_recovers_after_a_transient_failure
proves can_retry_last_mutation() transitions from false (mutation
in flight) to true after a transport-level failure on an ordinary
queue_heartbeat, exactly the 'regional outage' case; test_generic_mutation_retry_is_not_offered_for_unsafe_failures
proves a 409 (revision conflict) is never offered as a blind retry
and that retry_last_mutation() fails closed with ERR_INVALID_DATA
rather than resending a stale mutation. retry_last_mutation's literal
network dispatch (HTTPRequest.request()) is not exercised -- it needs
a live SceneTree that test_runner.tscn's synchronous single-_ready()
execution model cannot provide mid-suite; the two tests cover the
can_retry_last_mutation() decision boundary and the fail-closed path
instead, which is what's actually new here.
Verified against the real Godot 4.7.1 binary now that headless
testing has resumed: test_runner.tscn 214/214 clean (no crash, no
engine-level error), full make verify-multiplayer-local re-run clean,
zero new crash reports.
Remaining in §8.43: version-mismatch-specific messaging (a protocol
rejection currently surfaces only as the server's generic error
string), failed-reconnect UX, and §8.16's arena selection/long-running
worker integration.
Josh Creek