Several independent causes, all of which had to be right before the
Fleet could reach Ready.
The supervisor pointed --sdk-base-url at 127.0.0.1:9357, which is the
Agones sidecar's gRPC port; its HTTP surface is 9358, and that is what
AGONES_SDK_HTTP_PORT carries and what agones_sdk.gd reads. An HTTP
client against the gRPC port could never have worked, in kind or in
production.
The supervisor also treated the sidecar's first incomplete /gameserver
response as fatal. The sidecar accepts requests before the controller
populates status.address and status.ports, so this produced a restart
loop precisely during normal Agones startup. It now polls until the
endpoint is assigned or ReadyTimeout elapses.
server_boot.gd started ServerControl and the Agones SDK only under
--allocated-mode, but the kind smoke deliberately strips that flag, so
nothing served the readiness probe and the GameServer could never become
Ready. Lifecycle now keys on AGONES_SDK_HTTP_PORT, which Agones injects
into every managed container, while allocation and roster semantics stay
tied to --allocated-mode. The SDK node is added to the tree
non-deferred, since start_health() creates a Timer immediately.
Fleet: Agones assigns its own SDK service account and masks that token
from the game container while keeping it for the injected sidecar, so
the manifest must not pin serviceAccountName or
automountServiceAccountToken. Godot stores user:// under HOME, so HOME
points at the writable runtime volume to keep the root filesystem
read-only, and fsGroup makes that volume writable for the non-root user.
Namespace: Agones' Dynamic port policy injects a hostPort, which both
the baseline and restricted Pod Security Standards forbid, so the
workload namespace enforces privileged while continuing to audit and
warn against restricted.
NetworkPolicy: the injected sidecar reaches the Kubernetes API over
HTTPS, and NetworkPolicy applies to the whole Pod rather than to the
container whose token was masked.
The kind runner creates the namespace before Helm so Agones can install
its per-namespace SDK RBAC, scopes gameservers.namespaces to it, forces
the allocator and ping Services to ClusterIP because LoadBalancer
ingress never becomes ready in plain kind, and labels the node so the
production Fleet's on-demand/zone constraints are exercised rather than
edited out of the rendered manifest.
newAPIService never supplied SteamLogin, so POST /v1/session/steam
always returned 503 auth_unavailable in production. The only
implementation was cmd/testkit-api's fake, which derives an identity
from the ticket string itself and accepts anything -- so the passing
integration path was neither deployable nor secure. On the client side
the game started with an empty token and a loopback base URL, and no
production code called configure() or login_steam(); the menu entered
matchmaking directly, so every request failed ERR_UNAUTHORIZED before
reaching the network.
Add a real ISteamUserAuth/AuthenticateUserTicket adapter behind an
interface, so the production login path is testable with only the Valve
call stubbed. It rejects family-shared copies (the account playing does
not own the app) and, by default, VAC- or publisher-banned accounts, and
refuses malformed tickets locally rather than forwarding them.
Crucially it separates our faults from the player's: a Valve outage or a
revoked publisher key returns 503, not 401. Answering 401 would tell a
legitimate player their login failed and send them to fix an account
that is fine while the real fault went unnoticed. A banned identity now
returns 403 rather than a misleading 503.
Sign-in is configuration-gated on the publisher key and App ID: without
them the endpoint keeps returning 503, since silently accepting an
unverified ticket would be worse than refusing to authenticate. A
returning player keeps the player ID they already had, so ratings,
penalties and bans follow the account rather than the session.
Client side: acquire a web-API ticket through GodotSteam's async
signal -- requesting one returns a handle, not a ticket -- using the
existing dynamic-call pattern so stock Godot still parses the project.
The endpoint is configurable for release builds, and matchmaking
completes sign-in before it will queue.
Verified against real PostgreSQL; 232 Godot tests pass.
domain.validCandidate hard-requires a non-empty PredictedRTT map, but
CreateQueueTicket persisted an empty one and the only endpoint that
could fill it returned 503 in every real binary, because Service.Probe
was assigned nowhere outside api tests. No client-created ticket could
ever be selected by the matcher. The Godot client had no probe method at
all, so even a wired backend was unreachable from the game.
Four distinct defects had to be fixed for this path to work:
Nothing issued the nonce ProbeProvider was meant to compare against, so
the contract could not be satisfied even in principle. Add
POST /v1/probes/{region}/challenge, backed by a durable single-use
challenge -- durable because any replica may serve the answer for a
challenge another replica issued. RTT is the interval between issuing
and receiving, so no client-reported latency reaches placement.
CreateQueueTicket marshalled a nil map to JSON `null`, a JSONB scalar
rather than an object, and jsonb_set rejects that with "cannot set path
in scalar". RecordProbe would have failed at runtime even once wired.
Persist an object, and normalise non-object values in the update for
rows already written.
A nil ProbeRecorder made the handler report success while persisting
nothing, which silently leaves the ticket unmatchable. That is a
misconfiguration, not a successful probe; it now returns 503.
A successful probe updated PostgreSQL only. The candidate inserted at
enqueue time carries an empty RTT map, and the Redis keyspace has its
TTL continually refreshed, so the stale entry need never repair itself.
Refresh that player's projection after the probe commits.
Client side: add the challenge/answer round trip and have the
matchmaking screen collect evidence before creating a ticket, since
queueing first produces a search that can never match. Probing every
region fully is not required -- placement uses whichever regions
answered -- but queueing with none is refused rather than silently
stalling.
New integration test drives the real enqueue and probe paths and then
asks the actual matcher predicate, rather than hand-building a candidate
the way the unit tests do -- which is exactly why they missed this.
Also make the integration schema reset drop the whole public schema: the
enumerated table list silently broke with each new migration.
Prerequisite for wiring the allocator to publish rosters. The signing
key is a shared HMAC secret mounted into both the allocator and the
allocated game server; without a key ID, rotating it would invalidate
every authorisation already issued for an in-flight match, because a
server holding only the new key cannot verify a token signed with the
old one.
Add KeyID to JoinAuthorisation and append it to the canonical claim
bytes, so it is covered by the signature and cannot be repointed at a
different key than the one that actually signed. Allocated servers now
hold a set of currently-valid keys and select by ID: a rotation
publishes the new key alongside the old, and the old is dropped once no
live match can still reference it.
The key file becomes a JSON map of key ID to base64 key. A file of raw
key bytes is still accepted as a single key under the empty ID, which is
what an unrotated deployment and the kind fixture use.
Game/scripts/match_net.gd builds the canonical bytes independently, so
it changes in lockstep; the cross-language golden token in
test_match_net.gd is regenerated from the Go implementation and now
carries a key ID. Added tests cover accepting either key mid-rotation,
rejecting a retired key ID, and rejecting a token whose key ID was
swapped to name a key the server does hold.
Go suite and 223 Godot tests pass.
multiplayer-next.md was a 1662-line mix of standing architecture spec
and task-completion tracking, most of which was dense per-task DONE
evidence for finished Phases 0-6. Split it:
- MULTIPLAYER_SPEC.md (new): the locked architecture decisions, wire
format, server-side input handling, prediction/reconciliation,
latency/frame-rate budget, and match lifecycle state machine -
standing design reference, not task-tracked.
- multiplayer-next.md (trimmed 1662 -> ~370 lines): only outstanding
work remains - §0 status, §7 Phase 7/8 task tables condensed to
"what's left" per task, §8-11 reference material (refactoring notes,
gotchas, testing, flagged items). Phases 0-6 collapsed to a pointer
at git history instead of ~500 lines of DONE evidence.
Also:
- Repointed every `multiplayer-next.md §N` code comment (N 1-6) across
Game/scripts, Game/tools and Game/tests to MULTIPLAYER_SPEC.md, since
those sections moved. Task-number references (`task N.N`, §7-11)
correctly still point at multiplayer-next.md.
- Updated CLAUDE.md's doc index and docs/TECH_STACK.md's spec-section
citations to match.
- TODO.md: added a "what's left to actually finish multiplayer
(human-actionable)" checklist pulled from multiplayer-next.md §0 and
docs/MATCHMAKING.md - things that need a person (hardware, a design
decision, a Steam App ID, hands on a controller), not more agent code.
Closes §8.43's 'failed-reconnect UX' gap, found immediately after
wiring connect_to_assignment() itself: even with that fix in place, a
connection failure had nowhere to go. connect_to_assignment()'s
synchronous failures (assignment missing/expired, invalid endpoint,
NetworkManager.join() erroring immediately) only ever emitted
assignment_connection_failed -- a signal nothing in the client
listened to. state.phase would stay stuck at ASSIGNED, the UI would
keep showing "Your match server is ready" forever, with no way back
to a fresh search.
Worse, the likelier real-world failure mode had no handler at all:
NetworkManager.join() returns OK immediately once the attempt starts,
but the actual ENet handshake can still fail asynchronously afterward
(unreachable server, refused connection, ENet's own ~5s connect
timeout). This is exactly the gap main_menu.gd's own
_on_connection_failed exists to cover for the direct-join flow (see
its header comment) -- nothing covered the equivalent for a
matchmaking-driven connect.
ControlPlaneClient now connects both assignment_connection_failed and
NetworkManager.connection_failed (guarded to state.phase == CONNECTING,
so it never misattributes an unrelated direct-join failure to a
matchmaking search) to state.fail(...), so either failure mode now
surfaces as a failed search the player can actually retry from.
Verified: three new test_control_plane_client.gd tests cover the
synchronous failure path, the async NetworkManager.connection_failed
path (via a real end-to-end ASSIGNED -> CONNECTING flow), and that an
unrelated connection_failed outside CONNECTING is correctly ignored.
220/220 tests pass, stable across 3 repeated runs, no crash, no
engine-level error; full make verify-multiplayer-local and the
complete make verify-enet-integration suite (all five cases) both
clean; zero new crash reports throughout.
Also fixes a markdown table-integrity mistake introduced while
documenting this in the same edit pass: an earlier Edit call
accidentally duplicated a sentence and dropped the row's closing
'remains' clause in §8.43 -- caught and corrected before commit via
the usual pipe-count check.
Closes §8.43's 'version-mismatch-specific client messaging' gap. Per
the user's explicit go-ahead to design new server behavior for this
(rather than only wiring up something that already existed, the
pattern every other fix this session followed): before this, there
was no server-side protocol rejection at all. queue_create accepted
any protocol_version >= 1 unconditionally, so an outdated client could
only ever discover a mismatch by waiting in the queue forever
unmatched -- the matcher's own compatibility check requires every
formed player to share an identical protocol_version -- with no error
and no explanation given to the player.
Server: Service.MinProtocolVersion (opt-in, zero by default so every
existing caller keeps accepting protocol_version 1 unconditionally)
rejects a below-floor queue_create with 426 Upgrade Required /
client_outdated before the request ever reaches the candidate
provider. Wired via cmd/control-plane's new --min-protocol-version
flag (validated non-negative at startup).
Client: ControlPlaneClient recognises HTTPClient.RESPONSE_UPGRADE_REQUIRED
on queue_create specifically and sets a distinct 'Your client is out
of date -- please update to continue searching' message instead of
the server's raw generic error string, and clears _last_queue_create
so can_retry_queue_create() never offers 'Retry Search' for a failure
that retrying with the same build can never fix.
Verified: go build/vet/test -race clean across every server package.
TestQueueCreateEnforcesMinProtocolVersion covers below-floor rejection
(candidate provider never reached), exactly-at-floor acceptance, and
the error body naming client_outdated; TestQueueCreateMinProtocolVersionZeroIsDisabled
proves the opt-in default doesn't change behavior for every existing
caller. Godot: test_outdated_client_receives_a_distinct_message_and_no_retry_offer
proves the distinct message and suppressed retry offer. Full Godot
suite (217/217, 0 failed, no crash), full make verify-multiplayer-local
gate, zero new crash reports.
A player who completed the entire queue -> proposal -> allocate ->
assign pipeline would reach ASSIGNED, see "Your match server is
ready", and then simply sit there forever. connect_to_assignment()
existed in control_plane_client.gd, fully validated (checks the
assignment is available and fresh, splits and validates the
endpoint, carries the join authorisation via MatchNet's hello payload
rather than the URL) with its own assignment_connection_started/
assignment_connection_failed signals, and multiplayer-next.md's own
§8.41 row already described it as wired -- but grepping the whole
client found zero callers. Nothing anywhere in matchmaking.gd or
control_plane_client.gd itself ever invoked it.
ControlPlaneClient._connect_when_assigned() now calls it automatically
the moment state.phase reaches ASSIGNED. Wired into the single call
site every queue-shaped HTTP response already shares (heartbeat,
recover, and resync-triggered recover alike, since the WebSocket
match-lifecycle path always funnels into a REST resync first), so
both the ordinary poll path and the WebSocket-push path are covered
without a second call site to keep in sync. Two orderings are handled:
if the assignment fetch triggered earlier by ASSIGNMENT_READY has
already completed, it connects immediately; if not, it defers via
_pending_connect_match_id and resolves once the assignment becomes
available. _connect_attempted_match_id guards against a duplicate or
replayed ASSIGNED event reattempting the connection.
Verified against the real Godot 4.7.1 binary now that headless
testing has resumed: two new test_control_plane_client.gd tests cover
both orderings and the duplicate-attempt guard directly (216/216
total, 0 failed, no crash, no engine-level error, stable across
repeated runs); full make verify-multiplayer-local and the complete
make verify-enet-integration suite (all five cases, including the
3-process match) both pass clean; zero new crash reports throughout.
multiplayer-next.md's §8.41 row is corrected to describe what was
actually true (connect_to_assignment existed but was never called)
rather than repeating the prior, inaccurate 'already wired' claim.
Josh Creek