Every allocated GameServer reached Ready and was recycled by Agones ~20s
later. Health pings are the game process's job by design -- the
supervisor has no health implementation at all -- so a server that stops
pinging is exactly what Agones is built to reclaim.
start_health() armed a Timer on a node that might not be inside the
SceneTree. A Timer only ticks inside the tree, so the node reported
itself configured, sent nothing, and said nothing about it. It now
returns a bool, refuses loudly when unconfigured, and defers to _ready()
when called before parenting, so the SDK arms its own timer and no
caller has to get the ordering right. server_boot.gd defers the add like
every sibling does (§9 gotcha 27) and logs when AGONES_SDK_HTTP_PORT is
missing, which previously read identically to a healthy start.
Also bounded the in-flight latch: it is set across an await, so a request
that never completes would silence health permanently. Defence in depth
rather than an observed fault.
Tests target the contract rather than the mechanism: a test that parents
the SDK correctly and asserts pings passes with the bug present, because
the defect was in the wiring. The unit tests assert start_health()
cannot claim success out of tree, and were confirmed to fail against the
previous code. The smoke gains a counting sidecar and asserts a
*repeating* ping -- it reports "health pings in 3.0s = 1, want at least
2" when the loop is broken, which is the production symptom exactly. It
is also now actually run: nothing referenced it before.
Two diagnostic fixes, both of which changed conclusions during this work:
The kind gate only built the game-server image when the tag was absent,
so a local rerun silently verified whatever was built last. That is why
local runs and CI disagreed about the same commit. It now builds by
default, with KIND_REUSE_GAME_SERVER_IMAGE=1 as the opt-in fast path.
The failure dump logged only not-ready pods, and used --all-containers
with a shared tail. A GameServer recycled after reaching Ready leaves no
unready pod behind, and the Agones sidecar out-logs the game server, so
the relevant output was never captured. It now dumps every pod, per
container, current and previous, plus the GameServer and Fleet resources
-- Agones' own state machine is what rejects these.
Several independent causes, all of which had to be right before the
Fleet could reach Ready.
The supervisor pointed --sdk-base-url at 127.0.0.1:9357, which is the
Agones sidecar's gRPC port; its HTTP surface is 9358, and that is what
AGONES_SDK_HTTP_PORT carries and what agones_sdk.gd reads. An HTTP
client against the gRPC port could never have worked, in kind or in
production.
The supervisor also treated the sidecar's first incomplete /gameserver
response as fatal. The sidecar accepts requests before the controller
populates status.address and status.ports, so this produced a restart
loop precisely during normal Agones startup. It now polls until the
endpoint is assigned or ReadyTimeout elapses.
server_boot.gd started ServerControl and the Agones SDK only under
--allocated-mode, but the kind smoke deliberately strips that flag, so
nothing served the readiness probe and the GameServer could never become
Ready. Lifecycle now keys on AGONES_SDK_HTTP_PORT, which Agones injects
into every managed container, while allocation and roster semantics stay
tied to --allocated-mode. The SDK node is added to the tree
non-deferred, since start_health() creates a Timer immediately.
Fleet: Agones assigns its own SDK service account and masks that token
from the game container while keeping it for the injected sidecar, so
the manifest must not pin serviceAccountName or
automountServiceAccountToken. Godot stores user:// under HOME, so HOME
points at the writable runtime volume to keep the root filesystem
read-only, and fsGroup makes that volume writable for the non-root user.
Namespace: Agones' Dynamic port policy injects a hostPort, which both
the baseline and restricted Pod Security Standards forbid, so the
workload namespace enforces privileged while continuing to audit and
warn against restricted.
NetworkPolicy: the injected sidecar reaches the Kubernetes API over
HTTPS, and NetworkPolicy applies to the whole Pod rather than to the
container whose token was masked.
The kind runner creates the namespace before Helm so Agones can install
its per-namespace SDK RBAC, scopes gameservers.namespaces to it, forces
the allocator and ping Services to ClusterIP because LoadBalancer
ingress never becomes ready in plain kind, and labels the node so the
production Fleet's on-demand/zone constraints are exercised rather than
edited out of the rendered manifest.
Prerequisite for wiring the allocator to publish rosters. The signing
key is a shared HMAC secret mounted into both the allocator and the
allocated game server; without a key ID, rotating it would invalidate
every authorisation already issued for an in-flight match, because a
server holding only the new key cannot verify a token signed with the
old one.
Add KeyID to JoinAuthorisation and append it to the canonical claim
bytes, so it is covered by the signature and cannot be repointed at a
different key than the one that actually signed. Allocated servers now
hold a set of currently-valid keys and select by ID: a rotation
publishes the new key alongside the old, and the old is dropped once no
live match can still reference it.
The key file becomes a JSON map of key ID to base64 key. A file of raw
key bytes is still accepted as a single key under the empty ID, which is
what an unrotated deployment and the kind fixture use.
Game/scripts/match_net.gd builds the canonical bytes independently, so
it changes in lockstep; the cross-language golden token in
test_match_net.gd is regenerated from the Go implementation and now
carries a key ID. Added tests cover accepting either key mid-rotation,
rejecting a retired key ID, and rejecting a token whose key ID was
swapped to name a key the server does hold.
Go suite and 223 Godot tests pass.
Godot's ConfigFile writer does not round-trip comments in project.godot. An
observed rewrite deleted both `;` blocks outright and spliced the three-line
`#` block above run/main_scene.dedicated_server onto the setting's own line,
leaving it commented out — which would send dedicated builds to the
interactive main menu instead of server_boot.tscn, with nothing failing until
someone noticed a server process rendering a menu.
Move the explanations into the code that owns the settings (server_boot.gd for
the dedicated-server override, video_settings.gd for stretch mode and vsync)
so they cannot be destroyed by a rewrite, and leave project.godot holding only
assignments plus Godot's own regenerated header.
Add tests/cases/test_project_settings.gd as the backstop: the feature-override
assertions read project.godot as text and reject a line that has been folded
into a comment, since ProjectSettings resolves `key.<feature>` overrides at
load time and never exposes the suffixed key. Verified by reproducing the
exact corruption, which fails the test, and it also covers the Jolt physics
engine, the required autoloads, and that no test-hook autoload is ever shipped
registered.
server_boot.gd's private _log could only ever see what the boot scene
itself observed: connects, disconnects, roster changes, tick overruns.
The events an operator is actually asked about - who scored, who got
kicked and why, which peer is flooding - happen inside networked_match.gd
and match_sim.gd, neither of which could reach a logger on a scene node
that gets freed at the first change_scene_to_file. scripts/server_log.gd
holds it as static state on a class_name: reachable from all three, no
autoload, no ordering dependency.
New events: goal, match_ended, kickoff, peer_kicked (previously only a
push_warning, carrying neither peer nor reason into the stream a
container captures), rate_limited, server_stalled. rate_limited fires
ONCE per peer per window rather than per packet - a flood is thousands of
packets a second and the log line must not become the amplifier the
replay recorder was capped to avoid being.
Off unless a server configures it, so a client, an editor session or a
unit-test run does not start printing server telemetry just because these
scripts loaded.
Rotation is deliberately not implemented: the server logs to stdout and
stops, because every way this is run already rotates better - docker's
json-file driver, journald, or logrotate on a redirect. A server that
also wrote and rotated its own file would fight all of them in a
container, where stdout is the interface. SERVER.md (6.6) documents the
three configurations.
Five tests on the one piece with real logic - the one-line contract.
Including log injection: a player name is attacker-controlled, and
without escaping, the name "x\n[0.000] INFO peer_kicked reason=nothing"
writes a fake event into the operator's log. Newlines are escaped rather
than dropped so the attempt stays visible.
End-to-end verification of the new events comes with 6.5, which is what
first makes a server run a match at all.
6.1: "Linux Dedicated Server" preset (dedicated_server=true,
custom_features="dedicated_server") mirroring the existing training
preset, plus run/main_scene.dedicated_server so the server binary reaches
its own entry point with no flag. Builds: an 85MB Linux x86_64 binary,
gitignored like the training one.
6.3: scripts/server_config.gd declares every server flag once - name,
type, default, section, help - and one parser turns that into parsing,
type checking, range validation, config-file backing and --help. The
flags had grown to ~30 across server_boot.gd and networked_match.gd, each
parsed inline with begins_with, none documented, and an unrecognised flag
was SILENTLY IGNORED: --max-clientss=8 ran a server on the default cap
and said nothing. Unknown flags, missing values, wrong types, duplicates
and out-of-range values are now hard errors, reported all at once.
Precedence is command line > config file > default. server_boot.gd parses
strictly because it owns the whole command line; networked_match.gd reads
the same declaration leniently because it is one consumer of an argv the
smoke harnesses also fill with --role= and --drive-seconds=. Nothing is
lost - every server flag is declared, so the strict pass already caught
any typo before the match scene re-reads its own.
13 unit tests covering the precedence order, the typo rejection that
motivated this, --no-<bool> not double-listing in --help, and --help
documenting every flag asserted against the declaration rather than a
hand-kept list. Verified end to end: --help prints, a typo'd flag refuses
to start, and the plain/replay-log/late-joiner smoke scenarios still pass.
Lands tasks 1.0-1.8 of multiplayer-todo.md: the pure-function test runner,
net_codec (wire format quantizers/pack-unpack), NetworkManager (ENet
transport, manual polling, min-RTT clock sync), MatchNet (handshake,
protocol/tick-rate gating, roster with team+ready state), lobby.tscn (team
columns, switch team, ready toggle), server_boot.tscn (headless dedicated
server with structured logging and an overrun watchdog), and main_menu.gd's
Host/Join-by-IP UI (connecting overlay, cancel, bounded failure path).
Followed by an adversarial review (Opus subagent) that found and fixed two
real bugs - an unvalidated player_name broadcast that let one client's
oversized name head-of-line-block the reliable channel for everyone, and a
server-side roster leak across a host/re-host cycle - plus three gaps in
the test suite itself where a claim of "verified" wasn't actually backed
by what the test checked. All five two-process smoke tests plus the
pure-function suite are green with the strengthened assertions in place.
Josh Creek