fix(agones): stop a dead health loop from passing as a healthy server

Every allocated GameServer reached Ready and was recycled by Agones ~20s
later. Health pings are the game process's job by design -- the
supervisor has no health implementation at all -- so a server that stops
pinging is exactly what Agones is built to reclaim.

start_health() armed a Timer on a node that might not be inside the
SceneTree. A Timer only ticks inside the tree, so the node reported
itself configured, sent nothing, and said nothing about it. It now
returns a bool, refuses loudly when unconfigured, and defers to _ready()
when called before parenting, so the SDK arms its own timer and no
caller has to get the ordering right. server_boot.gd defers the add like
every sibling does (§9 gotcha 27) and logs when AGONES_SDK_HTTP_PORT is
missing, which previously read identically to a healthy start.

Also bounded the in-flight latch: it is set across an await, so a request
that never completes would silence health permanently. Defence in depth
rather than an observed fault.

Tests target the contract rather than the mechanism: a test that parents
the SDK correctly and asserts pings passes with the bug present, because
the defect was in the wiring. The unit tests assert start_health()
cannot claim success out of tree, and were confirmed to fail against the
previous code. The smoke gains a counting sidecar and asserts a
*repeating* ping -- it reports "health pings in 3.0s = 1, want at least
2" when the loop is broken, which is the production symptom exactly. It
is also now actually run: nothing referenced it before.

Two diagnostic fixes, both of which changed conclusions during this work:

The kind gate only built the game-server image when the tag was absent,
so a local rerun silently verified whatever was built last. That is why
local runs and CI disagreed about the same commit. It now builds by
default, with KIND_REUSE_GAME_SERVER_IMAGE=1 as the opt-in fast path.

The failure dump logged only not-ready pods, and used --all-containers
with a shared tail. A GameServer recycled after reaching Ready leaves no
unready pod behind, and the Agones sidecar out-logs the game server, so
the relevant output was never captured. It now dumps every pod, per
container, current and previous, plus the GameServer and Fleet resources
-- Agones' own state machine is what rejects these.
This commit is contained in:
Josh Creek
2026-09-05 21:35:50 +01:00
parent ca70568fad
commit 0de97381b7
6 changed files with 249 additions and 26 deletions
+11 -3
View File
@@ -91,11 +91,19 @@ func _ready() -> void:
if agones_managed:
_agones = AgonesSDKScript.new()
_agones.name = "AgonesSDK"
# Health creates and starts a Timer immediately, so the SDK node must be
# in the tree before start_health() runs.
get_tree().root.add_child(_agones)
# Configure before parenting, then request health and defer the add like
# every other node here (§9 gotcha 27: add_child() on get_tree().root
# from inside _ready() is refused because the tree is still attaching
# this very node, and the refusal is not catchable from GDScript). The
# SDK arms its own timer in _ready(), so nothing depends on the order
# these deferred calls happen to flush in.
if _agones.configure_from_environment():
_agones.start_health()
else:
# Never silent: without this the log looks identical to a healthy
# server right up until Agones recycles it.
printerr("cosmic-clash-server: AGONES_SDK_HTTP_PORT is missing or invalid; Agones health pings are disabled")
get_tree().root.add_child.call_deferred(_agones)
if allocated_mode:
var roster_file := String(config.get_value("join-authorisations-file"))
var key_file := String(config.get_value("join-authorisations-key-file"))