fix(agones): stop a dead health loop from passing as a healthy server

Every allocated GameServer reached Ready and was recycled by Agones ~20s
later. Health pings are the game process's job by design -- the
supervisor has no health implementation at all -- so a server that stops
pinging is exactly what Agones is built to reclaim.

start_health() armed a Timer on a node that might not be inside the
SceneTree. A Timer only ticks inside the tree, so the node reported
itself configured, sent nothing, and said nothing about it. It now
returns a bool, refuses loudly when unconfigured, and defers to _ready()
when called before parenting, so the SDK arms its own timer and no
caller has to get the ordering right. server_boot.gd defers the add like
every sibling does (§9 gotcha 27) and logs when AGONES_SDK_HTTP_PORT is
missing, which previously read identically to a healthy start.

Also bounded the in-flight latch: it is set across an await, so a request
that never completes would silence health permanently. Defence in depth
rather than an observed fault.

Tests target the contract rather than the mechanism: a test that parents
the SDK correctly and asserts pings passes with the bug present, because
the defect was in the wiring. The unit tests assert start_health()
cannot claim success out of tree, and were confirmed to fail against the
previous code. The smoke gains a counting sidecar and asserts a
*repeating* ping -- it reports "health pings in 3.0s = 1, want at least
2" when the loop is broken, which is the production symptom exactly. It
is also now actually run: nothing referenced it before.

Two diagnostic fixes, both of which changed conclusions during this work:

The kind gate only built the game-server image when the tag was absent,
so a local rerun silently verified whatever was built last. That is why
local runs and CI disagreed about the same commit. It now builds by
default, with KIND_REUSE_GAME_SERVER_IMAGE=1 as the opt-in fast path.

The failure dump logged only not-ready pods, and used --all-containers
with a shared tail. A GameServer recycled after reaching Ready leaves no
unready pod behind, and the Agones sidecar out-logs the game server, so
the relevant output was never captured. It now dumps every pod, per
container, current and previous, plus the GameServer and Fleet resources
-- Agones' own state machine is what rejects these.

scripts/verify_kind_agones.sh

@@ -40,20 +40,31 @@ dump_cluster_state() {
     # does not: FailedScheduling, ImagePullBackOff, readiness probe errors.
     echo "=== namespace ${ns}: recent events ===" >&2
     kubectl -n "$ns" get events --sort-by=.lastTimestamp 2>&1 | tail -40 >&2 || true
+    # Log EVERY pod, not only the not-ready ones. A GameServer that reaches
+    # Ready and is then recycled on a health check leaves no unready pod
+    # behind: the failures are already deleted and the survivors read 2/2
+    # Running, so filtering on readiness dumped nothing useful and the game
+    # server's own output went unseen for several CI runs.
     for pod in $(kubectl -n "$ns" get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null); do
       ready="$(kubectl -n "$ns" get pod "$pod" -o jsonpath='{.status.containerStatuses[*].ready}' 2>/dev/null || true)"
-      case "$ready" in
-        *false*|"")
-          echo "=== ${ns}/${pod} is not ready (ready=${ready:-unknown}) ===" >&2
-          kubectl -n "$ns" describe pod "$pod" 2>&1 | tail -35 >&2 || true
-          echo "--- ${ns}/${pod} logs (current) ---" >&2
-          kubectl -n "$ns" logs "$pod" --all-containers --tail=40 >&2 2>&1 || true
-          echo "--- ${ns}/${pod} logs (previous, if it restarted) ---" >&2
-          kubectl -n "$ns" logs "$pod" --all-containers --previous --tail=40 >&2 2>&1 || true
-          ;;
-      esac
+      echo "=== ${ns}/${pod} (ready=${ready:-unknown}) ===" >&2
+      kubectl -n "$ns" describe pod "$pod" 2>&1 | tail -35 >&2 || true
+      # Per container, not --all-containers: the Agones sidecar is far chattier
+      # than the game server, so a shared tail hides exactly the output needed,
+      # and --previous without -c resolves to a container that never restarted.
+      for container in $(kubectl -n "$ns" get pod "$pod" -o jsonpath='{range .spec.containers[*]}{.name}{"\n"}{end}' 2>/dev/null); do
+        echo "--- ${ns}/${pod}[${container}] logs (current) ---" >&2
+        kubectl -n "$ns" logs "$pod" -c "$container" --tail=60 >&2 2>&1 || true
+        echo "--- ${ns}/${pod}[${container}] logs (previous, if it restarted) ---" >&2
+        kubectl -n "$ns" logs "$pod" -c "$container" --previous --tail=60 >&2 2>&1 || true
+      done
     done
   done
+  # Agones' own view: a GameServer can be Unhealthy while its Pod looks fine,
+  # which is precisely the shape of a failed health check.
+  echo "=== Agones GameServers and Fleets ===" >&2
+  kubectl get gameservers --all-namespaces -o wide >&2 2>&1 || true
+  kubectl get fleets --all-namespaces -o wide >&2 2>&1 || true
   echo "=== helm releases ===" >&2
   helm list --all-namespaces >&2 2>&1 || true
 }
@@ -92,7 +103,14 @@ fi
 
 kind delete cluster --name "$cluster_name" >/dev/null 2>&1 || true
 
-if ! docker image inspect "$game_server_image" >/dev/null 2>&1; then
+# Build by default. Reusing whatever happens to be tagged locally silently
+# verifies stale code: a developer fixes the game server, reruns this gate, and
+# it exercises the previous build because the tag already exists. CI never hits
+# that because a fresh runner has no image, which is precisely how a local pass
+# and a CI failure can disagree about the same commit.
+if [[ "${KIND_REUSE_GAME_SERVER_IMAGE:-}" == 1 ]] && docker image inspect "$game_server_image" >/dev/null 2>&1; then
+  echo "Reusing existing $game_server_image (KIND_REUSE_GAME_SERVER_IMAGE=1); it may not contain local changes"
+else
   echo "Building $game_server_image from the pinned game-server target"
   docker build --target game-server -t "$game_server_image" .
 fi

scripts/verify_multiplayer_local.sh

@@ -48,6 +48,18 @@ echo "local multiplayer gate: bounded fuzz targets"
 echo "local multiplayer gate: Godot harness"
 run_godot_harness
 
+# The Agones SDK smoke needs a live SceneTree and awaits an HTTP round trip, so
+# it cannot live in test_runner.tscn -- that runner calls test methods without
+# awaiting. It covers the property the unit tests structurally cannot: that
+# start_health() produces a *repeating* ping, which is what Agones enforces and
+# whose absence silently recycled every allocated GameServer.
+echo "local multiplayer gate: Agones SDK smoke"
+if [[ -x "$godot_bin" ]]; then
+  "$godot_bin" --headless --path "$root_dir/Game" --script res://tests/agones_sdk_smoke.gd
+else
+  echo "local multiplayer gate: skipping Agones SDK smoke, Godot executable not found ($godot_bin)" >&2
+fi
+
 echo "local multiplayer gate: contracts and manifests"
 python3 -m json.tool "$root_dir/server/contracts/v1/openapi.json" >/dev/null
 # json.tool only proves the contract parses. test_contracts.py is what actually