Files
CosmicClash/scripts/verify_multiplayer_local.sh
T
Josh Creek 0de97381b7 fix(agones): stop a dead health loop from passing as a healthy server
Every allocated GameServer reached Ready and was recycled by Agones ~20s
later. Health pings are the game process's job by design -- the
supervisor has no health implementation at all -- so a server that stops
pinging is exactly what Agones is built to reclaim.

start_health() armed a Timer on a node that might not be inside the
SceneTree. A Timer only ticks inside the tree, so the node reported
itself configured, sent nothing, and said nothing about it. It now
returns a bool, refuses loudly when unconfigured, and defers to _ready()
when called before parenting, so the SDK arms its own timer and no
caller has to get the ordering right. server_boot.gd defers the add like
every sibling does (§9 gotcha 27) and logs when AGONES_SDK_HTTP_PORT is
missing, which previously read identically to a healthy start.

Also bounded the in-flight latch: it is set across an await, so a request
that never completes would silence health permanently. Defence in depth
rather than an observed fault.

Tests target the contract rather than the mechanism: a test that parents
the SDK correctly and asserts pings passes with the bug present, because
the defect was in the wiring. The unit tests assert start_health()
cannot claim success out of tree, and were confirmed to fail against the
previous code. The smoke gains a counting sidecar and asserts a
*repeating* ping -- it reports "health pings in 3.0s = 1, want at least
2" when the loop is broken, which is the production symptom exactly. It
is also now actually run: nothing referenced it before.

Two diagnostic fixes, both of which changed conclusions during this work:

The kind gate only built the game-server image when the tag was absent,
so a local rerun silently verified whatever was built last. That is why
local runs and CI disagreed about the same commit. It now builds by
default, with KIND_REUSE_GAME_SERVER_IMAGE=1 as the opt-in fast path.

The failure dump logged only not-ready pods, and used --all-containers
with a shared tail. A GameServer recycled after reaching Ready leaves no
unready pod behind, and the Agones sidecar out-logs the game server, so
the relevant output was never captured. It now dumps every pod, per
container, current and previous, plus the GameServer and Fleet resources
-- Agones' own state machine is what rejects these.
2026-09-05 21:35:50 +01:00

83 lines
3.9 KiB
Bash
Executable File

#!/usr/bin/env bash
set -euo pipefail
root_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
godot_bin="${GODOT_BIN:-/Applications/Godot.app/Contents/MacOS/Godot}"
godot_image="barichello/godot-ci@sha256:622e5ca81b54cd8038ecf7de5d157b47efc800d7cf635af2eec18a6aee4bab7e"
run_godot_harness() {
if [[ -x "$godot_bin" ]]; then
set +e
"$godot_bin" --headless --path "$root_dir/Game" res://tests/test_runner.tscn
local native_status=$?
set -e
if [[ "$native_status" -eq 0 ]]; then
return
fi
# A failing test exits 1 and must fail the gate. A signal exit (as seen
# with the host Metal/Vulkan stack) is an engine-host failure, so rerun
# the identical pinned Linux harness instead of losing all verification.
if [[ "$native_status" -lt 128 ]]; then
return "$native_status"
fi
echo "local multiplayer gate: native Godot crashed (status $native_status); using pinned headless fallback" >&2
fi
if ! command -v docker >/dev/null 2>&1 || ! docker info >/dev/null 2>&1; then
echo "local multiplayer gate: Godot executable not found ($godot_bin), and Docker is unavailable for the pinned headless fallback" >&2
return 2
fi
echo "local multiplayer gate: using pinned headless Godot container fallback"
docker run --rm --platform linux/amd64 \
--mount "type=bind,src=$root_dir,dst=/workspace" \
-w /workspace "$godot_image" \
godot --headless --path Game res://tests/test_runner.tscn
}
echo "local multiplayer gate: Go tests"
(cd "$root_dir/server" && go test ./...)
echo "local multiplayer gate: Go race and vet"
(cd "$root_dir/server" && go test -race ./...)
(cd "$root_dir/server" && go vet ./...)
echo "local multiplayer gate: bounded fuzz targets"
(cd "$root_dir/server" && go test ./domain -fuzz FuzzQueueCreateDoesNotPanic -fuzztime=2s)
(cd "$root_dir/server" && go test ./domain -fuzz FuzzResultDigestIsDeterministic -fuzztime=2s)
(cd "$root_dir/server" && go test ./domain -fuzz FuzzSyncEventApplicationDoesNotPanic -fuzztime=2s)
echo "local multiplayer gate: Godot harness"
run_godot_harness
# The Agones SDK smoke needs a live SceneTree and awaits an HTTP round trip, so
# it cannot live in test_runner.tscn -- that runner calls test methods without
# awaiting. It covers the property the unit tests structurally cannot: that
# start_health() produces a *repeating* ping, which is what Agones enforces and
# whose absence silently recycled every allocated GameServer.
echo "local multiplayer gate: Agones SDK smoke"
if [[ -x "$godot_bin" ]]; then
"$godot_bin" --headless --path "$root_dir/Game" --script res://tests/agones_sdk_smoke.gd
else
echo "local multiplayer gate: skipping Agones SDK smoke, Godot executable not found ($godot_bin)" >&2
fi
echo "local multiplayer gate: contracts and manifests"
python3 -m json.tool "$root_dir/server/contracts/v1/openapi.json" >/dev/null
# json.tool only proves the contract parses. test_contracts.py is what actually
# checks the operation IDs, envelopes and state vocabulary generated clients
# bind to; it was previously not run by any target, so a real mismatch between
# openapi.json and the suite sat undetected.
python3 "$root_dir/server/contracts/v1/test_contracts.py"
python3 "$root_dir/server/migrations/test_migration.py"
python3 "$root_dir/server/security/test_fleet_manifests.py"
python3 "$root_dir/server/security/test_compose_manifests.py"
python3 "$root_dir/server/security/test_kubernetes_policies.py"
python3 "$root_dir/server/security/test_supply_chain.py"
python3 "$root_dir/server/security/test_threat_model.py"
python3 "$root_dir/scripts/verify_observability_manifests.py"
# The checker above validates the checked-in manifests; this validates the
# checker itself still rejects a widened scrape scope.
python3 "$root_dir/server/security/test_observability_manifests.py"
python3 -m unittest "$root_dir/scripts/test_verify_agones_allocation_response.py"
echo "LOCAL MULTIPLAYER GATE PASS"