CosmicClash/AGONES-CI-INVESTIGATION.md

Investigate and fix the failing Agones Integration CI gate

Task

make verify-kind-agones (workflow .github/workflows/agones-integration.yml, script scripts/verify_kind_agones.sh) fails. Find the root cause and fix it so the gate passes on CI. Repo: jcreek/CosmicClash, branch feat/multiplayer, PR #30.

What is already known — do not re-derive this

The failure. helm upgrade --install agones ... --wait --timeout 5m fails with Error: context deadline exceeded. Immediately before, Helm reports:

resource Deployment/agones-system/agones-controller not ready. status: InProgress, message: Available: 0/1
resource Deployment/agones-system/agones-extensions not ready. status: InProgress, message: Available: 0/1
resource Deployment/agones-system/agones-allocator not ready. status: InProgress, message: Available: 0/1

So the cluster is created, the game-server image loads, and the Agones chart installs — but none of its Deployments become Available inside 5 minutes. The script never reaches the parts that exercise this repo's own manifests.

It is pre-existing. It fails identically at 089c127c, the branch head before recent work. It is not caused by the branch's changes. Do not assume a recent commit broke it.

It is not architecture-specific. It fails the same way on GitHub's ubuntu-24.04 amd64 runners and on an arm64 macOS developer machine. Agones 1.49.0 publishes both amd64 and arm64 images.

It is not a Helm kubeVersion rejection. Agones charts 1.49.0, 1.50.0 and 1.51.0 declare no kubeVersion constraint, so Helm is not refusing the Kubernetes version — the pods are being created and are not becoming ready.

Ruled out as a red herring: reproducing locally on a machine with heavy Docker usage produced FailedCreatePodSandBox: containerd connection reset, which is local resource pressure, not the CI cause. If you see that locally, clear Docker state and retry rather than chasing it.

There may be two distinct failures, not one. After docker system prune, a local run got past the Agones install cleanly (controller and allocator both reached "condition met") and failed later, at:

scripts/verify_kind_agones.sh:146
kubectl wait --for=jsonpath='{.status.ready}'=2 fleet/cosmic-clash-game -n cosmic-clash --timeout=5m
error: timed out waiting for the condition on fleets/cosmic-clash-game

So locally the Agones install is fine and the Fleet's game-server pods never become Ready; on CI the run never gets that far because the Agones install itself times out. Treat these as potentially separate problems: fixing the CI Agones timeout may simply expose the Fleet one underneath. Both need to pass.

The Fleet failure is the more suspicious of the two for recent work, because deploy/k8s/base/fleet.yaml changed: the join-signing key material moved from a single raw-bytes secret key (join-signing-key) to a JSON map (join-signing-keys.json), and the mount's items[].key moved with it. The script's kubectl create secret was updated to match and does succeed (secret/cosmic-clash-game-server created), so the obvious mismatch is not present -- but verify the pod actually mounts and starts rather than assuming. Note the script's sed also strips --allocated-mode and the roster path and blanks --control-plane-url, so the game server runs in a reduced mode here; check whether it is failing for a reason unrelated to the key at all.

Pinned versions (all in scripts/verify_kind_agones.sh)

Thing	Value	Override
Agones chart	1.49.0	AGONES_VERSION
kind node image	kindest/node:v1.33.1 (Kubernetes 1.33)	KIND_NODE_IMAGE
Cluster	single node, --wait 120s	KIND_CLUSTER_NAME
Runner	ubuntu-latest (ubuntu-24.04), 30 min timeout	—

The chart is installed with --set agones.controller.replicas=1, agones.extensions.replicas=1, agones.allocator.replicas=1, and agones.extensions.resources.{requests,limits}.ephemeral-storage lowered to 128Mi/512Mi. That ephemeral-storage override already exists because Agones 1.49 otherwise requests 10,100 MiB and will not schedule on a default kind node — there is a comment saying so. A similar resource-fit problem for the other Deployments is a strong hypothesis worth checking first.

Diagnostics are already in place

The script now dumps, on any failure and before the cluster is deleted: node capacity and conditions, pods in agones-system and cosmic-clash, recent events per namespace, and describe + current/previous logs for every not-ready pod. Set KIND_KEEP_ON_FAILURE=1 to retain the cluster for interactive inspection instead of deleting it.

Its first run revealed a bug in the diagnostics themselves: a kubectl cluster-info reachability guard suppressed the entire dump. That guard has been removed, so the dump now always runs on failure.

Start by reading that output, either from a CI run or a local run. The most likely candidates it will distinguish between:

1. Resource pressure — FailedScheduling ... Insufficient cpu/memory/ ephemeral-storage. Fix by lowering requests for the other Deployments the way extensions already is, or by giving the kind cluster more capacity.
2. Version incompatibility — Agones 1.49 against Kubernetes 1.33. Check Agones' release notes for its supported Kubernetes range; if 1.33 is outside it, either raise agones_version or lower kind_node_image. Confirm the pairing is one Agones actually tests.
3. Probe/readiness failure — pods Running but never Ready. The pod logs and describe output will show the failing probe.
4. Image pull — ImagePullBackOff on an Agones image.

Constraints

• Do not weaken the gate to make it pass. Removing --wait, extending the timeout to hide a real failure, or || true around the install are all wrong. If the cause is genuinely a timeout on slow-but-working startup, raising it is acceptable only with evidence that the pods do become Available, and the new value should be justified in a comment.
• Keep it a disposable, isolated cluster: it must not touch an existing cluster, and the EXIT trap must still remove the one it created.
• If you change a pinned version, pin the new one explicitly and say why in the commit message. Do not float to latest.
• CLAUDE.md applies: never create co-authored commits, never mention Claude.

Verification

• make verify-kind-agones passes locally (needs Docker, kind, kubectl, Helm).
• The Agones Integration workflow passes on PR #30. It is pull_request triggered with path filters on Dockerfile, Makefile, deploy/k8s/**, scripts/verify_kind_agones.sh, and its own workflow file — so a change to the script will trigger it.
• Do not regress the other seven workflows. Allocated Compose Smoke was also failing and has just been fixed; confirm it stays green.

Useful context

• multiplayer-next.md §7 task 8.49 describes what this gate is meant to prove.
• deploy/k8s/base/fleet.yaml is the Fleet the script applies after Agones is up, with a sed that swaps the release digest placeholder for the locally built image and strips --allocated-mode and the roster path (there is no control plane in this disposable cluster).
• The gate is a prerequisite for issue #17 (standing up a real cluster).