# Investigate and fix the failing Agones Integration CI gate

## Task

`make verify-kind-agones` (workflow `.github/workflows/agones-integration.yml`,
script `scripts/verify_kind_agones.sh`) fails. Find the root cause and fix it so
the gate passes on CI. Repo: `jcreek/CosmicClash`, branch `feat/multiplayer`,
PR #30.

## What is already known — do not re-derive this

**The failure.** `helm upgrade --install agones ... --wait --timeout 5m` fails
with `Error: context deadline exceeded`. Immediately before, Helm reports:

```
resource Deployment/agones-system/agones-controller not ready. status: InProgress, message: Available: 0/1
resource Deployment/agones-system/agones-extensions not ready. status: InProgress, message: Available: 0/1
resource Deployment/agones-system/agones-allocator not ready. status: InProgress, message: Available: 0/1
```

So the cluster is created, the game-server image loads, and the Agones chart
installs — but none of its Deployments become Available inside 5 minutes. The
script never reaches the parts that exercise this repo's own manifests.

**It is pre-existing.** It fails identically at `089c127c`, the branch head
before recent work. It is not caused by the branch's changes. Do not assume a
recent commit broke it.

**It is not architecture-specific.** It fails the same way on GitHub's
`ubuntu-24.04` amd64 runners and on an arm64 macOS developer machine. Agones
1.49.0 publishes both amd64 and arm64 images.

**It is not a Helm kubeVersion rejection.** Agones charts 1.49.0, 1.50.0 and
1.51.0 declare no `kubeVersion` constraint, so Helm is not refusing the
Kubernetes version — the pods are being created and are not becoming ready.

**Ruled out as a red herring:** reproducing locally on a machine with heavy
Docker usage produced `FailedCreatePodSandBox: containerd connection reset`,
which is local resource pressure, not the CI cause. If you see that locally,
clear Docker state and retry rather than chasing it.

**There may be two distinct failures, not one.** After `docker system prune`,
a local run got *past* the Agones install cleanly (controller and allocator
both reached "condition met") and failed later, at:

```
scripts/verify_kind_agones.sh:146
kubectl wait --for=jsonpath='{.status.ready}'=2 fleet/cosmic-clash-game -n cosmic-clash --timeout=5m
error: timed out waiting for the condition on fleets/cosmic-clash-game
```

So locally the Agones install is fine and the **Fleet's game-server pods never
become Ready**; on CI the run never gets that far because the Agones install
itself times out. Treat these as potentially separate problems: fixing the CI
Agones timeout may simply expose the Fleet one underneath. Both need to pass.

The Fleet failure is the more suspicious of the two for recent work, because
`deploy/k8s/base/fleet.yaml` changed: the join-signing key material moved from
a single raw-bytes secret key (`join-signing-key`) to a JSON map
(`join-signing-keys.json`), and the mount's `items[].key` moved with it. The
script's `kubectl create secret` was updated to match and does succeed
(`secret/cosmic-clash-game-server created`), so the obvious mismatch is not
present -- but verify the pod actually mounts and starts rather than assuming.
Note the script's `sed` also strips `--allocated-mode` and the roster path and
blanks `--control-plane-url`, so the game server runs in a reduced mode here;
check whether it is failing for a reason unrelated to the key at all.

## Pinned versions (all in `scripts/verify_kind_agones.sh`)

| Thing | Value | Override |
|---|---|---|
| Agones chart | `1.49.0` | `AGONES_VERSION` |
| kind node image | `kindest/node:v1.33.1` (Kubernetes 1.33) | `KIND_NODE_IMAGE` |
| Cluster | single node, `--wait 120s` | `KIND_CLUSTER_NAME` |
| Runner | `ubuntu-latest` (ubuntu-24.04), 30 min timeout | — |

The chart is installed with `--set agones.controller.replicas=1`,
`agones.extensions.replicas=1`, `agones.allocator.replicas=1`, and
`agones.extensions.resources.{requests,limits}.ephemeral-storage` lowered to
128Mi/512Mi. That ephemeral-storage override already exists because Agones 1.49
otherwise requests 10,100 MiB and will not schedule on a default kind node —
there is a comment saying so. **A similar resource-fit problem for the other
Deployments is a strong hypothesis worth checking first.**

## Diagnostics are already in place

The script now dumps, on any failure and before the cluster is deleted: node
capacity and conditions, pods in `agones-system` and `cosmic-clash`, recent
events per namespace, and describe + current/previous logs for every not-ready
pod. Set `KIND_KEEP_ON_FAILURE=1` to retain the cluster for interactive
inspection instead of deleting it.

Its first run revealed a bug in the diagnostics themselves: a
`kubectl cluster-info` reachability guard suppressed the entire dump. That
guard has been removed, so the dump now always runs on failure.

**Start by reading that output**, either from a CI run or a local run. The most
likely candidates it will distinguish between:

1. **Resource pressure** — `FailedScheduling ... Insufficient cpu/memory/
   ephemeral-storage`. Fix by lowering requests for the other Deployments the
   way extensions already is, or by giving the kind cluster more capacity.
2. **Version incompatibility** — Agones 1.49 against Kubernetes 1.33. Check
   Agones' release notes for its supported Kubernetes range; if 1.33 is outside
   it, either raise `agones_version` or lower `kind_node_image`. Confirm the
   pairing is one Agones actually tests.
3. **Probe/readiness failure** — pods Running but never Ready. The pod logs and
   describe output will show the failing probe.
4. **Image pull** — `ImagePullBackOff` on an Agones image.

## Constraints

- **Do not weaken the gate to make it pass.** Removing `--wait`, extending the
  timeout to hide a real failure, or `|| true` around the install are all wrong.
  If the cause is genuinely a timeout on slow-but-working startup, raising it
  is acceptable *only* with evidence that the pods do become Available, and the
  new value should be justified in a comment.
- Keep it a disposable, isolated cluster: it must not touch an existing cluster,
  and the EXIT trap must still remove the one it created.
- If you change a pinned version, pin the new one explicitly and say why in the
  commit message. Do not float to `latest`.
- `CLAUDE.md` applies: never create co-authored commits, never mention Claude.

## Verification

- `make verify-kind-agones` passes locally (needs Docker, kind, kubectl, Helm).
- The `Agones Integration` workflow passes on PR #30. It is `pull_request`
  triggered with path filters on `Dockerfile`, `Makefile`, `deploy/k8s/**`,
  `scripts/verify_kind_agones.sh`, and its own workflow file — so a change to
  the script will trigger it.
- Do not regress the other seven workflows. `Allocated Compose Smoke` was also
  failing and has just been fixed; confirm it stays green.

## Useful context

- `multiplayer-next.md` §7 task 8.49 describes what this gate is meant to prove.
- `deploy/k8s/base/fleet.yaml` is the Fleet the script applies after Agones is
  up, with a `sed` that swaps the release digest placeholder for the locally
  built image and strips `--allocated-mode` and the roster path (there is no
  control plane in this disposable cluster).
- The gate is a prerequisite for issue #17 (standing up a real cluster).