diff --git a/AGONES-CI-INVESTIGATION.md b/AGONES-CI-INVESTIGATION.md new file mode 100644 index 00000000..459dbcc9 --- /dev/null +++ b/AGONES-CI-INVESTIGATION.md @@ -0,0 +1,141 @@ +# Investigate and fix the failing Agones Integration CI gate + +## Task + +`make verify-kind-agones` (workflow `.github/workflows/agones-integration.yml`, +script `scripts/verify_kind_agones.sh`) fails. Find the root cause and fix it so +the gate passes on CI. Repo: `jcreek/CosmicClash`, branch `feat/multiplayer`, +PR #30. + +## What is already known — do not re-derive this + +**The failure.** `helm upgrade --install agones ... --wait --timeout 5m` fails +with `Error: context deadline exceeded`. Immediately before, Helm reports: + +``` +resource Deployment/agones-system/agones-controller not ready. status: InProgress, message: Available: 0/1 +resource Deployment/agones-system/agones-extensions not ready. status: InProgress, message: Available: 0/1 +resource Deployment/agones-system/agones-allocator not ready. status: InProgress, message: Available: 0/1 +``` + +So the cluster is created, the game-server image loads, and the Agones chart +installs — but none of its Deployments become Available inside 5 minutes. The +script never reaches the parts that exercise this repo's own manifests. + +**It is pre-existing.** It fails identically at `089c127c`, the branch head +before recent work. It is not caused by the branch's changes. Do not assume a +recent commit broke it. + +**It is not architecture-specific.** It fails the same way on GitHub's +`ubuntu-24.04` amd64 runners and on an arm64 macOS developer machine. Agones +1.49.0 publishes both amd64 and arm64 images. + +**It is not a Helm kubeVersion rejection.** Agones charts 1.49.0, 1.50.0 and +1.51.0 declare no `kubeVersion` constraint, so Helm is not refusing the +Kubernetes version — the pods are being created and are not becoming ready. + +**Ruled out as a red herring:** reproducing locally on a machine with heavy +Docker usage produced `FailedCreatePodSandBox: containerd connection reset`, +which is local resource pressure, not the CI cause. If you see that locally, +clear Docker state and retry rather than chasing it. + +**There may be two distinct failures, not one.** After `docker system prune`, +a local run got *past* the Agones install cleanly (controller and allocator +both reached "condition met") and failed later, at: + +``` +scripts/verify_kind_agones.sh:146 +kubectl wait --for=jsonpath='{.status.ready}'=2 fleet/cosmic-clash-game -n cosmic-clash --timeout=5m +error: timed out waiting for the condition on fleets/cosmic-clash-game +``` + +So locally the Agones install is fine and the **Fleet's game-server pods never +become Ready**; on CI the run never gets that far because the Agones install +itself times out. Treat these as potentially separate problems: fixing the CI +Agones timeout may simply expose the Fleet one underneath. Both need to pass. + +The Fleet failure is the more suspicious of the two for recent work, because +`deploy/k8s/base/fleet.yaml` changed: the join-signing key material moved from +a single raw-bytes secret key (`join-signing-key`) to a JSON map +(`join-signing-keys.json`), and the mount's `items[].key` moved with it. The +script's `kubectl create secret` was updated to match and does succeed +(`secret/cosmic-clash-game-server created`), so the obvious mismatch is not +present -- but verify the pod actually mounts and starts rather than assuming. +Note the script's `sed` also strips `--allocated-mode` and the roster path and +blanks `--control-plane-url`, so the game server runs in a reduced mode here; +check whether it is failing for a reason unrelated to the key at all. + +## Pinned versions (all in `scripts/verify_kind_agones.sh`) + +| Thing | Value | Override | +|---|---|---| +| Agones chart | `1.49.0` | `AGONES_VERSION` | +| kind node image | `kindest/node:v1.33.1` (Kubernetes 1.33) | `KIND_NODE_IMAGE` | +| Cluster | single node, `--wait 120s` | `KIND_CLUSTER_NAME` | +| Runner | `ubuntu-latest` (ubuntu-24.04), 30 min timeout | — | + +The chart is installed with `--set agones.controller.replicas=1`, +`agones.extensions.replicas=1`, `agones.allocator.replicas=1`, and +`agones.extensions.resources.{requests,limits}.ephemeral-storage` lowered to +128Mi/512Mi. That ephemeral-storage override already exists because Agones 1.49 +otherwise requests 10,100 MiB and will not schedule on a default kind node — +there is a comment saying so. **A similar resource-fit problem for the other +Deployments is a strong hypothesis worth checking first.** + +## Diagnostics are already in place + +The script now dumps, on any failure and before the cluster is deleted: node +capacity and conditions, pods in `agones-system` and `cosmic-clash`, recent +events per namespace, and describe + current/previous logs for every not-ready +pod. Set `KIND_KEEP_ON_FAILURE=1` to retain the cluster for interactive +inspection instead of deleting it. + +Its first run revealed a bug in the diagnostics themselves: a +`kubectl cluster-info` reachability guard suppressed the entire dump. That +guard has been removed, so the dump now always runs on failure. + +**Start by reading that output**, either from a CI run or a local run. The most +likely candidates it will distinguish between: + +1. **Resource pressure** — `FailedScheduling ... Insufficient cpu/memory/ + ephemeral-storage`. Fix by lowering requests for the other Deployments the + way extensions already is, or by giving the kind cluster more capacity. +2. **Version incompatibility** — Agones 1.49 against Kubernetes 1.33. Check + Agones' release notes for its supported Kubernetes range; if 1.33 is outside + it, either raise `agones_version` or lower `kind_node_image`. Confirm the + pairing is one Agones actually tests. +3. **Probe/readiness failure** — pods Running but never Ready. The pod logs and + describe output will show the failing probe. +4. **Image pull** — `ImagePullBackOff` on an Agones image. + +## Constraints + +- **Do not weaken the gate to make it pass.** Removing `--wait`, extending the + timeout to hide a real failure, or `|| true` around the install are all wrong. + If the cause is genuinely a timeout on slow-but-working startup, raising it + is acceptable *only* with evidence that the pods do become Available, and the + new value should be justified in a comment. +- Keep it a disposable, isolated cluster: it must not touch an existing cluster, + and the EXIT trap must still remove the one it created. +- If you change a pinned version, pin the new one explicitly and say why in the + commit message. Do not float to `latest`. +- `CLAUDE.md` applies: never create co-authored commits, never mention Claude. + +## Verification + +- `make verify-kind-agones` passes locally (needs Docker, kind, kubectl, Helm). +- The `Agones Integration` workflow passes on PR #30. It is `pull_request` + triggered with path filters on `Dockerfile`, `Makefile`, `deploy/k8s/**`, + `scripts/verify_kind_agones.sh`, and its own workflow file — so a change to + the script will trigger it. +- Do not regress the other seven workflows. `Allocated Compose Smoke` was also + failing and has just been fixed; confirm it stays green. + +## Useful context + +- `multiplayer-next.md` §7 task 8.49 describes what this gate is meant to prove. +- `deploy/k8s/base/fleet.yaml` is the Fleet the script applies after Agones is + up, with a `sed` that swaps the release digest placeholder for the locally + built image and strips `--allocated-mode` and the roster path (there is no + control plane in this disposable cluster). +- The gate is a prerequisite for issue #17 (standing up a real cluster). diff --git a/scripts/verify_kind_agones.sh b/scripts/verify_kind_agones.sh index b51decbd..79ea44c4 100755 --- a/scripts/verify_kind_agones.sh +++ b/scripts/verify_kind_agones.sh @@ -58,7 +58,11 @@ dump_cluster_state() { cleanup() { local status=$? - if [[ "$status" != 0 ]] && kubectl cluster-info --context "kind-${cluster_name}" >/dev/null 2>&1; then + # No reachability guard here: every command inside dump_cluster_state is + # already `|| true`, so a gone cluster costs a few harmless errors, whereas + # a guard that misjudges reachability silently suppresses the whole dump -- + # which is exactly what happened on its first run. + if [[ "$status" != 0 ]]; then dump_cluster_state fi if [[ "$status" != 0 && "${KIND_KEEP_ON_FAILURE:-}" == 1 ]]; then @@ -96,15 +100,17 @@ kind load docker-image "$game_server_image" --name "$cluster_name" helm repo add agones https://agones.dev/chart/stable >/dev/null helm repo update >/dev/null -# Agones 1.49 otherwise requests 10,100 MiB of ephemeral storage for its -# extensions pod, which exceeds a default single-node kind cluster before the -# Fleet can be exercised. These are smoke-only bounds; production resource -# sizing remains deployment-owned. +# Agones 1.49 otherwise requests 10,100 MiB of ephemeral storage for both its +# controller and extensions pods, which exceeds a default single-node kind +# cluster before the Fleet can be exercised. These are smoke-only bounds; +# production resource sizing remains deployment-owned. helm upgrade --install agones agones/agones \ --namespace agones-system --create-namespace \ --version "$agones_version" \ --set agones.crds.cleanup.enabled=true \ --set agones.controller.replicas=1 \ + --set agones.controller.resources.requests.ephemeral-storage=128Mi \ + --set agones.controller.resources.limits.ephemeral-storage=512Mi \ --set agones.extensions.replicas=1 \ --set agones.extensions.resources.requests.ephemeral-storage=128Mi \ --set agones.extensions.resources.limits.ephemeral-storage=512Mi \