mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
test(kind): always dump on failure, and record the second Agones failure
The dump added in 9ab1bec8 never ran. A `kubectl cluster-info`
reachability guard suppressed it, so its first exercise produced exactly
the silence it was written to prevent. Every command inside is already
`|| true`, so the guard bought nothing and cost the whole dump; removed.
That run did establish something the CI logs cannot: after clearing
local Docker pressure, the Agones install completes cleanly (controller
and allocator both reach "condition met") and the gate instead fails
later, waiting for the Fleet's game-server pods to become Ready. CI
never reaches that point because the Agones install times out first.
So there are likely two failures stacked, and fixing the CI timeout will
probably expose the Fleet one. Recorded in AGONES-CI-INVESTIGATION.md
along with the reasons the Fleet failure warrants suspicion -- fleet.yaml
changed its join-signing key mount from raw bytes to a JSON map this
branch -- and the reasons it may be unrelated.
This commit is contained in:
Josh Creek
@@ -0,0 +1,141 @@
|
||||
# Investigate and fix the failing Agones Integration CI gate
|
||||
|
||||
## Task
|
||||
|
||||
`make verify-kind-agones` (workflow `.github/workflows/agones-integration.yml`,
|
||||
script `scripts/verify_kind_agones.sh`) fails. Find the root cause and fix it so
|
||||
the gate passes on CI. Repo: `jcreek/CosmicClash`, branch `feat/multiplayer`,
|
||||
PR #30.
|
||||
|
||||
## What is already known — do not re-derive this
|
||||
|
||||
**The failure.** `helm upgrade --install agones ... --wait --timeout 5m` fails
|
||||
with `Error: context deadline exceeded`. Immediately before, Helm reports:
|
||||
|
||||
```
|
||||
resource Deployment/agones-system/agones-controller not ready. status: InProgress, message: Available: 0/1
|
||||
resource Deployment/agones-system/agones-extensions not ready. status: InProgress, message: Available: 0/1
|
||||
resource Deployment/agones-system/agones-allocator not ready. status: InProgress, message: Available: 0/1
|
||||
```
|
||||
|
||||
So the cluster is created, the game-server image loads, and the Agones chart
|
||||
installs — but none of its Deployments become Available inside 5 minutes. The
|
||||
script never reaches the parts that exercise this repo's own manifests.
|
||||
|
||||
**It is pre-existing.** It fails identically at `089c127c`, the branch head
|
||||
before recent work. It is not caused by the branch's changes. Do not assume a
|
||||
recent commit broke it.
|
||||
|
||||
**It is not architecture-specific.** It fails the same way on GitHub's
|
||||
`ubuntu-24.04` amd64 runners and on an arm64 macOS developer machine. Agones
|
||||
1.49.0 publishes both amd64 and arm64 images.
|
||||
|
||||
**It is not a Helm kubeVersion rejection.** Agones charts 1.49.0, 1.50.0 and
|
||||
1.51.0 declare no `kubeVersion` constraint, so Helm is not refusing the
|
||||
Kubernetes version — the pods are being created and are not becoming ready.
|
||||
|
||||
**Ruled out as a red herring:** reproducing locally on a machine with heavy
|
||||
Docker usage produced `FailedCreatePodSandBox: containerd connection reset`,
|
||||
which is local resource pressure, not the CI cause. If you see that locally,
|
||||
clear Docker state and retry rather than chasing it.
|
||||
|
||||
**There may be two distinct failures, not one.** After `docker system prune`,
|
||||
a local run got *past* the Agones install cleanly (controller and allocator
|
||||
both reached "condition met") and failed later, at:
|
||||
|
||||
```
|
||||
scripts/verify_kind_agones.sh:146
|
||||
kubectl wait --for=jsonpath='{.status.ready}'=2 fleet/cosmic-clash-game -n cosmic-clash --timeout=5m
|
||||
error: timed out waiting for the condition on fleets/cosmic-clash-game
|
||||
```
|
||||
|
||||
So locally the Agones install is fine and the **Fleet's game-server pods never
|
||||
become Ready**; on CI the run never gets that far because the Agones install
|
||||
itself times out. Treat these as potentially separate problems: fixing the CI
|
||||
Agones timeout may simply expose the Fleet one underneath. Both need to pass.
|
||||
|
||||
The Fleet failure is the more suspicious of the two for recent work, because
|
||||
`deploy/k8s/base/fleet.yaml` changed: the join-signing key material moved from
|
||||
a single raw-bytes secret key (`join-signing-key`) to a JSON map
|
||||
(`join-signing-keys.json`), and the mount's `items[].key` moved with it. The
|
||||
script's `kubectl create secret` was updated to match and does succeed
|
||||
(`secret/cosmic-clash-game-server created`), so the obvious mismatch is not
|
||||
present -- but verify the pod actually mounts and starts rather than assuming.
|
||||
Note the script's `sed` also strips `--allocated-mode` and the roster path and
|
||||
blanks `--control-plane-url`, so the game server runs in a reduced mode here;
|
||||
check whether it is failing for a reason unrelated to the key at all.
|
||||
|
||||
## Pinned versions (all in `scripts/verify_kind_agones.sh`)
|
||||
|
||||
| Thing | Value | Override |
|
||||
|---|---|---|
|
||||
| Agones chart | `1.49.0` | `AGONES_VERSION` |
|
||||
| kind node image | `kindest/node:v1.33.1` (Kubernetes 1.33) | `KIND_NODE_IMAGE` |
|
||||
| Cluster | single node, `--wait 120s` | `KIND_CLUSTER_NAME` |
|
||||
| Runner | `ubuntu-latest` (ubuntu-24.04), 30 min timeout | — |
|
||||
|
||||
The chart is installed with `--set agones.controller.replicas=1`,
|
||||
`agones.extensions.replicas=1`, `agones.allocator.replicas=1`, and
|
||||
`agones.extensions.resources.{requests,limits}.ephemeral-storage` lowered to
|
||||
128Mi/512Mi. That ephemeral-storage override already exists because Agones 1.49
|
||||
otherwise requests 10,100 MiB and will not schedule on a default kind node —
|
||||
there is a comment saying so. **A similar resource-fit problem for the other
|
||||
Deployments is a strong hypothesis worth checking first.**
|
||||
|
||||
## Diagnostics are already in place
|
||||
|
||||
The script now dumps, on any failure and before the cluster is deleted: node
|
||||
capacity and conditions, pods in `agones-system` and `cosmic-clash`, recent
|
||||
events per namespace, and describe + current/previous logs for every not-ready
|
||||
pod. Set `KIND_KEEP_ON_FAILURE=1` to retain the cluster for interactive
|
||||
inspection instead of deleting it.
|
||||
|
||||
Its first run revealed a bug in the diagnostics themselves: a
|
||||
`kubectl cluster-info` reachability guard suppressed the entire dump. That
|
||||
guard has been removed, so the dump now always runs on failure.
|
||||
|
||||
**Start by reading that output**, either from a CI run or a local run. The most
|
||||
likely candidates it will distinguish between:
|
||||
|
||||
1. **Resource pressure** — `FailedScheduling ... Insufficient cpu/memory/
|
||||
ephemeral-storage`. Fix by lowering requests for the other Deployments the
|
||||
way extensions already is, or by giving the kind cluster more capacity.
|
||||
2. **Version incompatibility** — Agones 1.49 against Kubernetes 1.33. Check
|
||||
Agones' release notes for its supported Kubernetes range; if 1.33 is outside
|
||||
it, either raise `agones_version` or lower `kind_node_image`. Confirm the
|
||||
pairing is one Agones actually tests.
|
||||
3. **Probe/readiness failure** — pods Running but never Ready. The pod logs and
|
||||
describe output will show the failing probe.
|
||||
4. **Image pull** — `ImagePullBackOff` on an Agones image.
|
||||
|
||||
## Constraints
|
||||
|
||||
- **Do not weaken the gate to make it pass.** Removing `--wait`, extending the
|
||||
timeout to hide a real failure, or `|| true` around the install are all wrong.
|
||||
If the cause is genuinely a timeout on slow-but-working startup, raising it
|
||||
is acceptable *only* with evidence that the pods do become Available, and the
|
||||
new value should be justified in a comment.
|
||||
- Keep it a disposable, isolated cluster: it must not touch an existing cluster,
|
||||
and the EXIT trap must still remove the one it created.
|
||||
- If you change a pinned version, pin the new one explicitly and say why in the
|
||||
commit message. Do not float to `latest`.
|
||||
- `CLAUDE.md` applies: never create co-authored commits, never mention Claude.
|
||||
|
||||
## Verification
|
||||
|
||||
- `make verify-kind-agones` passes locally (needs Docker, kind, kubectl, Helm).
|
||||
- The `Agones Integration` workflow passes on PR #30. It is `pull_request`
|
||||
triggered with path filters on `Dockerfile`, `Makefile`, `deploy/k8s/**`,
|
||||
`scripts/verify_kind_agones.sh`, and its own workflow file — so a change to
|
||||
the script will trigger it.
|
||||
- Do not regress the other seven workflows. `Allocated Compose Smoke` was also
|
||||
failing and has just been fixed; confirm it stays green.
|
||||
|
||||
## Useful context
|
||||
|
||||
- `multiplayer-next.md` §7 task 8.49 describes what this gate is meant to prove.
|
||||
- `deploy/k8s/base/fleet.yaml` is the Fleet the script applies after Agones is
|
||||
up, with a `sed` that swaps the release digest placeholder for the locally
|
||||
built image and strips `--allocated-mode` and the roster path (there is no
|
||||
control plane in this disposable cluster).
|
||||
- The gate is a prerequisite for issue #17 (standing up a real cluster).
|
||||
@@ -58,7 +58,11 @@ dump_cluster_state() {
|
||||
|
||||
cleanup() {
|
||||
local status=$?
|
||||
if [[ "$status" != 0 ]] && kubectl cluster-info --context "kind-${cluster_name}" >/dev/null 2>&1; then
|
||||
# No reachability guard here: every command inside dump_cluster_state is
|
||||
# already `|| true`, so a gone cluster costs a few harmless errors, whereas
|
||||
# a guard that misjudges reachability silently suppresses the whole dump --
|
||||
# which is exactly what happened on its first run.
|
||||
if [[ "$status" != 0 ]]; then
|
||||
dump_cluster_state
|
||||
fi
|
||||
if [[ "$status" != 0 && "${KIND_KEEP_ON_FAILURE:-}" == 1 ]]; then
|
||||
@@ -96,15 +100,17 @@ kind load docker-image "$game_server_image" --name "$cluster_name"
|
||||
|
||||
helm repo add agones https://agones.dev/chart/stable >/dev/null
|
||||
helm repo update >/dev/null
|
||||
# Agones 1.49 otherwise requests 10,100 MiB of ephemeral storage for its
|
||||
# extensions pod, which exceeds a default single-node kind cluster before the
|
||||
# Fleet can be exercised. These are smoke-only bounds; production resource
|
||||
# sizing remains deployment-owned.
|
||||
# Agones 1.49 otherwise requests 10,100 MiB of ephemeral storage for both its
|
||||
# controller and extensions pods, which exceeds a default single-node kind
|
||||
# cluster before the Fleet can be exercised. These are smoke-only bounds;
|
||||
# production resource sizing remains deployment-owned.
|
||||
helm upgrade --install agones agones/agones \
|
||||
--namespace agones-system --create-namespace \
|
||||
--version "$agones_version" \
|
||||
--set agones.crds.cleanup.enabled=true \
|
||||
--set agones.controller.replicas=1 \
|
||||
--set agones.controller.resources.requests.ephemeral-storage=128Mi \
|
||||
--set agones.controller.resources.limits.ephemeral-storage=512Mi \
|
||||
--set agones.extensions.replicas=1 \
|
||||
--set agones.extensions.resources.requests.ephemeral-storage=128Mi \
|
||||
--set agones.extensions.resources.limits.ephemeral-storage=512Mi \
|
||||
|
||||
Reference in New Issue
Block a user