CosmicClash/TRAINING.md

# Training the AI bot

Cosmic Clash bots are trained with reinforcement learning (self-play PPO): two
ships in a headless arena share one policy that learns by playing against
itself. Training runs in Python ([Godot RL Agents](https://github.com/edbeeching/godot_rl_agents)
bridge + Stable-Baselines3); the trained policy is exported to a small JSON
file and runs **inside the game** in pure GDScript — shipped bots need no
Python, no .NET, no network.

## How it fits together

- `Game/scenes/training.tscn` + `scripts/training_mode.gd` — headless self-play
  environment: two RL ships, randomized episode starts, goal rewards. Contains
  the vendored godot_rl_agents `Sync` node that talks TCP to the trainer.
- `scripts/ship_ai_controller.gd` — training-side bridge (observations,
  rewards, action mapping). `scripts/ship_observations.gd` is the *shared*
  observation builder — training and in-game inference must stay identical,
  so never fork it.
- `training/train.py` — PPO trainer; launches N parallel headless Godot
  instances (2 agents each) — from source by default, or from a pre-built
  binary via `--exported-binary` (see TRAINING_LINUX.md's "Exported-binary
  training" section; `training/export_linux.sh` builds it from
  `Game/export_presets.cfg`'s "Linux Training" preset).
- `training/export_policy.py` — SB3 checkpoint → JSON policy for the game.
- `scripts/ai_ship_controller.gd` + `scripts/policy_network.gd` — in-game
  inference (GDScript MLP forward pass).
- `training/evaluate.py` — pits two exported policies against each other and
  appends to `training/eval_history.json`.

## Hardware

The environment is our own headless Godot sim — fully cross-platform:

- **Any machine (e.g. the M4 Mac mini)**: fine for pipeline development,
  smoke runs, and short experiments. Env stepping is CPU-bound; the policy is
  a small MLP, so even CPU-only PPO updates are cheap.
- **Linux + NVIDIA GPU (e.g. the RTX 3090 box)**: recommended for real
  multi-hour/overnight runs. PyTorch CUDA works out of the box; more CPU
  cores also mean more parallel Godot instances (`--n-parallel`).

There is no hard GPU requirement (unlike Rocket League tooling) — a GPU
mainly speeds up learning updates on long runs.

For the Linux/3090 remote-training workflow (setup, throughput tuning,
auto-copying results back to the Mac, dashboard over the network), see
[TRAINING_LINUX.md](TRAINING_LINUX.md).

## Setup

Needs Python 3.10+ and a Godot 4.7 binary.

```bash
cd training
python3.12 -m venv .venv          # macOS: brew install python@3.12
.venv/bin/pip install -r requirements.txt
```

On Linux, download the Godot 4.7 Linux binary and point at it:

```bash
export GODOT_BIN=~/godot/Godot_v4.7.1-stable_linux.x86_64
```

(macOS default is `/Applications/Godot.app/Contents/MacOS/Godot`; override
with `GODOT_BIN` or `--godot_bin` if yours lives elsewhere.)

## Run a training session

Before starting a multi-hour curriculum run, execute the bounded policy and
orchestrator checks from the repository root:

```bash
make verify-training
```

The target changes into `training/` deliberately because the test modules
import sibling files such as `generation5.py` and `evaluate.py`.

```bash
cd training
.venv/bin/python train.py --experiment run01 --timesteps 20000000 --n-parallel 6 --speedup 16
```

- Checkpoints land in `training/checkpoints/run01/` every `--checkpoint-every`
  steps (default 100k), plus `final.zip` on exit (also written on Ctrl-C).
- Resume with `--resume checkpoints/run01/final.zip`.
- `--n-parallel` = Godot instances (2 agents each). Scale with CPU cores.
- `--speedup` = in-engine physics speedup. Raise until CPU saturates.
- `--wandb` mirrors logs to Weights & Biases (`pip install wandb` first).

Expect the smoke-run scale (~100k steps) to only learn crude ball-chasing;
real behaviour needs tens of millions of steps (hours on the 3090 box).

### Watch progress

```bash
.venv/bin/tensorboard --logdir training/logs
```

Key curves: `rollout/ep_rew_mean` (should trend up), `rollout/ep_len_mean`
(should trend *down* from 225 as goals end episodes early — 225 action steps
= the 30s episode timeout), `rollout/goal_rate` (fraction of recent episodes
that ended in an actual goal rather than timing out as a draw — the live
signal for "is the policy actually finishing more episodes by scoring",
since `ep_rew_mean` mixes that with dense reward-shaping (ball chasing/
touching) and doesn't isolate it).

### Reward/observation tuning

Reward weights are exported vars on `ShipAIController` (goal reward on
`TrainingMode`) — tune in `training.tscn`/scripts without touching the
trainer. If you change the *observation* layout (`ship_observations.gd`),
old checkpoints/exports become incompatible: retrain, and bump a note in
your experiment name.

## Export a checkpoint into the game

```bash
cd training
.venv/bin/python export_policy.py checkpoints/run01/final.zip ../Game/bots/hard.json
```

The exporter runs a parity check (JSON forward pass vs SB3 prediction) before
writing. Models live in `Game/bots/`.

## Evaluate progress between checkpoints

TensorBoard shows learning, but "is the new checkpoint actually *better*?"
needs head-to-head play:

```bash
.venv/bin/python export_policy.py checkpoints/run01/ppo_5000000_steps.zip /tmp/candidate.json
.venv/bin/python evaluate.py ../Game/bots/hard.json /tmp/candidate.json --episodes 40
```

Golden-goal episodes (first goal wins, timeout = draw), sides swapped halfway
for fairness, using the exact inference path that ships in-game. Every run
appends to `training/eval_history.json` — the long-term progress record.
Evaluate each new candidate against the previous promoted bot and a fixed
early reference to see absolute progress over time.

The evaluator launches Godot with the explicit headless display driver,
Compatibility renderer, dummy audio driver, and a temporary writable log path
so evaluation is reproducible on machines where the default renderer or
`user://` log location is unavailable. On 2026-09-01, the current Stage 6
export completed the full 100-episode paired evaluation against each of its
three league references on the independent seeds 19 and 43 (seed 1 was already
recorded in `eval_history.json`):

| Seed | Stage 3 reference | Stage 4 reference | Stage 5 reference |
| --- | ---: | ---: | ---: |
| 1 | 79–17–4 | 60–31–9 | 32–51–17 |
| 19 | 82–13–5 | 65–21–14 | 40–42–18 |
| 43 | 74–17–9 | 58–27–15 | 36–54–10 |

Cells are candidate wins–reference wins–draws. The candidate clears the
documented 15% reference-regression margin against Stage 3 and Stage 4 on all
three seeds, but fails against Stage 5 on seeds 1 and 43 and is effectively
even on seed 19. The physical-team splits stayed within the 20% diagnostic
ceiling. This is independent runtime evidence, not promotion evidence: Stage 6
remains open and requires another training attempt or an explicit human
decision after reviewing the Stage 5 regression.

If a model was trained with the locomotion mask on (curriculum stages 1, 2,
and 5 — see below), pass `--grounded-a`/`--grounded-b` for whichever side it's on.
The eval otherwise runs `AIShipController` fully unmasked regardless of how a
model was trained, so a grounded model's untrained vertical/pitch-roll output
reaches the ship as noise it never had to contend with during training —
this understates it, not a neutral comparison.

## Difficulty tiers

A bot is `(model, reaction_ticks, action_noise)` — configured on the Match
mode (`bot_model_path`, `bot_reaction_ticks`, `bot_action_noise` in
`match.tscn`) or any `AIShipController`:

- **Model**: the main lever. An early checkpoint *is* an easy bot — promote
  e.g. `easy.json` / `medium.json` / `hard.json` from different stages of one
  training run (verify the gaps with `evaluate.py`).
- **reaction_ticks** (default 8 = training cadence): higher = slower
  reactions, easier.
- **action_noise**: adds execution error, easier.

### Promoted bots (`Game/bots/promoted/`)

`Game/bots/*.json` is a flat, ever-growing dump of every experiment/curriculum
export — useful for `evaluate.py` and for A/B-ing arbitrary past checkpoints
against each other in the in-game Spectate dropdown (`main_menu.gd` lists
`Game/bots/` non-recursively, so anything one directory deeper is invisible
to it), but none of those filenames (`run07.json`, `curric-s3-no_draws.json`,
...) are meant to be *the* shipped bot — they get superseded constantly and
the automated curriculum pipeline (`run_training.sh`) only ever writes new
flat files there, never touching subdirectories.

`Game/bots/promoted/<tier>.json` is the small, curated, hand-maintained set
actually referenced by the shipped game — currently `easy.json` (promoted
2026-08-08 from generation 4's `20260806-1939-curric-s3-gauntlet`; this is
the 320M-step MultiDiscrete policy and the foundation for the planned
generation-5 curriculum below), `medium.json` (promoted 2026-08-17 from
generation 5's `20260816-2126-gen5-s4-handling-retry2` — Stage 4 attempt 3,
which the pipeline recorded as a *fail* on the 80% training-goal-rate gate at
0.7731, but which beats `easy.json` 65-22-13 in the 100-episode paired
evaluation, 87% non-draw and 12.6% physical-side imbalance, both inside the
Stage-4 bars; promoted by hand on gameplay feel, so its recorded
`decision: "fail"` in `generation5_state.json` is expected and not a
bookkeeping error) and `reference-grounded.json` (added for
generation 4 — a copy of generation 3's `curric-s5-aggression`, made before
the flat `Game/bots/` dump was scrapped for the redesign, kept as the
strongest grounded-era artifact and the fixed yardstick generations 1-3 were
all measured against; see "Generation 4"'s final report). `match.tscn`/
`spectate.tscn` point their `bot_model_path` exports here directly, so a
promoted file is never touched by training scripts, never overwritten by a
same-named future export, and never disturbed by pruning old experiment
files from the flat dump.

`hard.json` was promoted 2026-08-24 from generation 5's
`20260823-1734-gen5-s5-intercepts-retry2` — Stage 5 attempt 3, and like
`medium.json` above it is recorded as a *fail* in `generation5_state.json`.
Stage 5 blocked after three attempts on two telemetry floors: `goal_rate`
0.7369 against a 0.75 bar (marginal), and `productive_air_touch_fraction`
0.0001 against 0.005. The second is not a judgement about this policy — see
"Generation 5" below — the metric is quantised at 0.01 (one touch per ~100
episode logging window), so a 0.005 floor demands a productive air touch in
half of all windows, and no policy in the lineage has ever come close. On
everything else it is the strongest bot produced so far: `upright_fraction`
0.757 against a 0.40 floor (the pre-Round-6 lineage never exceeded 0.331),
`forward_motion_fraction` 0.479 against 0.20, and it beats `medium.json`
47-32-21 over 100 paired episodes.

Attempt 2 (`20260823-0258-gen5-s5-intercepts-retry1`) posts a much wider
margin against `medium.json` (63-23-14) and was the obvious alternative, but a
direct 100-episode head-to-head between the two finished 36-39 with 25 draws —
a dead heat, so the wider indirect margin does not reflect a real strength
difference. Attempt 3 was taken on the tiebreakers: it is the later checkpoint
(it resumed from attempt 2) and edges every telemetry metric. That head-to-head
also measured a 17% physical side imbalance (physical teams 0-1 = 29-46), which
looked worth investigating as a possible asymmetry in the arena or in
`ship_observations.gd`'s team-1 mirroring. **It is not — it is seed variance.**
A follow-up ran `hard.json` against *itself* (self-play, so any split is purely
positional and cannot be a strength difference) over 10 independent seeds at 30
episodes each: pooled 113-125 across 300 episodes, a 4.0% imbalance, sign test
p = 0.48, with team 1 ahead in only 3 of the 10 seeds. Per-seed imbalance
ranged from 0.0% to 43.3%, so swings far larger than the original observation
occur by chance at these episode counts.

The trap worth remembering: `evaluate.py --seed` defaults to 1, so every
evaluation in this file that did not pass `--seed` shares one paired
starting-state sequence, and seed 1 happens to favour team 1 (7-19 in the
self-play run above, 29-46 in the 100-episode head-to-head — same direction
because it is the same seed, not because it replicates). Two such runs are one
observation sampled twice, not independent confirmation. Vary the seed before
concluding anything from a side split. This also means the
`physical_side_imbalance_ceiling` gate in `generation5.py` is a single-seed
measurement and should be read as a coarse catastrophe check, not evidence
about side balance either way.

Every tier runs at full trained cadence (`reaction_ticks=8`, `action_noise=0`)
— the game does not manufacture difficulty gaps by handicapping a model. When
promoting, keep the tiers monotonic: a lower tier must never point at a policy
that beats the tier above it.

To promote a new bot into a tier: copy the chosen `Game/bots/<experiment>.json`
to `Game/bots/promoted/<tier>.json` (overwriting the old one), and note the
source experiment + date in this section. Do this for `medium.json`/
`hard.json` as later curriculum stages clear the bar against `easy.json` in
`evaluate.py`.

## Curriculum training

Training from scratch with self-play alone hands the network every skill
at once — finishing, defending, positioning, not stalling to a draw — off a
sparse ±40 goal reward. `train.py` has a `curriculum` flag group that stages
this the way you'd coach a human: score first, then also defend, then learn
that a draw is still a failure, and only then spend compute polishing general
movement. Each stage is a normal chained run — a new `--experiment` resumed
via `--resume checkpoints/<previous>/final.zip`, same as any other run —
just with different curriculum flags.

`curriculum.py` has run through four generations so far. Generation 1
(below) ran stages 1-6 to completion/block and is archived; generation 2
started a fresh stage 1 seeded from generation 1's last clean pass instead
of continuing to retry a stage that kept getting worse, but also failed 3
attempts; generation 3 replaced generation 2's single all-or-nothing unmask
stage with a gradual ramp, and also failed (worse, on its final attempt,
than either prior generation); generation 4 (the one `curriculum.py` actually
runs today) is a full redesign, not a further patch — see "Generation 4"
below, and "Generation 3" for why a fourth attempt at gating *when* the
policy could use full 3D controls was abandoned rather than retried again.

### Generation 1 (archived — see `curriculum_state_gen1.json`)

| Stage | Flags | What it teaches |
|---|---|---|
| 1 — score | `--opponent-mode inert --attack-goal-bias 1.0 --no-allow-vertical --no-allow-pitch-roll` | Team 1 is a do-nothing placeholder ship parked at its spawn (an effectively empty net); near-goal resets always target the goal the trainee attacks; the ship can't fly or pitch/roll, only drive and yaw. Isolated finishing practice. |
| 2 — defend too | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll` | Reintroduces a live opponent (self-play) and the default episode-start mix — the same near-goal state is now simultaneously a finishing chance for one side and a defensive save for the other. Locomotion stays grounded. |
| 3 — no draws | `--draw-penalty 5 --reset-std 0.3` | Training episodes are golden-goal (end at the *first* goal), so there's no in-episode goal-margin to penalize — `draw_penalty` is the closest available signal: a one-time penalty when an episode times out with no goal at all, on top of the existing per-tick `time_penalty`. Also lifts the locomotion mask (full 3D controls) by omitting `--allow-vertical`/`--allow-pitch-roll`; pair that with `--reset-std` since the policy never got a reward gradient on those axes before now, so expect a brief re-exploration wobble. |
| 4 — mechanics/refinement | *(no curriculum flags — plain `next_run.sh`)* | Stock self-play, full controls, default reward/start-state mix. This is what all runs before this feature already did. |
| 5 — aggression | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5` | **Resumes from stage 2 (`curric-s2-defend`), not stage 4** — see the regression note below. Retunes ball-pursuit reward weights (up from 0.02/0.002/0.4) for much more aggressive, constantly-chasing floor play, deliberately keeping the locomotion mask on so it can't reopen the stage-3 regression. Passed 2026-07-22 (41-47 vs grounded stage 2 — close, not yet a clear win). |
| 6 — unmask | `--opponent-mode self_play --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5 --airborne-penalty 0.003` | Re-opens full 3D controls on top of the aggression retune — this is the same grounded-checkpoint-to-full-3D transition that regressed stage 3, but this time paired with `airborne_penalty` (dense, scaled by height above the floor — see `ship_ai_controller.gd`) so the policy learns to *prefer* staying grounded through incentives instead of a hard mask, and can still pick up genuinely useful aerial/wall plays instead of never touching those axes. **Failed 3 attempts in a row** (25% → 20% → 15% win rate vs `curric-s5-aggression`) and blocked — see "Generation 2" below for what replaced it. |

> **Stages 3-4 regressed; stage 6 deliberately reopens the same transition
> with a mitigation.** The locomotion-mask inference bugfix (`8c15c46`)
> revealed that stage 3's evals up to that point had been running with an
> unfairly unmasked grounded reference. Re-evaluated fairly, `curric-s2-defend`
> (grounded) beats both `curric-s3-no_draws` (26-60) and `curric-s4-mechanics`
> (24-57) — lifting the locomotion mask to full 3D in stage 3 was a clear
> regression in floor play that self-play never earned back in 20M steps.
> Stage 5 sidesteps this by resuming and evaluating against stage 2 directly
> (`curriculum.py`'s `resume_from_experiment`/`reference_experiment` stage-dict
> overrides) instead of chaining through stages 3-4. Stage 6 is where full 3D
> flight comes back — not masked away this time, but discouraged via
> `airborne_penalty` and given ~12x the training time to settle. See
> `curriculum_state_gen1.json`'s log for the full eval numbers.

Stage 6 (`unmask`, `retry1`, `retry2`) all used identical flags —
`curriculum.py` always reuses `STAGES[stage_index]["flags"]` on retry, only
the resume checkpoint changes — so continued training just drifted the same
policy further rather than converging differently (25% → 20% → 15% win rate
vs `curric-s5-aggression`). After 3 failed attempts the script blocked for
human review; rather than pile up `retry4`, `retry5`, ... on a lineage that
kept getting worse, generation 2 (below) replaces it with a fresh stage 1.

### Generation 2 (archived — see `curriculum_state_gen2.json`)

`curriculum.py`'s `STAGES` list contained a single stage, `unmask`
(displays as stage 1 — `curric-s1-unmask`), which picks up exactly where
generation 1's regression analysis left off. It resumes directly from
`FOUNDATION_EXPERIMENT` (`curric-s5-aggression`'s own checkpoint — the last
stage that passed cleanly) via `resume_from_experiment`/`reference_experiment`
overrides, rather than re-running stages 1-5 or continuing generation 1's
drifted `retry2`:

`--opponent-mode self_play --velocity-to-ball-weight 0.08 --ball-distance-penalty 0.01 --ball-touch-reward 0.7 --airborne-penalty 0.003 --ball-velocity-to-goal-weight 0.06 --goal-reward 80 --draw-penalty 5`

Compared to generation 1's stage 6:
- `velocity_to_ball_weight` (0.05→0.08) and `ball_distance_penalty`
  (0.006→0.01) — the actual ball-chasing terms, unchanged since stage 5
  despite three failed attempts — plus `ball_touch_reward` (0.5→0.7).
- Two scoring-specific terms newly exposed via `train.py` (they already
  existed as `ship_ai_controller.gd`/`training_mode.gd` `@export`s, just not
  as CLI flags): `ball_velocity_to_goal_weight` (0.004 default → 0.06)
  rewards the ball actually moving toward the goal, not just being
  chased/touched; `goal_reward` (40 default → 80) is the terminal reward for
  scoring itself.
- `draw_penalty 5` (proven effective in stage 3 against passivity), which
  generation 1's stage 6 had never set — previously all carrot for scoring,
  no stick for never scoring.
- `reset_retry_checkpoint: True` on the stage dict, so if this stage itself
  fails and retries, `resume_checkpoint()` resets to `FOUNDATION_EXPERIMENT`
  again instead of drifting a failed attempt further — the specific bug that
  made generation 1's 3 retries monotonically worse instead of converging.

Deliberately not added: a cooldown/cap on `ball_velocity_to_goal_weight` to
guard against a bot farming near-misses (bouncing the ball toward goal
repeatedly without finishing) instead of actually scoring. Unlike the
touch-farming bug (see `ship_ai_controller.gd`'s `ball_touch_reward`
comments) this term is already direction-scaled by construction (it's a
velocity-toward-goal quantity, not an undirected contact count), so the risk
is theoretical rather than demonstrated. If this stage's eval shows high
`ball_velocity_to_goal_weight` accrual without a matching rise in actual
goals scored, that's the signal to add one.

Every experiment name `curriculum.py` generates is now timestamped
(`YYYYMMDD-HHMM-<name>`, e.g. `20260727-0930-curric-s1-unmask`), applied once
in `run_stage_attempt` — this keeps generation 2's names from colliding with
generation 1's plain ones (both checkpoint directories and TensorBoard run
names come straight from `--experiment`) and makes run order obvious in
TensorBoard without cross-referencing `curriculum_state.json`.

**Generation 2 also failed 3 attempts in a row**, landing at a stable
32% / 28% / 31% win rate vs `curric-s5-aggression` each time — the second
and third attempts each continued the *same* checkpoint lineage for another
full 240M steps with zero improvement, ruling out both the reward retune
above and "just needs more time" as fixes. Every attempt showed `train/std`
collapsing from ~0.30 to ~0.13-0.15 within the first ~10% of steps and never
recovering. See "Generation 3" below for the redesign this prompted.

### Generation 3 (current)

Generation 2's failures point at the *mechanism* of the transition, not the
reward weights: flipping `allow_vertical`/`allow_pitch_roll` from false to
true in one step let PPO's action-distribution std collapse on those axes
before the policy ever meaningfully explored them. Generation 3 replaces
that boolean mask with a float ramp (`vertical_ramp`/`pitch_roll_ramp` on
`ShipAIController`, 0.0-1.0, multiplying the axis's effect in `set_action`
instead of gating it) and spreads the transition across 4 stages instead of
1:

| Stage | `vertical-ramp`/`pitch-roll-ramp` | `airborne-penalty` | timesteps | gated |
|---|---|---|---|---|
| 1 — `unmask-ramp25` | 0.25 | 0.0 | 40M (~4h) | No — trains, checkpoints, always advances |
| 2 — `unmask-ramp50` | 0.5 | 0.001 | 40M (~4h) | No |
| 3 — `unmask-ramp75` | 0.75 | 0.002 | 40M (~4h) | No |
| 4 — `unmask` | 1.0 | 0.003 | 240M (~24h) | **Yes** — evaluated against `curric-s5-aggression`, same 15-point regression gate as every prior attempt |

The 3 warmup stages are deliberately ungated: they're waypoints en route to
the real, measured transition, not decisions in their own right, so
`curriculum.py`'s `main()` loop trains and checkpoints them and always
advances (no eval call, no retry logic — there's nothing to fail against).
Only the final `unmask` stage is evaluated, with the same reference bot,
opponent mode (`self_play`, not `frozen` — kept identical to every prior
attempt so a pass or fail cleanly isolates the ramp as the only variable),
and 240M-step budget as all 3 failed all-or-nothing attempts, for a direct
comparison. `airborne_penalty` ramps in step with the axes so it doesn't
fight a still-mostly-inert axis early on.

All the reward-shaping flags from generation 2's stage (`velocity-to-ball-weight`,
`ball-distance-penalty`, `ball-touch-reward`, `ball-velocity-to-goal-weight`,
`goal-reward`, `draw-penalty`) are unchanged and identical across all 4
stages, so the ramp is the sole studied variable.

`curriculum_state.json` was reset (generation 2's log archived to
`curriculum_state_gen2.json`) rather than continuing to log against a stage
list whose stage 0 no longer means what it used to.

**Open question, resolved 2026-08-04.** The gated `unmask` stage failed all
3 attempts: 29% → 30% → **24%** win rate vs `curric-s5-aggression` (the
third, worst by then) — landing in the same ~28-32% band the section above
flagged as "evidence the plateau isn't an exploration/collapse problem at
all." `train/std` collapsed from ~0.30 to ~0.13-0.15 within the first ~10%
of steps in every attempt of every generation regardless of hard-mask vs.
gradual-ramp mechanism, so gating *when* the axes were allowed to act never
addressed the actual cause. See "Generation 4" below for the redesign and
root-cause diagnosis this prompted, and `curriculum_state_gen3.json` for the
archived full log.

### Generation 4 (current) — action space redesign, not a further ramp patch

Three generations spent ~2 weeks trying different ways to gate *when* the
policy could use vertical thrust/pitch/roll on top of a continuous Gaussian
action space, and all three converged on the same failure: PPO's action
std collapsing within the first ~10% of steps and never recovering,
regardless of mechanism. Research into how self-play PPO bots that have
actually solved this class of problem (RLGym/RLBot's Necto/Nexto) approach
it turned up a structural difference — they don't gate control authority at
all; they train the full action space from step 1 using discrete/bucketed
actions, not a continuous Gaussian, plus reward/state-setter curriculum
instead of action masking.

**Root cause, verified against this project's own physics** (not assumed):
flight in this game is a *sustained set-point*, not an impulse. Ship mass
5.0, `vertical_thrust` 120 (`ship.gd`/`ship.tscn`), default gravity 9.8 m/s²
→ hovering requires *holding* `thrust.y ≈ 0.408` continuously. A Gaussian
whose mean sits near 0 and whose σ has collapsed to ~0.13 samples
`thrust.y ∈ [-0.4, 0.4]` — it can brush the hover value but can never *hold*
it long enough to earn the reward gradient that would move the mean. That's
a fixed point; no ramp on the axis's downstream *effect* (which is applied
*after* PPO samples the action) moves it, exactly as the "open question"
above speculated might be the case. Independently, this redesign also found
and fixed a real, previously-unnoticed bug unrelated to the action space:
`godot_rl`'s `godot_env.py` never marks an episode timeout as a truncation
(it returns the same `done` array for both term and trunc — see its own
`# TODO update API to term, trunc`), so PPO was bootstrapping `V(s_T)=0` on
every 30s draw in every generation to date instead of correctly estimating
the value of the state it timed out in.

**Action space**: switched to per-axis `MultiDiscrete` (7 heads, `nvec =
[5,5,5,5,5,5,2]`) instead of continuous `Box(7)` — see
`Game/scripts/ship_action_codec.gd`, the single source of truth for the
layout/decode shared by training and in-game inference. Not a single
lookup table (RLGym's approach for Rocket League's *coupled* car controls):
Cosmic Clash's 7 axes are near-independent thruster/torque channels, so a
curated combination table would throw away that factorization for no
benefit. `thrust_y`'s bins are deliberately asymmetric
(`-0.5, 0, 0.45, 0.75, 1.0`, vs. the symmetric `-1, -0.5, 0, 0.5, 1` on
every other axis) — a uniform-random policy over those 5 bins averages
0.34, just below the 0.408 hover point, so a fresh policy drifts gently
through the volume instead of pinning to the floor (symmetric bins) or
sticking to the ceiling (`ceiling_pull_strength` 11.5 > gravity 9.8). This
is the direct analogue of the RLGym/RLBot fix for the same failure mode
("add more jump actions to the discrete action parser"). `godot_rl`'s
`ActionSpaceProcessor` already emits `MultiDiscrete` with zero Python-side
changes when every action entry is `Discrete` — the only reason this
project's action space flattened to `Box(7)` before was that `turbo`
(binary) was mixed with continuous entries.

**Backward compatibility**: every export before generation 4 (e.g.
`Game/bots/promoted/easy.json`) has no `"action_space"` field in its JSON;
absence means `{"type": "continuous"}` and decodes through the exact same
path as before (`ShipActionCodec.from_continuous`, moved verbatim out of
`ai_ship_controller.gd`). `PolicyNetwork.gd`'s forward pass itself never
changed — only the caller's decode branches on the model's declared type.
`export_policy.py`'s parity check is now index-level for a `MultiDiscrete`
model (argmax per head's logit slice, compared against SB3's own
`deterministic=True` chosen index) rather than comparing clipped floats,
since a head-order mistake would otherwise train and export cleanly and
only surface as silently wrong in-game behaviour.

**No grounded stage.** Full action space live from step 1 — no successful
self-play RL bot in this problem class gates control authority, it's failed
9/9 attempts (3 generations × 3 attempts) here, and every prior generation's
checkpoints are a different, incompatible action/observation shape anyway
(nothing to resume from). 3 stages instead of a ramp:

| Stage | Opponent | Timesteps | Gated | What it teaches |
|---|---|---|---|---|
| 1 — `bootstrap` | `inert` | 40M (~4h) | No | Empty-net finishing from a random policy — no moving target, full action space from the start. |
| 2 — `selfplay` | `self_play` | 160M (~16h) | Yes, vs stage 1 | Where essentially all the learning happens. |
| 3 — `gauntlet` | `frozen` = stage 2's own export | 120M (~12h) | Yes, vs stage 2 | A stationary opponent for a low-variance measurement, and a check that self-play didn't converge to a fixed point that only beats itself. |

An "air drill" state-setter branch (`training_mode.gd`'s `air_drill_chance`,
new — ball spawned high, both ships spawned low and lateral, unsolvable
without climbing, kept clear of every wall so the RLGym-warned wall-bounce
exploit has no wall nearby to bounce off) runs at a constant rate across
*all* stages rather than being introduced late — gating *when* a skill gets
drilled would reproduce the exact "gate what the policy can do" pattern
that failed 3 generations running.

**Observations**: `ShipObservations.SIZE` grew 31 → 35 (own contact normal
+ an `in_contact` flag, appended — never inserted, see that file's
append-only invariant) so the value function can actually see the condition
`wall_contact_penalty` fires on, instead of predicting a reward with no
supporting signal.

**Reward shaping**: mostly unchanged — a farmability check on the existing
weights (`velocity_to_ball_weight`'s term telescopes to ~2.7 over a 20m
approach, well under `goal_reward`=80; not gameable) argues generation 2/3's
tuning was never the actual problem. Two changes: `airborne_penalty` is no
longer passed by any stage (previously ramped *up* in lockstep with the
axis generation 3 was trying to teach — directly adversarial to the goal of
genuine aerial play), and `tilt_penalty` dropped 4x (0.002 → 0.0005 default)
since an aerial approach to a high ball requires pitching. Deliberately
*not* added: a standalone air-touch reward — that's the exact exploit RLGym
warns about ("hits the ball off a wall high up instead of doing a real
aerial"); the air-drill state setter already makes aerial skill
instrumentally necessary to earn the existing ball-directed rewards.

**Exploration**: `--reset-std` (meaningless under `MultiDiscrete` — no
`log_std`) is replaced by `--reset-logits <scale>` (multiplies
`action_net`'s weights/bias, optionally scoped to specific heads via
`--reset-logits-heads`) for a deliberate post-diagnosis recovery, and more
importantly by `--entropy-floor` (`train.py`'s `EntropyFloorCallback`): a
*persistent* per-rollout controller nudging `ent_coef` to hold policy
entropy near a target that decays over the run, replacing the one-shot
`--reset-std` shock that reliably decayed away within ~10% of steps in
every prior generation with something that responds continuously instead of
once. `--ent-coef`'s default rose 0.0001 → 0.01 (tuned for `MultiDiscrete`'s
bounded ~10-nat entropy, not a Gaussian's unbounded differential entropy).
Per-head entropy (`train/entropy_head_<name>`) replaces the old aggregate
`train/std` scalar — it identifies *which* axis is collapsing instead of
one number for all seven.

**Validation before spending the full ~32h budget**: see the ladder below —
cheapest checks first (an offline action-space assertion, a headless Godot
boot, a 100k-step smoke run, export parity + an in-game round trip against
`easy.json`), then flight telemetry (`rollout/airborne_fraction`,
`mean_altitude`, `air_touch_fraction`, `vertical_thrust_mean` — leading
indicators visible from the first rollout instead of only in a win rate
measured a full run later), then a short controlled A/B (MultiDiscrete vs.
continuous, otherwise identical, ~20M steps each) before committing to the
full curriculum — every past generation bet a full day on an unfalsifiable
hypothesis, which is what made each failure expensive to diagnose.

### Air-touch metrics: which to gate on (changed 2026-08-24)

**Gate on `productive_air_touch_episode_fraction`.** It is 1.0 for an episode
containing at least one productive aerial and 0.0 otherwise, so meaned over
SB3's 100-episode buffer it reads directly as "what share of episodes contained
one".

**Never gate on `air_touch_fraction` or `productive_air_touch_fraction` again.**
Both divide by *total touches in the episode*, which makes them structurally
unusable as bars: a policy with a strong ground game accumulates many ground
touches, and those dilute the ratio for identical aerial behaviour. Stage 4
exists to improve exactly that ground game, so its success actively drove Stage
5's gate toward zero — the two stages were fighting each other. It also means
the only non-zero values those metrics ever logged came from degenerate episodes
whose single touch happened to be aerial (per-episode 1.0, hence the exactly
`0.0100` that was every run's maximum). They are kept only as continuity with
nine attempts of history.

`AIR_TOUCH_HEIGHT` also moved 5.0 → 3.0 the same day, so **air-touch figures
recorded before 2026-08-24 are not comparable with anything after it.** 5.0 was
never derived from anything; 3.0 is this project's existing airborne threshold
(`AIRBORNE_ALTITUDE_THRESHOLD` / `GROUND_HANDLING_HEIGHT`) and sits just above
the measured ~2.4m mean episode peak ball height. `_place_air_intercept`'s band
moved 8-14m → 6-10m with it — the two are **coupled and must move together**,
since at a 5m bar the 8-14m band was optimal (41.2% above-bar touches) and
lowering the band alone collapses it to 4.3%.

Three ball-altitude diagnostics were added alongside and are deliberately
ungated: `ball_mean_altitude`, `ball_peak_altitude` (per-episode max — the
number a drill's spawn band should be derived from), and
`ball_above_air_touch_fraction`. Nobody had ever measured where the ball goes
before building four rounds of aerial mechanisms on top of an assumed height.

1. `training/test_action_space.py` — offline, seconds. Catches a head-order
   mismatch, the single most likely silent killer (trains "fine" for 24h,
   produces garbage — e.g. pitch commands driving strafe thrusters — with no
   error).
2. `godot --headless --path Game res://scenes/training.tscn` with no
   trainer listening, 30s — catches `class_name`/observation-size
   regressions.
3. `.venv/bin/python train.py --experiment smoke --timesteps 100000
   --n-parallel 2` — confirms the `MultiDiscrete` handshake and new
   callback metrics emit.
4. `export_policy.py` on the smoke checkpoint (mandatory index-level parity
   check), then `evaluate.py <smoke>.json ../Game/bots/promoted/easy.json
   --episodes 4` — exercises the real GDScript decode path.
5. A short A/B: two 20M-step runs, identical except action space
   (`MultiDiscrete` vs. the old continuous `Box(7)`), comparing
   `rollout/airborne_fraction`. If discrete pulls meaningfully ahead, the
   32h curriculum is a justified bet; if both stay near zero, the
   hypothesis above is wrong and reward/compute explanations move to the
   front — cheaper than a 4th blind multi-day generation either way.

All curriculum flags default to leaving Godot's own `@export` defaults
alone (`train.py` only forwards a flag when you pass it), so ordinary runs
are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
`--opponent-model <path>` (for `frozen`), `--draw-penalty`,
`--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
`--air-drill-chance`, `--air-intercept-chance`, `--team-size`,
`--velocity-to-ball-weight`, `--forward-velocity-to-ball-weight`,
`--ball-distance-penalty`, `--ball-touch-reward`, `--airborne-penalty`,
`--tilt-penalty`, `--ground-tilt-penalty`, `--speed-reward-weight`,
`--ball-velocity-to-goal-weight`, `--goal-reward`, and
`--opponent-pool` with `--opponent-mode=league`.
(`--vertical-ramp`/`--pitch-roll-ramp` are gone — generation
4 has no locomotion mask/ramp to control.)

### Running it automatically

`training/curriculum.py` (started via `curriculum.sh`, same detached-tmux
pattern as `start_training.sh`) drives all stages end to end: for each
stage it runs `run_training.sh` (pull, train, export, commit+push) with that
stage's flags, then evaluates the resulting checkpoint against a reference
bot over 100 episodes — the previous stage's own passing export (stage 1 is
ungated, so this only applies to stages 2+). Once every stage passes, a
final (non-gating) report evaluates the result against both
`Game/bots/promoted/easy.json` (the shipped bot) and
`Game/bots/promoted/reference-grounded.json` (a copy of generation 3's
`curric-s5-aggression`, the strongest grounded-era artifact and the
yardstick generations 1-3 were all measured against) — those two numbers are
what actually answer "did generation 4 work?"

```bash
cd training
./curriculum.sh                                            # start/resume the curriculum
./curriculum.sh --seed-checkpoint checkpoints/some/final.zip # override stage 1's resume source for this run
```

The gate is deliberately lenient: it blocks a stage only on a **clear
regression** (the reference beating the candidate by 15+ points of win
rate), not "must show improvement." A 40-episode eval already misled us once
in this project — run11 was the first model to deliberately score a goal,
but its head-to-head eval read as a loss on sample noise alone. A strict
gate would have retried that stage forever for the wrong reason; a loose one
still catches a genuinely broken stage. Progress and every attempt's eval
result are logged to `curriculum_state.json` (committed alongside
`eval_history.json` after each attempt).

A stage gets up to 2 retries (3 attempts total) before the script stops and
asks for a human look — it will not retry indefinitely or advance past a
stage that keeps failing on its own. By default a retry resumes from that
stage's own previous attempt (no `reset_retry_checkpoint` stage override is
set in generation 4 — nothing yet suggests a retry needs to reset to a
clean upstream checkpoint the way generation 3's single `unmask` stage did;
add one if a stage's retries turn out to be drifting rather than
converging). Once you've looked at why a block happened (more timesteps? a
flag needs adjusting? the eval itself was misleading?), re-run with
`--force-retry` to try again or `--skip-to-next-stage` if you judge the
result good enough despite the gate.

Running a stage by hand (e.g. to experiment with flags before trusting the
orchestrator) still works exactly as the table above describes — just call
`next_run.sh`/`run_training.sh` directly with that stage's flags.

### Generation 5 follow-on

Generation 4's stage-3 export is the foundation rather than a throwaway
baseline: all generation-5 stages resume from
`checkpoints/20260806-1939-curric-s3-gauntlet/final.zip`. Its match results
are strong, but playtesting and its final telemetry expose the next learning
targets: it spends about 39% of play above the airborne threshold while only
about 0.04% of episode-level touches are aerial, and it often travels on its
side and strikes the ball with its roof. This is a successful scoring policy
that now needs control quality and a more productive use of flight.

Turbo remains forward-only for players and policies: it activates only with
positive forward thrust and multiplies the resulting combined thrust vector.
Generation 5 preserves the same control contract Stage 3 was trained under.

Generation 5 adds three episode telemetry signals to TensorBoard:
`upright_fraction` (low-altitude ticks with the
ship's up vector substantially upright), `forward_motion_fraction`
(low-altitude moving ticks whose planar velocity points broadly along the
nose), and `productive_air_touch_fraction` (touches above the aerial height
that send the ball toward the attack goal). The automatic gates are
deliberately conservative catastrophe floors; every stage records its final
500-rollout tail means in `generation5_state.json` so later threshold changes
can be based on evidence instead of a single watched match.

| Stage | Regime | Budget | Learning target | Advancement gate |
|---|---|---:|---|---|
| 4 — `handling` | Self-play, current balanced start mix | 40M (~4h) | Prefer upright, nose-led travel near the floor. Replace the orientation-agnostic speed bonus with low-altitude forward-motion shaping, and apply the stronger tilt cost only near the floor so pitch/roll remain free in genuine aerial play. | Before Stage 5: at least 80% training goal rate, at least 80% non-draw rate in the paired evaluation versus promoted Stage 3, no clear head-to-head regression, no more than 20% physical-side win imbalance, and the upright/forward-motion telemetry floors. |
| 5 — `intercepts` | Self-play with 40–50% improved air-intercept starts | 60M (~6h) | Convert existing vertical movement into useful aerial touches. Spawn a moving high ball on reachable attacking and defensive trajectories, away from walls, so contact is instrumental to scoring or saving rather than independently rewarded. | No clear regression versus Stage 4; productive aerial-touch telemetry must improve materially without reducing upright/forward-motion telemetry back to the Stage-3 baseline. |
| 6 — `league` | Live policy against a frozen opponent sampled per episode from Stage 3, Stage 4, and Stage 5 | 100M (~10h) | Prevent a narrow self-play equilibrium and consolidate ground handling, aerial interception, attack, and defence against distinct styles. | No clear head-to-head regression against any pool member plus conservative handling/aerial telemetry floors. Promote the passing result to `medium.json` after these recorded evaluations support it. |

Stage 7 teamplay remains deliberately unconfigured. The fixed roster
observation and `team_size` plumbing can run 2v2. The team-credit reward and
paired 2v2 evaluation prerequisites are implemented but remain opt-in:
`ShipAIController.team_touch_credit_weight` shares a bounded
fraction of a touch payout across same-team agents (default `0.0` preserves
all existing curricula), and `evaluate.py --team-size=2` runs the same policy
as a two-ship team with the existing paired side swap. Stage 7 stays
unconfigured until a recorded 2v2 evaluation establishes teamplay gates.

`training/generation5.py` implements Stages 4–6 separately from the completed
generation-4 orchestrator and state. It always begins Stage 4 from
`checkpoints/20260806-1939-curric-s3-gauntlet/final.zip`, then resumes each
later stage from its passing predecessor. `generation5.sh` runs it detached,
and retries/blocks use the same restart-safe pattern as the earlier
curriculum:

```bash
cd training
.venv/bin/python generation5.py --dry-run  # print and validate the next command only
./generation5.sh            # run/resume in tmux
tmux attach -t cosmic-generation5
cat generation5_state.json
```

Stage 4 removes the generic speed bonus, halves the old orientation-agnostic
closing reward, and adds a nose-led planar approach reward plus a tilt cost
that fades to zero by 3m altitude. Its scoring gates deliberately run before
Stage 5: becoming upright is not progress if the resulting policy stops
finishing goals. Stage 5 adds moving high-ball intercept
starts aimed toward real goals rather than a standalone air-touch reward.

**Stage 4 was closed by human override on 2026-08-17**, not by the automatic
gate. `20260816-2126-gen5-s4-handling-retry2` exhausted all three attempts and
the pipeline recorded `decision: "fail"`, on the training goal-rate floor alone
(0.7731 vs 0.80). Every other gate passed — 65-22-13 versus
`promoted/easy.json`, 87% non-draw against the 80% floor, 12.6% physical-side
imbalance against the 20% ceiling, and both handling telemetry floors clear
(`upright_fraction` 0.772 vs 0.45, `forward_motion_fraction` 0.315 vs 0.25) —
and the round's three attempts improved the training goal rate monotonically
(0.537 → 0.683 → 0.773). The same checkpoint also plays well enough by hand to
have been promoted to `medium.json`. `generation5_state.json` therefore has
that log entry's `decision` flipped to `"pass"` with a `decision_override`
block recording the original verdict and reasoning, and `stage_index`/
`attempt`/`status` advanced to Stage 5 attempt 1 — which is what
`passing_entry()` needs to resolve Stage 5's resume checkpoint and evaluation
reference, and what `league_pool()` will later need at Stage 6. Prefer this
edit over `--skip-to-next-stage`: that flag advances `stage_index` without
marking anything as passing, so the run dies immediately with `RuntimeError:
No passing generation-5 stage index 0`. Note that retry2 already clears Stage
5's own 0.75 goal-rate floor; the 0.80 Stage-4 figure was always the stricter
of the two.

**Stage 5 (`intercepts`) blocked after its own three attempts on 2026-08-18**,
all on the same single floor: `rollout/productive_air_touch_fraction` stayed
exactly 0.0 across a continuous 180M-step lineage (retries resume the
previous attempt's checkpoint, so this is one training run, not three), while
`air_touch_fraction` sat at noise level (0.00008 → 0.00006 → 0.00006) and
`goal_rate`/`upright_fraction`/`forward_motion_fraction` all kept improving on
the same budget — a dead-flat metric next to ones that keep moving, the same
missing-mechanism signature as Stage 4's original plateau, not a slow-learning
one. The cause: `forward_velocity_to_ball_weight`, the term that actually
taught ground pursuit, is hard-gated below `GROUND_HANDLING_HEIGHT` and does
nothing in the air, so Stage 5's `air_intercept_chance` was asking for aerial
pursuit with only the generic, orientation-agnostic `velocity_to_ball_weight`
(0.04) to learn it from. `air_approach_weight` (`ship_ai_controller.gd`) adds
the airborne mirror — nose-first 3D closing speed on the ball, active above
`GROUND_HANDLING_HEIGHT`, no uprightness multiplier since a real aerial
requires pitching away from level — set to 0.15 to match
`forward_velocity_to_ball_weight`'s proven magnitude, and folded into
`HANDLING_REWARD_FLAGS` so Stage 6 inherits it too. The three blocked attempts
were deleted and Stage 5 restarts from Stage 4's checkpoint with the new term,
same reasoning as every previous mechanism change: don't resume a policy
shaped by an absent term into one where it now exists.

**`air_approach_weight` alone did not fix it.** A further three attempts
(180M more steps, 360M cumulative across all six Stage-5 attempts, 2026-08-18
to 2026-08-19) closed with `productive_air_touch_fraction` still exactly 0.0
and `air_touch_fraction` still at noise level, while `goal_rate` kept clearing
its (lower) floor. Working out the actual physics instead of retuning another
number found the real gap: an unredirected `_place_air_intercept` ball
(spawned 6-12m up, aimed at a goal whose collision box sits at ~0-1.5m) sags
well short of the goal from gravity alone over the required flight
distance — it does not auto-score — so it just falls to the floor, where the
already-solved ground game collects the exact same `goal_reward`/
`ball_touch_reward` regardless of whether anything touched it in the air.
Nothing in the reward ever made a genuinely aerial touch worth more than
waiting the second or two for the ball to land, so `air_approach_weight`'s
dense closing-speed shaping had nothing to reinforce — a second,
independent missing-incentive gap in the same stage, not a training-duration
problem. `air_touch_bonus_weight` (`ship_ai_controller.gd`) closes it
directly: an event bonus on top of `ball_touch_reward`, paid only for a
touch that is both above `AIR_TOUCH_HEIGHT` and goal-directed, scaled by the
exact same alignment factor already gating the base touch reward —
conjunctive, not standalone, so it targets exactly the behaviour
`productive_air_touch_fraction` measures without reopening the RLGym
wall-bounce exploit the original "no standalone air-touch reward" decision
(above) was written to avoid: air-intercept/air-drill spawns are kept away
from every wall by construction. Set to 0.5 (roughly `ball_touch_reward`'s
own magnitude) and folded into `HANDLING_REWARD_FLAGS`. The three blocked
attempts were deleted and Stage 5 restarts from Stage 4's checkpoint again
with both terms active.

**Neither reward term was the problem — the drill was unsolvable.** The
third set of three attempts blocked on `productive_air_touch_fraction=0.0`
yet again, but this time the surrounding telemetry told a different story
from Rounds 7-8: the ship had measurably left the floor
(`airborne_fraction` 0.223 → 0.258, `mean_altitude` 2.59 → 3.25,
`vertical_thrust_mean` 0.004 → 0.063, `grounded_upright_fraction` 0.352 →
0.182), and watching a game confirmed it now chases and strikes the ball in
the air. The reward work had worked; the metric could not see it, because
`productive_air_touch_fraction` counts only touches with the *ball* above
`AIR_TOUCH_HEIGHT` (5 m), and `_place_air_intercept`'s spawn geometry never
produced a reachable one.

Simulating that spawn distribution against the ship's real flight envelope
(`vertical_thrust` 120 / `mass` 5 = 24 m/s², less 9.8 gravity, with
`drag_coefficient` 0.98/tick capping climb near 12 m/s) settles it
arithmetically. A ball spawned 6-12 m up moving 6-11 m/s is above 5 m for a
median of **0.80 s**, while the ship spawned 7-13 m behind it, 3-10 m below
it, and **at a dead stop**. An *ideal* interceptor — point mass, instant
attitude, no righting torque, isotropic thrust, zero reaction delay — makes
that touch in **0.00%** of episodes and reaches the ball at all before it
lands in 0.5%. Six attempts and 360M steps were spent optimising against an
event the environment could not produce.

The fix is in `_place_air_intercept` (see its `AIR_INTERCEPT_*` constants):
ball higher and slower, ship closer and already carrying planar speed toward
it. The dead-stop spawn was the single largest factor — a ship in real play
is already moving, and starting from rest spent most of the window just
building speed. The same simulation now puts an ideal interceptor at ~98%
reach and ~37% above 5 m, so the 0.005 floor has real headroom.
`AIR_TOUCH_HEIGHT` deliberately stays at 5.0: lowering the bar to meet a
broken drill would make the metric incomparable with earlier generations.

Unlike every earlier round this does **not** restart from Stage 4's
checkpoint. That rule exists because a changed reward function invalidates
the learned value function; here the reward function is untouched and only
the environment's state distribution moves, so the existing policy — which
already learned to fly — is exactly what should be pointed at a now-reachable
target. `generation5_state.json` carries a one-shot `resume_override` for
this, consumed the first time `resume_checkpoint()` uses it.

The general lesson, and the one worth carrying into later generations: when
a telemetry floor reads *exactly* zero while the behaviour it is meant to
measure is visibly happening, check that the environment can produce the
event at all before touching the reward function again. Cost of not checking
here: three rounds and 360M timesteps.

**The gate itself was wrong, and so was its bar (2026-08-24).** Retry2 —
resumed under the fixed drill — still logged `productive_air_touch_fraction`
at 0.0. Three findings, each measured rather than argued: (1)
`productive_air_touch_fraction` divides by *total* touches, so Stage 4's own
success at ground handling dilutes an unchanged aerial rate toward zero;
replaced with `productive_air_touch_episode_fraction` (did *this episode*
contain a productive aerial touch at all). (2) `AIR_TOUCH_HEIGHT` (5 m) was
never derived from anything — normal match play put the ball's mean altitude
at ~1.6 m and its average episode peak at only ~2.4 m, clearing 5 m barely 5%
of the time — so it was lowered to 3.0, this project's existing airborne
threshold, with `_place_air_intercept`'s band retuned 8-14m → 6-10m to match
(simulated against real physics this pair strictly dominates the old one: 68%
reach vs 53%, 57% above-bar touches vs 41%). (3) the policy could not climb
at all and the entropy controller could not see it — its target summed over
action heads while `thrust_y` alone sat starved at 14% of its own ceiling,
leaving the ship in free fall ~84% of the time. Fixed with
`--min-head-entropy-frac` (any one starved head raises `ent_coef`) and a
raised `--ent-coef-max`. Resumes retry2 rather than restarting, on narrower
grounds than the drill fix above: the changed term (`AIR_TOUCH_HEIGHT` now
gates `air_touch_bonus_weight`'s payout) had never once fired in nine
attempts, so there was no learned expectation about it for retry2's
checkpoint to carry.

**Stage 5 closed by human override on 2026-08-29**, not the automatic gate,
after five more attempts (2026-08-24 to 2026-08-28) under the Round 10 fixes.
`productive_air_touch_episode_fraction` read 0.00004, 0.00006, 0.00002,
0.00018, 0.00006 across them — no trend toward the 0.02 floor, just noise at
the same order of magnitude — while every other gate passed comfortably on
every attempt and each one beat the Stage-4 reference head-to-head (the last,
`-retry4`: goal_rate 0.796 vs 0.72, upright 0.778 vs 0.40, forward_motion
0.493 vs 0.20, eval 53-26-21 with balanced sides 29-11 / 24-15).
`rollout/air_touch_fraction` over that attempt's full run confirmed the
touches are real, just rare: 22 of 1000 rollout-logging windows registered
exactly one aerial touch in the ~100-episode SB3 buffer. The 0.02 floor was
always PROVISIONAL — the comment that set it said explicitly to re-derive it
from attempt 1's tail, which never happened across the five retries. Lowered
to 0.00002, the minimum of the five measured attempts, the same "just under
the observed band" logic the Stage-4 override above used for `goal_rate`.
`generation5_state.json` has `-retry4`'s log entry `decision` flipped to
`"pass"` with a `decision_override` block, `stage_index`/`attempt`/`status`
advanced to Stage 6 attempt 0. Stage 6's own `0.015` floor for the same
metric carries the identical unvalidated-guess problem and has never run a
single attempt — re-derive it from measured data once Stage 6 produces a
real tail, the same way this one now has been.

Stage 6's `league` opponent mode samples a historical exported policy at each
episode reset. Each later stage preserves the preceding shaping and adds one
new difficulty. Stage 6 now also reserves 10% each for wall-play and
pre-rebound states; these starts are generated by `TrainingMode` and are not
present in Stages 4–5. The generation-5 orchestrator evaluates every candidate
against every reference on three independent paired seeds (`1, 19, 43`) before
advancing; pass `--evaluation-seeds` only when deliberately running a
different, recorded experiment. This avoids promoting a policy from a single
side-biased starting-state sequence.

The physical-side gate is separate from the model-vs-model score. A paired
side swap can make an identical policy appear perfectly balanced overall even
when the Player 2 ship never functions. This caught the canonical action-frame
bug exposed by Stage 3's pitch/roll use: team 1 observations are rotated 180°
about Y, but rotation commands feed world-space torque, so team 1 pitch and
roll must be rotated back (X/Z signs inverted). Thrust remains unchanged
because it is applied through the ship's local basis.

## Self-play notes

By default both ships share the live policy (mirrored, team-relative
observations — see `ship_observations.gd`), so training is against the
current self. `--opponent-mode inert`/`frozen` (see Curriculum training
above) replace that with a placeholder or a fixed exported policy for one
side of a run; `frozen` is a single-fixed-model slice of full league play.
Fixed-opponent training against a *pool* of past checkpoints sampled per
episode (to avoid strategy collapse on long self-play runs) is still
deferred — see TODO.md.