CosmicClash/TRAINING.md

# Training the AI bot

Cosmic Clash bots are trained with reinforcement learning (self-play PPO): two
ships in a headless arena share one policy that learns by playing against
itself. Training runs in Python ([Godot RL Agents](https://github.com/edbeeching/godot_rl_agents)
bridge + Stable-Baselines3); the trained policy is exported to a small JSON
file and runs **inside the game** in pure GDScript — shipped bots need no
Python, no .NET, no network.

## How it fits together

- `Game/scenes/training.tscn` + `scripts/training_mode.gd` — headless self-play
  environment: two RL ships, randomized episode starts, goal rewards. Contains
  the vendored godot_rl_agents `Sync` node that talks TCP to the trainer.
- `scripts/ship_ai_controller.gd` — training-side bridge (observations,
  rewards, action mapping). `scripts/ship_observations.gd` is the *shared*
  observation builder — training and in-game inference must stay identical,
  so never fork it.
- `training/train.py` — PPO trainer; launches N parallel headless Godot
  instances (2 agents each) — from source by default, or from a pre-built
  binary via `--exported-binary` (see TRAINING_LINUX.md's "Exported-binary
  training" section; `training/export_linux.sh` builds it from
  `Game/export_presets.cfg`'s "Linux Training" preset).
- `training/export_policy.py` — SB3 checkpoint → JSON policy for the game.
- `scripts/ai_ship_controller.gd` + `scripts/policy_network.gd` — in-game
  inference (GDScript MLP forward pass).
- `training/evaluate.py` — pits two exported policies against each other and
  appends to `training/eval_history.json`.

## Hardware

The environment is our own headless Godot sim — fully cross-platform:

- **Any machine (e.g. the M4 Mac mini)**: fine for pipeline development,
  smoke runs, and short experiments. Env stepping is CPU-bound; the policy is
  a small MLP, so even CPU-only PPO updates are cheap.
- **Linux + NVIDIA GPU (e.g. the RTX 3090 box)**: recommended for real
  multi-hour/overnight runs. PyTorch CUDA works out of the box; more CPU
  cores also mean more parallel Godot instances (`--n-parallel`).

There is no hard GPU requirement (unlike Rocket League tooling) — a GPU
mainly speeds up learning updates on long runs.

For the Linux/3090 remote-training workflow (setup, throughput tuning,
auto-copying results back to the Mac, dashboard over the network), see
[TRAINING_LINUX.md](TRAINING_LINUX.md).

## Setup

Needs Python 3.10+ and a Godot 4.7 binary.

```bash
cd training
python3.12 -m venv .venv          # macOS: brew install python@3.12
.venv/bin/pip install -r requirements.txt
```

On Linux, download the Godot 4.7 Linux binary and point at it:

```bash
export GODOT_BIN=~/godot/Godot_v4.7.1-stable_linux.x86_64
```

(macOS default is `/Applications/Godot.app/Contents/MacOS/Godot`; override
with `GODOT_BIN` or `--godot_bin` if yours lives elsewhere.)

## Run a training session

```bash
cd training
.venv/bin/python train.py --experiment run01 --timesteps 20000000 --n-parallel 6 --speedup 16
```

- Checkpoints land in `training/checkpoints/run01/` every `--checkpoint-every`
  steps (default 100k), plus `final.zip` on exit (also written on Ctrl-C).
- Resume with `--resume checkpoints/run01/final.zip`.
- `--n-parallel` = Godot instances (2 agents each). Scale with CPU cores.
- `--speedup` = in-engine physics speedup. Raise until CPU saturates.
- `--wandb` mirrors logs to Weights & Biases (`pip install wandb` first).

Expect the smoke-run scale (~100k steps) to only learn crude ball-chasing;
real behaviour needs tens of millions of steps (hours on the 3090 box).

### Watch progress

```bash
.venv/bin/tensorboard --logdir training/logs
```

Key curves: `rollout/ep_rew_mean` (should trend up), `rollout/ep_len_mean`
(should trend *down* from 225 as goals end episodes early — 225 action steps
= the 30s episode timeout), `rollout/goal_rate` (fraction of recent episodes
that ended in an actual goal rather than timing out as a draw — the live
signal for "is the policy actually finishing more episodes by scoring",
since `ep_rew_mean` mixes that with dense reward-shaping (ball chasing/
touching) and doesn't isolate it).

### Reward/observation tuning

Reward weights are exported vars on `ShipAIController` (goal reward on
`TrainingMode`) — tune in `training.tscn`/scripts without touching the
trainer. If you change the *observation* layout (`ship_observations.gd`),
old checkpoints/exports become incompatible: retrain, and bump a note in
your experiment name.

## Export a checkpoint into the game

```bash
cd training
.venv/bin/python export_policy.py checkpoints/run01/final.zip ../Game/bots/hard.json
```

The exporter runs a parity check (JSON forward pass vs SB3 prediction) before
writing. Models live in `Game/bots/`.

## Evaluate progress between checkpoints

TensorBoard shows learning, but "is the new checkpoint actually *better*?"
needs head-to-head play:

```bash
.venv/bin/python export_policy.py checkpoints/run01/ppo_5000000_steps.zip /tmp/candidate.json
.venv/bin/python evaluate.py ../Game/bots/hard.json /tmp/candidate.json --episodes 40
```

Golden-goal episodes (first goal wins, timeout = draw), sides swapped halfway
for fairness, using the exact inference path that ships in-game. Every run
appends to `training/eval_history.json` — the long-term progress record.
Evaluate each new candidate against the previous promoted bot and a fixed
early reference to see absolute progress over time.

If a model was trained with the locomotion mask on (curriculum stages 1, 2,
and 5 — see below), pass `--grounded-a`/`--grounded-b` for whichever side it's on.
The eval otherwise runs `AIShipController` fully unmasked regardless of how a
model was trained, so a grounded model's untrained vertical/pitch-roll output
reaches the ship as noise it never had to contend with during training —
this understates it, not a neutral comparison.

## Difficulty tiers

A bot is `(model, reaction_ticks, action_noise)` — configured on the Match
mode (`bot_model_path`, `bot_reaction_ticks`, `bot_action_noise` in
`match.tscn`) or any `AIShipController`:

- **Model**: the main lever. An early checkpoint *is* an easy bot — promote
  e.g. `easy.json` / `medium.json` / `hard.json` from different stages of one
  training run (verify the gaps with `evaluate.py`).
- **reaction_ticks** (default 8 = training cadence): higher = slower
  reactions, easier.
- **action_noise**: adds execution error, easier.

### Promoted bots (`Game/bots/promoted/`)

`Game/bots/*.json` is a flat, ever-growing dump of every experiment/curriculum
export — useful for `evaluate.py` and for A/B-ing arbitrary past checkpoints
against each other in the in-game Spectate dropdown (`main_menu.gd` lists
`Game/bots/` non-recursively, so anything one directory deeper is invisible
to it), but none of those filenames (`run07.json`, `curric-s3-no_draws.json`,
...) are meant to be *the* shipped bot — they get superseded constantly and
the automated curriculum pipeline (`run_training.sh`) only ever writes new
flat files there, never touching subdirectories.

`Game/bots/promoted/<tier>.json` is the small, curated, hand-maintained set
actually referenced by the shipped game — currently just `easy.json`
(promoted 2026-07-24 from `curric-s6-unmask`, the strongest checkpoint at the
time). `match.tscn`/`spectate.tscn` point their `bot_model_path` exports here
directly, so a promoted file is never touched by training scripts, never
overwritten by a same-named future export, and never disturbed by pruning old
experiment files from the flat dump.

To promote a new bot into a tier: copy the chosen `Game/bots/<experiment>.json`
to `Game/bots/promoted/<tier>.json` (overwriting the old one), and note the
source experiment + date in this section. Do this for `medium.json`/
`hard.json` as later curriculum stages clear the bar against `easy.json` in
`evaluate.py`.

## Curriculum training

Training from scratch with self-play alone hands the network every skill
at once — finishing, defending, positioning, not stalling to a draw — off a
sparse ±40 goal reward. `train.py` has a `curriculum` flag group that stages
this the way you'd coach a human: score first, then also defend, then learn
that a draw is still a failure, and only then spend compute polishing general
movement. Each stage is a normal chained run — a new `--experiment` resumed
via `--resume checkpoints/<previous>/final.zip`, same as any other run —
just with different curriculum flags.

`curriculum.py` has run through three generations so far. Generation 1
(below) ran stages 1-6 to completion/block and is archived; generation 2
started a fresh stage 1 seeded from generation 1's last clean pass instead
of continuing to retry a stage that kept getting worse, but also failed 3
attempts; generation 3 (the one `curriculum.py` actually runs today)
replaces generation 2's single all-or-nothing unmask stage with a gradual
ramp — see "Generation 3" below.

### Generation 1 (archived — see `curriculum_state_gen1.json`)

| Stage | Flags | What it teaches |
|---|---|---|
| 1 — score | `--opponent-mode inert --attack-goal-bias 1.0 --no-allow-vertical --no-allow-pitch-roll` | Team 1 is a do-nothing placeholder ship parked at its spawn (an effectively empty net); near-goal resets always target the goal the trainee attacks; the ship can't fly or pitch/roll, only drive and yaw. Isolated finishing practice. |
| 2 — defend too | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll` | Reintroduces a live opponent (self-play) and the default episode-start mix — the same near-goal state is now simultaneously a finishing chance for one side and a defensive save for the other. Locomotion stays grounded. |
| 3 — no draws | `--draw-penalty 5 --reset-std 0.3` | Training episodes are golden-goal (end at the *first* goal), so there's no in-episode goal-margin to penalize — `draw_penalty` is the closest available signal: a one-time penalty when an episode times out with no goal at all, on top of the existing per-tick `time_penalty`. Also lifts the locomotion mask (full 3D controls) by omitting `--allow-vertical`/`--allow-pitch-roll`; pair that with `--reset-std` since the policy never got a reward gradient on those axes before now, so expect a brief re-exploration wobble. |
| 4 — mechanics/refinement | *(no curriculum flags — plain `next_run.sh`)* | Stock self-play, full controls, default reward/start-state mix. This is what all runs before this feature already did. |
| 5 — aggression | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5` | **Resumes from stage 2 (`curric-s2-defend`), not stage 4** — see the regression note below. Retunes ball-pursuit reward weights (up from 0.02/0.002/0.4) for much more aggressive, constantly-chasing floor play, deliberately keeping the locomotion mask on so it can't reopen the stage-3 regression. Passed 2026-07-22 (41-47 vs grounded stage 2 — close, not yet a clear win). |
| 6 — unmask | `--opponent-mode self_play --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5 --airborne-penalty 0.003` | Re-opens full 3D controls on top of the aggression retune — this is the same grounded-checkpoint-to-full-3D transition that regressed stage 3, but this time paired with `airborne_penalty` (dense, scaled by height above the floor — see `ship_ai_controller.gd`) so the policy learns to *prefer* staying grounded through incentives instead of a hard mask, and can still pick up genuinely useful aerial/wall plays instead of never touching those axes. **Failed 3 attempts in a row** (25% → 20% → 15% win rate vs `curric-s5-aggression`) and blocked — see "Generation 2" below for what replaced it. |

> **Stages 3-4 regressed; stage 6 deliberately reopens the same transition
> with a mitigation.** The locomotion-mask inference bugfix (`8c15c46`)
> revealed that stage 3's evals up to that point had been running with an
> unfairly unmasked grounded reference. Re-evaluated fairly, `curric-s2-defend`
> (grounded) beats both `curric-s3-no_draws` (26-60) and `curric-s4-mechanics`
> (24-57) — lifting the locomotion mask to full 3D in stage 3 was a clear
> regression in floor play that self-play never earned back in 20M steps.
> Stage 5 sidesteps this by resuming and evaluating against stage 2 directly
> (`curriculum.py`'s `resume_from_experiment`/`reference_experiment` stage-dict
> overrides) instead of chaining through stages 3-4. Stage 6 is where full 3D
> flight comes back — not masked away this time, but discouraged via
> `airborne_penalty` and given ~12x the training time to settle. See
> `curriculum_state_gen1.json`'s log for the full eval numbers.

Stage 6 (`unmask`, `retry1`, `retry2`) all used identical flags —
`curriculum.py` always reuses `STAGES[stage_index]["flags"]` on retry, only
the resume checkpoint changes — so continued training just drifted the same
policy further rather than converging differently (25% → 20% → 15% win rate
vs `curric-s5-aggression`). After 3 failed attempts the script blocked for
human review; rather than pile up `retry4`, `retry5`, ... on a lineage that
kept getting worse, generation 2 (below) replaces it with a fresh stage 1.

### Generation 2 (archived — see `curriculum_state_gen2.json`)

`curriculum.py`'s `STAGES` list contained a single stage, `unmask`
(displays as stage 1 — `curric-s1-unmask`), which picks up exactly where
generation 1's regression analysis left off. It resumes directly from
`FOUNDATION_EXPERIMENT` (`curric-s5-aggression`'s own checkpoint — the last
stage that passed cleanly) via `resume_from_experiment`/`reference_experiment`
overrides, rather than re-running stages 1-5 or continuing generation 1's
drifted `retry2`:

`--opponent-mode self_play --velocity-to-ball-weight 0.08 --ball-distance-penalty 0.01 --ball-touch-reward 0.7 --airborne-penalty 0.003 --ball-velocity-to-goal-weight 0.06 --goal-reward 80 --draw-penalty 5`

Compared to generation 1's stage 6:
- `velocity_to_ball_weight` (0.05→0.08) and `ball_distance_penalty`
  (0.006→0.01) — the actual ball-chasing terms, unchanged since stage 5
  despite three failed attempts — plus `ball_touch_reward` (0.5→0.7).
- Two scoring-specific terms newly exposed via `train.py` (they already
  existed as `ship_ai_controller.gd`/`training_mode.gd` `@export`s, just not
  as CLI flags): `ball_velocity_to_goal_weight` (0.004 default → 0.06)
  rewards the ball actually moving toward the goal, not just being
  chased/touched; `goal_reward` (40 default → 80) is the terminal reward for
  scoring itself.
- `draw_penalty 5` (proven effective in stage 3 against passivity), which
  generation 1's stage 6 had never set — previously all carrot for scoring,
  no stick for never scoring.
- `reset_retry_checkpoint: True` on the stage dict, so if this stage itself
  fails and retries, `resume_checkpoint()` resets to `FOUNDATION_EXPERIMENT`
  again instead of drifting a failed attempt further — the specific bug that
  made generation 1's 3 retries monotonically worse instead of converging.

Deliberately not added: a cooldown/cap on `ball_velocity_to_goal_weight` to
guard against a bot farming near-misses (bouncing the ball toward goal
repeatedly without finishing) instead of actually scoring. Unlike the
touch-farming bug (see `ship_ai_controller.gd`'s `ball_touch_reward`
comments) this term is already direction-scaled by construction (it's a
velocity-toward-goal quantity, not an undirected contact count), so the risk
is theoretical rather than demonstrated. If this stage's eval shows high
`ball_velocity_to_goal_weight` accrual without a matching rise in actual
goals scored, that's the signal to add one.

Every experiment name `curriculum.py` generates is now timestamped
(`YYYYMMDD-HHMM-<name>`, e.g. `20260727-0930-curric-s1-unmask`), applied once
in `run_stage_attempt` — this keeps generation 2's names from colliding with
generation 1's plain ones (both checkpoint directories and TensorBoard run
names come straight from `--experiment`) and makes run order obvious in
TensorBoard without cross-referencing `curriculum_state.json`.

**Generation 2 also failed 3 attempts in a row**, landing at a stable
32% / 28% / 31% win rate vs `curric-s5-aggression` each time — the second
and third attempts each continued the *same* checkpoint lineage for another
full 240M steps with zero improvement, ruling out both the reward retune
above and "just needs more time" as fixes. Every attempt showed `train/std`
collapsing from ~0.30 to ~0.13-0.15 within the first ~10% of steps and never
recovering. See "Generation 3" below for the redesign this prompted.

### Generation 3 (current)

Generation 2's failures point at the *mechanism* of the transition, not the
reward weights: flipping `allow_vertical`/`allow_pitch_roll` from false to
true in one step let PPO's action-distribution std collapse on those axes
before the policy ever meaningfully explored them. Generation 3 replaces
that boolean mask with a float ramp (`vertical_ramp`/`pitch_roll_ramp` on
`ShipAIController`, 0.0-1.0, multiplying the axis's effect in `set_action`
instead of gating it) and spreads the transition across 4 stages instead of
1:

| Stage | `vertical-ramp`/`pitch-roll-ramp` | `airborne-penalty` | timesteps | gated |
|---|---|---|---|---|
| 1 — `unmask-ramp25` | 0.25 | 0.0 | 40M (~4h) | No — trains, checkpoints, always advances |
| 2 — `unmask-ramp50` | 0.5 | 0.001 | 40M (~4h) | No |
| 3 — `unmask-ramp75` | 0.75 | 0.002 | 40M (~4h) | No |
| 4 — `unmask` | 1.0 | 0.003 | 240M (~24h) | **Yes** — evaluated against `curric-s5-aggression`, same 15-point regression gate as every prior attempt |

The 3 warmup stages are deliberately ungated: they're waypoints en route to
the real, measured transition, not decisions in their own right, so
`curriculum.py`'s `main()` loop trains and checkpoints them and always
advances (no eval call, no retry logic — there's nothing to fail against).
Only the final `unmask` stage is evaluated, with the same reference bot,
opponent mode (`self_play`, not `frozen` — kept identical to every prior
attempt so a pass or fail cleanly isolates the ramp as the only variable),
and 240M-step budget as all 3 failed all-or-nothing attempts, for a direct
comparison. `airborne_penalty` ramps in step with the axes so it doesn't
fight a still-mostly-inert axis early on.

All the reward-shaping flags from generation 2's stage (`velocity-to-ball-weight`,
`ball-distance-penalty`, `ball-touch-reward`, `ball-velocity-to-goal-weight`,
`goal-reward`, `draw-penalty`) are unchanged and identical across all 4
stages, so the ramp is the sole studied variable.

`curriculum_state.json` was reset (generation 2's log archived to
`curriculum_state_gen2.json`) rather than continuing to log against a stage
list whose stage 0 no longer means what it used to.

**Open question, not yet resolved by data:** the ramp scales the action's
effect in Godot, which runs *after* PPO samples the action — PPO's own
std-collapse dynamics don't directly see the ramp, only the reward it
produces. It's possible this doesn't prevent the collapse, or even makes it
happen faster at low ramp values (weaker reward signal on those axes gives
less incentive to keep exploring them). Watch `train/std` per stage in
TensorBoard rather than assuming the ramp is working. If the final gated
stage still lands ~28-32%, that's evidence the plateau isn't an
exploration/collapse problem at all — worth revisiting reward shaping, or
trying `--opponent-mode frozen --opponent-model <path>` during the warmup
stages (implemented, never yet exercised in this project) to remove
self-play's moving-target instability while the policy first learns to use
the new axes.

All curriculum flags default to leaving Godot's own `@export` defaults
alone (`train.py` only forwards a flag when you pass it), so ordinary runs
are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
`--opponent-model <path>` (for `frozen`), `--draw-penalty`,
`--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
`--vertical-ramp`, `--pitch-roll-ramp` (0.0-1.0 locomotion-unmask ramp),
`--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`,
`--airborne-penalty`, `--ball-velocity-to-goal-weight`, `--goal-reward`.

### Running it automatically

`training/curriculum.py` (started via `curriculum.sh`, same detached-tmux
pattern as `start_training.sh`) drives all stages end to end: for each
stage it runs `run_training.sh` (pull, train, export, commit+push) with that
stage's flags, then evaluates the resulting checkpoint against a reference
bot over 100 episodes — the fixed `rookie.json` baseline for a from-scratch
stage 1 (no `resume_from_experiment`/`reference_experiment` override on
`STAGES[0]`), the previous stage's promoted checkpoint by default for
stages 2+, or an explicit override in that stage's dict when it deliberately
skips a since-regressed branch (generation 1's stage 5) or seeds from a
fixed foundation checkpoint (generation 2's stage 1 — see above).

```bash
cd training
./curriculum.sh                                              # start/resume the curriculum
./curriculum.sh --seed-checkpoint checkpoints/run11/final.zip # override stage 1's resume source for this run
```

The gate is deliberately lenient: it blocks a stage only on a **clear
regression** (the reference beating the candidate by 15+ points of win
rate), not "must show improvement." A 40-episode eval already misled us once
in this project — run11 was the first model to deliberately score a goal,
but its head-to-head eval read as a loss on sample noise alone. A strict
gate would have retried that stage forever for the wrong reason; a loose one
still catches a genuinely broken stage. Progress and every attempt's eval
result are logged to `curriculum_state.json` (committed alongside
`eval_history.json` after each attempt).

A stage gets up to 2 retries (3 attempts total) before the script stops and
asks for a human look — it will not retry indefinitely or advance past a
stage that keeps failing on its own. By default a retry resumes from that
stage's own previous attempt with a fresh `--reset-std`; a stage can instead
set `reset_retry_checkpoint: True` (generation 2's stage 1 does) to always
reset to its normal resume source instead — see the generation 1 → 2
postmortem above for why blind same-checkpoint retries can make things
monotonically worse. Once you've looked at why a block happened (more
timesteps? a flag needs adjusting? the eval itself was misleading?), re-run
with `--force-retry` to try again or `--skip-to-next-stage` if you judge the
result good enough despite the gate.

Running a stage by hand (e.g. to experiment with flags before trusting the
orchestrator) still works exactly as the table above describes — just call
`next_run.sh`/`run_training.sh` directly with that stage's flags.

## Self-play notes

By default both ships share the live policy (mirrored, team-relative
observations — see `ship_observations.gd`), so training is against the
current self. `--opponent-mode inert`/`frozen` (see Curriculum training
above) replace that with a placeholder or a fixed exported policy for one
side of a run; `frozen` is a single-fixed-model slice of full league play.
Fixed-opponent training against a *pool* of past checkpoints sampled per
episode (to avoid strategy collapse on long self-play runs) is still
deferred — see TODO.md.