CosmicClash/TRAINING.md

Training the AI bot

Cosmic Clash bots are trained with reinforcement learning (self-play PPO): two ships in a headless arena share one policy that learns by playing against itself. Training runs in Python (Godot RL Agents bridge + Stable-Baselines3); the trained policy is exported to a small JSON file and runs inside the game in pure GDScript — shipped bots need no Python, no .NET, no network.

How it fits together

• Game/scenes/training.tscn + scripts/training_mode.gd — headless self-play environment: two RL ships, randomized episode starts, goal rewards. Contains the vendored godot_rl_agents Sync node that talks TCP to the trainer.
• scripts/ship_ai_controller.gd — training-side bridge (observations, rewards, action mapping). scripts/ship_observations.gd is the shared observation builder — training and in-game inference must stay identical, so never fork it.
• training/train.py — PPO trainer; launches N parallel headless Godot instances (2 agents each) — from source by default, or from a pre-built binary via --exported-binary (see TRAINING_LINUX.md's "Exported-binary training" section; training/export_linux.sh builds it from Game/export_presets.cfg's "Linux Training" preset).
• training/export_policy.py — SB3 checkpoint → JSON policy for the game.
• scripts/ai_ship_controller.gd + scripts/policy_network.gd — in-game inference (GDScript MLP forward pass).
• training/evaluate.py — pits two exported policies against each other and appends to training/eval_history.json.

Hardware

The environment is our own headless Godot sim — fully cross-platform:

• Any machine (e.g. the M4 Mac mini): fine for pipeline development, smoke runs, and short experiments. Env stepping is CPU-bound; the policy is a small MLP, so even CPU-only PPO updates are cheap.
• Linux + NVIDIA GPU (e.g. the RTX 3090 box): recommended for real multi-hour/overnight runs. PyTorch CUDA works out of the box; more CPU cores also mean more parallel Godot instances (--n-parallel).

There is no hard GPU requirement (unlike Rocket League tooling) — a GPU mainly speeds up learning updates on long runs.

For the Linux/3090 remote-training workflow (setup, throughput tuning, auto-copying results back to the Mac, dashboard over the network), see TRAINING_LINUX.md.

Setup

Needs Python 3.10+ and a Godot 4.7 binary.

cd training
python3.12 -m venv .venv          # macOS: brew install python@3.12
.venv/bin/pip install -r requirements.txt

On Linux, download the Godot 4.7 Linux binary and point at it:

export GODOT_BIN=~/godot/Godot_v4.7.1-stable_linux.x86_64

(macOS default is /Applications/Godot.app/Contents/MacOS/Godot; override with GODOT_BIN or --godot_bin if yours lives elsewhere.)

Run a training session

cd training
.venv/bin/python train.py --experiment run01 --timesteps 20000000 --n-parallel 6 --speedup 16

• Checkpoints land in training/checkpoints/run01/ every --checkpoint-every steps (default 100k), plus final.zip on exit (also written on Ctrl-C).
• Resume with --resume checkpoints/run01/final.zip.
• --n-parallel = Godot instances (2 agents each). Scale with CPU cores.
• --speedup = in-engine physics speedup. Raise until CPU saturates.
• --wandb mirrors logs to Weights & Biases (pip install wandb first).

Expect the smoke-run scale (~100k steps) to only learn crude ball-chasing; real behaviour needs tens of millions of steps (hours on the 3090 box).

Watch progress

.venv/bin/tensorboard --logdir training/logs

Key curves: rollout/ep_rew_mean (should trend up), rollout/ep_len_mean (should trend down from 225 as goals end episodes early — 225 action steps = the 30s episode timeout), rollout/goal_rate (fraction of recent episodes that ended in an actual goal rather than timing out as a draw — the live signal for "is the policy actually finishing more episodes by scoring", since ep_rew_mean mixes that with dense reward-shaping (ball chasing/ touching) and doesn't isolate it).

Reward/observation tuning

Reward weights are exported vars on ShipAIController (goal reward on TrainingMode) — tune in training.tscn/scripts without touching the trainer. If you change the observation layout (ship_observations.gd), old checkpoints/exports become incompatible: retrain, and bump a note in your experiment name.

Export a checkpoint into the game

cd training
.venv/bin/python export_policy.py checkpoints/run01/final.zip ../Game/bots/hard.json

The exporter runs a parity check (JSON forward pass vs SB3 prediction) before writing. Models live in Game/bots/.

Evaluate progress between checkpoints

TensorBoard shows learning, but "is the new checkpoint actually better?" needs head-to-head play:

.venv/bin/python export_policy.py checkpoints/run01/ppo_5000000_steps.zip /tmp/candidate.json
.venv/bin/python evaluate.py ../Game/bots/hard.json /tmp/candidate.json --episodes 40

Golden-goal episodes (first goal wins, timeout = draw), sides swapped halfway for fairness, using the exact inference path that ships in-game. Every run appends to training/eval_history.json — the long-term progress record. Evaluate each new candidate against the previous promoted bot and a fixed early reference to see absolute progress over time.

If a model was trained with the locomotion mask on (curriculum stages 1, 2, and 5 — see below), pass --grounded-a/--grounded-b for whichever side it's on. The eval otherwise runs AIShipController fully unmasked regardless of how a model was trained, so a grounded model's untrained vertical/pitch-roll output reaches the ship as noise it never had to contend with during training — this understates it, not a neutral comparison.

Difficulty tiers

A bot is (model, reaction_ticks, action_noise) — configured on the Match mode (bot_model_path, bot_reaction_ticks, bot_action_noise in match.tscn) or any AIShipController:

• Model: the main lever. An early checkpoint is an easy bot — promote e.g. easy.json / medium.json / hard.json from different stages of one training run (verify the gaps with evaluate.py).
• reaction_ticks (default 8 = training cadence): higher = slower reactions, easier.
• action_noise: adds execution error, easier.

Promoted bots (Game/bots/promoted/)

Game/bots/*.json is a flat, ever-growing dump of every experiment/curriculum export — useful for evaluate.py and for A/B-ing arbitrary past checkpoints against each other in the in-game Spectate dropdown (main_menu.gd lists Game/bots/ non-recursively, so anything one directory deeper is invisible to it), but none of those filenames (run07.json, curric-s3-no_draws.json, ...) are meant to be the shipped bot — they get superseded constantly and the automated curriculum pipeline (run_training.sh) only ever writes new flat files there, never touching subdirectories.

Game/bots/promoted/<tier>.json is the small, curated, hand-maintained set actually referenced by the shipped game — currently just easy.json (promoted 2026-07-24 from curric-s6-unmask, the strongest checkpoint at the time). match.tscn/spectate.tscn point their bot_model_path exports here directly, so a promoted file is never touched by training scripts, never overwritten by a same-named future export, and never disturbed by pruning old experiment files from the flat dump.

To promote a new bot into a tier: copy the chosen Game/bots/<experiment>.json to Game/bots/promoted/<tier>.json (overwriting the old one), and note the source experiment + date in this section. Do this for medium.json/ hard.json as later curriculum stages clear the bar against easy.json in evaluate.py.

Curriculum training

Training from scratch with self-play alone hands the network every skill at once — finishing, defending, positioning, not stalling to a draw — off a sparse ±40 goal reward. train.py has a curriculum flag group that stages this the way you'd coach a human: score first, then also defend, then learn that a draw is still a failure, and only then spend compute polishing general movement. Each stage is a normal chained run — a new --experiment resumed via --resume checkpoints/<previous>/final.zip, same as any other run — just with different curriculum flags.

curriculum.py has run through three generations so far. Generation 1 (below) ran stages 1-6 to completion/block and is archived; generation 2 started a fresh stage 1 seeded from generation 1's last clean pass instead of continuing to retry a stage that kept getting worse, but also failed 3 attempts; generation 3 (the one curriculum.py actually runs today) replaces generation 2's single all-or-nothing unmask stage with a gradual ramp — see "Generation 3" below.

Generation 1 (archived — see curriculum_state_gen1.json)

Stage	Flags	What it teaches
1 — score	--opponent-mode inert --attack-goal-bias 1.0 --no-allow-vertical --no-allow-pitch-roll	Team 1 is a do-nothing placeholder ship parked at its spawn (an effectively empty net); near-goal resets always target the goal the trainee attacks; the ship can't fly or pitch/roll, only drive and yaw. Isolated finishing practice.
2 — defend too	--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll	Reintroduces a live opponent (self-play) and the default episode-start mix — the same near-goal state is now simultaneously a finishing chance for one side and a defensive save for the other. Locomotion stays grounded.
3 — no draws	--draw-penalty 5 --reset-std 0.3	Training episodes are golden-goal (end at the first goal), so there's no in-episode goal-margin to penalize — draw_penalty is the closest available signal: a one-time penalty when an episode times out with no goal at all, on top of the existing per-tick time_penalty. Also lifts the locomotion mask (full 3D controls) by omitting --allow-vertical/--allow-pitch-roll; pair that with --reset-std since the policy never got a reward gradient on those axes before now, so expect a brief re-exploration wobble.
4 — mechanics/refinement	(no curriculum flags — plain next_run.sh)	Stock self-play, full controls, default reward/start-state mix. This is what all runs before this feature already did.
5 — aggression	--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5	Resumes from stage 2 (curric-s2-defend), not stage 4 — see the regression note below. Retunes ball-pursuit reward weights (up from 0.02/0.002/0.4) for much more aggressive, constantly-chasing floor play, deliberately keeping the locomotion mask on so it can't reopen the stage-3 regression. Passed 2026-07-22 (41-47 vs grounded stage 2 — close, not yet a clear win).
6 — unmask	--opponent-mode self_play --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5 --airborne-penalty 0.003	Re-opens full 3D controls on top of the aggression retune — this is the same grounded-checkpoint-to-full-3D transition that regressed stage 3, but this time paired with airborne_penalty (dense, scaled by height above the floor — see ship_ai_controller.gd) so the policy learns to prefer staying grounded through incentives instead of a hard mask, and can still pick up genuinely useful aerial/wall plays instead of never touching those axes. Failed 3 attempts in a row (25% → 20% → 15% win rate vs curric-s5-aggression) and blocked — see "Generation 2" below for what replaced it.

Stages 3-4 regressed; stage 6 deliberately reopens the same transition with a mitigation. The locomotion-mask inference bugfix (8c15c46) revealed that stage 3's evals up to that point had been running with an unfairly unmasked grounded reference. Re-evaluated fairly, curric-s2-defend (grounded) beats both curric-s3-no_draws (26-60) and curric-s4-mechanics (24-57) — lifting the locomotion mask to full 3D in stage 3 was a clear regression in floor play that self-play never earned back in 20M steps. Stage 5 sidesteps this by resuming and evaluating against stage 2 directly (curriculum.py's resume_from_experiment/reference_experiment stage-dict overrides) instead of chaining through stages 3-4. Stage 6 is where full 3D flight comes back — not masked away this time, but discouraged via airborne_penalty and given ~12x the training time to settle. See curriculum_state_gen1.json's log for the full eval numbers.

Stage 6 (unmask, retry1, retry2) all used identical flags — curriculum.py always reuses STAGES[stage_index]["flags"] on retry, only the resume checkpoint changes — so continued training just drifted the same policy further rather than converging differently (25% → 20% → 15% win rate vs curric-s5-aggression). After 3 failed attempts the script blocked for human review; rather than pile up retry4, retry5, ... on a lineage that kept getting worse, generation 2 (below) replaces it with a fresh stage 1.

Generation 2 (archived — see curriculum_state_gen2.json)

curriculum.py's STAGES list contained a single stage, unmask (displays as stage 1 — curric-s1-unmask), which picks up exactly where generation 1's regression analysis left off. It resumes directly from FOUNDATION_EXPERIMENT (curric-s5-aggression's own checkpoint — the last stage that passed cleanly) via resume_from_experiment/reference_experiment overrides, rather than re-running stages 1-5 or continuing generation 1's drifted retry2:

--opponent-mode self_play --velocity-to-ball-weight 0.08 --ball-distance-penalty 0.01 --ball-touch-reward 0.7 --airborne-penalty 0.003 --ball-velocity-to-goal-weight 0.06 --goal-reward 80 --draw-penalty 5

Compared to generation 1's stage 6:

• velocity_to_ball_weight (0.05→0.08) and ball_distance_penalty (0.006→0.01) — the actual ball-chasing terms, unchanged since stage 5 despite three failed attempts — plus ball_touch_reward (0.5→0.7).
• Two scoring-specific terms newly exposed via train.py (they already existed as ship_ai_controller.gd/training_mode.gd @exports, just not as CLI flags): ball_velocity_to_goal_weight (0.004 default → 0.06) rewards the ball actually moving toward the goal, not just being chased/touched; goal_reward (40 default → 80) is the terminal reward for scoring itself.
• draw_penalty 5 (proven effective in stage 3 against passivity), which generation 1's stage 6 had never set — previously all carrot for scoring, no stick for never scoring.
• reset_retry_checkpoint: True on the stage dict, so if this stage itself fails and retries, resume_checkpoint() resets to FOUNDATION_EXPERIMENT again instead of drifting a failed attempt further — the specific bug that made generation 1's 3 retries monotonically worse instead of converging.

Deliberately not added: a cooldown/cap on ball_velocity_to_goal_weight to guard against a bot farming near-misses (bouncing the ball toward goal repeatedly without finishing) instead of actually scoring. Unlike the touch-farming bug (see ship_ai_controller.gd's ball_touch_reward comments) this term is already direction-scaled by construction (it's a velocity-toward-goal quantity, not an undirected contact count), so the risk is theoretical rather than demonstrated. If this stage's eval shows high ball_velocity_to_goal_weight accrual without a matching rise in actual goals scored, that's the signal to add one.

Every experiment name curriculum.py generates is now timestamped (YYYYMMDD-HHMM-<name>, e.g. 20260727-0930-curric-s1-unmask), applied once in run_stage_attempt — this keeps generation 2's names from colliding with generation 1's plain ones (both checkpoint directories and TensorBoard run names come straight from --experiment) and makes run order obvious in TensorBoard without cross-referencing curriculum_state.json.

Generation 2 also failed 3 attempts in a row, landing at a stable 32% / 28% / 31% win rate vs curric-s5-aggression each time — the second and third attempts each continued the same checkpoint lineage for another full 240M steps with zero improvement, ruling out both the reward retune above and "just needs more time" as fixes. Every attempt showed train/std collapsing from ~0.30 to ~0.13-0.15 within the first ~10% of steps and never recovering. See "Generation 3" below for the redesign this prompted.

Generation 3 (current)

Generation 2's failures point at the mechanism of the transition, not the reward weights: flipping allow_vertical/allow_pitch_roll from false to true in one step let PPO's action-distribution std collapse on those axes before the policy ever meaningfully explored them. Generation 3 replaces that boolean mask with a float ramp (vertical_ramp/pitch_roll_ramp on ShipAIController, 0.0-1.0, multiplying the axis's effect in set_action instead of gating it) and spreads the transition across 4 stages instead of 1:

Stage	vertical-ramp/pitch-roll-ramp	airborne-penalty	timesteps	gated
1 — unmask-ramp25	0.25	0.0	40M (~4h)	No — trains, checkpoints, always advances
2 — unmask-ramp50	0.5	0.001	40M (~4h)	No
3 — unmask-ramp75	0.75	0.002	40M (~4h)	No
4 — unmask	1.0	0.003	240M (~24h)	Yes — evaluated against curric-s5-aggression, same 15-point regression gate as every prior attempt

The 3 warmup stages are deliberately ungated: they're waypoints en route to the real, measured transition, not decisions in their own right, so curriculum.py's main() loop trains and checkpoints them and always advances (no eval call, no retry logic — there's nothing to fail against). Only the final unmask stage is evaluated, with the same reference bot, opponent mode (self_play, not frozen — kept identical to every prior attempt so a pass or fail cleanly isolates the ramp as the only variable), and 240M-step budget as all 3 failed all-or-nothing attempts, for a direct comparison. airborne_penalty ramps in step with the axes so it doesn't fight a still-mostly-inert axis early on.

All the reward-shaping flags from generation 2's stage (velocity-to-ball-weight, ball-distance-penalty, ball-touch-reward, ball-velocity-to-goal-weight, goal-reward, draw-penalty) are unchanged and identical across all 4 stages, so the ramp is the sole studied variable.

curriculum_state.json was reset (generation 2's log archived to curriculum_state_gen2.json) rather than continuing to log against a stage list whose stage 0 no longer means what it used to.

Open question, not yet resolved by data: the ramp scales the action's effect in Godot, which runs after PPO samples the action — PPO's own std-collapse dynamics don't directly see the ramp, only the reward it produces. It's possible this doesn't prevent the collapse, or even makes it happen faster at low ramp values (weaker reward signal on those axes gives less incentive to keep exploring them). Watch train/std per stage in TensorBoard rather than assuming the ramp is working. If the final gated stage still lands ~28-32%, that's evidence the plateau isn't an exploration/collapse problem at all — worth revisiting reward shaping, or trying --opponent-mode frozen --opponent-model <path> during the warmup stages (implemented, never yet exercised in this project) to remove self-play's moving-target instability while the policy first learns to use the new axes.

All curriculum flags default to leaving Godot's own @export defaults alone (train.py only forwards a flag when you pass it), so ordinary runs are unaffected. Full flag list: --opponent-mode {self_play,inert,frozen}, --opponent-model <path> (for frozen), --draw-penalty, --attack-goal-bias, --kickoff-chance, --near-goal-chance, --vertical-ramp, --pitch-roll-ramp (0.0-1.0 locomotion-unmask ramp), --velocity-to-ball-weight, --ball-distance-penalty, --ball-touch-reward, --airborne-penalty, --ball-velocity-to-goal-weight, --goal-reward.

Running it automatically

training/curriculum.py (started via curriculum.sh, same detached-tmux pattern as start_training.sh) drives all stages end to end: for each stage it runs run_training.sh (pull, train, export, commit+push) with that stage's flags, then evaluates the resulting checkpoint against a reference bot over 100 episodes — the fixed rookie.json baseline for a from-scratch stage 1 (no resume_from_experiment/reference_experiment override on STAGES[0]), the previous stage's promoted checkpoint by default for stages 2+, or an explicit override in that stage's dict when it deliberately skips a since-regressed branch (generation 1's stage 5) or seeds from a fixed foundation checkpoint (generation 2's stage 1 — see above).

cd training
./curriculum.sh                                              # start/resume the curriculum
./curriculum.sh --seed-checkpoint checkpoints/run11/final.zip # override stage 1's resume source for this run

The gate is deliberately lenient: it blocks a stage only on a clear regression (the reference beating the candidate by 15+ points of win rate), not "must show improvement." A 40-episode eval already misled us once in this project — run11 was the first model to deliberately score a goal, but its head-to-head eval read as a loss on sample noise alone. A strict gate would have retried that stage forever for the wrong reason; a loose one still catches a genuinely broken stage. Progress and every attempt's eval result are logged to curriculum_state.json (committed alongside eval_history.json after each attempt).

A stage gets up to 2 retries (3 attempts total) before the script stops and asks for a human look — it will not retry indefinitely or advance past a stage that keeps failing on its own. By default a retry resumes from that stage's own previous attempt with a fresh --reset-std; a stage can instead set reset_retry_checkpoint: True (generation 2's stage 1 does) to always reset to its normal resume source instead — see the generation 1 → 2 postmortem above for why blind same-checkpoint retries can make things monotonically worse. Once you've looked at why a block happened (more timesteps? a flag needs adjusting? the eval itself was misleading?), re-run with --force-retry to try again or --skip-to-next-stage if you judge the result good enough despite the gate.

Running a stage by hand (e.g. to experiment with flags before trusting the orchestrator) still works exactly as the table above describes — just call next_run.sh/run_training.sh directly with that stage's flags.

Self-play notes

By default both ships share the live policy (mirrored, team-relative observations — see ship_observations.gd), so training is against the current self. --opponent-mode inert/frozen (see Curriculum training above) replace that with a placeholder or a fixed exported policy for one side of a run; frozen is a single-fixed-model slice of full league play. Fixed-opponent training against a pool of past checkpoints sampled per episode (to avoid strategy collapse on long self-play runs) is still deferred — see TODO.md.