CosmicClash/TRAINING.md

# Training the AI bot

Cosmic Clash bots are trained with reinforcement learning (self-play PPO): two
ships in a headless arena share one policy that learns by playing against
itself. Training runs in Python ([Godot RL Agents](https://github.com/edbeeching/godot_rl_agents)
bridge + Stable-Baselines3); the trained policy is exported to a small JSON
file and runs **inside the game** in pure GDScript — shipped bots need no
Python, no .NET, no network.

## How it fits together

- `Game/scenes/training.tscn` + `scripts/training_mode.gd` — headless self-play
  environment: two RL ships, randomized episode starts, goal rewards. Contains
  the vendored godot_rl_agents `Sync` node that talks TCP to the trainer.
- `scripts/ship_ai_controller.gd` — training-side bridge (observations,
  rewards, action mapping). `scripts/ship_observations.gd` is the *shared*
  observation builder — training and in-game inference must stay identical,
  so never fork it.
- `training/train.py` — PPO trainer; launches N parallel headless Godot
  instances (2 agents each).
- `training/export_policy.py` — SB3 checkpoint → JSON policy for the game.
- `scripts/ai_ship_controller.gd` + `scripts/policy_network.gd` — in-game
  inference (GDScript MLP forward pass).
- `training/evaluate.py` — pits two exported policies against each other and
  appends to `training/eval_history.json`.

## Hardware

The environment is our own headless Godot sim — fully cross-platform:

- **Any machine (e.g. the M4 Mac mini)**: fine for pipeline development,
  smoke runs, and short experiments. Env stepping is CPU-bound; the policy is
  a small MLP, so even CPU-only PPO updates are cheap.
- **Linux + NVIDIA GPU (e.g. the RTX 3090 box)**: recommended for real
  multi-hour/overnight runs. PyTorch CUDA works out of the box; more CPU
  cores also mean more parallel Godot instances (`--n-parallel`).

There is no hard GPU requirement (unlike Rocket League tooling) — a GPU
mainly speeds up learning updates on long runs.

## Setup

Needs Python 3.10+ and a Godot 4.7 binary.

```bash
cd training
python3.12 -m venv .venv          # macOS: brew install python@3.12
.venv/bin/pip install -r requirements.txt
```

On Linux, download the Godot 4.7 Linux binary and point at it:

```bash
export GODOT_BIN=~/godot/Godot_v4.7.1-stable_linux.x86_64
```

(macOS default is `/Applications/Godot.app/Contents/MacOS/Godot`; override
with `GODOT_BIN` or `--godot_bin` if yours lives elsewhere.)

## Run a training session

```bash
cd training
.venv/bin/python train.py --experiment run01 --timesteps 20000000 --n-parallel 6 --speedup 16
```

- Checkpoints land in `training/checkpoints/run01/` every `--checkpoint-every`
  steps (default 100k), plus `final.zip` on exit (also written on Ctrl-C).
- Resume with `--resume checkpoints/run01/final.zip`.
- `--n-parallel` = Godot instances (2 agents each). Scale with CPU cores.
- `--speedup` = in-engine physics speedup. Raise until CPU saturates.
- `--wandb` mirrors logs to Weights & Biases (`pip install wandb` first).

Expect the smoke-run scale (~100k steps) to only learn crude ball-chasing;
real behaviour needs tens of millions of steps (hours on the 3090 box).

### Watch progress

```bash
.venv/bin/tensorboard --logdir training/logs
```

Key curves: `rollout/ep_rew_mean` (should trend up), `rollout/ep_len_mean`
(should trend *down* from 225 as goals end episodes early — 225 action steps
= the 30s episode timeout).

### Reward/observation tuning

Reward weights are exported vars on `ShipAIController` (goal reward on
`TrainingMode`) — tune in `training.tscn`/scripts without touching the
trainer. If you change the *observation* layout (`ship_observations.gd`),
old checkpoints/exports become incompatible: retrain, and bump a note in
your experiment name.

## Export a checkpoint into the game

```bash
cd training
.venv/bin/python export_policy.py checkpoints/run01/final.zip ../Game/bots/hard.json
```

The exporter runs a parity check (JSON forward pass vs SB3 prediction) before
writing. Models live in `Game/bots/`.

## Evaluate progress between checkpoints

TensorBoard shows learning, but "is the new checkpoint actually *better*?"
needs head-to-head play:

```bash
.venv/bin/python export_policy.py checkpoints/run01/ppo_5000000_steps.zip /tmp/candidate.json
.venv/bin/python evaluate.py ../Game/bots/hard.json /tmp/candidate.json --episodes 40
```

Golden-goal episodes (first goal wins, timeout = draw), sides swapped halfway
for fairness, using the exact inference path that ships in-game. Every run
appends to `training/eval_history.json` — the long-term progress record.
Evaluate each new candidate against the previous promoted bot and a fixed
early reference to see absolute progress over time.

## Difficulty tiers

A bot is `(model, reaction_ticks, action_noise)` — configured on the Match
mode (`bot_model_path`, `bot_reaction_ticks`, `bot_action_noise` in
`match.tscn`) or any `AIShipController`:

- **Model**: the main lever. An early checkpoint *is* an easy bot — promote
  e.g. `easy.json` / `medium.json` / `hard.json` from different stages of one
  training run (verify the gaps with `evaluate.py`).
- **reaction_ticks** (default 8 = training cadence): higher = slower
  reactions, easier.
- **action_noise**: adds execution error, easier.

## Self-play notes

Both ships share the live policy (mirrored, team-relative observations — see
`ship_observations.gd`), so training is always against the current self.
Fixed-opponent training against frozen checkpoints (league play, to avoid
strategy collapse on long runs) is deferred — see TODO.md; the pieces
(exported JSON bots + `AIShipController`) already exist.