Training the AI bot

Cosmic Clash bots are trained with reinforcement learning (self-play PPO): two ships in a headless arena share one policy that learns by playing against itself. Training runs in Python (Godot RL Agents bridge + Stable-Baselines3); the trained policy is exported to a small JSON file and runs inside the game in pure GDScript — shipped bots need no Python, no .NET, no network.

How it fits together

Game/scenes/training.tscn + scripts/training_mode.gd — headless self-play environment: two RL ships, randomized episode starts, goal rewards. Contains the vendored godot_rl_agents Sync node that talks TCP to the trainer.
scripts/ship_ai_controller.gd — training-side bridge (observations, rewards, action mapping). scripts/ship_observations.gd is the shared observation builder — training and in-game inference must stay identical, so never fork it.
training/train.py — PPO trainer; launches N parallel headless Godot instances (2 agents each).
training/export_policy.py — SB3 checkpoint → JSON policy for the game.
scripts/ai_ship_controller.gd + scripts/policy_network.gd — in-game inference (GDScript MLP forward pass).
training/evaluate.py — pits two exported policies against each other and appends to training/eval_history.json.

Hardware

The environment is our own headless Godot sim — fully cross-platform:

Any machine (e.g. the M4 Mac mini): fine for pipeline development, smoke runs, and short experiments. Env stepping is CPU-bound; the policy is a small MLP, so even CPU-only PPO updates are cheap.
Linux + NVIDIA GPU (e.g. the RTX 3090 box): recommended for real multi-hour/overnight runs. PyTorch CUDA works out of the box; more CPU cores also mean more parallel Godot instances (--n-parallel).

There is no hard GPU requirement (unlike Rocket League tooling) — a GPU mainly speeds up learning updates on long runs.

Setup

Needs Python 3.10+ and a Godot 4.7 binary.

cd training
python3.12 -m venv .venv          # macOS: brew install python@3.12
.venv/bin/pip install -r requirements.txt

On Linux, download the Godot 4.7 Linux binary and point at it:

export GODOT_BIN=~/godot/Godot_v4.7.1-stable_linux.x86_64

(macOS default is /Applications/Godot.app/Contents/MacOS/Godot; override with GODOT_BIN or --godot_bin if yours lives elsewhere.)

Run a training session

cd training
.venv/bin/python train.py --experiment run01 --timesteps 20000000 --n-parallel 6 --speedup 16

Checkpoints land in training/checkpoints/run01/ every --checkpoint-every steps (default 100k), plus final.zip on exit (also written on Ctrl-C).
Resume with --resume checkpoints/run01/final.zip.
--n-parallel = Godot instances (2 agents each). Scale with CPU cores.
--speedup = in-engine physics speedup. Raise until CPU saturates.
--wandb mirrors logs to Weights & Biases (pip install wandb first).

Expect the smoke-run scale (~100k steps) to only learn crude ball-chasing; real behaviour needs tens of millions of steps (hours on the 3090 box).

Watch progress

.venv/bin/tensorboard --logdir training/logs

Key curves: rollout/ep_rew_mean (should trend up), rollout/ep_len_mean (should trend down from 225 as goals end episodes early — 225 action steps = the 30s episode timeout).

Reward/observation tuning

Reward weights are exported vars on ShipAIController (goal reward on TrainingMode) — tune in training.tscn/scripts without touching the trainer. If you change the observation layout (ship_observations.gd), old checkpoints/exports become incompatible: retrain, and bump a note in your experiment name.

Export a checkpoint into the game

cd training
.venv/bin/python export_policy.py checkpoints/run01/final.zip ../Game/bots/hard.json

The exporter runs a parity check (JSON forward pass vs SB3 prediction) before writing. Models live in Game/bots/.

Evaluate progress between checkpoints

TensorBoard shows learning, but "is the new checkpoint actually better?" needs head-to-head play:

.venv/bin/python export_policy.py checkpoints/run01/ppo_5000000_steps.zip /tmp/candidate.json
.venv/bin/python evaluate.py ../Game/bots/hard.json /tmp/candidate.json --episodes 40

Golden-goal episodes (first goal wins, timeout = draw), sides swapped halfway for fairness, using the exact inference path that ships in-game. Every run appends to training/eval_history.json — the long-term progress record. Evaluate each new candidate against the previous promoted bot and a fixed early reference to see absolute progress over time.

Difficulty tiers

A bot is (model, reaction_ticks, action_noise) — configured on the Match mode (bot_model_path, bot_reaction_ticks, bot_action_noise in match.tscn) or any AIShipController:

Model: the main lever. An early checkpoint is an easy bot — promote e.g. easy.json / medium.json / hard.json from different stages of one training run (verify the gaps with evaluate.py).
reaction_ticks (default 8 = training cadence): higher = slower reactions, easier.
action_noise: adds execution error, easier.

Self-play notes

Both ships share the live policy (mirrored, team-relative observations — see ship_observations.gd), so training is always against the current self. Fixed-opponent training against frozen checkpoints (league play, to avoid strategy collapse on long runs) is deferred — see TODO.md; the pieces (exported JSON bots + AIShipController) already exist.

5.6 KiB Raw Blame History