feat(*): add staged curriculum training with an automated stage-by-stage orchestrator

This commit is contained in:
Josh Creek
2026-07-21 12:38:51 +01:00
parent 1d539cc8c7
commit 8e3fafcc8b
7 changed files with 581 additions and 18 deletions
+70 -5
View File
@@ -135,10 +135,75 @@ mode (`bot_model_path`, `bot_reaction_ticks`, `bot_action_noise` in
reactions, easier.
- **action_noise**: adds execution error, easier.
## Curriculum training
Training from scratch with self-play alone hands the network every skill
at once — finishing, defending, positioning, not stalling to a draw — off a
sparse ±40 goal reward. `train.py` has a `curriculum` flag group that stages
this the way you'd coach a human: score first, then also defend, then learn
that a draw is still a failure, and only then spend compute polishing general
movement. Each stage is a normal chained run — a new `--experiment` resumed
via `--resume checkpoints/<previous>/final.zip`, same as any other run —
just with different curriculum flags.
| Stage | Flags | What it teaches |
|---|---|---|
| 1 — score | `--opponent-mode inert --attack-goal-bias 1.0 --no-allow-vertical --no-allow-pitch-roll` | Team 1 is a do-nothing placeholder ship parked at its spawn (an effectively empty net); near-goal resets always target the goal the trainee attacks; the ship can't fly or pitch/roll, only drive and yaw. Isolated finishing practice. |
| 2 — defend too | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll` | Reintroduces a live opponent (self-play) and the default episode-start mix — the same near-goal state is now simultaneously a finishing chance for one side and a defensive save for the other. Locomotion stays grounded. |
| 3 — no draws | `--draw-penalty 5 --reset-std 0.3` | Training episodes are golden-goal (end at the *first* goal), so there's no in-episode goal-margin to penalize — `draw_penalty` is the closest available signal: a one-time penalty when an episode times out with no goal at all, on top of the existing per-tick `time_penalty`. Also lifts the locomotion mask (full 3D controls) by omitting `--allow-vertical`/`--allow-pitch-roll`; pair that with `--reset-std` since the policy never got a reward gradient on those axes before now, so expect a brief re-exploration wobble. |
| 4 — mechanics/refinement | *(no curriculum flags — plain `next_run.sh`)* | Stock self-play, full controls, default reward/start-state mix. This is what all runs before this feature already did. |
All curriculum flags default to leaving Godot's own `@export` defaults
alone (`train.py` only forwards a flag when you pass it), so ordinary runs
are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
`--opponent-model <path>` (for `frozen`), `--draw-penalty`,
`--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
`--allow-vertical`/`--no-allow-vertical`, `--allow-pitch-roll`/`--no-allow-pitch-roll`.
### Running it automatically
`training/curriculum.py` (started via `curriculum.sh`, same detached-tmux
pattern as `start_training.sh`) drives all four stages end to end: for each
stage it runs `run_training.sh` (pull, train, export, commit+push) with that
stage's flags, then evaluates the resulting checkpoint against a reference
bot — the fixed `rookie.json` baseline for stage 1, or the previous stage's
promoted checkpoint for stages 2-4 — over 100 episodes.
```bash
cd training
./curriculum.sh # start/resume the curriculum
./curriculum.sh --seed-checkpoint checkpoints/run11/final.zip # seed stage 1 instead of a fresh policy
```
The gate is deliberately lenient: it blocks a stage only on a **clear
regression** (the reference beating the candidate by 15+ points of win
rate), not "must show improvement." A 40-episode eval already misled us once
in this project — run11 was the first model to deliberately score a goal,
but its head-to-head eval read as a loss on sample noise alone. A strict
gate would have retried that stage forever for the wrong reason; a loose one
still catches a genuinely broken stage. Progress and every attempt's eval
result are logged to `curriculum_state.json` (committed alongside
`eval_history.json` after each attempt).
A stage gets up to 2 retries (3 attempts total, each resuming from that
stage's own previous attempt with a fresh `--reset-std`) before the script
stops and asks for a human look — it will not retry indefinitely or advance
past a stage that keeps failing on its own. Once you've looked at why (more
timesteps? a flag needs adjusting? the eval itself was misleading?), re-run
with `--force-retry` to try again or `--skip-to-next-stage` if you judge the
result good enough despite the gate.
Running a stage by hand (e.g. to experiment with flags before trusting the
orchestrator) still works exactly as the table above describes — just call
`next_run.sh`/`run_training.sh` directly with that stage's flags.
## Self-play notes
Both ships share the live policy (mirrored, team-relative observations — see
`ship_observations.gd`), so training is always against the current self.
Fixed-opponent training against frozen checkpoints (league play, to avoid
strategy collapse on long runs) is deferred — see TODO.md; the pieces
(exported JSON bots + `AIShipController`) already exist.
By default both ships share the live policy (mirrored, team-relative
observations — see `ship_observations.gd`), so training is against the
current self. `--opponent-mode inert`/`frozen` (see Curriculum training
above) replace that with a placeholder or a fixed exported policy for one
side of a run; `frozen` is a single-fixed-model slice of full league play.
Fixed-opponent training against a *pool* of past checkpoints sampled per
episode (to avoid strategy collapse on long self-play runs) is still
deferred — see TODO.md.