feat(training): add generation 5 curriculum

This commit is contained in:
Josh Creek
2026-08-08 14:56:17 +01:00
parent 33952b3cd0
commit 341a67f6da
10 changed files with 832 additions and 19 deletions
+84 -9
View File
@@ -162,11 +162,9 @@ flat files there, never touching subdirectories.
`Game/bots/promoted/<tier>.json` is the small, curated, hand-maintained set
actually referenced by the shipped game — currently `easy.json` (promoted
2026-07-24 from `curric-s6-unmask`, the strongest checkpoint at the
time — note `curric-s6-unmask` was itself generation 1's *failed* unmask
stage, so `easy.json` is weaker than `reference-grounded.json` below; a
strong generation 4 result should promote a real replacement, plus
`medium.json`/`hard.json`) and `reference-grounded.json` (added for
2026-08-08 from generation 4's `20260806-1939-curric-s3-gauntlet`; this is
the 320M-step MultiDiscrete policy and the foundation for the planned
generation-5 curriculum below) and `reference-grounded.json` (added for
generation 4 — a copy of generation 3's `curric-s5-aggression`, made before
the flat `Game/bots/` dump was scrapped for the redesign, kept as the
strongest grounded-era artifact and the fixed yardstick generations 1-3 were
@@ -176,6 +174,11 @@ promoted file is never touched by training scripts, never overwritten by a
same-named future export, and never disturbed by pruning old experiment
files from the flat dump.
Until distinct `medium.json` and `hard.json` policies earn promotion, the
three menu tiers all run this same `easy.json` policy at its full trained
cadence (`reaction_ticks=8`, `action_noise=0`). The tiers are labels only;
the game does not manufacture difficulty gaps by handicapping this model.
To promote a new bot into a tier: copy the chosen `Game/bots/<experiment>.json`
to `Game/bots/promoted/<tier>.json` (overwriting the old one), and note the
source experiment + date in this section. Do this for `medium.json`/
@@ -493,10 +496,13 @@ alone (`train.py` only forwards a flag when you pass it), so ordinary runs
are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
`--opponent-model <path>` (for `frozen`), `--draw-penalty`,
`--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
`--air-drill-chance` (generation 4's state-setter aerial curriculum),
`--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`,
`--airborne-penalty`, `--tilt-penalty`, `--ball-velocity-to-goal-weight`,
`--goal-reward`. (`--vertical-ramp`/`--pitch-roll-ramp` are gone — generation
`--air-drill-chance`, `--air-intercept-chance`, `--team-size`,
`--velocity-to-ball-weight`, `--forward-velocity-to-ball-weight`,
`--ball-distance-penalty`, `--ball-touch-reward`, `--airborne-penalty`,
`--tilt-penalty`, `--ground-tilt-penalty`, `--speed-reward-weight`,
`--ball-velocity-to-goal-weight`, `--goal-reward`, and
`--opponent-pool` with `--opponent-mode=league`.
(`--vertical-ramp`/`--pitch-roll-ramp` are gone — generation
4 has no locomotion mask/ramp to control.)
### Running it automatically
@@ -546,6 +552,75 @@ Running a stage by hand (e.g. to experiment with flags before trusting the
orchestrator) still works exactly as the table above describes — just call
`next_run.sh`/`run_training.sh` directly with that stage's flags.
### Generation 5 follow-on
Generation 4's stage-3 export is the foundation rather than a throwaway
baseline: all generation-5 stages resume from
`checkpoints/20260806-1939-curric-s3-gauntlet/final.zip`. Its match results
are strong, but playtesting and its final telemetry expose the next learning
targets: it spends about 39% of play above the airborne threshold while only
about 0.04% of episode-level touches are aerial, and it often travels on its
side and strikes the ball with its roof. This is a successful scoring policy
that now needs control quality and a more productive use of flight.
Turbo remains forward-only for players and policies: it activates only with
positive forward thrust and multiplies the resulting combined thrust vector.
Generation 5 preserves the same control contract Stage 3 was trained under.
Generation 5 adds three episode telemetry signals to TensorBoard:
`upright_fraction` (low-altitude ticks with the
ship's up vector substantially upright), `forward_motion_fraction`
(low-altitude moving ticks whose planar velocity points broadly along the
nose), and `productive_air_touch_fraction` (touches above the aerial height
that send the ball toward the attack goal). The automatic gates are
deliberately conservative catastrophe floors; every stage records its final
500-rollout tail means in `generation5_state.json` so later threshold changes
can be based on evidence instead of a single watched match.
| Stage | Regime | Budget | Learning target | Advancement gate |
|---|---|---:|---|---|
| 4 — `handling` | Self-play, current balanced start mix | 40M (~4h) | Prefer upright, nose-led travel near the floor. Replace the orientation-agnostic speed bonus with low-altitude forward-motion shaping, and apply the stronger tilt cost only near the floor so pitch/roll remain free in genuine aerial play. | Before Stage 5: at least 80% training goal rate, at least 80% non-draw rate in the paired evaluation versus promoted Stage 3, no clear head-to-head regression, no more than 20% physical-side win imbalance, and the upright/forward-motion telemetry floors. |
| 5 — `intercepts` | Self-play with 4050% improved air-intercept starts | 60M (~6h) | Convert existing vertical movement into useful aerial touches. Spawn a moving high ball on reachable attacking and defensive trajectories, away from walls, so contact is instrumental to scoring or saving rather than independently rewarded. | No clear regression versus Stage 4; productive aerial-touch telemetry must improve materially without reducing upright/forward-motion telemetry back to the Stage-3 baseline. |
| 6 — `league` | Live policy against a frozen opponent sampled per episode from Stage 3, Stage 4, and Stage 5 | 100M (~10h) | Prevent a narrow self-play equilibrium and consolidate ground handling, aerial interception, attack, and defence against distinct styles. | No clear head-to-head regression against any pool member plus conservative handling/aerial telemetry floors. Promote the passing result to `medium.json` after these recorded evaluations support it. |
Stage 7 teamplay remains deliberately unconfigured. The fixed roster
observation and `team_size` plumbing can run 2v2, but there is no paired 2v2
evaluation or team-credit reward yet; spending 120M steps without those gates
would make a pass meaningless.
`training/generation5.py` implements Stages 46 separately from the completed
generation-4 orchestrator and state. It always begins Stage 4 from
`checkpoints/20260806-1939-curric-s3-gauntlet/final.zip`, then resumes each
later stage from its passing predecessor. `generation5.sh` runs it detached,
and retries/blocks use the same restart-safe pattern as the earlier
curriculum:
```bash
cd training
.venv/bin/python generation5.py --dry-run # print and validate the next command only
./generation5.sh # run/resume in tmux
tmux attach -t cosmic-generation5
cat generation5_state.json
```
Stage 4 removes the generic speed bonus, halves the old orientation-agnostic
closing reward, and adds a nose-led planar approach reward plus a tilt cost
that fades to zero by 3m altitude. Its scoring gates deliberately run before
Stage 5: becoming upright is not progress if the resulting policy stops
finishing goals. Stage 5 adds moving high-ball intercept
starts aimed toward real goals rather than a standalone air-touch reward.
Stage 6's `league` opponent mode samples a historical exported policy at each
episode reset. Each later stage preserves the preceding shaping and adds one
new difficulty.
The physical-side gate is separate from the model-vs-model score. A paired
side swap can make an identical policy appear perfectly balanced overall even
when the Player 2 ship never functions. This caught the canonical action-frame
bug exposed by Stage 3's pitch/roll use: team 1 observations are rotated 180°
about Y, but rotation commands feed world-space torque, so team 1 pitch and
roll must be rotated back (X/Z signs inverted). Thrust remains unchanged
because it is applied through the ship's local basis.
## Self-play notes
By default both ships share the live policy (mirrored, team-relative