feat(training): add airborne_penalty and a stage-6 "unmask" curriculum run

Stage 5 (aggression) passed (41-47 vs grounded curric-s2-defend, within
the lenient gate but not yet a clear win). Rather than keep the locomotion
mask on indefinitely, stage 6 reopens full 3D controls on top of the
aggression retune and pairs it with a new dense airborne_penalty (scaled
by height above the floor) so the policy learns to prefer staying grounded
through incentives instead of a hard mask — same regime shift that
regressed stage 3, but this time with a mitigation and ~12x the training
time (~240M timesteps / ~24h vs ~20M / ~2h) to actually re-converge
instead of stalling mid-shift.

airborne_penalty follows the existing SHIP_AI_OVERRIDES pattern: default
0 (off) on ship_ai_controller.gd, exposed via train.py's new
--airborne-penalty flag, added to training_mode.gd's allow-list. Also adds
a per-stage timesteps override in curriculum.py (STAGES[n]["timesteps"])
since this is the first stage to need a different budget than the rest.
This commit is contained in:
Josh Creek
2026-07-22 21:22:27 +01:00
parent 2de3085f13
commit 1afdc301ab
5 changed files with 71 additions and 17 deletions
+17 -15
View File
@@ -159,21 +159,22 @@ just with different curriculum flags.
| 2 — defend too | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll` | Reintroduces a live opponent (self-play) and the default episode-start mix — the same near-goal state is now simultaneously a finishing chance for one side and a defensive save for the other. Locomotion stays grounded. |
| 3 — no draws | `--draw-penalty 5 --reset-std 0.3` | Training episodes are golden-goal (end at the *first* goal), so there's no in-episode goal-margin to penalize — `draw_penalty` is the closest available signal: a one-time penalty when an episode times out with no goal at all, on top of the existing per-tick `time_penalty`. Also lifts the locomotion mask (full 3D controls) by omitting `--allow-vertical`/`--allow-pitch-roll`; pair that with `--reset-std` since the policy never got a reward gradient on those axes before now, so expect a brief re-exploration wobble. |
| 4 — mechanics/refinement | *(no curriculum flags — plain `next_run.sh`)* | Stock self-play, full controls, default reward/start-state mix. This is what all runs before this feature already did. |
| 5 — aggression | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5` | **Resumes from stage 2 (`curric-s2-defend`), not stage 4** — see the regression note below. Retunes ball-pursuit reward weights (up from 0.02/0.002/0.4) for much more aggressive, constantly-chasing floor play, deliberately keeping the locomotion mask on so it can't reopen the stage-3 regression. |
| 5 — aggression | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5` | **Resumes from stage 2 (`curric-s2-defend`), not stage 4** — see the regression note below. Retunes ball-pursuit reward weights (up from 0.02/0.002/0.4) for much more aggressive, constantly-chasing floor play, deliberately keeping the locomotion mask on so it can't reopen the stage-3 regression. Passed 2026-07-22 (41-47 vs grounded stage 2 — close, not yet a clear win). |
| 6 — unmask | `--opponent-mode self_play --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5 --airborne-penalty 0.003` | Re-opens full 3D controls on top of the aggression retune — this is the same grounded-checkpoint-to-full-3D transition that regressed stage 3, but this time paired with `airborne_penalty` (dense, scaled by height above the floor — see `ship_ai_controller.gd`) so the policy learns to *prefer* staying grounded through incentives instead of a hard mask, and can still pick up genuinely useful aerial/wall plays instead of never touching those axes. Runs much longer (~240M timesteps / ~24h vs every prior stage's ~20M/~2h) to actually re-converge through the regime shift instead of stalling mid-way like stage 3 did in a fifth of the time. |
> **Stages 3-4 regressed and are parked.** The locomotion-mask inference bugfix
> (`8c15c46`) revealed that stage 3's evals up to that point had been running
> with an unfairly unmasked grounded reference. Re-evaluated fairly,
> `curric-s2-defend` (grounded) beats both `curric-s3-no_draws` (26-60) and
> `curric-s4-mechanics` (24-57) — lifting the locomotion mask to full 3D in
> stage 3 was a clear regression in floor play that self-play never earned
> back. Stage 5 sidesteps this by resuming and evaluating against stage 2
> directly (`curriculum.py`'s `resume_from_experiment`/`reference_experiment`
> stage-dict overrides) instead of chaining through stages 3-4. Full 3D
> flight is parked as a separate initiative — see TODO.md — that will need a
> redesigned unmasking approach (more timesteps and/or reward rebalancing) so
> it doesn't cost floor fundamentals again. See `curriculum_state.json`'s
> stage-2/stage-3 log entries for the full eval numbers.
> **Stages 3-4 regressed; stage 6 deliberately reopens the same transition
> with a mitigation.** The locomotion-mask inference bugfix (`8c15c46`)
> revealed that stage 3's evals up to that point had been running with an
> unfairly unmasked grounded reference. Re-evaluated fairly, `curric-s2-defend`
> (grounded) beats both `curric-s3-no_draws` (26-60) and `curric-s4-mechanics`
> (24-57) — lifting the locomotion mask to full 3D in stage 3 was a clear
> regression in floor play that self-play never earned back in 20M steps.
> Stage 5 sidesteps this by resuming and evaluating against stage 2 directly
> (`curriculum.py`'s `resume_from_experiment`/`reference_experiment` stage-dict
> overrides) instead of chaining through stages 3-4. Stage 6 is where full 3D
> flight comes back — not masked away this time, but discouraged via
> `airborne_penalty` and given ~12x the training time to settle. See
> `curriculum_state.json`'s log for the full eval numbers.
All curriculum flags default to leaving Godot's own `@export` defaults
alone (`train.py` only forwards a flag when you pass it), so ordinary runs
@@ -181,7 +182,8 @@ are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
`--opponent-model <path>` (for `frozen`), `--draw-penalty`,
`--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
`--allow-vertical`/`--no-allow-vertical`, `--allow-pitch-roll`/`--no-allow-pitch-roll`,
`--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`.
`--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`,
`--airborne-penalty`.
### Running it automatically