mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-14 01:22:17 +00:00
feat(training): add airborne_penalty and a stage-6 "unmask" curriculum run
Stage 5 (aggression) passed (41-47 vs grounded curric-s2-defend, within the lenient gate but not yet a clear win). Rather than keep the locomotion mask on indefinitely, stage 6 reopens full 3D controls on top of the aggression retune and pairs it with a new dense airborne_penalty (scaled by height above the floor) so the policy learns to prefer staying grounded through incentives instead of a hard mask — same regime shift that regressed stage 3, but this time with a mitigation and ~12x the training time (~240M timesteps / ~24h vs ~20M / ~2h) to actually re-converge instead of stalling mid-shift. airborne_penalty follows the existing SHIP_AI_OVERRIDES pattern: default 0 (off) on ship_ai_controller.gd, exposed via train.py's new --airborne-penalty flag, added to training_mode.gd's allow-list. Also adds a per-stage timesteps override in curriculum.py (STAGES[n]["timesteps"]) since this is the first stage to need a different budget than the rest.
This commit is contained in:
+17
-15
@@ -159,21 +159,22 @@ just with different curriculum flags.
|
||||
| 2 — defend too | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll` | Reintroduces a live opponent (self-play) and the default episode-start mix — the same near-goal state is now simultaneously a finishing chance for one side and a defensive save for the other. Locomotion stays grounded. |
|
||||
| 3 — no draws | `--draw-penalty 5 --reset-std 0.3` | Training episodes are golden-goal (end at the *first* goal), so there's no in-episode goal-margin to penalize — `draw_penalty` is the closest available signal: a one-time penalty when an episode times out with no goal at all, on top of the existing per-tick `time_penalty`. Also lifts the locomotion mask (full 3D controls) by omitting `--allow-vertical`/`--allow-pitch-roll`; pair that with `--reset-std` since the policy never got a reward gradient on those axes before now, so expect a brief re-exploration wobble. |
|
||||
| 4 — mechanics/refinement | *(no curriculum flags — plain `next_run.sh`)* | Stock self-play, full controls, default reward/start-state mix. This is what all runs before this feature already did. |
|
||||
| 5 — aggression | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5` | **Resumes from stage 2 (`curric-s2-defend`), not stage 4** — see the regression note below. Retunes ball-pursuit reward weights (up from 0.02/0.002/0.4) for much more aggressive, constantly-chasing floor play, deliberately keeping the locomotion mask on so it can't reopen the stage-3 regression. |
|
||||
| 5 — aggression | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5` | **Resumes from stage 2 (`curric-s2-defend`), not stage 4** — see the regression note below. Retunes ball-pursuit reward weights (up from 0.02/0.002/0.4) for much more aggressive, constantly-chasing floor play, deliberately keeping the locomotion mask on so it can't reopen the stage-3 regression. Passed 2026-07-22 (41-47 vs grounded stage 2 — close, not yet a clear win). |
|
||||
| 6 — unmask | `--opponent-mode self_play --velocity-to-ball-weight 0.05 --ball-distance-penalty 0.006 --ball-touch-reward 0.5 --airborne-penalty 0.003` | Re-opens full 3D controls on top of the aggression retune — this is the same grounded-checkpoint-to-full-3D transition that regressed stage 3, but this time paired with `airborne_penalty` (dense, scaled by height above the floor — see `ship_ai_controller.gd`) so the policy learns to *prefer* staying grounded through incentives instead of a hard mask, and can still pick up genuinely useful aerial/wall plays instead of never touching those axes. Runs much longer (~240M timesteps / ~24h vs every prior stage's ~20M/~2h) to actually re-converge through the regime shift instead of stalling mid-way like stage 3 did in a fifth of the time. |
|
||||
|
||||
> **Stages 3-4 regressed and are parked.** The locomotion-mask inference bugfix
|
||||
> (`8c15c46`) revealed that stage 3's evals up to that point had been running
|
||||
> with an unfairly unmasked grounded reference. Re-evaluated fairly,
|
||||
> `curric-s2-defend` (grounded) beats both `curric-s3-no_draws` (26-60) and
|
||||
> `curric-s4-mechanics` (24-57) — lifting the locomotion mask to full 3D in
|
||||
> stage 3 was a clear regression in floor play that self-play never earned
|
||||
> back. Stage 5 sidesteps this by resuming and evaluating against stage 2
|
||||
> directly (`curriculum.py`'s `resume_from_experiment`/`reference_experiment`
|
||||
> stage-dict overrides) instead of chaining through stages 3-4. Full 3D
|
||||
> flight is parked as a separate initiative — see TODO.md — that will need a
|
||||
> redesigned unmasking approach (more timesteps and/or reward rebalancing) so
|
||||
> it doesn't cost floor fundamentals again. See `curriculum_state.json`'s
|
||||
> stage-2/stage-3 log entries for the full eval numbers.
|
||||
> **Stages 3-4 regressed; stage 6 deliberately reopens the same transition
|
||||
> with a mitigation.** The locomotion-mask inference bugfix (`8c15c46`)
|
||||
> revealed that stage 3's evals up to that point had been running with an
|
||||
> unfairly unmasked grounded reference. Re-evaluated fairly, `curric-s2-defend`
|
||||
> (grounded) beats both `curric-s3-no_draws` (26-60) and `curric-s4-mechanics`
|
||||
> (24-57) — lifting the locomotion mask to full 3D in stage 3 was a clear
|
||||
> regression in floor play that self-play never earned back in 20M steps.
|
||||
> Stage 5 sidesteps this by resuming and evaluating against stage 2 directly
|
||||
> (`curriculum.py`'s `resume_from_experiment`/`reference_experiment` stage-dict
|
||||
> overrides) instead of chaining through stages 3-4. Stage 6 is where full 3D
|
||||
> flight comes back — not masked away this time, but discouraged via
|
||||
> `airborne_penalty` and given ~12x the training time to settle. See
|
||||
> `curriculum_state.json`'s log for the full eval numbers.
|
||||
|
||||
All curriculum flags default to leaving Godot's own `@export` defaults
|
||||
alone (`train.py` only forwards a flag when you pass it), so ordinary runs
|
||||
@@ -181,7 +182,8 @@ are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
|
||||
`--opponent-model <path>` (for `frozen`), `--draw-penalty`,
|
||||
`--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
|
||||
`--allow-vertical`/`--no-allow-vertical`, `--allow-pitch-roll`/`--no-allow-pitch-roll`,
|
||||
`--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`.
|
||||
`--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`,
|
||||
`--airborne-penalty`.
|
||||
|
||||
### Running it automatically
|
||||
|
||||
|
||||
Reference in New Issue
Block a user