mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-14 07:22:05 +00:00
feat(*): add staged curriculum training with an automated stage-by-stage orchestrator
This commit is contained in:
+70
-5
@@ -135,10 +135,75 @@ mode (`bot_model_path`, `bot_reaction_ticks`, `bot_action_noise` in
|
||||
reactions, easier.
|
||||
- **action_noise**: adds execution error, easier.
|
||||
|
||||
## Curriculum training
|
||||
|
||||
Training from scratch with self-play alone hands the network every skill
|
||||
at once — finishing, defending, positioning, not stalling to a draw — off a
|
||||
sparse ±40 goal reward. `train.py` has a `curriculum` flag group that stages
|
||||
this the way you'd coach a human: score first, then also defend, then learn
|
||||
that a draw is still a failure, and only then spend compute polishing general
|
||||
movement. Each stage is a normal chained run — a new `--experiment` resumed
|
||||
via `--resume checkpoints/<previous>/final.zip`, same as any other run —
|
||||
just with different curriculum flags.
|
||||
|
||||
| Stage | Flags | What it teaches |
|
||||
|---|---|---|
|
||||
| 1 — score | `--opponent-mode inert --attack-goal-bias 1.0 --no-allow-vertical --no-allow-pitch-roll` | Team 1 is a do-nothing placeholder ship parked at its spawn (an effectively empty net); near-goal resets always target the goal the trainee attacks; the ship can't fly or pitch/roll, only drive and yaw. Isolated finishing practice. |
|
||||
| 2 — defend too | `--opponent-mode self_play --no-allow-vertical --no-allow-pitch-roll` | Reintroduces a live opponent (self-play) and the default episode-start mix — the same near-goal state is now simultaneously a finishing chance for one side and a defensive save for the other. Locomotion stays grounded. |
|
||||
| 3 — no draws | `--draw-penalty 5 --reset-std 0.3` | Training episodes are golden-goal (end at the *first* goal), so there's no in-episode goal-margin to penalize — `draw_penalty` is the closest available signal: a one-time penalty when an episode times out with no goal at all, on top of the existing per-tick `time_penalty`. Also lifts the locomotion mask (full 3D controls) by omitting `--allow-vertical`/`--allow-pitch-roll`; pair that with `--reset-std` since the policy never got a reward gradient on those axes before now, so expect a brief re-exploration wobble. |
|
||||
| 4 — mechanics/refinement | *(no curriculum flags — plain `next_run.sh`)* | Stock self-play, full controls, default reward/start-state mix. This is what all runs before this feature already did. |
|
||||
|
||||
All curriculum flags default to leaving Godot's own `@export` defaults
|
||||
alone (`train.py` only forwards a flag when you pass it), so ordinary runs
|
||||
are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
|
||||
`--opponent-model <path>` (for `frozen`), `--draw-penalty`,
|
||||
`--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
|
||||
`--allow-vertical`/`--no-allow-vertical`, `--allow-pitch-roll`/`--no-allow-pitch-roll`.
|
||||
|
||||
### Running it automatically
|
||||
|
||||
`training/curriculum.py` (started via `curriculum.sh`, same detached-tmux
|
||||
pattern as `start_training.sh`) drives all four stages end to end: for each
|
||||
stage it runs `run_training.sh` (pull, train, export, commit+push) with that
|
||||
stage's flags, then evaluates the resulting checkpoint against a reference
|
||||
bot — the fixed `rookie.json` baseline for stage 1, or the previous stage's
|
||||
promoted checkpoint for stages 2-4 — over 100 episodes.
|
||||
|
||||
```bash
|
||||
cd training
|
||||
./curriculum.sh # start/resume the curriculum
|
||||
./curriculum.sh --seed-checkpoint checkpoints/run11/final.zip # seed stage 1 instead of a fresh policy
|
||||
```
|
||||
|
||||
The gate is deliberately lenient: it blocks a stage only on a **clear
|
||||
regression** (the reference beating the candidate by 15+ points of win
|
||||
rate), not "must show improvement." A 40-episode eval already misled us once
|
||||
in this project — run11 was the first model to deliberately score a goal,
|
||||
but its head-to-head eval read as a loss on sample noise alone. A strict
|
||||
gate would have retried that stage forever for the wrong reason; a loose one
|
||||
still catches a genuinely broken stage. Progress and every attempt's eval
|
||||
result are logged to `curriculum_state.json` (committed alongside
|
||||
`eval_history.json` after each attempt).
|
||||
|
||||
A stage gets up to 2 retries (3 attempts total, each resuming from that
|
||||
stage's own previous attempt with a fresh `--reset-std`) before the script
|
||||
stops and asks for a human look — it will not retry indefinitely or advance
|
||||
past a stage that keeps failing on its own. Once you've looked at why (more
|
||||
timesteps? a flag needs adjusting? the eval itself was misleading?), re-run
|
||||
with `--force-retry` to try again or `--skip-to-next-stage` if you judge the
|
||||
result good enough despite the gate.
|
||||
|
||||
Running a stage by hand (e.g. to experiment with flags before trusting the
|
||||
orchestrator) still works exactly as the table above describes — just call
|
||||
`next_run.sh`/`run_training.sh` directly with that stage's flags.
|
||||
|
||||
## Self-play notes
|
||||
|
||||
Both ships share the live policy (mirrored, team-relative observations — see
|
||||
`ship_observations.gd`), so training is always against the current self.
|
||||
Fixed-opponent training against frozen checkpoints (league play, to avoid
|
||||
strategy collapse on long runs) is deferred — see TODO.md; the pieces
|
||||
(exported JSON bots + `AIShipController`) already exist.
|
||||
By default both ships share the live policy (mirrored, team-relative
|
||||
observations — see `ship_observations.gd`), so training is against the
|
||||
current self. `--opponent-mode inert`/`frozen` (see Curriculum training
|
||||
above) replace that with a placeholder or a fixed exported policy for one
|
||||
side of a run; `frozen` is a single-fixed-model slice of full league play.
|
||||
Fixed-opponent training against a *pool* of past checkpoints sampled per
|
||||
episode (to avoid strategy collapse on long self-play runs) is still
|
||||
deferred — see TODO.md.
|
||||
|
||||
Reference in New Issue
Block a user