feat(training): curriculum generation 4 — MultiDiscrete action space redesign

Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.

Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.

- Game/scripts/ship_action_codec.gd (new): single source of truth for a
  per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
  by training and in-game inference, replacing the continuous Gaussian.
  thrust_y's bins are deliberately asymmetric so a random policy drifts
  through the volume instead of floor-pinning. Legacy continuous decode
  (ai_ship_controller.gd's old logic) preserved verbatim so every
  pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
  unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
  so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
  vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
  lowered 4x (aerial approaches require pitching); flight telemetry
  (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
  and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
  high, ships low, kept clear of walls) so aerial practice is forced by the
  environment instead of relying on reward-driven exploration alone; snapshot
  terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
  SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
  now-meaningless --reset-std; new EntropyFloorCallback (a persistent
  per-rollout ent_coef controller replacing the one-shot std-reset shock)
  and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
  --tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
  index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
  grounded stage, full action space live from step 1; deletes generation
  1-3's checkpoint-lineage machinery (nothing to resume from); final report
  evaluates against both promoted/easy.json and the new
  promoted/reference-grounded.json (a copy of curric-s5-aggression, the
  strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
  intermediate checkpoint files a single stage was writing (~500MB ->
  ~0.2MB per run); requirements.txt pinned (behaviour here now depends on
  specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
  mismatch before it silently corrupts 24h of training.

Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.

See TRAINING.md's "Generation 4" section for the full design writeup.

TRAINING.md

@@ -161,12 +161,20 @@ the automated curriculum pipeline (`run_training.sh`) only ever writes new
 flat files there, never touching subdirectories.
 
 `Game/bots/promoted/<tier>.json` is the small, curated, hand-maintained set
-actually referenced by the shipped game — currently just `easy.json`
-(promoted 2026-07-24 from `curric-s6-unmask`, the strongest checkpoint at the
-time). `match.tscn`/`spectate.tscn` point their `bot_model_path` exports here
-directly, so a promoted file is never touched by training scripts, never
-overwritten by a same-named future export, and never disturbed by pruning old
-experiment files from the flat dump.
+actually referenced by the shipped game — currently `easy.json` (promoted
+2026-07-24 from `curric-s6-unmask`, the strongest checkpoint at the
+time — note `curric-s6-unmask` was itself generation 1's *failed* unmask
+stage, so `easy.json` is weaker than `reference-grounded.json` below; a
+strong generation 4 result should promote a real replacement, plus
+`medium.json`/`hard.json`) and `reference-grounded.json` (added for
+generation 4 — a copy of generation 3's `curric-s5-aggression`, made before
+the flat `Game/bots/` dump was scrapped for the redesign, kept as the
+strongest grounded-era artifact and the fixed yardstick generations 1-3 were
+all measured against; see "Generation 4"'s final report). `match.tscn`/
+`spectate.tscn` point their `bot_model_path` exports here directly, so a
+promoted file is never touched by training scripts, never overwritten by a
+same-named future export, and never disturbed by pruning old experiment
+files from the flat dump.
 
 To promote a new bot into a tier: copy the chosen `Game/bots/<experiment>.json`
 to `Game/bots/promoted/<tier>.json` (overwriting the old one), and note the
@@ -185,13 +193,16 @@ movement. Each stage is a normal chained run — a new `--experiment` resumed
 via `--resume checkpoints/<previous>/final.zip`, same as any other run —
 just with different curriculum flags.
 
-`curriculum.py` has run through three generations so far. Generation 1
+`curriculum.py` has run through four generations so far. Generation 1
 (below) ran stages 1-6 to completion/block and is archived; generation 2
 started a fresh stage 1 seeded from generation 1's last clean pass instead
 of continuing to retry a stage that kept getting worse, but also failed 3
-attempts; generation 3 (the one `curriculum.py` actually runs today)
-replaces generation 2's single all-or-nothing unmask stage with a gradual
-ramp — see "Generation 3" below.
+attempts; generation 3 replaced generation 2's single all-or-nothing unmask
+stage with a gradual ramp, and also failed (worse, on its final attempt,
+than either prior generation); generation 4 (the one `curriculum.py` actually
+runs today) is a full redesign, not a further patch — see "Generation 4"
+below, and "Generation 3" for why a fourth attempt at gating *when* the
+policy could use full 3D controls was abandoned rather than retried again.
 
 ### Generation 1 (archived — see `curriculum_state_gen1.json`)
 
@@ -319,28 +330,174 @@ stages, so the ramp is the sole studied variable.
 `curriculum_state_gen2.json`) rather than continuing to log against a stage
 list whose stage 0 no longer means what it used to.
 
-**Open question, not yet resolved by data:** the ramp scales the action's
-effect in Godot, which runs *after* PPO samples the action — PPO's own
-std-collapse dynamics don't directly see the ramp, only the reward it
-produces. It's possible this doesn't prevent the collapse, or even makes it
-happen faster at low ramp values (weaker reward signal on those axes gives
-less incentive to keep exploring them). Watch `train/std` per stage in
-TensorBoard rather than assuming the ramp is working. If the final gated
-stage still lands ~28-32%, that's evidence the plateau isn't an
-exploration/collapse problem at all — worth revisiting reward shaping, or
-trying `--opponent-mode frozen --opponent-model <path>` during the warmup
-stages (implemented, never yet exercised in this project) to remove
-self-play's moving-target instability while the policy first learns to use
-the new axes.
+**Open question, resolved 2026-08-04.** The gated `unmask` stage failed all
+3 attempts: 29% → 30% → **24%** win rate vs `curric-s5-aggression` (the
+third, worst by then) — landing in the same ~28-32% band the section above
+flagged as "evidence the plateau isn't an exploration/collapse problem at
+all." `train/std` collapsed from ~0.30 to ~0.13-0.15 within the first ~10%
+of steps in every attempt of every generation regardless of hard-mask vs.
+gradual-ramp mechanism, so gating *when* the axes were allowed to act never
+addressed the actual cause. See "Generation 4" below for the redesign and
+root-cause diagnosis this prompted, and `curriculum_state_gen3.json` for the
+archived full log.
+
+### Generation 4 (current) — action space redesign, not a further ramp patch
+
+Three generations spent ~2 weeks trying different ways to gate *when* the
+policy could use vertical thrust/pitch/roll on top of a continuous Gaussian
+action space, and all three converged on the same failure: PPO's action
+std collapsing within the first ~10% of steps and never recovering,
+regardless of mechanism. Research into how self-play PPO bots that have
+actually solved this class of problem (RLGym/RLBot's Necto/Nexto) approach
+it turned up a structural difference — they don't gate control authority at
+all; they train the full action space from step 1 using discrete/bucketed
+actions, not a continuous Gaussian, plus reward/state-setter curriculum
+instead of action masking.
+
+**Root cause, verified against this project's own physics** (not assumed):
+flight in this game is a *sustained set-point*, not an impulse. Ship mass
+5.0, `vertical_thrust` 120 (`ship.gd`/`ship.tscn`), default gravity 9.8 m/s²
+→ hovering requires *holding* `thrust.y ≈ 0.408` continuously. A Gaussian
+whose mean sits near 0 and whose σ has collapsed to ~0.13 samples
+`thrust.y ∈ [-0.4, 0.4]` — it can brush the hover value but can never *hold*
+it long enough to earn the reward gradient that would move the mean. That's
+a fixed point; no ramp on the axis's downstream *effect* (which is applied
+*after* PPO samples the action) moves it, exactly as the "open question"
+above speculated might be the case. Independently, this redesign also found
+and fixed a real, previously-unnoticed bug unrelated to the action space:
+`godot_rl`'s `godot_env.py` never marks an episode timeout as a truncation
+(it returns the same `done` array for both term and trunc — see its own
+`# TODO update API to term, trunc`), so PPO was bootstrapping `V(s_T)=0` on
+every 30s draw in every generation to date instead of correctly estimating
+the value of the state it timed out in.
+
+**Action space**: switched to per-axis `MultiDiscrete` (7 heads, `nvec =
+[5,5,5,5,5,5,2]`) instead of continuous `Box(7)` — see
+`Game/scripts/ship_action_codec.gd`, the single source of truth for the
+layout/decode shared by training and in-game inference. Not a single
+lookup table (RLGym's approach for Rocket League's *coupled* car controls):
+Cosmic Clash's 7 axes are near-independent thruster/torque channels, so a
+curated combination table would throw away that factorization for no
+benefit. `thrust_y`'s bins are deliberately asymmetric
+(`-0.5, 0, 0.45, 0.75, 1.0`, vs. the symmetric `-1, -0.5, 0, 0.5, 1` on
+every other axis) — a uniform-random policy over those 5 bins averages
+0.34, just below the 0.408 hover point, so a fresh policy drifts gently
+through the volume instead of pinning to the floor (symmetric bins) or
+sticking to the ceiling (`ceiling_pull_strength` 11.5 > gravity 9.8). This
+is the direct analogue of the RLGym/RLBot fix for the same failure mode
+("add more jump actions to the discrete action parser"). `godot_rl`'s
+`ActionSpaceProcessor` already emits `MultiDiscrete` with zero Python-side
+changes when every action entry is `Discrete` — the only reason this
+project's action space flattened to `Box(7)` before was that `turbo`
+(binary) was mixed with continuous entries.
+
+**Backward compatibility**: every export before generation 4 (e.g.
+`Game/bots/promoted/easy.json`) has no `"action_space"` field in its JSON;
+absence means `{"type": "continuous"}` and decodes through the exact same
+path as before (`ShipActionCodec.from_continuous`, moved verbatim out of
+`ai_ship_controller.gd`). `PolicyNetwork.gd`'s forward pass itself never
+changed — only the caller's decode branches on the model's declared type.
+`export_policy.py`'s parity check is now index-level for a `MultiDiscrete`
+model (argmax per head's logit slice, compared against SB3's own
+`deterministic=True` chosen index) rather than comparing clipped floats,
+since a head-order mistake would otherwise train and export cleanly and
+only surface as silently wrong in-game behaviour.
+
+**No grounded stage.** Full action space live from step 1 — no successful
+self-play RL bot in this problem class gates control authority, it's failed
+9/9 attempts (3 generations × 3 attempts) here, and every prior generation's
+checkpoints are a different, incompatible action/observation shape anyway
+(nothing to resume from). 3 stages instead of a ramp:
+
+| Stage | Opponent | Timesteps | Gated | What it teaches |
+|---|---|---|---|---|
+| 1 — `bootstrap` | `inert` | 40M (~4h) | No | Empty-net finishing from a random policy — no moving target, full action space from the start. |
+| 2 — `selfplay` | `self_play` | 160M (~16h) | Yes, vs stage 1 | Where essentially all the learning happens. |
+| 3 — `gauntlet` | `frozen` = stage 2's own export | 120M (~12h) | Yes, vs stage 2 | A stationary opponent for a low-variance measurement, and a check that self-play didn't converge to a fixed point that only beats itself. |
+
+An "air drill" state-setter branch (`training_mode.gd`'s `air_drill_chance`,
+new — ball spawned high, both ships spawned low and lateral, unsolvable
+without climbing, kept clear of every wall so the RLGym-warned wall-bounce
+exploit has no wall nearby to bounce off) runs at a constant rate across
+*all* stages rather than being introduced late — gating *when* a skill gets
+drilled would reproduce the exact "gate what the policy can do" pattern
+that failed 3 generations running.
+
+**Observations**: `ShipObservations.SIZE` grew 31 → 35 (own contact normal
++ an `in_contact` flag, appended — never inserted, see that file's
+append-only invariant) so the value function can actually see the condition
+`wall_contact_penalty` fires on, instead of predicting a reward with no
+supporting signal.
+
+**Reward shaping**: mostly unchanged — a farmability check on the existing
+weights (`velocity_to_ball_weight`'s term telescopes to ~2.7 over a 20m
+approach, well under `goal_reward`=80; not gameable) argues generation 2/3's
+tuning was never the actual problem. Two changes: `airborne_penalty` is no
+longer passed by any stage (previously ramped *up* in lockstep with the
+axis generation 3 was trying to teach — directly adversarial to the goal of
+genuine aerial play), and `tilt_penalty` dropped 4x (0.002 → 0.0005 default)
+since an aerial approach to a high ball requires pitching. Deliberately
+*not* added: a standalone air-touch reward — that's the exact exploit RLGym
+warns about ("hits the ball off a wall high up instead of doing a real
+aerial"); the air-drill state setter already makes aerial skill
+instrumentally necessary to earn the existing ball-directed rewards.
+
+**Exploration**: `--reset-std` (meaningless under `MultiDiscrete` — no
+`log_std`) is replaced by `--reset-logits <scale>` (multiplies
+`action_net`'s weights/bias, optionally scoped to specific heads via
+`--reset-logits-heads`) for a deliberate post-diagnosis recovery, and more
+importantly by `--entropy-floor` (`train.py`'s `EntropyFloorCallback`): a
+*persistent* per-rollout controller nudging `ent_coef` to hold policy
+entropy near a target that decays over the run, replacing the one-shot
+`--reset-std` shock that reliably decayed away within ~10% of steps in
+every prior generation with something that responds continuously instead of
+once. `--ent-coef`'s default rose 0.0001 → 0.01 (tuned for `MultiDiscrete`'s
+bounded ~10-nat entropy, not a Gaussian's unbounded differential entropy).
+Per-head entropy (`train/entropy_head_<name>`) replaces the old aggregate
+`train/std` scalar — it identifies *which* axis is collapsing instead of
+one number for all seven.
+
+**Validation before spending the full ~32h budget**: see the ladder below —
+cheapest checks first (an offline action-space assertion, a headless Godot
+boot, a 100k-step smoke run, export parity + an in-game round trip against
+`easy.json`), then flight telemetry (`rollout/airborne_fraction`,
+`mean_altitude`, `air_touch_fraction`, `vertical_thrust_mean` — leading
+indicators visible from the first rollout instead of only in a win rate
+measured a full run later), then a short controlled A/B (MultiDiscrete vs.
+continuous, otherwise identical, ~20M steps each) before committing to the
+full curriculum — every past generation bet a full day on an unfalsifiable
+hypothesis, which is what made each failure expensive to diagnose.
+
+1. `training/test_action_space.py` — offline, seconds. Catches a head-order
+   mismatch, the single most likely silent killer (trains "fine" for 24h,
+   produces garbage — e.g. pitch commands driving strafe thrusters — with no
+   error).
+2. `godot --headless --path Game res://scenes/training.tscn` with no
+   trainer listening, 30s — catches `class_name`/observation-size
+   regressions.
+3. `.venv/bin/python train.py --experiment smoke --timesteps 100000
+   --n-parallel 2` — confirms the `MultiDiscrete` handshake and new
+   callback metrics emit.
+4. `export_policy.py` on the smoke checkpoint (mandatory index-level parity
+   check), then `evaluate.py <smoke>.json ../Game/bots/promoted/easy.json
+   --episodes 4` — exercises the real GDScript decode path.
+5. A short A/B: two 20M-step runs, identical except action space
+   (`MultiDiscrete` vs. the old continuous `Box(7)`), comparing
+   `rollout/airborne_fraction`. If discrete pulls meaningfully ahead, the
+   32h curriculum is a justified bet; if both stay near zero, the
+   hypothesis above is wrong and reward/compute explanations move to the
+   front — cheaper than a 4th blind multi-day generation either way.
 
 All curriculum flags default to leaving Godot's own `@export` defaults
 alone (`train.py` only forwards a flag when you pass it), so ordinary runs
 are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
 `--opponent-model <path>` (for `frozen`), `--draw-penalty`,
 `--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
-`--vertical-ramp`, `--pitch-roll-ramp` (0.0-1.0 locomotion-unmask ramp),
+`--air-drill-chance` (generation 4's state-setter aerial curriculum),
 `--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`,
-`--airborne-penalty`, `--ball-velocity-to-goal-weight`, `--goal-reward`.
+`--airborne-penalty`, `--tilt-penalty`, `--ball-velocity-to-goal-weight`,
+`--goal-reward`. (`--vertical-ramp`/`--pitch-roll-ramp` are gone — generation
+4 has no locomotion mask/ramp to control.)
 
 ### Running it automatically
 
@@ -348,17 +505,19 @@ are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
 pattern as `start_training.sh`) drives all stages end to end: for each
 stage it runs `run_training.sh` (pull, train, export, commit+push) with that
 stage's flags, then evaluates the resulting checkpoint against a reference
-bot over 100 episodes — the fixed `rookie.json` baseline for a from-scratch
-stage 1 (no `resume_from_experiment`/`reference_experiment` override on
-`STAGES[0]`), the previous stage's promoted checkpoint by default for
-stages 2+, or an explicit override in that stage's dict when it deliberately
-skips a since-regressed branch (generation 1's stage 5) or seeds from a
-fixed foundation checkpoint (generation 2's stage 1 — see above).
+bot over 100 episodes — the previous stage's own passing export (stage 1 is
+ungated, so this only applies to stages 2+). Once every stage passes, a
+final (non-gating) report evaluates the result against both
+`Game/bots/promoted/easy.json` (the shipped bot) and
+`Game/bots/promoted/reference-grounded.json` (a copy of generation 3's
+`curric-s5-aggression`, the strongest grounded-era artifact and the
+yardstick generations 1-3 were all measured against) — those two numbers are
+what actually answer "did generation 4 work?"
 
 ```bash
 cd training
-./curriculum.sh                                              # start/resume the curriculum
-./curriculum.sh --seed-checkpoint checkpoints/run11/final.zip # override stage 1's resume source for this run
+./curriculum.sh                                            # start/resume the curriculum
+./curriculum.sh --seed-checkpoint checkpoints/some/final.zip # override stage 1's resume source for this run
 ```
 
 The gate is deliberately lenient: it blocks a stage only on a **clear
@@ -374,13 +533,13 @@ result are logged to `curriculum_state.json` (committed alongside
 A stage gets up to 2 retries (3 attempts total) before the script stops and
 asks for a human look — it will not retry indefinitely or advance past a
 stage that keeps failing on its own. By default a retry resumes from that
-stage's own previous attempt with a fresh `--reset-std`; a stage can instead
-set `reset_retry_checkpoint: True` (generation 2's stage 1 does) to always
-reset to its normal resume source instead — see the generation 1 → 2
-postmortem above for why blind same-checkpoint retries can make things
-monotonically worse. Once you've looked at why a block happened (more
-timesteps? a flag needs adjusting? the eval itself was misleading?), re-run
-with `--force-retry` to try again or `--skip-to-next-stage` if you judge the
+stage's own previous attempt (no `reset_retry_checkpoint` stage override is
+set in generation 4 — nothing yet suggests a retry needs to reset to a
+clean upstream checkpoint the way generation 3's single `unmask` stage did;
+add one if a stage's retries turn out to be drifting rather than
+converging). Once you've looked at why a block happened (more timesteps? a
+flag needs adjusting? the eval itself was misleading?), re-run with
+`--force-retry` to try again or `--skip-to-next-stage` if you judge the
 result good enough despite the gate.
 
 Running a stage by hand (e.g. to experiment with flags before trusting the