chore(training): curriculum progress after 20260731-2149-curric-s1-unmask-ramp25

Generation 2's single "unmask" stage (flip vertical/pitch-roll locomotion
from grounded-only to full 3D in one step) failed 3 independent 240M-step
attempts, landing at a stable 32% / 28% / 31% win rate vs curric-s5-aggression
each time -- not noise, and not fixable by more training time (attempts 2-3
each continued the same checkpoint lineage for another full 240M steps with
zero improvement). Every attempt shows train/std collapsing from ~0.30 to
~0.13-0.15 within the first ~10% of steps and never recovering: the policy
locks the newly-opened axes back down before ever meaningfully exploring
them.

Replaces the boolean allow_vertical/allow_pitch_roll mask on ShipAIController
with float vertical_ramp/pitch_roll_ramp multipliers (0.0-1.0), scaling axis
effect in set_action() instead of gating it outright -- the action space
never changes shape, so checkpoints stay resumable across ramp values. The
single unmask stage in curriculum.py becomes 4: three ungated warmup stages
(25%/50%/75% authority, airborne_penalty ramping in step) that train,
checkpoint, and always advance with no eval gate, then the measured stage at
full authority -- same reference, opponent mode, and 240M budget as the 3
failed attempts, for a direct comparison. Adds a "gated" flag/branch to
main()'s loop for the ungated stages.

This is generation 3 of the curriculum; generation 2's state is archived to
curriculum_state_gen2.json (mirroring the earlier gen1 -> gen2 archival) and
curriculum_state.json resets fresh, since its stage 0 no longer means what it
used to. See TRAINING.md's "Generation 3" section for the full postmortem,
stage table, and the open question about whether scaling action effect in
Godot (which PPO's own entropy/exploration math never sees) actually
addresses the collapse.

Game/scripts/ship_ai_controller.gd

@@ -79,15 +79,24 @@ extends AIController3D
 # unaffected; the floor-lock curriculum stage turns it on.
 @export var airborne_penalty := 0.0
 
-# Locomotion curriculum: when false, the corresponding action axes are
+# Locomotion curriculum: scales how much of the corresponding action axes
-# discarded in set_action before reaching the ship, so the ship stays
+# actually reaches the ship, from 0.0 (fully discarded, grounded-only) to
-# grounded and only yaws — basic scoring/defending doesn't need 3D flight.
+# 1.0 (full effect) — this scales the *effect* of thrust.y/rotation.x/
-# This masks the *effect* of thrust.y/rotation.x/rotation.z, not the action
+# rotation.z in set_action, not the action space's shape: the policy always
-# space's shape: the policy still outputs values for these axes (still
+# outputs values for these axes (always contributing to PPO's entropy/log-
-# contributing to PPO's entropy/log-prob), they're just discarded here, so
+# prob), they're just attenuated here, so checkpoints stay resumable across
-# checkpoints stay resumable once a later curriculum stage re-enables them.
+# ramp values.
-@export var allow_vertical := true
+#
-@export var allow_pitch_roll := true
+# A hard 0/1 flip (the original bool mask) let PPO's action-distribution
+# std collapse to ~0.13-0.15 within the first ~10% of steps, before the
+# policy ever meaningfully explored the newly-unmasked axes — 3 independent
+# 240M-step attempts at the all-or-nothing flip all landed at a stable
+# ~28-32% win rate vs curric-s5-aggression (see TRAINING.md's generation 3
+# section). A gradual ramp across several short curriculum stages, each
+# resuming from the previous ramp value's checkpoint, lets the policy adopt
+# each axis incrementally instead of all at once.
+@export_range(0.0, 1.0) var vertical_ramp := 1.0
+@export_range(0.0, 1.0) var pitch_roll_ramp := 1.0
 
 # Contact normals with y above this are floor contact (exempt from the wall
 # penalty); below it they read as wall (sideways) or ceiling (downward).
@@ -163,9 +172,9 @@ func get_action_space() -> Dictionary:
 func set_action(action) -> void:
 	var thrust: Array = action["thrust"]
 	var rot: Array = action["rotation"]
-	var thrust_y: float = thrust[1] if allow_vertical else 0.0
+	var thrust_y: float = thrust[1] * vertical_ramp
-	var pitch: float = rot[0] if allow_pitch_roll else 0.0
+	var pitch: float = rot[0] * pitch_roll_ramp
-	var roll: float = rot[2] if allow_pitch_roll else 0.0
+	var roll: float = rot[2] * pitch_roll_ramp
 	rl_controller.action.thrust = Vector3(thrust[0], thrust_y, thrust[2])
 	rl_controller.action.rotation = Vector3(pitch, rot[1], roll)
 	rl_controller.action.turbo = int(action["turbo"]) == 1

Game/scripts/training_mode.gd

@@ -188,7 +188,7 @@ const SHIP_AI_OVERRIDES := [
 	"ball_touch_reward", "ball_touch_cooldown_ticks", "ball_touch_direction_floor",
 	"velocity_to_ball_weight", "ball_velocity_to_goal_weight", "ball_distance_penalty",
 	"wall_contact_penalty", "tilt_penalty", "speed_reward_weight", "time_penalty",
-	"airborne_penalty", "allow_vertical", "allow_pitch_roll",
+	"airborne_penalty", "vertical_ramp", "pitch_roll_ramp",
 ]
 
 
@@ -239,7 +239,7 @@ func _ai_default(name: String) -> Variant:
 		"speed_reward_weight": return 0.004
 		"time_penalty": return 0.001
 		"airborne_penalty": return 0.0
-		"allow_vertical", "allow_pitch_roll": return true
+		"vertical_ramp", "pitch_roll_ramp": return 1.0
 		_: return null
 
 

TODO.md

@@ -8,7 +8,7 @@ The training pipeline is built — see `TRAINING.md` (self-play PPO via the vend
 
 - [x] Promote a first tier: `curric-s6-unmask` copied into `Game/bots/promoted/easy.json` as the shipped "easy" bot (see TRAINING.md's "Promoted bots" section) — `match.tscn`/`spectate.tscn` now default there instead of `run05.json`.
 - [ ] Long training runs on the Linux/3090 box to produce actually-good bots; promote further checkpoints into `Game/bots/promoted/` as `medium`/`hard` tiers once they clear `easy.json` in `evaluate.py`.
-- [x] Staged curriculum (score → defend → avoid draws → full mechanics) via `train.py`'s `--opponent-mode`/`--draw-penalty`/`--attack-goal-bias`/`--allow-vertical`/`--allow-pitch-roll` flags — see TRAINING.md's "Curriculum training" section. `--opponent-mode=frozen` is a single-fixed-model slice of the league idea below, not the full sampled pool.
+- [x] Staged curriculum (score → defend → avoid draws → full mechanics) via `train.py`'s `--opponent-mode`/`--draw-penalty`/`--attack-goal-bias`/`--vertical-ramp`/`--pitch-roll-ramp` flags — see TRAINING.md's "Curriculum training" section. `--opponent-mode=frozen` is a single-fixed-model slice of the league idea below, not the full sampled pool.
 - [ ] Frozen-opponent league: train the live policy against a *pool* of past exported checkpoints, sampled per-episode (today's `--opponent-mode=frozen` only supports one fixed model per run) to prevent self-play strategy collapse on long runs.
 - [ ] Richer state setter / curriculum: aerial states, wall plays, rebound scenarios as skill grows (beyond the score/defend/draw staging already in place).
 - [ ] Main-menu difficulty picker (Match already takes `bot_model_path`/`bot_reaction_ticks`/`bot_action_noise` exports).

TRAINING.md

@@ -185,11 +185,13 @@ movement. Each stage is a normal chained run — a new `--experiment` resumed
 via `--resume checkpoints/<previous>/final.zip`, same as any other run —
 just with different curriculum flags.
 
-`curriculum.py` has run through two generations so far. Generation 1 (below)
+`curriculum.py` has run through three generations so far. Generation 1
-ran stages 1-6 to completion/block and is archived; generation 2 (the one
+(below) ran stages 1-6 to completion/block and is archived; generation 2
-`curriculum.py` actually runs today) starts a fresh stage 1 seeded from
+started a fresh stage 1 seeded from generation 1's last clean pass instead
-generation 1's last clean pass instead of continuing to retry a stage that
+of continuing to retry a stage that kept getting worse, but also failed 3
-kept getting worse — see "Generation 2" below.
+attempts; generation 3 (the one `curriculum.py` actually runs today)
+replaces generation 2's single all-or-nothing unmask stage with a gradual
+ramp — see "Generation 3" below.
 
 ### Generation 1 (archived — see `curriculum_state_gen1.json`)
 
@@ -224,9 +226,9 @@ vs `curric-s5-aggression`). After 3 failed attempts the script blocked for
 human review; rather than pile up `retry4`, `retry5`, ... on a lineage that
 kept getting worse, generation 2 (below) replaces it with a fresh stage 1.
 
-### Generation 2 (current)
+### Generation 2 (archived — see `curriculum_state_gen2.json`)
 
-`curriculum.py`'s live `STAGES` list now contains a single stage, `unmask`
+`curriculum.py`'s `STAGES` list contained a single stage, `unmask`
 (displays as stage 1 — `curric-s1-unmask`), which picks up exactly where
 generation 1's regression analysis left off. It resumes directly from
 `FOUNDATION_EXPERIMENT` (`curric-s5-aggression`'s own checkpoint — the last
@@ -271,12 +273,72 @@ generation 1's plain ones (both checkpoint directories and TensorBoard run
 names come straight from `--experiment`) and makes run order obvious in
 TensorBoard without cross-referencing `curriculum_state.json`.
 
+**Generation 2 also failed 3 attempts in a row**, landing at a stable
+32% / 28% / 31% win rate vs `curric-s5-aggression` each time — the second
+and third attempts each continued the *same* checkpoint lineage for another
+full 240M steps with zero improvement, ruling out both the reward retune
+above and "just needs more time" as fixes. Every attempt showed `train/std`
+collapsing from ~0.30 to ~0.13-0.15 within the first ~10% of steps and never
+recovering. See "Generation 3" below for the redesign this prompted.
+
+### Generation 3 (current)
+
+Generation 2's failures point at the *mechanism* of the transition, not the
+reward weights: flipping `allow_vertical`/`allow_pitch_roll` from false to
+true in one step let PPO's action-distribution std collapse on those axes
+before the policy ever meaningfully explored them. Generation 3 replaces
+that boolean mask with a float ramp (`vertical_ramp`/`pitch_roll_ramp` on
+`ShipAIController`, 0.0-1.0, multiplying the axis's effect in `set_action`
+instead of gating it) and spreads the transition across 4 stages instead of
+1:
+
+| Stage | `vertical-ramp`/`pitch-roll-ramp` | `airborne-penalty` | timesteps | gated |
+|---|---|---|---|---|
+| 1 — `unmask-ramp25` | 0.25 | 0.0 | 40M (~4h) | No — trains, checkpoints, always advances |
+| 2 — `unmask-ramp50` | 0.5 | 0.001 | 40M (~4h) | No |
+| 3 — `unmask-ramp75` | 0.75 | 0.002 | 40M (~4h) | No |
+| 4 — `unmask` | 1.0 | 0.003 | 240M (~24h) | **Yes** — evaluated against `curric-s5-aggression`, same 15-point regression gate as every prior attempt |
+
+The 3 warmup stages are deliberately ungated: they're waypoints en route to
+the real, measured transition, not decisions in their own right, so
+`curriculum.py`'s `main()` loop trains and checkpoints them and always
+advances (no eval call, no retry logic — there's nothing to fail against).
+Only the final `unmask` stage is evaluated, with the same reference bot,
+opponent mode (`self_play`, not `frozen` — kept identical to every prior
+attempt so a pass or fail cleanly isolates the ramp as the only variable),
+and 240M-step budget as all 3 failed all-or-nothing attempts, for a direct
+comparison. `airborne_penalty` ramps in step with the axes so it doesn't
+fight a still-mostly-inert axis early on.
+
+All the reward-shaping flags from generation 2's stage (`velocity-to-ball-weight`,
+`ball-distance-penalty`, `ball-touch-reward`, `ball-velocity-to-goal-weight`,
+`goal-reward`, `draw-penalty`) are unchanged and identical across all 4
+stages, so the ramp is the sole studied variable.
+
+`curriculum_state.json` was reset (generation 2's log archived to
+`curriculum_state_gen2.json`) rather than continuing to log against a stage
+list whose stage 0 no longer means what it used to.
+
+**Open question, not yet resolved by data:** the ramp scales the action's
+effect in Godot, which runs *after* PPO samples the action — PPO's own
+std-collapse dynamics don't directly see the ramp, only the reward it
+produces. It's possible this doesn't prevent the collapse, or even makes it
+happen faster at low ramp values (weaker reward signal on those axes gives
+less incentive to keep exploring them). Watch `train/std` per stage in
+TensorBoard rather than assuming the ramp is working. If the final gated
+stage still lands ~28-32%, that's evidence the plateau isn't an
+exploration/collapse problem at all — worth revisiting reward shaping, or
+trying `--opponent-mode frozen --opponent-model <path>` during the warmup
+stages (implemented, never yet exercised in this project) to remove
+self-play's moving-target instability while the policy first learns to use
+the new axes.
+
 All curriculum flags default to leaving Godot's own `@export` defaults
 alone (`train.py` only forwards a flag when you pass it), so ordinary runs
 are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
 `--opponent-model <path>` (for `frozen`), `--draw-penalty`,
 `--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
-`--allow-vertical`/`--no-allow-vertical`, `--allow-pitch-roll`/`--no-allow-pitch-roll`,
+`--vertical-ramp`, `--pitch-roll-ramp` (0.0-1.0 locomotion-unmask ramp),
 `--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`,
 `--airborne-penalty`, `--ball-velocity-to-goal-weight`, `--goal-reward`.