feat(training): Replace all-or-nothing unmask with a gradual ramp

Generation 2's single "unmask" stage (flip vertical/pitch-roll locomotion
from grounded-only to full 3D in one step) failed 3 independent 240M-step
attempts, landing at a stable 32% / 28% / 31% win rate vs curric-s5-aggression
each time -- not noise, and not fixable by more training time (attempts 2-3
each continued the same checkpoint lineage for another full 240M steps with
zero improvement). Every attempt shows train/std collapsing from ~0.30 to
~0.13-0.15 within the first ~10% of steps and never recovering: the policy
locks the newly-opened axes back down before ever meaningfully exploring
them.

Replaces the boolean allow_vertical/allow_pitch_roll mask on ShipAIController
with float vertical_ramp/pitch_roll_ramp multipliers (0.0-1.0), scaling axis
effect in set_action() instead of gating it outright -- the action space
never changes shape, so checkpoints stay resumable across ramp values. The
single unmask stage in curriculum.py becomes 4: three ungated warmup stages
(25%/50%/75% authority, airborne_penalty ramping in step) that train,
checkpoint, and always advance with no eval gate, then the measured stage at
full authority -- same reference, opponent mode, and 240M budget as the 3
failed attempts, for a direct comparison. Adds a "gated" flag/branch to
main()'s loop for the ungated stages.

This is generation 3 of the curriculum; generation 2's state is archived to
curriculum_state_gen2.json (mirroring the earlier gen1 -> gen2 archival) and
curriculum_state.json resets fresh, since its stage 0 no longer means what it
used to. See TRAINING.md's "Generation 3" section for the full postmortem,
stage table, and the open question about whether scaling action effect in
Godot (which PPO's own entropy/exploration math never sees) actually
addresses the collapse.
This commit is contained in:
Josh Creek
2026-07-31 21:47:53 +01:00
parent 0759e1514b
commit 3fd1c00895
8 changed files with 287 additions and 147 deletions
+21 -12
View File
@@ -79,15 +79,24 @@ extends AIController3D
# unaffected; the floor-lock curriculum stage turns it on.
@export var airborne_penalty := 0.0
# Locomotion curriculum: when false, the corresponding action axes are
# discarded in set_action before reaching the ship, so the ship stays
# grounded and only yaws — basic scoring/defending doesn't need 3D flight.
# This masks the *effect* of thrust.y/rotation.x/rotation.z, not the action
# space's shape: the policy still outputs values for these axes (still
# contributing to PPO's entropy/log-prob), they're just discarded here, so
# checkpoints stay resumable once a later curriculum stage re-enables them.
@export var allow_vertical := true
@export var allow_pitch_roll := true
# Locomotion curriculum: scales how much of the corresponding action axes
# actually reaches the ship, from 0.0 (fully discarded, grounded-only) to
# 1.0 (full effect) — this scales the *effect* of thrust.y/rotation.x/
# rotation.z in set_action, not the action space's shape: the policy always
# outputs values for these axes (always contributing to PPO's entropy/log-
# prob), they're just attenuated here, so checkpoints stay resumable across
# ramp values.
#
# A hard 0/1 flip (the original bool mask) let PPO's action-distribution
# std collapse to ~0.13-0.15 within the first ~10% of steps, before the
# policy ever meaningfully explored the newly-unmasked axes — 3 independent
# 240M-step attempts at the all-or-nothing flip all landed at a stable
# ~28-32% win rate vs curric-s5-aggression (see TRAINING.md's generation 3
# section). A gradual ramp across several short curriculum stages, each
# resuming from the previous ramp value's checkpoint, lets the policy adopt
# each axis incrementally instead of all at once.
@export_range(0.0, 1.0) var vertical_ramp := 1.0
@export_range(0.0, 1.0) var pitch_roll_ramp := 1.0
# Contact normals with y above this are floor contact (exempt from the wall
# penalty); below it they read as wall (sideways) or ceiling (downward).
@@ -163,9 +172,9 @@ func get_action_space() -> Dictionary:
func set_action(action) -> void:
var thrust: Array = action["thrust"]
var rot: Array = action["rotation"]
var thrust_y: float = thrust[1] if allow_vertical else 0.0
var pitch: float = rot[0] if allow_pitch_roll else 0.0
var roll: float = rot[2] if allow_pitch_roll else 0.0
var thrust_y: float = thrust[1] * vertical_ramp
var pitch: float = rot[0] * pitch_roll_ramp
var roll: float = rot[2] * pitch_roll_ramp
rl_controller.action.thrust = Vector3(thrust[0], thrust_y, thrust[2])
rl_controller.action.rotation = Vector3(pitch, rot[1], roll)
rl_controller.action.turbo = int(action["turbo"]) == 1
+2 -2
View File
@@ -188,7 +188,7 @@ const SHIP_AI_OVERRIDES := [
"ball_touch_reward", "ball_touch_cooldown_ticks", "ball_touch_direction_floor",
"velocity_to_ball_weight", "ball_velocity_to_goal_weight", "ball_distance_penalty",
"wall_contact_penalty", "tilt_penalty", "speed_reward_weight", "time_penalty",
"airborne_penalty", "allow_vertical", "allow_pitch_roll",
"airborne_penalty", "vertical_ramp", "pitch_roll_ramp",
]
@@ -239,7 +239,7 @@ func _ai_default(name: String) -> Variant:
"speed_reward_weight": return 0.004
"time_penalty": return 0.001
"airborne_penalty": return 0.0
"allow_vertical", "allow_pitch_roll": return true
"vertical_ramp", "pitch_roll_ramp": return 1.0
_: return null