feat(training): curriculum generation 4 — MultiDiscrete action space redesign

Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.

Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.

- Game/scripts/ship_action_codec.gd (new): single source of truth for a
  per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
  by training and in-game inference, replacing the continuous Gaussian.
  thrust_y's bins are deliberately asymmetric so a random policy drifts
  through the volume instead of floor-pinning. Legacy continuous decode
  (ai_ship_controller.gd's old logic) preserved verbatim so every
  pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
  unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
  so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
  vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
  lowered 4x (aerial approaches require pitching); flight telemetry
  (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
  and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
  high, ships low, kept clear of walls) so aerial practice is forced by the
  environment instead of relying on reward-driven exploration alone; snapshot
  terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
  SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
  now-meaningless --reset-std; new EntropyFloorCallback (a persistent
  per-rollout ent_coef controller replacing the one-shot std-reset shock)
  and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
  --tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
  index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
  grounded stage, full action space live from step 1; deletes generation
  1-3's checkpoint-lineage machinery (nothing to resume from); final report
  evaluates against both promoted/easy.json and the new
  promoted/reference-grounded.json (a copy of curric-s5-aggression, the
  strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
  intermediate checkpoint files a single stage was writing (~500MB ->
  ~0.2MB per run); requirements.txt pinned (behaviour here now depends on
  specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
  mismatch before it silently corrupts 24h of training.

Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.

See TRAINING.md's "Generation 4" section for the full design writeup.

Game/scripts/training_mode.gd

@@ -54,6 +54,16 @@ extends GameMode
 # trainee's near-goal resets are always finishing chances, not a coin flip
 # between attacking and defending an empty net.
 @export_range(0.0, 1.0) var attack_goal_bias := 0.5
+# Fourth episode-start branch (after kickoff/near-goal, before the fully-
+# random fallback): ball spawned high, both ships spawned low and lateral —
+# unsolvable without climbing. Default 0 (off) so ordinary runs are
+# unaffected. Added for curriculum generation 4: the existing random branch
+# already samples ship/ball Y across the full arena height, but that only
+# randomizes the *initial* state — under gravity+drag a floor-pinned policy
+# sinks back to the floor in ~1.5s, so the *stationary* state distribution
+# stayed floor-pinned even though the initial one wasn't. See
+# _place_air_drill.
+@export_range(0.0, 1.0) var air_drill_chance := 0.0
 
 # Placement bounds for randomized episode starts, derived from the standard
 # enclosure (ArenaBoundary). The inset keeps a randomly oriented ship (1x1x4
@@ -188,7 +198,7 @@ func _parse_eval_args() -> void:
 # of silently matching an unrelated inherited export.
 const TRAINING_MODE_OVERRIDES := [
 	"goal_reward", "draw_penalty", "kickoff_state_chance",
-	"ball_near_goal_chance", "attack_goal_bias",
+	"ball_near_goal_chance", "attack_goal_bias", "air_drill_chance",
 ]
 # ShipAIController @export names a curriculum run may override, read as
 # --ai_<name>=<value> to avoid colliding with the names above.
@@ -196,7 +206,7 @@ const SHIP_AI_OVERRIDES := [
 	"ball_touch_reward", "ball_touch_cooldown_ticks", "ball_touch_direction_floor",
 	"velocity_to_ball_weight", "ball_velocity_to_goal_weight", "ball_distance_penalty",
 	"wall_contact_penalty", "tilt_penalty", "speed_reward_weight", "time_penalty",
-	"airborne_penalty", "vertical_ramp", "pitch_roll_ramp",
+	"airborne_penalty",
 ]
 
 
@@ -243,11 +253,10 @@ func _ai_default(name: String) -> Variant:
 		"ball_velocity_to_goal_weight": return 0.004
 		"ball_distance_penalty": return 0.002
 		"wall_contact_penalty": return 0.0025
-		"tilt_penalty": return 0.002
+		"tilt_penalty": return 0.0005
 		"speed_reward_weight": return 0.004
 		"time_penalty": return 0.001
 		"airborne_penalty": return 0.0
-		"vertical_ramp", "pitch_roll_ramp": return 1.0
 		_: return null
 
 
@@ -299,6 +308,14 @@ func _physics_process(_delta):
 				agent.reward -= draw_penalty
 				agent.done = true
 				agent.goal_scored_this_episode = false
+				# Snapshot BEFORE _reset_episode() below, which moves the ship/
+				# ball and would otherwise make this the post-reset state, not
+				# the terminal one PPO needs to bootstrap V(s) from (see
+				# ShipAIController.get_info / cosmic_env.py's truncation remap).
+				# A goal (_on_goal_scored) does NOT do this — a goal is a
+				# genuine terminal, V(s)=0 is correct there.
+				agent.truncated_this_episode = true
+				agent.terminal_obs = ShipObservations.build(agent.ship, agent.opponent, agent.ball, agent.attack_goal_position)
 		_reset_episode()
 		return
 
@@ -330,6 +347,7 @@ func _on_goal_scored(conceding_team: int) -> void:
 		agent.reward += goal_reward if agent.ship.team != conceding_team else -goal_reward
 		agent.done = true
 		agent.goal_scored_this_episode = true
+		agent.truncated_this_episode = false  # genuine terminal, not a timeout
 	_reset_episode()
 
 
@@ -363,6 +381,8 @@ func _reset_episode() -> void:
 	elif roll < kickoff_state_chance + ball_near_goal_chance:
 		_place_ships_random()
 		_place_ball_near_goal()
+	elif roll < kickoff_state_chance + ball_near_goal_chance + air_drill_chance:
+		_place_air_drill()
 	else:
 		_place_ships_random()
 		_place_ball_random()
@@ -373,6 +393,46 @@ func _place_ball_random() -> void:
 	_place_body(ball, Transform3D(Basis.IDENTITY, _random_position()), velocity, Vector3.ZERO)
 
 
+# Extra clearance for the air drill's ball placement specifically — well
+# beyond SPAWN_INSET, and well beyond the ball's own radius. The ball (unlike
+# _random_position) has no collision-avoidance resample, so this is the
+# anti-exploit measure: the RLGym wall-bounce exploit ("hits the ball off a
+# wall high up instead of doing a real aerial") needs a wall to bounce off,
+# so simply not generating ball states anywhere near one removes the exploit
+# from the training distribution entirely, rather than trying to price it
+# out via reward shaping.
+const AIR_DRILL_BALL_WALL_CLEARANCE := 5.0
+
+# Air drill state (see air_drill_chance): ball spawned high, both ships
+# spawned low and lateral, so the state is unsolvable without climbing.
+func _place_air_drill() -> void:
+	var ball_half_x := ArenaBoundary.INNER_HALF_X - AIR_DRILL_BALL_WALL_CLEARANCE
+	var ball_half_z := ArenaBoundary.GOAL_LINE_Z - AIR_DRILL_BALL_WALL_CLEARANCE
+	var ball_position := Vector3(
+		randf_range(-ball_half_x, ball_half_x),
+		randf_range(ArenaBoundary.INNER_HEIGHT * 0.45, FIELD_MAX_Y),
+		randf_range(-ball_half_z, ball_half_z)
+	)
+	var ball_velocity := _random_direction() * randf_range(0.0, MAX_RANDOM_BALL_SPEED * 0.5)
+	_place_body(ball, Transform3D(Basis.IDENTITY, ball_position), ball_velocity, Vector3.ZERO)
+
+	for ship in ships:
+		if ship in _inert_ships:
+			continue
+		var lateral_offset := Vector3(randf_range(-1, 1), 0.0, randf_range(-1, 1))
+		lateral_offset = lateral_offset.normalized() if lateral_offset.length_squared() > 0.001 else Vector3.FORWARD
+		lateral_offset *= randf_range(6.0, 14.0)
+		var ship_position := Vector3(
+			clampf(ball_position.x + lateral_offset.x, -FIELD_HALF_X, FIELD_HALF_X),
+			randf_range(FIELD_MIN_Y, 4.0),
+			clampf(ball_position.z + lateral_offset.z, -FIELD_HALF_Z, FIELD_HALF_Z)
+		)
+		var orientation := Basis.from_euler(Vector3(
+			randf_range(-0.4, 0.4), randf_range(-PI, PI), randf_range(-0.4, 0.4)
+		))
+		_place_body(ship, Transform3D(orientation, ship_position), Vector3.ZERO, Vector3.ZERO)
+
+
 # Attacking/defending drill states: ball close to a goal, moving toward it.
 # Which goal is picked is biased by attack_goal_bias (0.5 = uniform between
 # both, matching historical behaviour; 1.0 = always the goal team 0 attacks).