feat(training): curriculum generation 4 — MultiDiscrete action space redesign

Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.

Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.

- Game/scripts/ship_action_codec.gd (new): single source of truth for a
  per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
  by training and in-game inference, replacing the continuous Gaussian.
  thrust_y's bins are deliberately asymmetric so a random policy drifts
  through the volume instead of floor-pinning. Legacy continuous decode
  (ai_ship_controller.gd's old logic) preserved verbatim so every
  pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
  unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
  so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
  vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
  lowered 4x (aerial approaches require pitching); flight telemetry
  (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
  and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
  high, ships low, kept clear of walls) so aerial practice is forced by the
  environment instead of relying on reward-driven exploration alone; snapshot
  terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
  SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
  now-meaningless --reset-std; new EntropyFloorCallback (a persistent
  per-rollout ent_coef controller replacing the one-shot std-reset shock)
  and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
  --tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
  index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
  grounded stage, full action space live from step 1; deletes generation
  1-3's checkpoint-lineage machinery (nothing to resume from); final report
  evaluates against both promoted/easy.json and the new
  promoted/reference-grounded.json (a copy of curric-s5-aggression, the
  strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
  intermediate checkpoint files a single stage was writing (~500MB ->
  ~0.2MB per run); requirements.txt pinned (behaviour here now depends on
  specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
  mismatch before it silently corrupts 24h of training.

Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.

See TRAINING.md's "Generation 4" section for the full design writeup.

.gitignore

@@ -6,6 +6,13 @@ training/.venv/
 training/smoke_run.log
 training/__pycache__/
 
+# Intermediate PPO checkpoints: only final.zip is ever committed (see
+# run_training.sh) — --resume only ever points at final.zip, and a single
+# experiment's intermediate checkpoints were 2401 files / ~500MB, of which
+# final.zip was ~0.2MB. This is the training-results-survive-any-machine
+# property from ~2500x less data, not a relaxation of it.
+training/checkpoints/*/ppo_*_steps.zip
+
 # Exported training binary: a regenerable build artifact (rebuilt by
 # export_linux.sh / run_training.sh), not a training result.
 training/build/

Game/scripts/ai_ship_controller.gd

@@ -17,13 +17,16 @@ extends ShipController
 # Uniform noise magnitude added to each action axis (0 = play at full skill).
 @export_range(0.0, 1.0) var action_noise: float = 0.0
 
-# Must mirror whatever the model was actually trained with (see
+# Only meaningful for a "continuous"-action_space model (see
-# ShipAIController's identical exports on the training side, curriculum
+# ShipActionCodec) — i.e. one exported before curriculum generation 4, such
-# stages 1-2 in TRAINING.md). A model trained grounded (mask on) never got a
+# as Game/bots/promoted/reference-grounded.json. Must mirror whatever the
-# reward gradient on these axes, so its raw output there is untrained noise —
+# model was actually trained with: a model trained grounded (mask on) never
-# leaving this true for such a model doesn't make it fly well, it just lets
+# got a reward gradient on these axes, so its raw output there is untrained
-# that noise reach the ship instead of being discarded like it was in
+# noise — leaving this true for such a model doesn't make it fly well, it
-# training. Set false to match a grounded-trained model's actual behaviour.
+# just lets that noise reach the ship instead of being discarded like it was
+# in training. Set false to match a grounded-trained model's actual
+# behaviour. Generation-4-onward (multi_discrete) models train the full
+# action space from the start, so these flags are ignored for them.
 @export var allow_vertical := true
 @export var allow_pitch_roll := true
 
@@ -59,26 +62,18 @@ func get_action() -> ShipAction:
 func _decide() -> void:
 	var obs := ShipObservations.build(_ship, _opponent, _ball, _attack_goal_position)
 	var out := _policy.forward(obs)
-	# Output layout is the trainer's flattened action space (Box(7)), which
+	# See ShipActionCodec for the decode — the single source of truth shared
-	# gymnasium orders by SORTED key name — rotation xyz, thrust xyz, turbo
+	# with the training side, so this must never reimplement layout/ordering
-	# (> 0 means on) — NOT ShipAction's thrust-first declaration order.
+	# locally (see that file's header for why).
-	_action.rotation = Vector3(
+	if _policy.action_space.get("type", "continuous") == "continuous":
-		_axis(out[0]) if allow_pitch_roll else 0.0,
+		_action = ShipActionCodec.from_continuous(out, action_noise)
-		_axis(out[1]),
+		if not allow_pitch_roll:
-		_axis(out[2]) if allow_pitch_roll else 0.0
+			_action.rotation.x = 0.0
-	)
+			_action.rotation.z = 0.0
-	_action.thrust = Vector3(
+		if not allow_vertical:
-		_axis(out[3]),
+			_action.thrust.y = 0.0
-		_axis(out[4]) if allow_vertical else 0.0,
+	else:
-		_axis(out[5])
+		_action = ShipActionCodec.from_logits(out, action_noise)
-	)
-	_action.turbo = out[6] > 0.0
-
-
-func _axis(value: float) -> float:
-	if action_noise > 0.0:
-		value += randf_range(-action_noise, action_noise)
-	return clampf(value, -1.0, 1.0)
 
 
 # Find ship/ball/opponent/goal once everything is spawned. ShipAction axes

Game/scripts/policy_network.gd

@@ -14,9 +14,18 @@ extends RefCounted
 #     {"weights": [[out x in floats]], "biases": [out floats], "activation": "tanh" | "linear"},
 #     ...
 #   ]
+#   "action_space": {"type": "multi_discrete", "heads": [{"name","bins"}, ...]}  // optional
 # }
+#
+# "action_space" is absent from every model exported before curriculum
+# generation 4 (e.g. Game/bots/promoted/easy.json) — absence means
+# {"type": "continuous"}, decoded via ShipActionCodec.from_continuous, the
+# same flattened-Box(7)-mean-output path this class has always produced.
+# This class itself never changes behaviour based on it; only the caller
+# (AIShipController._decide) branches on action_space["type"].
 
 var input_size: int = 0
+var action_space: Dictionary = {"type": "continuous"}
 var _layers: Array = []
 
 
@@ -32,6 +41,7 @@ static func load_from_file(path: String) -> PolicyNetwork:
 
 	var net := PolicyNetwork.new()
 	net.input_size = int(data.get("input_size", 0))
+	net.action_space = data.get("action_space", {"type": "continuous"})
 	for layer in data["layers"]:
 		# Flatten each layer's weights into a PackedFloat64Array for speed
 		var out_size: int = layer["biases"].size()
@@ -55,7 +65,14 @@ static func load_from_file(path: String) -> PolicyNetwork:
 
 
 func forward(observation: Array) -> Array:
-	var x := PackedFloat64Array(observation)
+	if observation.size() < input_size:
+		push_error("PolicyNetwork: observation has %d values, model expects %d" % [observation.size(), input_size])
+	# Slice rather than trust the caller: ShipObservations.SIZE only ever
+	# grows (append-only), so an older/smaller model must still decode
+	# correctly against a newer, longer observation vector — the extra
+	# trailing values it never trained on are simply dropped here rather
+	# than corrupting the first layer's dot product by accident.
+	var x := PackedFloat64Array(observation.slice(0, input_size))
 	for layer in _layers:
 		var in_size: int = layer["in_size"]
 		var out_size: int = layer["out_size"]

Game/scripts/ship.gd

@@ -119,6 +119,16 @@ func _ready():
 			controller = child
 			break
 
+	# Always on (moved here from ShipAIController.setup, which only enabled
+	# it for training-side ships): ShipObservations now reads own-contact
+	# state (see its "contact" section) for every ship, training or shipped,
+	# so the RigidBody3D contact list must exist unconditionally rather than
+	# only for whichever ship happened to be a training agent. Cheap — a
+	# short per-tick contact list from the physics engine, not a rendering
+	# cost like the headless skips just below.
+	contact_monitor = true
+	max_contacts_reported = 8
+
 	_apply_team_color()
 
 	_boundary = get_tree().get_first_node_in_group("arena_boundary")

Game/scripts/ship_action_codec.gd

@@ -0,0 +1,128 @@
+class_name ShipActionCodec
+extends RefCounted
+
+# Single source of truth for the RL action layout — shared by training
+# (ShipAIController.get_action_space/set_action) and in-game inference
+# (AIShipController._decide via PolicyNetwork) so a trained policy's action
+# output is decoded identically in both contexts. Mirrors ShipObservations'
+# "do not fork this logic" role for observations; the train/inference seam
+# broke once before over exactly this kind of divergence (commit 8c15c46).
+#
+# Curriculum generation 4 replaces the old continuous Gaussian action space
+# (Box(7), see the "continuous" path below) with a per-axis MultiDiscrete
+# space: PPO's Gaussian std reliably collapsed to ~0.13-0.15 within the first
+# ~10% of every training run across 3 generations and never recovered, which
+# made a *sustained* set-point (e.g. hovering, thrust.y ~= 0.408 given this
+# ship's mass/thrust — see TRAINING.md) essentially unreachable: the
+# collapsed distribution can brush the hover value but never hold it long
+# enough to accumulate the reward signal that would move the mean. A
+# discrete bin is a single, atomic, repeatable choice with non-zero
+# probability under any softmax, which does not have that failure mode.
+#
+# HEADS order is deliberately gymnasium's *sorted* key order (verified:
+# "rot_x" < "rot_y" < "rot_z" < "thrust_x" < "thrust_y" < "thrust_z" <
+# "turbo") — godot_rl's ActionSpaceProcessor builds the Tuple action space
+# from a gymnasium Dict, which sorts keys regardless of insertion order, so
+# this order is what SB3/PPO actually samples/trains against and what
+# set_action() receives keyed by. Do not reorder without re-verifying that
+# sort order.
+const HEADS := [
+	{"name": "rot_x", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+	{"name": "rot_y", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+	{"name": "rot_z", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+	{"name": "thrust_x", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+	# Deliberately asymmetric: hovering this ship (mass 5.0, vertical_thrust
+	# 120, default gravity 9.8 m/s^2 — see ship.gd/ship.tscn) requires a
+	# sustained thrust.y ~= 0.408. Uniform-random selection over these 5 bins
+	# averages 0.34 — just below neutral buoyancy, so a fresh policy drifts
+	# gently through the volume instead of pinning to the floor (symmetric
+	# bins) or sticking to the ceiling (ceiling_pull_strength 11.5 > gravity
+	# 9.8, so the ceiling is easy to over-shoot into). This is the direct
+	# analogue of the RLGym/RLBot community fix for the same failure mode
+	# ("add more jump actions to the discrete action parser").
+	{"name": "thrust_y", "bins": [-0.5, 0.0, 0.45, 0.75, 1.0]},
+	{"name": "thrust_z", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+	{"name": "turbo", "bins": [0.0, 1.0]},
+]
+
+
+static func action_space_dict() -> Dictionary:
+	var space := {}
+	for head in HEADS:
+		space[head["name"]] = {"size": head["bins"].size(), "action_type": "discrete"}
+	return space
+
+
+# Training side: `action` is the Dictionary godot_rl's Sync node hands
+# set_action() — one entry per HEADS key, each an int (or int-valued float)
+# bin index in [0, bins.size()).
+static func from_indices(action: Dictionary) -> ShipAction:
+	var result := ShipAction.new()
+	var values := {}
+	for head in HEADS:
+		var index: int = clampi(int(round(float(action[head["name"]]))), 0, head["bins"].size() - 1)
+		values[head["name"]] = head["bins"][index]
+	result.rotation = Vector3(values["rot_x"], values["rot_y"], values["rot_z"])
+	result.thrust = Vector3(values["thrust_x"], values["thrust_y"], values["thrust_z"])
+	result.turbo = values["turbo"] > 0.0
+	return result
+
+
+# In-game inference for a MultiDiscrete-trained export: `logits` is the raw
+# policy_network.gd output — 32 floats (5+5+5+5+5+5+2), one contiguous slice
+# per head in HEADS order (matches export_policy.py's action_net layer,
+# which concatenates SB3's per-head categorical logits in that same order).
+# argmax within each slice picks that head's bin, same as SB3's
+# MultiCategoricalDistribution.mode() under deterministic inference.
+static func from_logits(logits: Array, noise: float) -> ShipAction:
+	var result := ShipAction.new()
+	var values := {}
+	var offset := 0
+	for head in HEADS:
+		var bins: Array = head["bins"]
+		var index := 0
+		if noise > 0.0 and randf() < noise:
+			# eps-random-bin: the discrete analogue of continuous action_noise
+			# (see ai_ship_controller.gd) — degrades gracefully and keeps the
+			# same 0..1 monotonic difficulty semantics as the continuous path.
+			index = randi() % bins.size()
+		else:
+			var best_value: float = logits[offset]
+			for i in range(1, bins.size()):
+				if logits[offset + i] > best_value:
+					best_value = logits[offset + i]
+					index = i
+		values[head["name"]] = bins[index]
+		offset += bins.size()
+	result.rotation = Vector3(values["rot_x"], values["rot_y"], values["rot_z"])
+	result.thrust = Vector3(values["thrust_x"], values["thrust_y"], values["thrust_z"])
+	result.turbo = values["turbo"] > 0.0
+	return result
+
+
+# Legacy continuous decode — moved verbatim from ai_ship_controller.gd so
+# every model exported before generation 4 (no "action_space" block in its
+# JSON, e.g. Game/bots/promoted/easy.json) keeps behaving byte-identically.
+# `out` is the trainer's flattened Box(7) output, gymnasium-sorted: rotation
+# xyz, thrust xyz, turbo (> 0 means on) — NOT ShipAction's thrust-first
+# declaration order.
+static func from_continuous(out: Array, noise: float) -> ShipAction:
+	var result := ShipAction.new()
+	result.rotation = Vector3(
+		_continuous_axis(out[0], noise),
+		_continuous_axis(out[1], noise),
+		_continuous_axis(out[2], noise)
+	)
+	result.thrust = Vector3(
+		_continuous_axis(out[3], noise),
+		_continuous_axis(out[4], noise),
+		_continuous_axis(out[5], noise)
+	)
+	result.turbo = out[6] > 0.0
+	return result
+
+
+static func _continuous_axis(value: float, noise: float) -> float:
+	if noise > 0.0:
+		value += randf_range(-noise, noise)
+	return clampf(value, -1.0, 1.0)

Game/scripts/ship_action_codec.gd.uid

@@ -0,0 +1 @@
+uid://cvpbp3mj58ejd

Game/scripts/ship_ai_controller.gd

@@ -7,10 +7,11 @@ extends AIController3D
 # the trainer are written into an RLShipController, which the ship pulls like
 # any other controller.
 #
-# Action space is ShipAction verbatim: 6 continuous axes (thrust xyz,
+# Action space/layout is owned by ShipActionCodec (get_action_space/
-# rotation xyz, each -1..1) + binary turbo. ShipAction axes are ship-local
+# set_action just delegate to it) — see that file for the per-axis
-# (body frame), so they need no team mirroring — only observations do
+# MultiDiscrete design and why. ShipAction axes are ship-local (body frame),
-# (see ShipObservations.canon).
+# so they need no team mirroring — only observations do (see
+# ShipObservations.canon).
 
 # Reward shaping weights. Dense terms accrue per physics tick (60 sim-ticks
 # per sim-second); event terms fire once. Exported so tuning needs no code
@@ -54,9 +55,13 @@ extends AIController3D
 # relative to the (then far weaker) ball-seeking shaping.
 @export var wall_contact_penalty := 0.0025
 # Per-tick penalty for not being upright, scaled by tilt: 0 when flat, full
-# value (-0.12/s) when inverted. A penalty rather than an upright bonus so a
+# value when inverted. A penalty rather than an upright bonus so a flat, idle
-# flat, idle ship farms nothing.
+# ship farms nothing. Lowered 4x for curriculum generation 4 (was 0.002,
-@export var tilt_penalty := 0.002
+# -0.12/s): a genuine aerial approach to a high ball requires pitching, and
+# the old value quietly opposed the exact behaviour generation 4 is trying
+# to teach. Not removed outright — an always-inverted bot still looks bad in
+# a shipped game.
+@export var tilt_penalty := 0.0005
 # Per-tick bonus for own speed: 0 stationary, full value (+0.24/s) at
 # max_speed. Run07 lesson: after the kickoff flurry both ships parked next to
 # a cornered ball — with every other dense term near zero there, standing
@@ -79,27 +84,15 @@ extends AIController3D
 # unaffected; the floor-lock curriculum stage turns it on.
 @export var airborne_penalty := 0.0
 
-# Locomotion curriculum: scales how much of the corresponding action axes
+# Height above which a touch counts toward air_touch_fraction telemetry
-# actually reaches the ship, from 0.0 (fully discarded, grounded-only) to
+# (see get_info) — not a reward term itself, see set_action/get_info's
-# 1.0 (full effect) — this scales the *effect* of thrust.y/rotation.x/
+# comments on why generation 4 deliberately does not add a standalone
-# rotation.z in set_action, not the action space's shape: the policy always
+# air-touch reward.
-# outputs values for these axes (always contributing to PPO's entropy/log-
+const AIR_TOUCH_HEIGHT := 5.0
-# prob), they're just attenuated here, so checkpoints stay resumable across
-# ramp values.
-#
-# A hard 0/1 flip (the original bool mask) let PPO's action-distribution
-# std collapse to ~0.13-0.15 within the first ~10% of steps, before the
-# policy ever meaningfully explored the newly-unmasked axes — 3 independent
-# 240M-step attempts at the all-or-nothing flip all landed at a stable
-# ~28-32% win rate vs curric-s5-aggression (see TRAINING.md's generation 3
-# section). A gradual ramp across several short curriculum stages, each
-# resuming from the previous ramp value's checkpoint, lets the policy adopt
-# each axis incrementally instead of all at once.
-@export_range(0.0, 1.0) var vertical_ramp := 1.0
-@export_range(0.0, 1.0) var pitch_roll_ramp := 1.0
 
 # Contact normals with y above this are floor contact (exempt from the wall
 # penalty); below it they read as wall (sideways) or ceiling (downward).
+# Mirrors ShipObservations.FLOOR_NORMAL_MIN_Y (see that file's comment).
 const FLOOR_NORMAL_MIN_Y := 0.7
 
 # Longest possible ship-to-ball separation: the enclosure's interior diagonal.
@@ -125,8 +118,32 @@ var attack_goal_position: Vector3
 # (true on goal, false on timeout) rather than toggle, so no reset is needed.
 var goal_scored_this_episode := false
 
+# Set only in the timeout branch (TrainingMode._physics_process), never on a
+# goal — a goal is a genuine terminal (V(s)=0 is correct there); a timeout
+# is an artificial episode boundary the value function should be bootstrapped
+# through instead (see get_info). Same "always overwritten by both call
+# sites, never cleared in reset()" pattern as goal_scored_this_episode above,
+# for the same reason.
+var truncated_this_episode := false
+var terminal_obs: Array = []
+
 var _ticks_since_ball_touch := 1 << 30  # large so the first touch always pays
 
+# Flight telemetry (see get_info) — leading indicators for whether the
+# policy is actually using its vertical/pitch-roll authority, visible from
+# the first rollout instead of only in a win-rate number measured a full
+# training run later. Accumulated per-tick, reset() zeroes them each episode;
+# get_info() reports the running fraction/mean so the *final* tick of an
+# episode (the one VecMonitor's info_keywords captures) holds the whole
+# episode's aggregate.
+const AIRBORNE_ALTITUDE_THRESHOLD := 3.0
+var _telemetry_ticks := 0
+var _airborne_ticks := 0
+var _altitude_sum := 0.0
+var _thrust_y_sum := 0.0
+var _touches := 0
+var _air_touches := 0
+
 
 # Wire up references after the ship is spawned. `attack_goal` is the goal
 # this ship scores into (goal.team == opponent's team).
@@ -138,10 +155,8 @@ func setup(p_ship: Ship, p_rl_controller: RLShipController, p_ball: RigidBody3D,
 	attack_goal_position = p_attack_goal_position
 	init(ship)
 
-	# Contact monitoring for the ball-touch reward (training-only cost;
+	# ship.contact_monitor is on unconditionally (see ship.gd) since
-	# the shipped game leaves contact_monitor off).
+	# ShipObservations now reads it for every ship, not just training agents.
-	ship.contact_monitor = true
-	ship.max_contacts_reported = 8
 	ship.body_entered.connect(_on_ship_body_entered)
 
 
@@ -153,36 +168,48 @@ func get_reward() -> float:
 	return reward
 
 
-# Symmetric across both self-play agents: reports whether this episode ended
+# Symmetric across both self-play agents. "goal_scored": whether this
-# in a goal at all, not which team scored — a clean "goal rate" signal
+# episode ended in a goal at all (not which team) — a clean "goal rate"
-# distinct from rollout/ep_rew_mean, which mixes this with dense shaping
+# signal distinct from rollout/ep_rew_mean, which mixes this with dense
-# (ball chasing/touching). See train.py's GoalRateCallback.
+# shaping (ball chasing/touching); see train.py's GoalRateCallback.
+# "truncated"/"terminal_obs": only present on a timeout tick — remapped by
+# cosmic_env.py into SB3's expected "TimeLimit.truncated"/
+# "terminal_observation" keys so PPO bootstraps V(s) through episode
+# timeouts instead of treating every 30s draw as a true terminal state (a
+# real, previously-unnoticed bug independent of the action-space work — see
+# TRAINING.md). Flight telemetry fields are leading indicators for
+# generation 4's core hypothesis (see train.py's FlightTelemetryCallback).
 func get_info() -> Dictionary:
-	return {"goal_scored": goal_scored_this_episode}
+	var info := {"goal_scored": goal_scored_this_episode}
+	if truncated_this_episode:
+		info["truncated"] = true
+		info["terminal_obs"] = terminal_obs
+	if _telemetry_ticks > 0:
+		info["airborne_fraction"] = float(_airborne_ticks) / _telemetry_ticks
+		info["mean_altitude"] = _altitude_sum / _telemetry_ticks
+		info["vertical_thrust_mean"] = _thrust_y_sum / _telemetry_ticks
+	if _touches > 0:
+		info["air_touch_fraction"] = float(_air_touches) / _touches
+	return info
 
 
 func get_action_space() -> Dictionary:
-	return {
+	return ShipActionCodec.action_space_dict()
-		"thrust": {"size": 3, "action_type": "continuous"},
-		"rotation": {"size": 3, "action_type": "continuous"},
-		"turbo": {"size": 2, "action_type": "discrete"},
-	}
 
 
 func set_action(action) -> void:
-	var thrust: Array = action["thrust"]
+	rl_controller.action = ShipActionCodec.from_indices(action)
-	var rot: Array = action["rotation"]
-	var thrust_y: float = thrust[1] * vertical_ramp
-	var pitch: float = rot[0] * pitch_roll_ramp
-	var roll: float = rot[2] * pitch_roll_ramp
-	rl_controller.action.thrust = Vector3(thrust[0], thrust_y, thrust[2])
-	rl_controller.action.rotation = Vector3(pitch, rot[1], roll)
-	rl_controller.action.turbo = int(action["turbo"]) == 1
 
 
 func reset():
 	super()
 	_ticks_since_ball_touch = 1 << 30
+	_telemetry_ticks = 0
+	_airborne_ticks = 0
+	_altitude_sum = 0.0
+	_thrust_y_sum = 0.0
+	_touches = 0
+	_air_touches = 0
 
 
 func _physics_process(delta):
@@ -238,19 +265,17 @@ func _physics_process(delta):
 		var height := maxf(ship.global_position.y, 0.0)
 		reward -= airborne_penalty * height / ArenaBoundary.INNER_HEIGHT
 
+	# Flight telemetry accumulation (see get_info) — not reward, just
+	# observation of what the policy is actually doing this tick.
+	_telemetry_ticks += 1
+	_altitude_sum += ship.global_position.y
+	if ship.global_position.y > AIRBORNE_ALTITUDE_THRESHOLD:
+		_airborne_ticks += 1
+	_thrust_y_sum += rl_controller.action.thrust.y
+
 
 func _wall_or_ceiling_contact() -> bool:
-	var state := PhysicsServer3D.body_get_direct_state(ship.get_rid())
+	return ShipObservations.contact_normal(ship) != Vector3.ZERO
-	if state == null:
-		return false
-	for i in state.get_contact_count():
-		if not state.get_contact_collider_object(i) is ArenaBoundary:
-			continue
-		# Normal points from the surface into the ship: floor ≈ +Y (exempt),
-		# anything flatter or downward is a wall or the ceiling.
-		if state.get_contact_local_normal(i).y < FLOOR_NORMAL_MIN_Y:
-			return true
-	return false
 
 
 func _on_ship_body_entered(body: Node) -> void:
@@ -264,3 +289,12 @@ func _on_ship_body_entered(body: Node) -> void:
 		alignment = clampf(ball.linear_velocity.normalized().dot(to_goal.normalized()), 0.0, 1.0)
 	reward += ball_touch_reward * lerpf(ball_touch_direction_floor, 1.0, alignment)
 	_ticks_since_ball_touch = 0
+
+	# Telemetry only (see get_info's air_touch_fraction) — not a reward term.
+	# Generation 4 deliberately doesn't reward high touches directly (see
+	# TRAINING.md's "why no air-touch reward" note); this just measures
+	# whether the air-drill state setter is producing genuine aerial
+	# contests, so a future decision to add one is data-driven.
+	_touches += 1
+	if ball.global_position.y > AIR_TOUCH_HEIGHT:
+		_air_touches += 1

Game/scripts/ship_observations.gd

@@ -19,8 +19,20 @@ const POSITION_SCALE := Vector3(20.0, 10.0, 20.0)
 const BALL_SPEED_SCALE := 30.0
 const GOAL_DISTANCE_SCALE := 40.0
 
-# Number of floats build() returns; the policy input size.
+# Contact normals with y above this are floor contact; below it they read as
-const SIZE := 31
+# wall (sideways) or ceiling (downward) — mirrors
+# ShipAIController.FLOOR_NORMAL_MIN_Y (kept here too since ShipAIController's
+# wall_contact_penalty and this observation feature must agree on what
+# counts as "in contact" for the reward/observation to stay consistent).
+const FLOOR_NORMAL_MIN_Y := 0.7
+
+# Number of floats build() returns; the policy input size. APPEND-ONLY: new
+# features go on the end and existing indices never move, so an old exported
+# model (whose network was trained against a shorter SIZE) still decodes its
+# first N inputs identically when SIZE grows — see PolicyNetwork.forward's
+# input_size slice/guard. Do not retune an *existing* index without
+# retraining every model in Game/bots/.
+const SIZE := 35
 
 
 # 180° rotation about Y for team 1; identity for team 0. A proper rotation
@@ -62,6 +74,17 @@ static func build(ship: Ship, opponent: Ship, ball: RigidBody3D, attack_goal_pos
 	_append(obs, canon(goal_rel, team) / POSITION_SCALE)
 	obs.append(goal_rel.length() / GOAL_DISTANCE_SCALE)
 
+	# Own contact state (appended — see SIZE's append-only invariant).
+	# Added for generation 4: ShipAIController's wall_contact_penalty used to
+	# fire on a condition the observation vector couldn't see coming,
+	# leaving the value function to predict a reward with no supporting
+	# signal. Also gives the policy a direct "am I resting on a surface"
+	# signal it can use to push off (a real aerial mechanic), distinct from
+	# inferring it indirectly from position/up-vector.
+	var normal := contact_normal(ship)
+	_append(obs, canon(normal, team))
+	obs.append(1.0 if normal != Vector3.ZERO else 0.0)
+
 	return obs
 
 
@@ -69,3 +92,22 @@ static func _append(obs: Array, v: Vector3) -> void:
 	obs.append(v.x)
 	obs.append(v.y)
 	obs.append(v.z)
+
+
+# Aggregate wall/ceiling contact normal (zero if none, or if only floor
+# contact — floor contact is excluded so "in_contact" means "touching
+# something other than the ground it's expected to rest on", matching
+# ShipAIController.wall_contact_penalty's own floor exemption). Requires
+# ship.contact_monitor (see ship.gd's _ready — on unconditionally for every
+# ship so training and in-game inference see identical observations).
+static func contact_normal(ship: Ship) -> Vector3:
+	var state := PhysicsServer3D.body_get_direct_state(ship.get_rid())
+	if state == null:
+		return Vector3.ZERO
+	for i in state.get_contact_count():
+		if not state.get_contact_collider_object(i) is ArenaBoundary:
+			continue
+		var normal := state.get_contact_local_normal(i)
+		if normal.y < FLOOR_NORMAL_MIN_Y:
+			return normal
+	return Vector3.ZERO

Game/scripts/training_mode.gd

@@ -54,6 +54,16 @@ extends GameMode
 # trainee's near-goal resets are always finishing chances, not a coin flip
 # between attacking and defending an empty net.
 @export_range(0.0, 1.0) var attack_goal_bias := 0.5
+# Fourth episode-start branch (after kickoff/near-goal, before the fully-
+# random fallback): ball spawned high, both ships spawned low and lateral —
+# unsolvable without climbing. Default 0 (off) so ordinary runs are
+# unaffected. Added for curriculum generation 4: the existing random branch
+# already samples ship/ball Y across the full arena height, but that only
+# randomizes the *initial* state — under gravity+drag a floor-pinned policy
+# sinks back to the floor in ~1.5s, so the *stationary* state distribution
+# stayed floor-pinned even though the initial one wasn't. See
+# _place_air_drill.
+@export_range(0.0, 1.0) var air_drill_chance := 0.0
 
 # Placement bounds for randomized episode starts, derived from the standard
 # enclosure (ArenaBoundary). The inset keeps a randomly oriented ship (1x1x4
@@ -188,7 +198,7 @@ func _parse_eval_args() -> void:
 # of silently matching an unrelated inherited export.
 const TRAINING_MODE_OVERRIDES := [
 	"goal_reward", "draw_penalty", "kickoff_state_chance",
-	"ball_near_goal_chance", "attack_goal_bias",
+	"ball_near_goal_chance", "attack_goal_bias", "air_drill_chance",
 ]
 # ShipAIController @export names a curriculum run may override, read as
 # --ai_<name>=<value> to avoid colliding with the names above.
@@ -196,7 +206,7 @@ const SHIP_AI_OVERRIDES := [
 	"ball_touch_reward", "ball_touch_cooldown_ticks", "ball_touch_direction_floor",
 	"velocity_to_ball_weight", "ball_velocity_to_goal_weight", "ball_distance_penalty",
 	"wall_contact_penalty", "tilt_penalty", "speed_reward_weight", "time_penalty",
-	"airborne_penalty", "vertical_ramp", "pitch_roll_ramp",
+	"airborne_penalty",
 ]
 
 
@@ -243,11 +253,10 @@ func _ai_default(name: String) -> Variant:
 		"ball_velocity_to_goal_weight": return 0.004
 		"ball_distance_penalty": return 0.002
 		"wall_contact_penalty": return 0.0025
-		"tilt_penalty": return 0.002
+		"tilt_penalty": return 0.0005
 		"speed_reward_weight": return 0.004
 		"time_penalty": return 0.001
 		"airborne_penalty": return 0.0
-		"vertical_ramp", "pitch_roll_ramp": return 1.0
 		_: return null
 
 
@@ -299,6 +308,14 @@ func _physics_process(_delta):
 				agent.reward -= draw_penalty
 				agent.done = true
 				agent.goal_scored_this_episode = false
+				# Snapshot BEFORE _reset_episode() below, which moves the ship/
+				# ball and would otherwise make this the post-reset state, not
+				# the terminal one PPO needs to bootstrap V(s) from (see
+				# ShipAIController.get_info / cosmic_env.py's truncation remap).
+				# A goal (_on_goal_scored) does NOT do this — a goal is a
+				# genuine terminal, V(s)=0 is correct there.
+				agent.truncated_this_episode = true
+				agent.terminal_obs = ShipObservations.build(agent.ship, agent.opponent, agent.ball, agent.attack_goal_position)
 		_reset_episode()
 		return
 
@@ -330,6 +347,7 @@ func _on_goal_scored(conceding_team: int) -> void:
 		agent.reward += goal_reward if agent.ship.team != conceding_team else -goal_reward
 		agent.done = true
 		agent.goal_scored_this_episode = true
+		agent.truncated_this_episode = false  # genuine terminal, not a timeout
 	_reset_episode()
 
 
@@ -363,6 +381,8 @@ func _reset_episode() -> void:
 	elif roll < kickoff_state_chance + ball_near_goal_chance:
 		_place_ships_random()
 		_place_ball_near_goal()
+	elif roll < kickoff_state_chance + ball_near_goal_chance + air_drill_chance:
+		_place_air_drill()
 	else:
 		_place_ships_random()
 		_place_ball_random()
@@ -373,6 +393,46 @@ func _place_ball_random() -> void:
 	_place_body(ball, Transform3D(Basis.IDENTITY, _random_position()), velocity, Vector3.ZERO)
 
 
+# Extra clearance for the air drill's ball placement specifically — well
+# beyond SPAWN_INSET, and well beyond the ball's own radius. The ball (unlike
+# _random_position) has no collision-avoidance resample, so this is the
+# anti-exploit measure: the RLGym wall-bounce exploit ("hits the ball off a
+# wall high up instead of doing a real aerial") needs a wall to bounce off,
+# so simply not generating ball states anywhere near one removes the exploit
+# from the training distribution entirely, rather than trying to price it
+# out via reward shaping.
+const AIR_DRILL_BALL_WALL_CLEARANCE := 5.0
+
+# Air drill state (see air_drill_chance): ball spawned high, both ships
+# spawned low and lateral, so the state is unsolvable without climbing.
+func _place_air_drill() -> void:
+	var ball_half_x := ArenaBoundary.INNER_HALF_X - AIR_DRILL_BALL_WALL_CLEARANCE
+	var ball_half_z := ArenaBoundary.GOAL_LINE_Z - AIR_DRILL_BALL_WALL_CLEARANCE
+	var ball_position := Vector3(
+		randf_range(-ball_half_x, ball_half_x),
+		randf_range(ArenaBoundary.INNER_HEIGHT * 0.45, FIELD_MAX_Y),
+		randf_range(-ball_half_z, ball_half_z)
+	)
+	var ball_velocity := _random_direction() * randf_range(0.0, MAX_RANDOM_BALL_SPEED * 0.5)
+	_place_body(ball, Transform3D(Basis.IDENTITY, ball_position), ball_velocity, Vector3.ZERO)
+
+	for ship in ships:
+		if ship in _inert_ships:
+			continue
+		var lateral_offset := Vector3(randf_range(-1, 1), 0.0, randf_range(-1, 1))
+		lateral_offset = lateral_offset.normalized() if lateral_offset.length_squared() > 0.001 else Vector3.FORWARD
+		lateral_offset *= randf_range(6.0, 14.0)
+		var ship_position := Vector3(
+			clampf(ball_position.x + lateral_offset.x, -FIELD_HALF_X, FIELD_HALF_X),
+			randf_range(FIELD_MIN_Y, 4.0),
+			clampf(ball_position.z + lateral_offset.z, -FIELD_HALF_Z, FIELD_HALF_Z)
+		)
+		var orientation := Basis.from_euler(Vector3(
+			randf_range(-0.4, 0.4), randf_range(-PI, PI), randf_range(-0.4, 0.4)
+		))
+		_place_body(ship, Transform3D(orientation, ship_position), Vector3.ZERO, Vector3.ZERO)
+
+
 # Attacking/defending drill states: ball close to a goal, moving toward it.
 # Which goal is picked is biased by attack_goal_bias (0.5 = uniform between
 # both, matching historical behaviour; 1.0 = always the goal team 0 attacks).

TRAINING.md

@@ -161,12 +161,20 @@ the automated curriculum pipeline (`run_training.sh`) only ever writes new
 flat files there, never touching subdirectories.
 
 `Game/bots/promoted/<tier>.json` is the small, curated, hand-maintained set
-actually referenced by the shipped game — currently just `easy.json`
+actually referenced by the shipped game — currently `easy.json` (promoted
-(promoted 2026-07-24 from `curric-s6-unmask`, the strongest checkpoint at the
+2026-07-24 from `curric-s6-unmask`, the strongest checkpoint at the
-time). `match.tscn`/`spectate.tscn` point their `bot_model_path` exports here
+time — note `curric-s6-unmask` was itself generation 1's *failed* unmask
-directly, so a promoted file is never touched by training scripts, never
+stage, so `easy.json` is weaker than `reference-grounded.json` below; a
-overwritten by a same-named future export, and never disturbed by pruning old
+strong generation 4 result should promote a real replacement, plus
-experiment files from the flat dump.
+`medium.json`/`hard.json`) and `reference-grounded.json` (added for
+generation 4 — a copy of generation 3's `curric-s5-aggression`, made before
+the flat `Game/bots/` dump was scrapped for the redesign, kept as the
+strongest grounded-era artifact and the fixed yardstick generations 1-3 were
+all measured against; see "Generation 4"'s final report). `match.tscn`/
+`spectate.tscn` point their `bot_model_path` exports here directly, so a
+promoted file is never touched by training scripts, never overwritten by a
+same-named future export, and never disturbed by pruning old experiment
+files from the flat dump.
 
 To promote a new bot into a tier: copy the chosen `Game/bots/<experiment>.json`
 to `Game/bots/promoted/<tier>.json` (overwriting the old one), and note the
@@ -185,13 +193,16 @@ movement. Each stage is a normal chained run — a new `--experiment` resumed
 via `--resume checkpoints/<previous>/final.zip`, same as any other run —
 just with different curriculum flags.
 
-`curriculum.py` has run through three generations so far. Generation 1
+`curriculum.py` has run through four generations so far. Generation 1
 (below) ran stages 1-6 to completion/block and is archived; generation 2
 started a fresh stage 1 seeded from generation 1's last clean pass instead
 of continuing to retry a stage that kept getting worse, but also failed 3
-attempts; generation 3 (the one `curriculum.py` actually runs today)
+attempts; generation 3 replaced generation 2's single all-or-nothing unmask
-replaces generation 2's single all-or-nothing unmask stage with a gradual
+stage with a gradual ramp, and also failed (worse, on its final attempt,
-ramp — see "Generation 3" below.
+than either prior generation); generation 4 (the one `curriculum.py` actually
+runs today) is a full redesign, not a further patch — see "Generation 4"
+below, and "Generation 3" for why a fourth attempt at gating *when* the
+policy could use full 3D controls was abandoned rather than retried again.
 
 ### Generation 1 (archived — see `curriculum_state_gen1.json`)
 
@@ -319,28 +330,174 @@ stages, so the ramp is the sole studied variable.
 `curriculum_state_gen2.json`) rather than continuing to log against a stage
 list whose stage 0 no longer means what it used to.
 
-**Open question, not yet resolved by data:** the ramp scales the action's
+**Open question, resolved 2026-08-04.** The gated `unmask` stage failed all
-effect in Godot, which runs *after* PPO samples the action — PPO's own
+3 attempts: 29% → 30% → **24%** win rate vs `curric-s5-aggression` (the
-std-collapse dynamics don't directly see the ramp, only the reward it
+third, worst by then) — landing in the same ~28-32% band the section above
-produces. It's possible this doesn't prevent the collapse, or even makes it
+flagged as "evidence the plateau isn't an exploration/collapse problem at
-happen faster at low ramp values (weaker reward signal on those axes gives
+all." `train/std` collapsed from ~0.30 to ~0.13-0.15 within the first ~10%
-less incentive to keep exploring them). Watch `train/std` per stage in
+of steps in every attempt of every generation regardless of hard-mask vs.
-TensorBoard rather than assuming the ramp is working. If the final gated
+gradual-ramp mechanism, so gating *when* the axes were allowed to act never
-stage still lands ~28-32%, that's evidence the plateau isn't an
+addressed the actual cause. See "Generation 4" below for the redesign and
-exploration/collapse problem at all — worth revisiting reward shaping, or
+root-cause diagnosis this prompted, and `curriculum_state_gen3.json` for the
-trying `--opponent-mode frozen --opponent-model <path>` during the warmup
+archived full log.
-stages (implemented, never yet exercised in this project) to remove
+
-self-play's moving-target instability while the policy first learns to use
+### Generation 4 (current) — action space redesign, not a further ramp patch
-the new axes.
+
+Three generations spent ~2 weeks trying different ways to gate *when* the
+policy could use vertical thrust/pitch/roll on top of a continuous Gaussian
+action space, and all three converged on the same failure: PPO's action
+std collapsing within the first ~10% of steps and never recovering,
+regardless of mechanism. Research into how self-play PPO bots that have
+actually solved this class of problem (RLGym/RLBot's Necto/Nexto) approach
+it turned up a structural difference — they don't gate control authority at
+all; they train the full action space from step 1 using discrete/bucketed
+actions, not a continuous Gaussian, plus reward/state-setter curriculum
+instead of action masking.
+
+**Root cause, verified against this project's own physics** (not assumed):
+flight in this game is a *sustained set-point*, not an impulse. Ship mass
+5.0, `vertical_thrust` 120 (`ship.gd`/`ship.tscn`), default gravity 9.8 m/s²
+→ hovering requires *holding* `thrust.y ≈ 0.408` continuously. A Gaussian
+whose mean sits near 0 and whose σ has collapsed to ~0.13 samples
+`thrust.y ∈ [-0.4, 0.4]` — it can brush the hover value but can never *hold*
+it long enough to earn the reward gradient that would move the mean. That's
+a fixed point; no ramp on the axis's downstream *effect* (which is applied
+*after* PPO samples the action) moves it, exactly as the "open question"
+above speculated might be the case. Independently, this redesign also found
+and fixed a real, previously-unnoticed bug unrelated to the action space:
+`godot_rl`'s `godot_env.py` never marks an episode timeout as a truncation
+(it returns the same `done` array for both term and trunc — see its own
+`# TODO update API to term, trunc`), so PPO was bootstrapping `V(s_T)=0` on
+every 30s draw in every generation to date instead of correctly estimating
+the value of the state it timed out in.
+
+**Action space**: switched to per-axis `MultiDiscrete` (7 heads, `nvec =
+[5,5,5,5,5,5,2]`) instead of continuous `Box(7)` — see
+`Game/scripts/ship_action_codec.gd`, the single source of truth for the
+layout/decode shared by training and in-game inference. Not a single
+lookup table (RLGym's approach for Rocket League's *coupled* car controls):
+Cosmic Clash's 7 axes are near-independent thruster/torque channels, so a
+curated combination table would throw away that factorization for no
+benefit. `thrust_y`'s bins are deliberately asymmetric
+(`-0.5, 0, 0.45, 0.75, 1.0`, vs. the symmetric `-1, -0.5, 0, 0.5, 1` on
+every other axis) — a uniform-random policy over those 5 bins averages
+0.34, just below the 0.408 hover point, so a fresh policy drifts gently
+through the volume instead of pinning to the floor (symmetric bins) or
+sticking to the ceiling (`ceiling_pull_strength` 11.5 > gravity 9.8). This
+is the direct analogue of the RLGym/RLBot fix for the same failure mode
+("add more jump actions to the discrete action parser"). `godot_rl`'s
+`ActionSpaceProcessor` already emits `MultiDiscrete` with zero Python-side
+changes when every action entry is `Discrete` — the only reason this
+project's action space flattened to `Box(7)` before was that `turbo`
+(binary) was mixed with continuous entries.
+
+**Backward compatibility**: every export before generation 4 (e.g.
+`Game/bots/promoted/easy.json`) has no `"action_space"` field in its JSON;
+absence means `{"type": "continuous"}` and decodes through the exact same
+path as before (`ShipActionCodec.from_continuous`, moved verbatim out of
+`ai_ship_controller.gd`). `PolicyNetwork.gd`'s forward pass itself never
+changed — only the caller's decode branches on the model's declared type.
+`export_policy.py`'s parity check is now index-level for a `MultiDiscrete`
+model (argmax per head's logit slice, compared against SB3's own
+`deterministic=True` chosen index) rather than comparing clipped floats,
+since a head-order mistake would otherwise train and export cleanly and
+only surface as silently wrong in-game behaviour.
+
+**No grounded stage.** Full action space live from step 1 — no successful
+self-play RL bot in this problem class gates control authority, it's failed
+9/9 attempts (3 generations × 3 attempts) here, and every prior generation's
+checkpoints are a different, incompatible action/observation shape anyway
+(nothing to resume from). 3 stages instead of a ramp:
+
+| Stage | Opponent | Timesteps | Gated | What it teaches |
+|---|---|---|---|---|
+| 1 — `bootstrap` | `inert` | 40M (~4h) | No | Empty-net finishing from a random policy — no moving target, full action space from the start. |
+| 2 — `selfplay` | `self_play` | 160M (~16h) | Yes, vs stage 1 | Where essentially all the learning happens. |
+| 3 — `gauntlet` | `frozen` = stage 2's own export | 120M (~12h) | Yes, vs stage 2 | A stationary opponent for a low-variance measurement, and a check that self-play didn't converge to a fixed point that only beats itself. |
+
+An "air drill" state-setter branch (`training_mode.gd`'s `air_drill_chance`,
+new — ball spawned high, both ships spawned low and lateral, unsolvable
+without climbing, kept clear of every wall so the RLGym-warned wall-bounce
+exploit has no wall nearby to bounce off) runs at a constant rate across
+*all* stages rather than being introduced late — gating *when* a skill gets
+drilled would reproduce the exact "gate what the policy can do" pattern
+that failed 3 generations running.
+
+**Observations**: `ShipObservations.SIZE` grew 31 → 35 (own contact normal
++ an `in_contact` flag, appended — never inserted, see that file's
+append-only invariant) so the value function can actually see the condition
+`wall_contact_penalty` fires on, instead of predicting a reward with no
+supporting signal.
+
+**Reward shaping**: mostly unchanged — a farmability check on the existing
+weights (`velocity_to_ball_weight`'s term telescopes to ~2.7 over a 20m
+approach, well under `goal_reward`=80; not gameable) argues generation 2/3's
+tuning was never the actual problem. Two changes: `airborne_penalty` is no
+longer passed by any stage (previously ramped *up* in lockstep with the
+axis generation 3 was trying to teach — directly adversarial to the goal of
+genuine aerial play), and `tilt_penalty` dropped 4x (0.002 → 0.0005 default)
+since an aerial approach to a high ball requires pitching. Deliberately
+*not* added: a standalone air-touch reward — that's the exact exploit RLGym
+warns about ("hits the ball off a wall high up instead of doing a real
+aerial"); the air-drill state setter already makes aerial skill
+instrumentally necessary to earn the existing ball-directed rewards.
+
+**Exploration**: `--reset-std` (meaningless under `MultiDiscrete` — no
+`log_std`) is replaced by `--reset-logits <scale>` (multiplies
+`action_net`'s weights/bias, optionally scoped to specific heads via
+`--reset-logits-heads`) for a deliberate post-diagnosis recovery, and more
+importantly by `--entropy-floor` (`train.py`'s `EntropyFloorCallback`): a
+*persistent* per-rollout controller nudging `ent_coef` to hold policy
+entropy near a target that decays over the run, replacing the one-shot
+`--reset-std` shock that reliably decayed away within ~10% of steps in
+every prior generation with something that responds continuously instead of
+once. `--ent-coef`'s default rose 0.0001 → 0.01 (tuned for `MultiDiscrete`'s
+bounded ~10-nat entropy, not a Gaussian's unbounded differential entropy).
+Per-head entropy (`train/entropy_head_<name>`) replaces the old aggregate
+`train/std` scalar — it identifies *which* axis is collapsing instead of
+one number for all seven.
+
+**Validation before spending the full ~32h budget**: see the ladder below —
+cheapest checks first (an offline action-space assertion, a headless Godot
+boot, a 100k-step smoke run, export parity + an in-game round trip against
+`easy.json`), then flight telemetry (`rollout/airborne_fraction`,
+`mean_altitude`, `air_touch_fraction`, `vertical_thrust_mean` — leading
+indicators visible from the first rollout instead of only in a win rate
+measured a full run later), then a short controlled A/B (MultiDiscrete vs.
+continuous, otherwise identical, ~20M steps each) before committing to the
+full curriculum — every past generation bet a full day on an unfalsifiable
+hypothesis, which is what made each failure expensive to diagnose.
+
+1. `training/test_action_space.py` — offline, seconds. Catches a head-order
+   mismatch, the single most likely silent killer (trains "fine" for 24h,
+   produces garbage — e.g. pitch commands driving strafe thrusters — with no
+   error).
+2. `godot --headless --path Game res://scenes/training.tscn` with no
+   trainer listening, 30s — catches `class_name`/observation-size
+   regressions.
+3. `.venv/bin/python train.py --experiment smoke --timesteps 100000
+   --n-parallel 2` — confirms the `MultiDiscrete` handshake and new
+   callback metrics emit.
+4. `export_policy.py` on the smoke checkpoint (mandatory index-level parity
+   check), then `evaluate.py <smoke>.json ../Game/bots/promoted/easy.json
+   --episodes 4` — exercises the real GDScript decode path.
+5. A short A/B: two 20M-step runs, identical except action space
+   (`MultiDiscrete` vs. the old continuous `Box(7)`), comparing
+   `rollout/airborne_fraction`. If discrete pulls meaningfully ahead, the
+   32h curriculum is a justified bet; if both stay near zero, the
+   hypothesis above is wrong and reward/compute explanations move to the
+   front — cheaper than a 4th blind multi-day generation either way.
 
 All curriculum flags default to leaving Godot's own `@export` defaults
 alone (`train.py` only forwards a flag when you pass it), so ordinary runs
 are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
 `--opponent-model <path>` (for `frozen`), `--draw-penalty`,
 `--attack-goal-bias`, `--kickoff-chance`, `--near-goal-chance`,
-`--vertical-ramp`, `--pitch-roll-ramp` (0.0-1.0 locomotion-unmask ramp),
+`--air-drill-chance` (generation 4's state-setter aerial curriculum),
 `--velocity-to-ball-weight`, `--ball-distance-penalty`, `--ball-touch-reward`,
-`--airborne-penalty`, `--ball-velocity-to-goal-weight`, `--goal-reward`.
+`--airborne-penalty`, `--tilt-penalty`, `--ball-velocity-to-goal-weight`,
+`--goal-reward`. (`--vertical-ramp`/`--pitch-roll-ramp` are gone — generation
+4 has no locomotion mask/ramp to control.)
 
 ### Running it automatically
 
@@ -348,17 +505,19 @@ are unaffected. Full flag list: `--opponent-mode {self_play,inert,frozen}`,
 pattern as `start_training.sh`) drives all stages end to end: for each
 stage it runs `run_training.sh` (pull, train, export, commit+push) with that
 stage's flags, then evaluates the resulting checkpoint against a reference
-bot over 100 episodes — the fixed `rookie.json` baseline for a from-scratch
+bot over 100 episodes — the previous stage's own passing export (stage 1 is
-stage 1 (no `resume_from_experiment`/`reference_experiment` override on
+ungated, so this only applies to stages 2+). Once every stage passes, a
-`STAGES[0]`), the previous stage's promoted checkpoint by default for
+final (non-gating) report evaluates the result against both
-stages 2+, or an explicit override in that stage's dict when it deliberately
+`Game/bots/promoted/easy.json` (the shipped bot) and
-skips a since-regressed branch (generation 1's stage 5) or seeds from a
+`Game/bots/promoted/reference-grounded.json` (a copy of generation 3's
-fixed foundation checkpoint (generation 2's stage 1 — see above).
+`curric-s5-aggression`, the strongest grounded-era artifact and the
+yardstick generations 1-3 were all measured against) — those two numbers are
+what actually answer "did generation 4 work?"
 
 ```bash
 cd training
 ./curriculum.sh                                            # start/resume the curriculum
-./curriculum.sh --seed-checkpoint checkpoints/run11/final.zip # override stage 1's resume source for this run
+./curriculum.sh --seed-checkpoint checkpoints/some/final.zip # override stage 1's resume source for this run
 ```
 
 The gate is deliberately lenient: it blocks a stage only on a **clear
@@ -374,13 +533,13 @@ result are logged to `curriculum_state.json` (committed alongside
 A stage gets up to 2 retries (3 attempts total) before the script stops and
 asks for a human look — it will not retry indefinitely or advance past a
 stage that keeps failing on its own. By default a retry resumes from that
-stage's own previous attempt with a fresh `--reset-std`; a stage can instead
+stage's own previous attempt (no `reset_retry_checkpoint` stage override is
-set `reset_retry_checkpoint: True` (generation 2's stage 1 does) to always
+set in generation 4 — nothing yet suggests a retry needs to reset to a
-reset to its normal resume source instead — see the generation 1 → 2
+clean upstream checkpoint the way generation 3's single `unmask` stage did;
-postmortem above for why blind same-checkpoint retries can make things
+add one if a stage's retries turn out to be drifting rather than
-monotonically worse. Once you've looked at why a block happened (more
+converging). Once you've looked at why a block happened (more timesteps? a
-timesteps? a flag needs adjusting? the eval itself was misleading?), re-run
+flag needs adjusting? the eval itself was misleading?), re-run with
-with `--force-retry` to try again or `--skip-to-next-stage` if you judge the
+`--force-retry` to try again or `--skip-to-next-stage` if you judge the
 result good enough despite the gate.
 
 Running a stage by hand (e.g. to experiment with flags before trusting the

training/cosmic_env.py

@@ -10,6 +10,7 @@ learning policy: self-play by construction.
 import pathlib
 import subprocess
 
+import numpy as np
 from godot_rl.core.godot_env import GodotEnv
 from godot_rl.wrappers.stable_baselines_wrapper import StableBaselinesGodotEnv
 
@@ -72,8 +73,13 @@ class CosmicClashEnv(GodotEnv):
 class CosmicClashVecEnv(StableBaselinesGodotEnv):
     """SB3 VecEnv over N parallel CosmicClashEnv instances.
 
-    convert_action_space=True flattens the env's (Box(6), Discrete(2)) action
+    convert_action_space=True: godot_rl's ActionSpaceProcessor reports a
-    space into a single Box(7): thrust xyz, rotation xyz, turbo (>0 means on).
+    gym.spaces.MultiDiscrete when every per-axis action entry is Discrete
+    (see ShipActionCodec/ShipAIController.get_action_space) — nvec
+    [5,5,5,5,5,5,2] for rotation xyz, thrust xyz, turbo, in that
+    gymnasium-sorted key order. No conversion logic here needs to change for
+    that; this class's only functional addition is the truncation-info
+    remap below.
     """
 
     def __init__(self, godot_bin: str, n_parallel: int = 1, seed: int = 0, port: int = GodotEnv.DEFAULT_PORT, **kwargs):
@@ -90,3 +96,27 @@ class CosmicClashVecEnv(StableBaselinesGodotEnv):
         self.n_parallel = n_parallel
         self._check_valid_action_space()
         self.results = None
+
+    def step(self, action):
+        """Remap ShipAIController.get_info()'s "truncated"/"terminal_obs"
+        into the keys SB3's on_policy_algorithm looks for
+        ("TimeLimit.truncated"/"terminal_observation") so PPO bootstraps
+        V(s) through an episode timeout instead of treating every 30s draw
+        as a true terminal state.
+
+        Godot_rl's own godot_env.py never sets either key (it returns the
+        same `done` array for both term and trunc, "# TODO update API to
+        term, trunc") and StableBaselinesGodotEnv.step() only ever returns
+        that single collapsed `dones` array to SB3 — so without this, PPO
+        has no way to distinguish "episode ended because a goal was scored"
+        (a genuine terminal, V(s)=0 is correct) from "episode ended because
+        the 30s clock ran out" (an artificial boundary that should be
+        bootstrapped through), and was silently treating every draw as the
+        former in every curriculum generation to date.
+        """
+        obs, rewards, dones, infos = super().step(action)
+        for info in infos:
+            if info.pop("truncated", False):
+                info["TimeLimit.truncated"] = True
+                info["terminal_observation"] = {"obs": np.array(info.pop("terminal_obs"), dtype=np.float32)}
+        return obs, rewards, dones, infos

training/curriculum.py

@@ -18,32 +18,49 @@ forever for the wrong reason. When a stage does fail MAX_RETRIES times in a
 row, the script stops and asks for a human look rather than retrying
 indefinitely or silently advancing past a bad stage.
 
-This is generation 3 of the curriculum. Generation 1 (6 stages: score,
+This is generation 4 of the curriculum — a full redesign, not a patch.
-defend, no_draws, mechanics, aggression, unmask) ran 2026-07-21 through
+Generations 1-3 (archived in curriculum_state_gen1.json/_gen2.json/_gen3.json)
-2026-07-26 and is archived in curriculum_state_gen1.json — its final stage
+all tried teaching full 3D flight by training grounded first and then
-("unmask", full 3D flight on top of the aggression retune) failed 3 straight
+opening up vertical/pitch-roll authority (a hard 0/1 mask in gen 1/2, a
-attempts, monotonically worsening (25% -> 20% -> 15% win rate vs
+gradual float ramp in gen 3) on top of a continuous Gaussian action space.
-curric-s5-aggression) because every retry resumed the same drifting
+All three failed: gen 1's hard mask went 25% -> 20% -> 15% win rate across 3
-checkpoint under identical flags instead of actually changing anything.
+attempts; gen 2's single-flip retune landed at a stable 32%/28%/31%; gen 3's
-Generation 2 (archived in curriculum_state_gen2.json) started a fresh
+gradual ramp landed at 29%/30%/24% — actually the worst of the three by its
-single "unmask" stage seeded directly from curric-s5-aggression's own
+final attempt. Every attempt showed the same signature regardless of
-checkpoint (FOUNDATION_EXPERIMENT below) with retuned reward weights — it
+mechanism: PPO's Gaussian action-distribution std collapsed from ~0.30 to
-also failed 3 attempts, landing at a stable 32% / 28% / 31% win rate each
+~0.13-0.15 within the first ~10% of steps and never recovered. The root
-time, ruling out both "retune the reward weights" and "just give it more
+cause: hovering this ship (mass 5.0, vertical_thrust 120, default gravity
-time" as fixes. Generation 3 replaces the single all-or-nothing unmask
+9.8 — see ship.gd) requires *holding* thrust.y ~= 0.408 continuously; a
-flip with a gradual ramp (4 stages: unmask-ramp25/50/75, then unmask at
+collapsed near-zero-mean Gaussian can brush that value but never sustain it
-full authority) — see TRAINING.md's "Generation 3" section for the full
+long enough to earn the reward gradient that would move the mean. No amount
-postmortem and design.
+of gating *when* the axis is allowed to act fixes a problem in *how* the
+policy represents a decision on it.
+
+Generation 4 (see TRAINING.md and Game/scripts/ship_action_codec.gd)
+replaces the action space itself with per-axis MultiDiscrete bins instead of
+a continuous Gaussian, trains the full action space from step 1 with no
+grounded stage at all (no successful self-play RL bot in this problem class
+gates control authority — see the RLGym/RLBot research cited in
+TRAINING.md), and adds a state-setter "air drill" episode-start branch
+(training_mode.gd's air_drill_chance) to force aerial practice instead of
+relying on reward-driven exploration alone. All of generation 1-3's
+checkpoint-lineage machinery (FOUNDATION_EXPERIMENT, locomotion-groundedness
+tracking, resume/reference overrides for skipping a regressed branch) is
+gone because there is nothing to resume from: every prior checkpoint is a
+different, incompatible action/observation shape. The two strongest prior
+artifacts are kept as fixed evaluation references instead (see
+PROMOTED_EASY/PROMOTED_REFERENCE_GROUNDED below) — they remain playable
+opponents forever via PolicyNetwork's format-versioned JSON even though
+their own checkpoints and generation are gone.
 
 Every experiment name this script generates is timestamped
 (YYYYMMDD-HHMM-<name>, applied once in run_stage_attempt) so runs stay
 unique across restarts/generations and sort chronologically in TensorBoard
-and checkpoints/ — plain names like "curric-s1-score" from generation 1
+and checkpoints/.
-would otherwise collide with generation 2's own stage 1.
 
 Usage:
     .venv/bin/python curriculum.py                  # run/resume the curriculum
-    .venv/bin/python curriculum.py --seed-checkpoint checkpoints/run11/final.zip
+    .venv/bin/python curriculum.py --seed-checkpoint checkpoints/some/final.zip
     .venv/bin/python curriculum.py --force-retry     # after fixing something, retry the blocked stage
     .venv/bin/python curriculum.py --skip-to-next-stage  # human judgment call: good enough, move on anyway
 
@@ -61,20 +78,14 @@ from datetime import datetime
 TRAINING_DIR = pathlib.Path(__file__).resolve().parent
 STATE_PATH = TRAINING_DIR / "curriculum_state.json"
 EVAL_HISTORY_PATH = TRAINING_DIR / "eval_history.json"
-ROOKIE_REFERENCE = TRAINING_DIR.parent / "Game" / "bots" / "rookie.json"
 
-# Generation 1's last cleanly-passing checkpoint (see curriculum_state_gen1.json)
+# Fixed evaluation references — never touched by training scripts (see
-# — generations 2 and 3 both build on this directly instead of re-running
+# TRAINING.md's "Promoted bots" section) — kept forever as playable
-# stages 1-5.
+# opponents via PolicyNetwork's format-versioned JSON even after their own
-FOUNDATION_EXPERIMENT = "curric-s5-aggression"
+# checkpoints/generation are gone. The final report (not a gate) evaluates
-
+# generation 4's result against both.
-# Groundedness (locomotion-mask state) for experiments that predate this
+PROMOTED_EASY = TRAINING_DIR.parent / "Game" / "bots" / "promoted" / "easy.json"
-# generation's own log, so _grounded_for_experiment can still answer for
+PROMOTED_REFERENCE_GROUNDED = TRAINING_DIR.parent / "Game" / "bots" / "promoted" / "reference-grounded.json"
-# them — see that function.
-LEGACY_GROUNDED = {
-    "rookie": False,
-    FOUNDATION_EXPERIMENT: True,
-}
 
 MAX_RETRIES = 2
 EVAL_EPISODES = 100
@@ -84,118 +95,108 @@ EVAL_EPISODES = 100
 # module docstring) — advance rather than retry.
 REGRESSION_MARGIN = 0.15
 
-# Standing flags applied to every attempt, mirroring next_run.sh: reset-std
+# Standing flags applied to every attempt. Generation 3's "--reset-std 0.3"
-# reopens exploration every attempt (harmless on fresh starts — train.py
+# (a one-shot shock, and meaningless anyway under MultiDiscrete — there is
-# only applies it on --resume), ent-coef keeps it from re-collapsing.
+# no log_std) is gone; EntropyFloorCallback (see train.py) is a continuous
-STANDING_ARGS = ["--reset-std", "0.3", "--ent-coef", "0.001"]
+# controller instead, which every generation's TensorBoard data argues is
+# what was actually needed (a single reset at attempt start reliably decayed
+# away within ~10% of steps, every time). --ent-coef raised an order of
+# magnitude from generation 3's 0.001: that value was tuned for a Gaussian's
+# unbounded differential entropy, not MultiDiscrete's bounded (~10-nat)
+# entropy.
+STANDING_ARGS = ["--ent-coef", "0.01", "--entropy-floor"]
 
-# Ball-chasing/scoring reward flags shared by every unmask-ramp stage
+# Reward-shaping flags shared by every stage so the studied variables (state
-# (generation 3 — see below): identical across all 4 stages so the ramp
+# mix, opponent mode) stay isolated — carried forward unchanged from
-# itself is the only studied variable. Lifted from generation 2's single
+# generation 2/3, which the reward-farmability analysis in TRAINING.md
-# "unmask" attempt (raised from stage 5's 0.05/0.006/0.5/0.02/60 defaults).
+# confirmed were never the actual problem. draw_penalty and airborne_penalty
-_UNMASK_RAMP_SHARED_FLAGS = [
+# are deliberately NOT overridden here (both default to 0.0 in
-    "--opponent-mode", "self_play",
+# training_mode.gd/ship_ai_controller.gd): generation 3's draw_penalty=5 and
+# airborne_penalty ramping up in lockstep with the unmask ramp were both
+# grounded-era, anti-flight pressures that have no place in a curriculum
+# whose entire point is teaching flight.
+_SHARED_REWARD_FLAGS = [
     "--velocity-to-ball-weight", "0.08",
     "--ball-distance-penalty", "0.01",
     "--ball-touch-reward", "0.7",
     "--ball-velocity-to-goal-weight", "0.06",
     "--goal-reward", "80",
-    "--draw-penalty", "5",
 ]
 
+# A stage dict may additionally set "abort_if": {"metric": "rollout/airborne_
+# fraction", "below": 0.05, "at_steps": N} to end that attempt early if a
+# flight-telemetry metric (see train.py's FlightTelemetryCallback) hasn't
+# cleared a bar by N *absolute* PPO timesteps (model.num_timesteps keeps
+# accumulating across --resume, so N must account for whatever this stage
+# inherits from its predecessor, not just this stage's own budget).
+# Deliberately unset on every stage below for now — rung 5 of TRAINING.md's
+# validation ladder (a short controlled A/B) should establish what a
+# sensible threshold actually looks like before any stage bets a real 12h+
+# budget on a guessed one.
 STAGES = [
     {
-        "name": "unmask-ramp25",
+        "name": "bootstrap",
-        # Generation 3, step 1/4 of a gradual locomotion-unmask ramp — see
+        # Stage 1/3: empty-net finishing practice from a random policy — no
-        # TRAINING.md's "Generation 3" section for the full postmortem.
+        # live opponent, so the full action space's first behaviour to
-        # Generation 2's single all-or-nothing "unmask" stage (flip
+        # emerge is "fly to ball, push it toward the net" without a moving
-        # vertical_ramp/pitch_roll_ramp 0 -> 1 in one step) failed 3
+        # target complicating credit assignment. Generation 1's own stage 1
-        # independent 240M-step attempts in a row, landing at a stable
+        # (also inert-opponent, also empty-net) was the one stage across all
-        # 32% / 28% / 31% win rate vs curric-s5-aggression each time — not
+        # 3 prior generations that unambiguously passed on its first
-        # noise (attempts 2-3 each gave the *same* checkpoint lineage
+        # attempt — reusing that shape here, just with the full action space
-        # another full 240M steps with zero improvement) and not fixable by
+        # live instead of yaw-only.
-        # more time. Every attempt shows train/std collapsing from ~0.30 to
-        # ~0.13-0.15 within the first ~10% of steps and never recovering —
-        # the policy locks the newly-opened axes back down before ever
-        # meaningfully exploring them.
-        #
-        # This stage instead scales vertical_ramp/pitch_roll_ramp to 25%
-        # authority. Ungated (see "gated" below and main()'s loop): this is
-        # a waypoint, not a measured transition — no eval runs, no
-        # regression gate applies, it always advances after training.
-        # airborne_penalty is off (0.0) here: at 25% authority the axis
-        # barely does anything yet, so there's nothing to discourage.
         "flags": [
-            *_UNMASK_RAMP_SHARED_FLAGS,
+            "--opponent-mode", "inert",
-            "--vertical-ramp", "0.25",
+            "--attack-goal-bias", "1.0",
-            "--pitch-roll-ramp", "0.25",
+            "--kickoff-chance", "0.10",
-            "--airborne-penalty", "0.0",
+            "--near-goal-chance", "0.50",
+            "--air-drill-chance", "0.20",
+            *_SHARED_REWARD_FLAGS,
         ],
-        "gated": False,
+        "gated": False,  # ungated waypoint: trains, checkpoints, always advances — no eval
-        "timesteps": 40_000_000,  # ~4h at the standing n-parallel/speedup (20M took ~2h)
+        "timesteps": 40_000_000,  # ~4h at the standing n-parallel/speedup
-        # Stage 0 MUST set this explicitly — resume_checkpoint()'s stage-0
-        # branch returns None (train from scratch) without it.
-        "resume_from_experiment": FOUNDATION_EXPERIMENT,
     },
     {
-        "name": "unmask-ramp50",
+        "name": "selfplay",
-        # Step 2/4: 50% authority. airborne_penalty at 1/3 of its final
+        # Stage 2/3: this is where essentially all of the actual learning
-        # value — enough to start discouraging unproductive altitude, not
+        # happens. Self-play (not frozen) as the main regime — it's what
-        # enough to fight the still-partial vertical axis outright.
+        # scales and what Necto/Nexto-class bots actually use; a frozen
-        # resume_from_experiment deliberately omitted: chains from
+        # target this early would cap skill at "exploits one specific bot"
-        # ramp25's pass via _resume_source_experiment's default
+        # instead of a moving, improving target. air_drill_chance stays on
-        # (_passing_experiment_for_stage) — do not add an override here.
+        # at a constant rate throughout (not introduced as a later stage) —
+        # gating *when* a skill is drilled reproduces the exact "gate what
+        # the policy is allowed to do" pattern that failed 3 generations in
+        # a row; only the state mix should vary between stages, never what
+        # the policy can act on.
         "flags": [
-            *_UNMASK_RAMP_SHARED_FLAGS,
+            "--opponent-mode", "self_play",
-            "--vertical-ramp", "0.5",
+            "--kickoff-chance", "0.15",
-            "--pitch-roll-ramp", "0.5",
+            "--near-goal-chance", "0.25",
-            "--airborne-penalty", "0.001",
+            "--air-drill-chance", "0.25",
+            *_SHARED_REWARD_FLAGS,
         ],
-        "gated": False,
+        "gated": True,
-        "timesteps": 40_000_000,
+        "timesteps": 160_000_000,  # ~16h
     },
     {
-        "name": "unmask-ramp75",
+        "name": "gauntlet",
-        # Step 3/4: 75% authority, airborne_penalty at 2/3 of its final
+        # Stage 3/3: a stationary opponent (this stage's own predecessor's
-        # value. Also chains automatically — no resume_from_experiment.
+        # export) gives a low-variance measurement — important when the gate
+        # is a 100-episode sample with a lenient 15-point margin — and
+        # catches a self-play fixed point: a policy that only learned to
+        # beat itself will look fine in stage 2 and stall here.
+        # opponent_model_from_previous_stage resolves --opponent-model at
+        # run time to whatever stage 2's own passing export turns out to be
+        # (see run_stage_attempt) rather than a hardcoded name.
         "flags": [
-            *_UNMASK_RAMP_SHARED_FLAGS,
+            "--opponent-mode", "frozen",
-            "--vertical-ramp", "0.75",
+            "--kickoff-chance", "0.15",
-            "--pitch-roll-ramp", "0.75",
+            "--near-goal-chance", "0.25",
-            "--airborne-penalty", "0.002",
+            "--air-drill-chance", "0.25",
+            *_SHARED_REWARD_FLAGS,
         ],
-        "gated": False,
+        "gated": True,
-        "timesteps": 40_000_000,
+        "timesteps": 120_000_000,  # ~12h
-    },
+        "opponent_model_from_previous_stage": True,
-    {
-        "name": "unmask",
-        # Step 4/4, the measured transition: full ramp (100% authority),
-        # airborne_penalty at its full value — behaviourally and eval-wise
-        # identical to generation 2's "unmask" stage's flags/config, so
-        # this stage's result is a direct, apples-to-apples comparison
-        # against the 3 failed all-or-nothing attempts (same reference,
-        # same opponent mode, same budget). Gated (default True): evaluated
-        # against FOUNDATION_EXPERIMENT exactly like every prior attempt.
-        #
-        # No resume_from_experiment here (deliberately, unlike generation
-        # 2's single-stage version) — this stage chains from ramp75's own
-        # checkpoint via the default resume path, not back to
-        # FOUNDATION_EXPERIMENT; only reference_experiment (the *eval*
-        # opponent) stays FOUNDATION_EXPERIMENT.
-        "flags": [
-            *_UNMASK_RAMP_SHARED_FLAGS,
-            "--vertical-ramp", "1.0",
-            "--pitch-roll-ramp", "1.0",
-            "--airborne-penalty", "0.003",
-        ],
-        "grounded": False,
-        "timesteps": 240_000_000,  # unchanged from the 3 failed attempts — same budget for a controlled comparison
-        "reference_experiment": FOUNDATION_EXPERIMENT,
-        # On retry, reset to the clean ramp75 checkpoint rather than
-        # compounding a failed full-ramp attempt's own drift — mirrors the
-        # generation 1 -> 2 postmortem (blind same-checkpoint retries only
-        # made things worse).
-        "reset_retry_checkpoint": True,
     },
 ]
 
@@ -227,50 +228,6 @@ def _logged_experiment_name(stage_index: int, attempt: int) -> str:
     raise RuntimeError(f"No logged experiment for stage {stage_index} attempt {attempt}")
 
 
-def resume_checkpoint(stage_index: int, attempt: int, seed_checkpoint: str | None) -> str | None:
-    if attempt > 0 and not STAGES[stage_index].get("reset_retry_checkpoint"):
-        # Retry: keep training the same stage's own last attempt.
-        prev = _logged_experiment_name(stage_index, attempt - 1)
-        return str(TRAINING_DIR / "checkpoints" / prev / "final.zip")
-    if stage_index == 0 and seed_checkpoint:
-        return seed_checkpoint
-    if stage_index == 0 and not STAGES[0].get("resume_from_experiment"):
-        # Deliberately fresh by default: the curriculum exists because
-        # resuming self-play across a regime change (run10, run11) didn't
-        # work, so a from-scratch stage 1 starts from a random policy under
-        # its own regime unless --seed-checkpoint or resume_from_experiment
-        # says otherwise.
-        return None
-    # Either a later stage chaining off its predecessor, or
-    # reset_retry_checkpoint: this stage's own retries have been drifting
-    # rather than converging (see the "unmask" stage's comment) — resume
-    # from the stage's normal resume source instead of compounding the last
-    # failed attempt's drift.
-    prev_experiment = _resume_source_experiment(stage_index)
-    return str(TRAINING_DIR / "checkpoints" / prev_experiment / "final.zip")
-
-
-def reference_bot(stage_index: int) -> str:
-    if stage_index == 0 and not STAGES[0].get("reference_experiment"):
-        return str(ROOKIE_REFERENCE)
-    prev_experiment = _reference_source_experiment(stage_index)
-    return str(TRAINING_DIR.parent / "Game" / "bots" / f"{prev_experiment}.json")
-
-
-# A stage normally chains off "whatever passed at the previous index," but a
-# stage can instead name an explicit resume_from_experiment/reference_experiment
-# to skip a since-regressed branch, or (stage 0) to seed from a fixed
-# foundation checkpoint instead of a from-scratch policy.
-def _resume_source_experiment(stage_index: int) -> str:
-    override = STAGES[stage_index].get("resume_from_experiment")
-    return override if override else _passing_experiment_for_stage(stage_index - 1)
-
-
-def _reference_source_experiment(stage_index: int) -> str:
-    override = STAGES[stage_index].get("reference_experiment")
-    return override if override else _passing_experiment_for_stage(stage_index - 1)
-
-
 def _passing_experiment_for_stage(stage_index: int) -> str:
     state = load_state()
     for entry in state["log"]:
@@ -279,14 +236,31 @@ def _passing_experiment_for_stage(stage_index: int) -> str:
     raise RuntimeError(f"No passing attempt recorded for stage {stage_index} ({STAGES[stage_index]['name']})")
 
 
-def _grounded_for_experiment(experiment: str) -> bool:
+def resume_checkpoint(stage_index: int, attempt: int, seed_checkpoint: str | None) -> str | None:
-    if experiment in LEGACY_GROUNDED:
+    if attempt > 0:
-        return LEGACY_GROUNDED[experiment]
+        # Retry: keep training the same stage's own last attempt. No
-    state = load_state()
+        # per-stage "reset to a clean upstream checkpoint" override in
-    for entry in state["log"]:
+        # generation 4 (unlike generation 3's "unmask" stage) — nothing yet
-        if entry["experiment"] == experiment:
+        # suggests a generation-4 retry needs that; add one if a stage's
-            return STAGES[entry["stage_index"]]["grounded"]
+        # retries turn out to be drifting rather than converging.
-    raise ValueError(f"Unknown experiment for groundedness lookup: {experiment}")
+        prev = _logged_experiment_name(stage_index, attempt - 1)
+        return str(TRAINING_DIR / "checkpoints" / prev / "final.zip")
+    if stage_index == 0:
+        # Deliberately fresh unless --seed-checkpoint says otherwise: full
+        # action space live from step 1, nothing to inherit — every prior
+        # generation's checkpoints are a different, incompatible
+        # action/observation shape (see module docstring).
+        return seed_checkpoint
+    prev_experiment = _passing_experiment_for_stage(stage_index - 1)
+    return str(TRAINING_DIR / "checkpoints" / prev_experiment / "final.zip")
+
+
+def reference_bot(stage_index: int) -> str:
+    """Only called for gated stages (stage 0 is ungated) — the previous
+    stage's own passing export, exactly like every prior generation's
+    default chaining."""
+    prev_experiment = _passing_experiment_for_stage(stage_index - 1)
+    return str(TRAINING_DIR.parent / "Game" / "bots" / f"{prev_experiment}.json")
 
 
 def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
@@ -294,9 +268,6 @@ def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
     # chronologically in TensorBoard/checkpoints — see module docstring.
     exp = f"{datetime.now().strftime('%Y%m%d-%H%M')}-{experiment_name(stage_index, attempt)}"
     resume = resume_checkpoint(stage_index, attempt, args.seed_checkpoint)
-    # A stage can override the run's timesteps budget (see "floor-lock",
-    # which deliberately runs much longer than the ~20M/~2h every stage so
-    # far has used); otherwise it falls back to curriculum.py's own --timesteps.
     timesteps = STAGES[stage_index].get("timesteps", args.timesteps)
     cmd = [
         "./run_training.sh", exp,
@@ -308,6 +279,17 @@ def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
     ]
     if resume:
         cmd += ["--resume", resume]
+    if STAGES[stage_index].get("opponent_model_from_previous_stage"):
+        prev_experiment = _passing_experiment_for_stage(stage_index - 1)
+        opponent_model = TRAINING_DIR.parent / "Game" / "bots" / f"{prev_experiment}.json"
+        cmd += ["--opponent-model", str(opponent_model)]
+    abort_if = STAGES[stage_index].get("abort_if")
+    if abort_if:
+        cmd += [
+            "--abort-metric", abort_if["metric"],
+            "--abort-below", str(abort_if["below"]),
+            "--abort-at-steps", str(abort_if["at_steps"]),
+        ]
     print(f"\n=== Stage {stage_index + 1}/{len(STAGES)} ({STAGES[stage_index]['name']}), "
           f"attempt {attempt + 1}/{MAX_RETRIES + 1}: {exp} ===")
     print(" ".join(cmd))
@@ -315,22 +297,9 @@ def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
     return exp
 
 
-def reference_grounded(stage_index: int) -> bool:
+def evaluate_attempt(experiment: str, reference: str, episodes: int) -> dict:
-    if stage_index == 0 and not STAGES[0].get("reference_experiment"):
-        # rookie.json predates the locomotion mask entirely — always full 3D.
-        return False
-    return _grounded_for_experiment(_reference_source_experiment(stage_index))
-
-
-def evaluate_attempt(experiment: str, reference: str, episodes: int, stage_index: int) -> dict:
     candidate = TRAINING_DIR.parent / "Game" / "bots" / f"{experiment}.json"
     cmd = [".venv/bin/python", "evaluate.py", str(candidate), reference, "--episodes", str(episodes)]
-    # Must match how each side was actually trained — see ai_ship_controller.gd's
-    # allow_vertical/allow_pitch_roll and evaluate.py's --grounded-a/-b.
-    if STAGES[stage_index]["grounded"]:
-        cmd.append("--grounded-a")
-    if reference_grounded(stage_index):
-        cmd.append("--grounded-b")
     print(" ".join(cmd))
     subprocess.run(cmd, cwd=TRAINING_DIR, check=True)
     history = json.loads(EVAL_HISTORY_PATH.read_text())
@@ -345,6 +314,33 @@ def decide(record: dict) -> str:
     return "pass"
 
 
+def final_report(experiment: str) -> None:
+    """Not a gate — the two numbers that actually answer "did generation 4
+    work?" (see TRAINING.md). promoted/easy.json is the shipped bot;
+    promoted/reference-grounded.json (a copy of generation 3's
+    curric-s5-aggression, made before the flat Game/bots/ dump was scrapped)
+    is the strongest grounded-era artifact and the yardstick generations 1-3
+    were all measured against. reference-grounded.json was trained with the
+    locomotion mask on, so needs --grounded-b; easy.json was itself promoted
+    from a *failed* unmask stage (curric-s6-unmask) and is full 3D like
+    every generation-4 candidate, so needs no flag."""
+    candidate = TRAINING_DIR.parent / "Game" / "bots" / f"{experiment}.json"
+    print("\n=== Curriculum complete — final report (informational, not a gate) ===")
+    for label, reference, extra_flags in [
+        ("promoted/easy.json (shipped bot)", PROMOTED_EASY, []),
+        ("promoted/reference-grounded.json (strongest grounded-era bot)", PROMOTED_REFERENCE_GROUNDED, ["--grounded-b"]),
+    ]:
+        if not reference.exists():
+            print(f"  vs {label}: skipped, file not found")
+            continue
+        cmd = [".venv/bin/python", "evaluate.py", str(candidate), str(reference), "--episodes", str(EVAL_EPISODES), *extra_flags]
+        print(" ".join(cmd))
+        subprocess.run(cmd, cwd=TRAINING_DIR, check=True)
+        record = json.loads(EVAL_HISTORY_PATH.read_text())[-1]
+        print(f"  vs {label}: {record['wins_a']}-{record['wins_b']} ({record['draws']} draws), "
+              f"win rate {record['win_rate_a']:.0%}")
+
+
 def commit_progress(experiment: str) -> None:
     subprocess.run(["git", "add", "curriculum_state.json", "eval_history.json"], cwd=TRAINING_DIR, check=True)
     result = subprocess.run(["git", "diff", "--cached", "--quiet"], cwd=TRAINING_DIR)
@@ -364,8 +360,7 @@ def main():
     parser.add_argument("--speedup", type=int, default=16)
     parser.add_argument(
         "--seed-checkpoint", default=None,
-        help="Resume stage 1 from this checkpoint instead of its default resume source "
+        help="Resume stage 1 from this checkpoint instead of training from scratch",
-        "(FOUNDATION_EXPERIMENT's checkpoint)",
     )
     parser.add_argument("--force-retry", action="store_true", help="Retry a blocked stage after human review")
     parser.add_argument("--skip-to-next-stage", action="store_true", help="Human judgment call: treat the blocked stage as good enough, advance anyway")
@@ -413,14 +408,13 @@ def main():
         last_experiment = experiment
 
         if not STAGES[stage_index].get("gated", True):
-            # Ungated ramp waypoint (see the unmask-ramp2X stages): trains,
+            # Ungated waypoint (see the bootstrap stage): trains,
             # checkpoints, and always advances — no eval, no regression
-            # gate, nothing to retry against. See TRAINING.md's
+            # gate, nothing to retry against.
-            # "Generation 3" section.
+            print(f"{experiment}: ungated waypoint — skipping eval, advancing unconditionally")
-            print(f"{experiment}: ungated ramp waypoint — skipping eval, advancing unconditionally")
             state["log"].append({
                 "stage_index": stage_index, "experiment": experiment, "attempt": attempt,
-                "decision": "pass", "note": "ungated ramp waypoint (no eval)",
+                "decision": "pass", "note": "ungated waypoint (no eval)",
             })
             state["stage_index"] += 1
             state["attempt"] = 0
@@ -430,7 +424,7 @@ def main():
             continue
 
         reference = reference_bot(stage_index)
-        record = evaluate_attempt(experiment, reference, EVAL_EPISODES, stage_index)
+        record = evaluate_attempt(experiment, reference, EVAL_EPISODES)
         decision = decide(record)
 
         print(f"{experiment}: candidate {record['wins_a']}-{record['wins_b']} reference "
@@ -469,6 +463,7 @@ def main():
         # None only if the loop above never ran at all (e.g. re-invoking
         # after the curriculum was already "done") — nothing new to commit
         # in that case.
+        final_report(last_experiment)
         commit_progress(last_experiment)
 
 

training/eval_history.json

@@ -178,5 +178,15 @@
     "wins_b": 56,
     "draws": 14,
     "win_rate_a": 0.3
+  },
+  {
+    "timestamp": "2026-08-04T21:26:41+00:00",
+    "model_a": "/Users/jcreek/Documents/repos/GitHub/CosmicClash/Game/bots/20260803-1829-curric-s4-unmask-retry2.json",
+    "model_b": "/Users/jcreek/Documents/repos/GitHub/CosmicClash/Game/bots/curric-s5-aggression.json",
+    "episodes": 100,
+    "wins_a": 24,
+    "wins_b": 56,
+    "draws": 20,
+    "win_rate_a": 0.24
   }
 ]

training/export_policy.py

@@ -1,9 +1,15 @@
 """Export a trained SB3 checkpoint to the JSON format PolicyNetwork.gd loads.
 
-The exported file contains the deterministic policy MLP (obs -> action means);
+The exported file contains the deterministic policy MLP. For a MultiDiscrete
-the game clamps outputs to [-1, 1] and treats the last value as turbo (> 0).
+(curriculum generation 4+) model, the raw output is 32 per-head logits
-A parity self-check compares the JSON forward pass against SB3's own
+decoded via ShipActionCodec.from_logits (argmax per head, mapped through
-deterministic prediction before writing.
+ACTION_HEADS' bin values below) and an "action_space" block is written to
+the JSON so the game knows to decode it that way. For an older continuous
+model, output is 7 action means, clamped to [-1, 1] and the last value
+treated as turbo (> 0) — no "action_space" block, matching every export
+before generation 4 (e.g. Game/bots/promoted/easy.json). A parity self-check
+compares the JSON forward pass against SB3's own deterministic prediction
+before writing, in either case.
 
 Example:
     .venv/bin/python export_policy.py checkpoints/smoke/final.zip ../Game/bots/rookie.json
@@ -13,10 +19,28 @@ import argparse
 import json
 import pathlib
 
+import gymnasium as gym
 import numpy as np
 import torch
 from stable_baselines3 import PPO
 
+# MUST exactly match Game/scripts/ship_action_codec.gd's HEADS (name, order,
+# and bin values) — this is what gets written into every generation-4
+# export's "action_space" block, and PolicyNetwork.gd/AIShipController never
+# re-derive it, they just decode against whatever's in the file. Sizes are
+# cross-checked against the live model's action_space.nvec below (a real
+# assertion), but bin *values* have no automated cross-language check —
+# treat any edit to either file as requiring the other.
+ACTION_HEADS = [
+    {"name": "rot_x", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "rot_y", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "rot_z", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "thrust_x", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "thrust_y", "bins": [-0.5, 0.0, 0.45, 0.75, 1.0]},
+    {"name": "thrust_z", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "turbo", "bins": [0.0, 1.0]},
+]
+
 
 def linear_to_layer(linear: torch.nn.Linear, activation: str) -> dict:
     return {
@@ -66,9 +90,40 @@ def main():
     policy = model.policy
     layers = extract_layers(policy)
     input_size = model.observation_space["obs"].shape[0]
+    is_multi_discrete = isinstance(model.action_space, gym.spaces.MultiDiscrete)
 
-    # Parity check: JSON forward pass must match SB3's deterministic action
+    output_data = {"input_size": int(input_size), "layers": layers}
     rng = np.random.default_rng(0)
+
+    if is_multi_discrete:
+        head_sizes = [len(head["bins"]) for head in ACTION_HEADS]
+        nvec = [int(n) for n in model.action_space.nvec]
+        assert nvec == head_sizes, (
+            f"model action_space.nvec {nvec} doesn't match ACTION_HEADS sizes {head_sizes} — "
+            "update ACTION_HEADS to match ship_action_codec.gd's HEADS"
+        )
+        output_data["action_space"] = {"type": "multi_discrete", "heads": ACTION_HEADS}
+
+        # Index-level parity check: deterministic=True now returns one
+        # argmax index per head (not a float to clip), so compare argmax of
+        # the JSON forward pass's raw logits, sliced per head, against SB3's
+        # own chosen indices — a head-order mistake here would otherwise
+        # train/export cleanly and only surface as silently wrong in-game
+        # behaviour (e.g. pitch commands driving strafe thrusters).
+        offsets = []
+        running = 0
+        for size in head_sizes:
+            offsets.append((running, running + size))
+            running += size
+        for _ in range(16):
+            obs = rng.uniform(-1, 1, input_size).astype(np.float32)
+            expected, _ = model.predict({"obs": obs}, deterministic=True)
+            logits = json_forward(layers, obs)
+            actual = np.array([int(np.argmax(logits[start:end])) for start, end in offsets])
+            assert np.array_equal(actual, expected), f"parity check failed: {actual} vs {expected}"
+    else:
+        # Legacy continuous parity check, unchanged: JSON forward pass must
+        # match SB3's deterministic action mean.
         for _ in range(16):
             obs = rng.uniform(-1, 1, input_size).astype(np.float32)
             expected, _ = model.predict({"obs": obs}, deterministic=True)
@@ -78,8 +133,12 @@ def main():
     output = pathlib.Path(args.output)
     output.parent.mkdir(parents=True, exist_ok=True)
     with open(output, "w") as f:
-        json.dump({"input_size": int(input_size), "layers": layers}, f)
+        json.dump(output_data, f)
-    print(f"Exported {args.checkpoint} -> {output} (input size {input_size}, {len(layers)} layers, parity OK)")
+    action_space_label = "multi_discrete" if is_multi_discrete else "continuous"
+    print(
+        f"Exported {args.checkpoint} -> {output} (input size {input_size}, {len(layers)} layers, "
+        f"action_space={action_space_label}, parity OK)"
+    )
 
 
 if __name__ == "__main__":

training/requirements.txt

@@ -1,5 +1,19 @@
-godot-rl
+# Pinned as of curriculum generation 4: this codebase now depends on
-stable-baselines3
+# specific library internals (godot_rl's ActionSpaceProcessor discrete-only
-tensorboard
+# -> MultiDiscrete branch, stable_baselines3's MultiCategoricalDistribution
+# logit layout — see Game/scripts/ship_action_codec.gd and train.py), not
+# just documented public APIs. An unpinned reinstall (e.g. via
+# setup_linux.sh on the remote training box) could silently resolve a newer
+# version that changes that behaviour without any error, which would be a
+# very expensive thing to discover partway through a 12h+ curriculum stage.
+# torch/gymnasium are pinned too even though they're transitive deps of the
+# two above, for the same reason (torch's log_std/action_net tensor
+# shapes, gymnasium's Dict space key-sorting behaviour that
+# ShipActionCodec's HEADS order relies on).
+godot-rl==0.8.2
+stable-baselines3==2.4.0
+torch==2.13.0
+gymnasium==1.0.0
+tensorboard==2.21.0
 # Optional, for --wandb logging:
 # wandb

training/run_training.sh

@@ -48,7 +48,12 @@ fi
 # Export for in-game use (parity-checked); models live in Game/bots/
 .venv/bin/python export_policy.py "checkpoints/$EXP/final.zip" "../Game/bots/$EXP.json"
 
-git add -A checkpoints logs eval_history.json "../Game/bots"
+# Only final.zip, not the intermediate ppo_*_steps.zip checkpoints (.gitignore
+# excludes them) — --resume only ever points at final.zip, so the "training
+# never stranded on one machine" property is fully preserved at ~0.2MB/run
+# instead of ~500MB/run (a single generation-3 experiment dir was 2401 files/
+# 506MB, of which final.zip was 221KB).
+git add "checkpoints/$EXP/final.zip" logs eval_history.json "../Game/bots"
 if git diff --cached --quiet; then
     echo "Nothing new to commit"
 else

training/test_action_space.py

@@ -0,0 +1,105 @@
+"""Rung 0 of TRAINING.md's validation ladder: offline, no Godot, seconds to
+run. Catches the single most likely silent killer in the generation-4
+action-space redesign — a head-order mismatch between the Python trainer and
+Game/scripts/ship_action_codec.gd's HEADS. If they disagree, training still
+runs happily for 24h+ (pitch commands driving strafe thrusters, say) and
+only surfaces as inexplicably-bad behaviour, not an error. This can't
+directly parse the GDScript file, but it locks the two real, checkable
+invariants an order mismatch would actually depend on: that gymnasium's Dict
+key-sorting produces the exact order ship_action_codec.gd's HEADS is written
+in, and that godot_rl's ActionSpaceProcessor converts that into the
+MultiDiscrete nvec the trainer expects. export_policy.py's own index-level
+parity check plus a real in-game round trip (rung 3) are what catch anything
+this can't.
+
+Usage:
+    .venv/bin/python test_action_space.py
+"""
+
+import sys
+
+import gymnasium as gym
+import numpy as np
+from godot_rl.core.utils import ActionSpaceProcessor
+
+from export_policy import ACTION_HEADS
+
+# The order Game/scripts/ship_action_codec.gd's HEADS is written in — kept
+# here as a literal, independent restatement (not derived from ACTION_HEADS)
+# so this test can actually catch export_policy.py's own list being edited
+# out of order too, not just catch nothing because both sides changed
+# together.
+EXPECTED_ORDER = ["rot_x", "rot_y", "rot_z", "thrust_x", "thrust_y", "thrust_z", "turbo"]
+
+
+def check_action_heads_match_expected_order() -> None:
+    names = [head["name"] for head in ACTION_HEADS]
+    assert names == EXPECTED_ORDER, (
+        f"export_policy.ACTION_HEADS order {names} != expected {EXPECTED_ORDER} — "
+        "this must match Game/scripts/ship_action_codec.gd's HEADS exactly"
+    )
+
+
+def check_gymnasium_sorts_to_expected_order() -> None:
+    # Build the Dict deliberately out of order (reversed) to prove it's
+    # gymnasium's sort doing the work here, not insertion order — this is
+    # exactly what godot_env.py does with the dict Godot sends over the wire
+    # (see godot_env.py's from_dict, which builds a spaces.Dict from
+    # ShipAIController.get_action_space()'s Dictionary).
+    sizes = {head["name"]: len(head["bins"]) for head in ACTION_HEADS}
+    reversed_dict = gym.spaces.Dict({name: gym.spaces.Discrete(sizes[name]) for name in reversed(EXPECTED_ORDER)})
+    sorted_names = list(reversed_dict.keys())
+    assert sorted_names == EXPECTED_ORDER, (
+        f"gymnasium.spaces.Dict sorted {sorted_names}, expected {EXPECTED_ORDER} — "
+        "if this changed, every export from this generation onward would be silently "
+        "mis-ordered relative to ship_action_codec.gd"
+    )
+
+
+def check_action_space_processor_produces_expected_multi_discrete() -> None:
+    sizes = [len(head["bins"]) for head in ACTION_HEADS]
+    tuple_space = gym.spaces.Tuple([gym.spaces.Discrete(n) for n in sizes])
+    processor = ActionSpaceProcessor(tuple_space, convert=True)
+    assert isinstance(processor.action_space, gym.spaces.MultiDiscrete), (
+        f"expected MultiDiscrete, got {type(processor.action_space)} — the all-discrete branch "
+        "in godot_rl's ActionSpaceProcessor may have changed (see requirements.txt's pin note)"
+    )
+    assert list(processor.action_space.nvec) == sizes, (
+        f"MultiDiscrete nvec {list(processor.action_space.nvec)} != expected {sizes}"
+    )
+
+
+def check_round_trip_preserves_per_head_values() -> None:
+    # An integer action per env, one column per head in EXPECTED_ORDER —
+    # confirms to_original_dist splits a MultiDiscrete action back into the
+    # same per-head order it was built from (this is what set_action() on
+    # the Godot side receives, keyed by head name).
+    sizes = [len(head["bins"]) for head in ACTION_HEADS]
+    tuple_space = gym.spaces.Tuple([gym.spaces.Discrete(n) for n in sizes])
+    processor = ActionSpaceProcessor(tuple_space, convert=True)
+
+    n_envs = 3
+    rng = np.random.default_rng(0)
+    action = np.stack([rng.integers(0, n, size=n_envs) for n in sizes], axis=1).astype(np.int64)
+    original = processor.to_original_dist(action)
+    assert len(original) == len(sizes)
+    for head_index, expected_column in enumerate(action.T):
+        np.testing.assert_array_equal(np.asarray(original[head_index]), expected_column)
+
+
+def main() -> int:
+    checks = [
+        check_action_heads_match_expected_order,
+        check_gymnasium_sorts_to_expected_order,
+        check_action_space_processor_produces_expected_multi_discrete,
+        check_round_trip_preserves_per_head_values,
+    ]
+    for check in checks:
+        check()
+        print(f"PASS: {check.__name__}")
+    print(f"\nAll {len(checks)} action-space checks passed.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

training/train.py

@@ -15,6 +15,7 @@ import argparse
 import os
 import pathlib
 
+from gymnasium import spaces
 from stable_baselines3 import PPO
 from stable_baselines3.common.callbacks import BaseCallback, CheckpointCallback
 from stable_baselines3.common.utils import safe_mean
@@ -25,6 +26,12 @@ from cosmic_env import CosmicClashVecEnv
 TRAINING_DIR = pathlib.Path(__file__).resolve().parent
 DEFAULT_GODOT_MACOS = "/Applications/Godot.app/Contents/MacOS/Godot"
 
+# Must match Game/scripts/ship_action_codec.gd's HEADS order exactly (both
+# are independently the gymnasium-sorted key order of the same 7 names) —
+# training/test_action_space.py's rung-0 check asserts this. Used only for
+# per-head entropy logging/reset-logits head selection below.
+ACTION_HEAD_NAMES = ["rot_x", "rot_y", "rot_z", "thrust_x", "thrust_y", "thrust_z", "turbo"]
+
 
 class GoalRateCallback(BaseCallback):
     """Logs rollout/goal_rate: the fraction of completed episodes in the
@@ -54,6 +61,154 @@ class GoalRateCallback(BaseCallback):
             self.logger.record("rollout/goal_rate", safe_mean(rates))
 
 
+class FlightTelemetryCallback(BaseCallback):
+    """Logs rollout/{airborne_fraction,mean_altitude,air_touch_fraction,
+    vertical_thrust_mean} — leading indicators for curriculum generation 4's
+    core hypothesis (a discrete action space lets the policy actually hold a
+    sustained vertical set-point, e.g. hovering), visible from the very
+    first rollout instead of only in a win-rate number measured a full
+    24h+ run later, which is what made every past generation's failure mode
+    expensive to diagnose. Requires VecMonitor(..., info_keywords=(...,
+    "airborne_fraction", "mean_altitude", "air_touch_fraction",
+    "vertical_thrust_mean")) — see ShipAIController.get_info."""
+
+    _KEYS = ("airborne_fraction", "mean_altitude", "air_touch_fraction", "vertical_thrust_mean")
+
+    def _on_step(self) -> bool:
+        return True
+
+    def _on_rollout_end(self) -> None:
+        if len(self.model.ep_info_buffer) == 0:
+            return
+        for key in self._KEYS:
+            values = [ep_info[key] for ep_info in self.model.ep_info_buffer if key in ep_info]
+            if values:
+                self.logger.record(f"rollout/{key}", safe_mean(values))
+
+
+class EntropyFloorCallback(BaseCallback):
+    """Replaces the old one-shot `--reset-std` shock (meaningless under
+    MultiDiscrete — there is no log_std) with a persistent controller.
+    Three curriculum generations' TensorBoard runs all show the same
+    signature: exploration (train/std, under the previous continuous
+    Gaussian) collapsing within the first ~10% of steps and never
+    recovering from a single reset applied at attempt start. A controller
+    that responds every rollout instead of once should not have that decay-
+    and-stay-collapsed failure mode.
+
+    Reads mean policy entropy each rollout (recomputed from a fresh
+    minibatch via the same RolloutBuffer.get() plumbing PPO's own train()
+    uses, since _on_rollout_end fires before that iteration's train() call)
+    and nudges model.ent_coef multiplicatively toward a target that decays
+    linearly from target_start_frac to target_end_frac of the action
+    space's maximum possible entropy (sum of ln(n) over each MultiDiscrete
+    head) over the run. PPO reads self.ent_coef fresh inside train() each
+    update, so mutating it here from a callback takes effect on the very
+    next update with no subclassing needed. No-ops (does nothing) for a
+    non-MultiDiscrete action space, e.g. a continuous-action A/B run.
+
+    Also logs train/entropy_head_<name> per action head — the direct
+    replacement for the old aggregate train/std scalar, and strictly more
+    useful: it identifies *which* axis is collapsing instead of one number
+    for all seven.
+    """
+
+    def __init__(
+        self,
+        total_timesteps: int,
+        target_start_frac: float = 0.55,
+        target_end_frac: float = 0.20,
+        adjust_rate: float = 1.02,
+        ent_coef_bounds: tuple[float, float] = (1e-4, 0.05),
+    ):
+        super().__init__()
+        self.total_timesteps = total_timesteps
+        self.target_start_frac = target_start_frac
+        self.target_end_frac = target_end_frac
+        self.adjust_rate = adjust_rate
+        self.ent_coef_bounds = ent_coef_bounds
+        self._is_multi_discrete = False
+        self._h_max = 0.0
+        self._start_timesteps = 0
+
+    def _on_training_start(self) -> None:
+        import numpy as np
+
+        self._is_multi_discrete = isinstance(self.model.action_space, spaces.MultiDiscrete)
+        if self._is_multi_discrete:
+            self._h_max = float(np.sum(np.log(self.model.action_space.nvec)))
+        # this call's own timesteps budget, not the resumed total — model.
+        # num_timesteps keeps accumulating across --resume calls, but
+        # total_timesteps below is this invocation's --timesteps.
+        self._start_timesteps = self.model.num_timesteps
+
+    def _on_step(self) -> bool:
+        return True
+
+    def _on_rollout_end(self) -> None:
+        if not self._is_multi_discrete:
+            return
+        import torch as th
+
+        batch = next(self.model.rollout_buffer.get(batch_size=self.model.batch_size))
+        with th.no_grad():
+            distribution = self.model.policy.get_distribution(batch.observations)
+        per_head = getattr(distribution, "distribution", None)
+        if per_head is None:
+            return
+
+        entropies = [dist.entropy().mean().item() for dist in per_head]
+        for name, entropy in zip(ACTION_HEAD_NAMES, entropies):
+            self.logger.record(f"train/entropy_head_{name}", entropy)
+
+        mean_entropy = sum(entropies)
+        progress = min((self.model.num_timesteps - self._start_timesteps) / self.total_timesteps, 1.0)
+        target_frac = self.target_start_frac + (self.target_end_frac - self.target_start_frac) * progress
+        target = target_frac * self._h_max
+        if mean_entropy < target:
+            self.model.ent_coef = min(self.model.ent_coef * self.adjust_rate, self.ent_coef_bounds[1])
+        else:
+            self.model.ent_coef = max(self.model.ent_coef / self.adjust_rate, self.ent_coef_bounds[0])
+        self.logger.record("train/ent_coef_adaptive", self.model.ent_coef)
+
+
+class AbortIfCallback(BaseCallback):
+    """Optional kill criterion (see curriculum.py's per-stage `abort_if`):
+    ends model.learn() early once `metric` (a rollout/* key logged by
+    FlightTelemetryCallback — must run earlier in the callback list so the
+    value exists by the time this checks it) is below `below` at or past
+    `at_steps`. Stops via SB3's own "_on_step returning False halts
+    training" contract rather than an exception, so the enclosing
+    try/finally in main() still runs and saves/exports/commits whatever
+    checkpoint exists — an aborted stage still leaves a usable, logged
+    artifact instead of either running a doomed stage to completion
+    unattended or leaving one stranded and uncommitted.
+    """
+
+    def __init__(self, metric: str, below: float, at_steps: int):
+        super().__init__()
+        self.metric = metric
+        self.below = below
+        self.at_steps = at_steps
+        self._checked = False
+        self._should_stop = False
+
+    def _on_step(self) -> bool:
+        return not self._should_stop
+
+    def _on_rollout_end(self) -> None:
+        if self._checked or self.model.num_timesteps < self.at_steps:
+            return
+        self._checked = True
+        value = self.logger.name_to_value.get(self.metric)
+        if value is not None and value < self.below:
+            print(
+                f"AbortIfCallback: {self.metric}={value:.4f} < {self.below} "
+                f"at {self.model.num_timesteps} steps — stopping early"
+            )
+            self._should_stop = True
+
+
 def parse_args():
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument(
@@ -75,18 +230,57 @@ def parse_args():
     parser.add_argument("--port", type=int, default=11008, help="Base TCP port (one per instance)")
     parser.add_argument("--seed", type=int, default=0)
     parser.add_argument("--resume", default=None, help="Checkpoint .zip to resume from")
-    parser.add_argument("--ent-coef", type=float, default=0.0001, help="Entropy bonus coefficient (applied on resume too)")
+    parser.add_argument(
+        "--ent-coef",
+        type=float,
+        default=0.01,
+        help="Entropy bonus coefficient (applied on resume too). Raised from 0.0001 for curriculum "
+        "generation 4: that value was tuned for a continuous Gaussian's differential entropy "
+        "(unbounded, can go negative); MultiDiscrete entropy is bounded (~10 nats for this action "
+        "space) and needs an order of magnitude more coefficient to matter. See --entropy-floor.",
+    )
     parser.add_argument("--n-steps", type=int, default=256, help="Rollout length per env between updates (applied on resume too)")
     parser.add_argument("--batch-size", type=int, default=256, help="PPO minibatch size (applied on resume too)")
     parser.add_argument(
-        "--reset-std",
+        "--reset-logits",
         type=float,
         default=None,
-        help="On resume, reset the policy action std to this value (recovers exploration after entropy collapse)",
+        help="On resume, multiply the policy's action_net weights/bias by this scale (e.g. 0.1), "
+        "pulling every head's softmax back toward uniform without discarding learned features — "
+        "the MultiDiscrete analogue of the old continuous --reset-std. Combine with "
+        "--reset-logits-heads to reset only specific heads.",
+    )
+    parser.add_argument(
+        "--reset-logits-heads",
+        default=None,
+        help=f"Comma-separated subset of {ACTION_HEAD_NAMES} to apply --reset-logits to (default: all heads)",
+    )
+    parser.add_argument(
+        "--entropy-floor",
+        action="store_true",
+        help="Enable EntropyFloorCallback: a persistent per-rollout controller nudging ent_coef to "
+        "hold policy entropy near a decaying target, replacing the one-shot --reset-std/"
+        "--reset-logits shock as the primary exploration mechanism (that flag remains for "
+        "resume-time recovery after a diagnosed collapse; this runs continuously).",
+    )
+    parser.add_argument(
+        "--checkpoint-every", type=int, default=10_000_000,
+        help="Timesteps between checkpoints. Raised from 100_000 for curriculum generation 4: at the "
+        "old value a single 240M-step stage wrote ~2400 intermediate checkpoint files (only final.zip "
+        "is ever committed, see .gitignore/run_training.sh, but they still accumulate in the working "
+        "tree during the run).",
     )
-    parser.add_argument("--checkpoint-every", type=int, default=100_000, help="Timesteps between checkpoints")
     parser.add_argument("--viz", action="store_true", help="Show game windows (debugging; slow)")
     parser.add_argument("--wandb", action="store_true", help="Also log to Weights & Biases")
+    parser.add_argument(
+        "--abort-metric", default=None,
+        help="Optional kill criterion (see curriculum.py's per-stage abort_if): a rollout/* metric name to watch",
+    )
+    parser.add_argument("--abort-below", type=float, default=None, help="Stop early if --abort-metric drops below this")
+    parser.add_argument(
+        "--abort-at-steps", type=int, default=None,
+        help="Don't check --abort-metric until at least this many timesteps have elapsed",
+    )
 
     curriculum = parser.add_argument_group(
         "curriculum", "Stage the training run — see TRAINING.md's Curriculum training section"
@@ -112,12 +306,13 @@ def parse_args():
     curriculum.add_argument("--kickoff-chance", type=float, default=None, help="Overrides kickoff_state_chance")
     curriculum.add_argument("--near-goal-chance", type=float, default=None, help="Overrides ball_near_goal_chance")
     curriculum.add_argument(
-        "--vertical-ramp", type=float, default=None,
+        "--air-drill-chance", type=float, default=None,
-        help="0.0-1.0: fraction of vertical thrust that reaches the ship (locomotion-unmask ramp; default 1.0)",
+        help="Overrides air_drill_chance: ball spawned high, both ships spawned low and lateral — "
+        "unsolvable without climbing (curriculum generation 4's state-setter aerial curriculum)",
     )
     curriculum.add_argument(
-        "--pitch-roll-ramp", type=float, default=None,
+        "--tilt-penalty", type=float, default=None,
-        help="0.0-1.0: fraction of pitch/roll rotation that reaches the ship (locomotion-unmask ramp; default 1.0)",
+        help="Overrides ShipAIController.tilt_penalty (dense per-tick cost scaled by non-upright tilt)",
     )
     curriculum.add_argument(
         "--velocity-to-ball-weight", type=float, default=None,
@@ -158,8 +353,8 @@ def _curriculum_kwargs(args) -> dict:
         "attack_goal_bias": args.attack_goal_bias,
         "kickoff_state_chance": args.kickoff_chance,
         "ball_near_goal_chance": args.near_goal_chance,
-        "ai_vertical_ramp": args.vertical_ramp,
+        "air_drill_chance": args.air_drill_chance,
-        "ai_pitch_roll_ramp": args.pitch_roll_ramp,
+        "ai_tilt_penalty": args.tilt_penalty,
         "ai_velocity_to_ball_weight": args.velocity_to_ball_weight,
         "ai_ball_distance_penalty": args.ball_distance_penalty,
         "ai_ball_touch_reward": args.ball_touch_reward,
@@ -192,7 +387,16 @@ def main():
         speedup=args.speedup,
         **_curriculum_kwargs(args),
     )
-    env = VecMonitor(env, info_keywords=("goal_scored",))
+    env = VecMonitor(
+        env,
+        info_keywords=(
+            "goal_scored",
+            "airborne_fraction",
+            "mean_altitude",
+            "air_touch_fraction",
+            "vertical_thrust_mean",
+        ),
+    )
 
     if args.resume:
         model = PPO.load(
@@ -207,14 +411,22 @@ def main():
             f"Resumed from {args.resume} at {model.num_timesteps} timesteps "
             f"(ent_coef={args.ent_coef}, n_steps={args.n_steps}, batch_size={args.batch_size})"
         )
-        if args.reset_std is not None:
+        if args.reset_logits is not None:
-            import math
-
             import torch
 
+            heads = args.reset_logits_heads.split(",") if args.reset_logits_heads else ACTION_HEAD_NAMES
+            nvec = list(model.action_space.nvec)
+            offset = 0
+            offsets = {}
+            for name, size in zip(ACTION_HEAD_NAMES, nvec):
+                offsets[name] = (offset, offset + size)
+                offset += size
             with torch.no_grad():
-                model.policy.log_std.fill_(math.log(args.reset_std))
+                for name in heads:
-            print(f"Reset policy action std to {args.reset_std}")
+                    start, end = offsets[name]
+                    model.policy.action_net.weight[start:end].mul_(args.reset_logits)
+                    model.policy.action_net.bias[start:end].mul_(args.reset_logits)
+            print(f"Reset action_net logits for heads {heads} by scale {args.reset_logits}")
     else:
         model = PPO(
             "MultiInputPolicy",
@@ -232,12 +444,18 @@ def main():
         save_path=str(checkpoint_dir),
         name_prefix="ppo",
     )
-    goal_rate_callback = GoalRateCallback()
+    # Order matters for AbortIfCallback (must run after FlightTelemetryCallback
+    # so the rollout/* metric it watches has already been logged this round).
+    callbacks = [checkpoint_callback, GoalRateCallback(), FlightTelemetryCallback()]
+    if args.entropy_floor:
+        callbacks.append(EntropyFloorCallback(total_timesteps=args.timesteps))
+    if args.abort_metric is not None and args.abort_below is not None and args.abort_at_steps is not None:
+        callbacks.append(AbortIfCallback(args.abort_metric, args.abort_below, args.abort_at_steps))
 
     try:
         model.learn(
             args.timesteps,
-            callback=[checkpoint_callback, goal_rate_callback],
+            callback=callbacks,
             tb_log_name=args.experiment,
             reset_num_timesteps=not args.resume,
         )