feat(training): curriculum generation 4 — MultiDiscrete action space redesign

Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.

Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.

- Game/scripts/ship_action_codec.gd (new): single source of truth for a
  per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
  by training and in-game inference, replacing the continuous Gaussian.
  thrust_y's bins are deliberately asymmetric so a random policy drifts
  through the volume instead of floor-pinning. Legacy continuous decode
  (ai_ship_controller.gd's old logic) preserved verbatim so every
  pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
  unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
  so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
  vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
  lowered 4x (aerial approaches require pitching); flight telemetry
  (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
  and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
  high, ships low, kept clear of walls) so aerial practice is forced by the
  environment instead of relying on reward-driven exploration alone; snapshot
  terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
  SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
  now-meaningless --reset-std; new EntropyFloorCallback (a persistent
  per-rollout ent_coef controller replacing the one-shot std-reset shock)
  and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
  --tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
  index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
  grounded stage, full action space live from step 1; deletes generation
  1-3's checkpoint-lineage machinery (nothing to resume from); final report
  evaluates against both promoted/easy.json and the new
  promoted/reference-grounded.json (a copy of curric-s5-aggression, the
  strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
  intermediate checkpoint files a single stage was writing (~500MB ->
  ~0.2MB per run); requirements.txt pinned (behaviour here now depends on
  specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
  mismatch before it silently corrupts 24h of training.

Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.

See TRAINING.md's "Generation 4" section for the full design writeup.

training/export_policy.py

@@ -1,9 +1,15 @@
 """Export a trained SB3 checkpoint to the JSON format PolicyNetwork.gd loads.
 
-The exported file contains the deterministic policy MLP (obs -> action means);
-the game clamps outputs to [-1, 1] and treats the last value as turbo (> 0).
-A parity self-check compares the JSON forward pass against SB3's own
-deterministic prediction before writing.
+The exported file contains the deterministic policy MLP. For a MultiDiscrete
+(curriculum generation 4+) model, the raw output is 32 per-head logits
+decoded via ShipActionCodec.from_logits (argmax per head, mapped through
+ACTION_HEADS' bin values below) and an "action_space" block is written to
+the JSON so the game knows to decode it that way. For an older continuous
+model, output is 7 action means, clamped to [-1, 1] and the last value
+treated as turbo (> 0) — no "action_space" block, matching every export
+before generation 4 (e.g. Game/bots/promoted/easy.json). A parity self-check
+compares the JSON forward pass against SB3's own deterministic prediction
+before writing, in either case.
 
 Example:
     .venv/bin/python export_policy.py checkpoints/smoke/final.zip ../Game/bots/rookie.json
@@ -13,10 +19,28 @@ import argparse
 import json
 import pathlib
 
+import gymnasium as gym
 import numpy as np
 import torch
 from stable_baselines3 import PPO
 
+# MUST exactly match Game/scripts/ship_action_codec.gd's HEADS (name, order,
+# and bin values) — this is what gets written into every generation-4
+# export's "action_space" block, and PolicyNetwork.gd/AIShipController never
+# re-derive it, they just decode against whatever's in the file. Sizes are
+# cross-checked against the live model's action_space.nvec below (a real
+# assertion), but bin *values* have no automated cross-language check —
+# treat any edit to either file as requiring the other.
+ACTION_HEADS = [
+    {"name": "rot_x", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "rot_y", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "rot_z", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "thrust_x", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "thrust_y", "bins": [-0.5, 0.0, 0.45, 0.75, 1.0]},
+    {"name": "thrust_z", "bins": [-1.0, -0.5, 0.0, 0.5, 1.0]},
+    {"name": "turbo", "bins": [0.0, 1.0]},
+]
+
 
 def linear_to_layer(linear: torch.nn.Linear, activation: str) -> dict:
     return {
@@ -66,20 +90,55 @@ def main():
     policy = model.policy
     layers = extract_layers(policy)
     input_size = model.observation_space["obs"].shape[0]
+    is_multi_discrete = isinstance(model.action_space, gym.spaces.MultiDiscrete)
 
-    # Parity check: JSON forward pass must match SB3's deterministic action
+    output_data = {"input_size": int(input_size), "layers": layers}
     rng = np.random.default_rng(0)
-    for _ in range(16):
-        obs = rng.uniform(-1, 1, input_size).astype(np.float32)
-        expected, _ = model.predict({"obs": obs}, deterministic=True)
-        actual = np.clip(json_forward(layers, obs), -1.0, 1.0)
-        assert np.allclose(actual, expected, atol=1e-5), f"parity check failed: {actual} vs {expected}"
+
+    if is_multi_discrete:
+        head_sizes = [len(head["bins"]) for head in ACTION_HEADS]
+        nvec = [int(n) for n in model.action_space.nvec]
+        assert nvec == head_sizes, (
+            f"model action_space.nvec {nvec} doesn't match ACTION_HEADS sizes {head_sizes} — "
+            "update ACTION_HEADS to match ship_action_codec.gd's HEADS"
+        )
+        output_data["action_space"] = {"type": "multi_discrete", "heads": ACTION_HEADS}
+
+        # Index-level parity check: deterministic=True now returns one
+        # argmax index per head (not a float to clip), so compare argmax of
+        # the JSON forward pass's raw logits, sliced per head, against SB3's
+        # own chosen indices — a head-order mistake here would otherwise
+        # train/export cleanly and only surface as silently wrong in-game
+        # behaviour (e.g. pitch commands driving strafe thrusters).
+        offsets = []
+        running = 0
+        for size in head_sizes:
+            offsets.append((running, running + size))
+            running += size
+        for _ in range(16):
+            obs = rng.uniform(-1, 1, input_size).astype(np.float32)
+            expected, _ = model.predict({"obs": obs}, deterministic=True)
+            logits = json_forward(layers, obs)
+            actual = np.array([int(np.argmax(logits[start:end])) for start, end in offsets])
+            assert np.array_equal(actual, expected), f"parity check failed: {actual} vs {expected}"
+    else:
+        # Legacy continuous parity check, unchanged: JSON forward pass must
+        # match SB3's deterministic action mean.
+        for _ in range(16):
+            obs = rng.uniform(-1, 1, input_size).astype(np.float32)
+            expected, _ = model.predict({"obs": obs}, deterministic=True)
+            actual = np.clip(json_forward(layers, obs), -1.0, 1.0)
+            assert np.allclose(actual, expected, atol=1e-5), f"parity check failed: {actual} vs {expected}"
 
     output = pathlib.Path(args.output)
     output.parent.mkdir(parents=True, exist_ok=True)
     with open(output, "w") as f:
-        json.dump({"input_size": int(input_size), "layers": layers}, f)
-    print(f"Exported {args.checkpoint} -> {output} (input size {input_size}, {len(layers)} layers, parity OK)")
+        json.dump(output_data, f)
+    action_space_label = "multi_discrete" if is_multi_discrete else "continuous"
+    print(
+        f"Exported {args.checkpoint} -> {output} (input size {input_size}, {len(layers)} layers, "
+        f"action_space={action_space_label}, parity OK)"
+    )
 
 
 if __name__ == "__main__":