mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-13 04:12:03 +00:00
feat(training): curriculum generation 4 — MultiDiscrete action space redesign
Three curriculum generations (2026-07-21 through 2026-08-04) all tried gating *when* the policy could use vertical thrust/pitch-roll on top of a continuous Gaussian action space, and all three failed the same way: PPO's action-distribution std collapsed within ~10% of steps and never recovered, landing at a 15-32% win rate vs the grounded reference regardless of mechanism (hard mask, then a gradual ramp). Generation 3's final attempt just landed at 24% — the worst of the three. Root cause, verified against this project's own physics: hovering this ship requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust 120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value but never sustain it long enough to earn the reward gradient that would move the mean — no amount of gating *when* the axis acts fixes a problem in *how* the policy represents a decision on it. This also independently found and fixes a real bug: godot_rl never marks an episode timeout as a truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every generation to date. - Game/scripts/ship_action_codec.gd (new): single source of truth for a per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared by training and in-game inference, replacing the continuous Gaussian. thrust_y's bins are deliberately asymmetric so a random policy drifts through the volume instead of floor-pinning. Legacy continuous decode (ai_ship_controller.gd's old logic) preserved verbatim so every pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working unchanged via an optional "action_space" JSON field. - ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only) so the value function can see what wall_contact_penalty fires on. - ship_ai_controller.gd: action space/decode via the codec; drop the vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default lowered 4x (aerial approaches require pitching); flight telemetry (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean) and truncation-snapshot fields on get_info(). - training_mode.gd: new air_drill_chance state-setter branch (ball spawned high, ships low, kept clear of walls) so aerial practice is forced by the environment instead of relying on reward-driven exploration alone; snapshot terminal observations before a timeout reset for the truncation fix. - cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into SB3's TimeLimit.truncated/terminal_observation keys. - train.py: --reset-logits (+ --reset-logits-heads) replaces the now-meaningless --reset-std; new EntropyFloorCallback (a persistent per-rollout ent_coef controller replacing the one-shot std-reset shock) and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/ --tilt-penalty flags; optional AbortIfCallback kill-criterion. - export_policy.py: writes the action_space block for MultiDiscrete models; index-level parity check (argmax per head) instead of comparing floats. - curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no grounded stage, full action space live from step 1; deletes generation 1-3's checkpoint-lineage machinery (nothing to resume from); final report evaluates against both promoted/easy.json and the new promoted/reference-grounded.json (a copy of curric-s5-aggression, the strongest grounded-era artifact, kept as a fixed yardstick). - run_training.sh/.gitignore: commit only final.zip, not the ~2400 intermediate checkpoint files a single stage was writing (~500MB -> ~0.2MB per run); requirements.txt pinned (behaviour here now depends on specific library internals, not just public APIs). - test_action_space.py (new): offline rung-0 check catching a head-order mismatch before it silently corrupts 24h of training. Validated: GDScript compiles clean (Godot --headless --import + script validation), free_play.tscn and training.tscn both boot headless without errors, offline action-space assertions pass. Not yet run: the actual smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4" section, before committing to the full ~32h curriculum. See TRAINING.md's "Generation 4" section for the full design writeup.
This commit is contained in:
@@ -14,9 +14,18 @@ extends RefCounted
|
||||
# {"weights": [[out x in floats]], "biases": [out floats], "activation": "tanh" | "linear"},
|
||||
# ...
|
||||
# ]
|
||||
# "action_space": {"type": "multi_discrete", "heads": [{"name","bins"}, ...]} // optional
|
||||
# }
|
||||
#
|
||||
# "action_space" is absent from every model exported before curriculum
|
||||
# generation 4 (e.g. Game/bots/promoted/easy.json) — absence means
|
||||
# {"type": "continuous"}, decoded via ShipActionCodec.from_continuous, the
|
||||
# same flattened-Box(7)-mean-output path this class has always produced.
|
||||
# This class itself never changes behaviour based on it; only the caller
|
||||
# (AIShipController._decide) branches on action_space["type"].
|
||||
|
||||
var input_size: int = 0
|
||||
var action_space: Dictionary = {"type": "continuous"}
|
||||
var _layers: Array = []
|
||||
|
||||
|
||||
@@ -32,6 +41,7 @@ static func load_from_file(path: String) -> PolicyNetwork:
|
||||
|
||||
var net := PolicyNetwork.new()
|
||||
net.input_size = int(data.get("input_size", 0))
|
||||
net.action_space = data.get("action_space", {"type": "continuous"})
|
||||
for layer in data["layers"]:
|
||||
# Flatten each layer's weights into a PackedFloat64Array for speed
|
||||
var out_size: int = layer["biases"].size()
|
||||
@@ -55,7 +65,14 @@ static func load_from_file(path: String) -> PolicyNetwork:
|
||||
|
||||
|
||||
func forward(observation: Array) -> Array:
|
||||
var x := PackedFloat64Array(observation)
|
||||
if observation.size() < input_size:
|
||||
push_error("PolicyNetwork: observation has %d values, model expects %d" % [observation.size(), input_size])
|
||||
# Slice rather than trust the caller: ShipObservations.SIZE only ever
|
||||
# grows (append-only), so an older/smaller model must still decode
|
||||
# correctly against a newer, longer observation vector — the extra
|
||||
# trailing values it never trained on are simply dropped here rather
|
||||
# than corrupting the first layer's dot product by accident.
|
||||
var x := PackedFloat64Array(observation.slice(0, input_size))
|
||||
for layer in _layers:
|
||||
var in_size: int = layer["in_size"]
|
||||
var out_size: int = layer["out_size"]
|
||||
|
||||
Reference in New Issue
Block a user