mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
Josh Creek
1811e9333e
Three curriculum generations (2026-07-21 through 2026-08-04) all tried gating *when* the policy could use vertical thrust/pitch-roll on top of a continuous Gaussian action space, and all three failed the same way: PPO's action-distribution std collapsed within ~10% of steps and never recovered, landing at a 15-32% win rate vs the grounded reference regardless of mechanism (hard mask, then a gradual ramp). Generation 3's final attempt just landed at 24% — the worst of the three. Root cause, verified against this project's own physics: hovering this ship requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust 120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value but never sustain it long enough to earn the reward gradient that would move the mean — no amount of gating *when* the axis acts fixes a problem in *how* the policy represents a decision on it. This also independently found and fixes a real bug: godot_rl never marks an episode timeout as a truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every generation to date. - Game/scripts/ship_action_codec.gd (new): single source of truth for a per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared by training and in-game inference, replacing the continuous Gaussian. thrust_y's bins are deliberately asymmetric so a random policy drifts through the volume instead of floor-pinning. Legacy continuous decode (ai_ship_controller.gd's old logic) preserved verbatim so every pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working unchanged via an optional "action_space" JSON field. - ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only) so the value function can see what wall_contact_penalty fires on. - ship_ai_controller.gd: action space/decode via the codec; drop the vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default lowered 4x (aerial approaches require pitching); flight telemetry (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean) and truncation-snapshot fields on get_info(). - training_mode.gd: new air_drill_chance state-setter branch (ball spawned high, ships low, kept clear of walls) so aerial practice is forced by the environment instead of relying on reward-driven exploration alone; snapshot terminal observations before a timeout reset for the truncation fix. - cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into SB3's TimeLimit.truncated/terminal_observation keys. - train.py: --reset-logits (+ --reset-logits-heads) replaces the now-meaningless --reset-std; new EntropyFloorCallback (a persistent per-rollout ent_coef controller replacing the one-shot std-reset shock) and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/ --tilt-penalty flags; optional AbortIfCallback kill-criterion. - export_policy.py: writes the action_space block for MultiDiscrete models; index-level parity check (argmax per head) instead of comparing floats. - curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no grounded stage, full action space live from step 1; deletes generation 1-3's checkpoint-lineage machinery (nothing to resume from); final report evaluates against both promoted/easy.json and the new promoted/reference-grounded.json (a copy of curric-s5-aggression, the strongest grounded-era artifact, kept as a fixed yardstick). - run_training.sh/.gitignore: commit only final.zip, not the ~2400 intermediate checkpoint files a single stage was writing (~500MB -> ~0.2MB per run); requirements.txt pinned (behaviour here now depends on specific library internals, not just public APIs). - test_action_space.py (new): offline rung-0 check catching a head-order mismatch before it silently corrupts 24h of training. Validated: GDScript compiles clean (Godot --headless --import + script validation), free_play.tscn and training.tscn both boot headless without errors, offline action-space assertions pass. Not yet run: the actual smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4" section, before committing to the full ~32h curriculum. See TRAINING.md's "Generation 4" section for the full design writeup.
63 lines
2.8 KiB
Bash
Executable File
63 lines
2.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Train, export the policy, and commit every artifact to git so no training
|
|
# is ever stranded on one machine. Idempotent and re-runnable: pulls latest
|
|
# before training, commits only when there is something new, and a Ctrl-C'd
|
|
# run still exports and commits (final.zip is written on the way out).
|
|
#
|
|
# Usage: ./run_training.sh <experiment> [train.py args...]
|
|
# e.g.: ./run_training.sh run02 --timesteps 20000000 --n-parallel 14 --speedup 16 \
|
|
# --resume checkpoints/run01/final.zip --ent-coef 0.001 --reset-std 0.3
|
|
set -uo pipefail
|
|
|
|
cd "$(dirname "$0")"
|
|
EXP="${1:?usage: run_training.sh <experiment> [train.py args...]}"
|
|
shift
|
|
|
|
# Train on the latest code and checkpoints from any machine
|
|
git pull --rebase
|
|
|
|
# Rebuild the exported training binary against the code we just pulled, so a
|
|
# --exported-binary run never trains on a stale snapshot. Only when the
|
|
# caller has already opted into the exported-binary flow (training/build/
|
|
# exists from a prior export) — a source-run caller pays no extra latency.
|
|
# Fails the whole run rather than silently falling through to train.py
|
|
# against a stale or partially-rebuilt binary (export_linux.sh sets -e, but
|
|
# that only exits *it*, not this script).
|
|
EXTRA_ARGS=()
|
|
if [ -d build ]; then
|
|
./export_linux.sh || { echo "export_linux.sh failed — aborting" >&2; exit 1; }
|
|
# Standing entry points (next_run.sh, curriculum.py) don't know about the
|
|
# exported binary and never pass --exported-binary themselves; default it
|
|
# here so opting in (by ever running export_linux.sh once) actually gets
|
|
# used, not just kept up to date. An explicit --exported-binary in "$@"
|
|
# still wins (argparse: later occurrence overrides).
|
|
EXTRA_ARGS=(--exported-binary build/CosmicClash.x86_64)
|
|
fi
|
|
|
|
# Let Ctrl-C stop train.py without killing this script, so the export and
|
|
# commit below still run
|
|
trap ':' INT
|
|
.venv/bin/python train.py --experiment "$EXP" "${EXTRA_ARGS[@]}" "$@"
|
|
trap - INT
|
|
|
|
if [ ! -f "checkpoints/$EXP/final.zip" ]; then
|
|
echo "No checkpoints/$EXP/final.zip — nothing to export or commit" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Export for in-game use (parity-checked); models live in Game/bots/
|
|
.venv/bin/python export_policy.py "checkpoints/$EXP/final.zip" "../Game/bots/$EXP.json"
|
|
|
|
# Only final.zip, not the intermediate ppo_*_steps.zip checkpoints (.gitignore
|
|
# excludes them) — --resume only ever points at final.zip, so the "training
|
|
# never stranded on one machine" property is fully preserved at ~0.2MB/run
|
|
# instead of ~500MB/run (a single generation-3 experiment dir was 2401 files/
|
|
# 506MB, of which final.zip was 221KB).
|
|
git add "checkpoints/$EXP/final.zip" logs eval_history.json "../Game/bots"
|
|
if git diff --cached --quiet; then
|
|
echo "Nothing new to commit"
|
|
else
|
|
git commit -m "chore(training): Add $EXP checkpoints, logs, and exported policy"
|
|
git push
|
|
fi
|