Round 4 changed three things at once and two of them cut upright pressure:
grounded_upright_reward went to 0 and ground_tilt_penalty was cut 2.5x,
while the new uprightness multiplier only pays below GROUND_HANDLING_HEIGHT
*and* while moving forward *and* facing the ball - a far narrower slice of
ticks than the penalty it was meant to replace. Net pressure fell and
upright_fraction fell with it (0.268 -> 0.239 -> 0.238, the lowest of any
round). Restore ground_tilt_penalty to 0.05 and change nothing else, so
this is a genuine single-variable test of multiplier plus full tilt
pressure.
The conjunctive mechanism itself held up: forward_motion_fraction reached
its best sustained value (0.242) without goal_rate sagging, ep_rew_mean
turned positive for the first time (+0.28), and eval win rate hit 49% with
no reward hacking.
Also adds grounded_upright_fraction: a diagnostic, deliberately ungated
metric measuring uprightness over real floor-contact ticks instead of
sub-3m ticks. upright_fraction has never exceeded 0.331 across four rounds
and ~560M steps without cheating, and its denominator is dominated by
ballistic transit (airborne_fraction ~0.45, mean_altitude ~4.4m) where
attitude is not meaningfully controllable - so it likely cannot measure
what the 0.45 floor was meant to capture. Re-baseline that floor from what
this reports rather than from another round of reshaping.
Rounds 2 and 3 showed that tuning grounded_upright_reward's magnitude only
slides along a tradeoff instead of resolving it: at 0.015 upright_fraction
climbed to 0.331 while goal_rate sagged to 0.542 (then farmed outright at
0.696/0.366), and at 0.004 goal_rate climbed 0.569->0.604 while
upright_fraction went flat at ~0.26. An additive uprightness bonus is an
alternative to playing well, so the policy just picks whichever is cheaper
and no magnitude buys both behaviours.
Change the mechanism rather than the number: grounded_upright_reward drops
to 0, and uprightness becomes a multiplier inside the nose-led approach
term, which already requires moving forward at the ball. Parked-and-upright
and fast-but-sideways now both pay zero; only upright, forward, nose-on to
the ball pays full. forward-velocity-to-ball rises 0.06 -> 0.15 to offset
the ~2-3x expected-value cut from the new factor, and ground-tilt-penalty
drops 0.05 -> 0.02 now that uprightness is paid positively during play.
Delete the three blocked attempts and reset state to restart from the
Stage-3 foundation.
grounded_upright_reward at 0.015 overshot: four force-retries pushed
upright_fraction from 0.265 to a plateauing 0.331, then the fifth jumped it
to 0.696 (55% over the 0.45 floor) while goal_rate collapsed 0.542->0.366
and forward_motion_fraction fell 0.244->0.184 (vertical_thrust_mean went
negative) - the policy learned to sit pinned upright and farm the bonus
instead of chasing the ball. It was sized "comparable to
time_penalty/ball_distance_penalty" but at 0.015/tick it was actually above
ball_distance_penalty's 0.01/tick worst case, so idling near the ball beat
playing. Cut to 0.004/tick (episode ceiling ~7.2, below
ball_distance_penalty's ~18 worst case). Delete the five blocked attempts
and reset generation5_state.json so the next run starts fresh from the
Stage-3 foundation rather than continuing from the farming checkpoint.
Stage 4's upright/forward-motion telemetry plateaued flat across all three
blocked attempts because ground_tilt_penalty (0.003) was too weak to matter
and nothing penalized sideways/reverse motion at all. Raise
ground_tilt_penalty to 0.05 and add a new non_forward_penalty term
(ship_ai_controller.gd) that directly costs non-forward planar velocity near
the floor, independent of the ball. Delete the three blocked attempts'
checkpoints/logs/exports and reset generation5_state.json so the next run
starts fresh from the Stage-3 foundation checkpoint instead of continuing
from the drifted retry2 weights.
All checkpoints/logs/exported policies here are a continuous-Gaussian,
31-input action/observation shape that generation 4's MultiDiscrete
redesign is structurally incompatible with -- nothing to resume from (see
the prior commit and TRAINING.md's "Generation 4" section). Game/bots/promoted/
(easy.json, and the new reference-grounded.json copied from
curric-s5-aggression before this) is untouched -- both remain valid,
playable evaluation opponents forever via PolicyNetwork's format-versioned
JSON despite their own checkpoints/generation being gone.
- training/checkpoints/*, training/logs/* removed (~3.5GB of working tree,
all generation 1-3 experiment runs).
- Game/bots/*.json flat dump removed (superseded exports; main_menu.gd's
Spectate dropdown will just be empty until the first generation-4 export).
- curriculum_state.json -> curriculum_state_gen3.json, archived alongside
the existing _gen1/_gen2 logs (all three are referenced as postmortem
evidence in TRAINING.md/curriculum.py). A fresh curriculum_state.json
will be created on the next curriculum.py run (load_state() already
handles a missing file).
training/eval_history.json is deliberately NOT reset -- it's the one
continuous cross-generation progress record.
NOT YET PUSHED: this needs the remote Linux training box quiesced first
(kill any active tmux session, confirm it's synced to origin) so its own
run_training.sh doesn't race a still-running job's final commit against
this deletion.
Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.
Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.
- Game/scripts/ship_action_codec.gd (new): single source of truth for a
per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
by training and in-game inference, replacing the continuous Gaussian.
thrust_y's bins are deliberately asymmetric so a random policy drifts
through the volume instead of floor-pinning. Legacy continuous decode
(ai_ship_controller.gd's old logic) preserved verbatim so every
pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
lowered 4x (aerial approaches require pitching); flight telemetry
(airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
high, ships low, kept clear of walls) so aerial practice is forced by the
environment instead of relying on reward-driven exploration alone; snapshot
terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
now-meaningless --reset-std; new EntropyFloorCallback (a persistent
per-rollout ent_coef controller replacing the one-shot std-reset shock)
and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
--tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
grounded stage, full action space live from step 1; deletes generation
1-3's checkpoint-lineage machinery (nothing to resume from); final report
evaluates against both promoted/easy.json and the new
promoted/reference-grounded.json (a copy of curric-s5-aggression, the
strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
intermediate checkpoint files a single stage was writing (~500MB ->
~0.2MB per run); requirements.txt pinned (behaviour here now depends on
specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
mismatch before it silently corrupts 24h of training.
Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.
See TRAINING.md's "Generation 4" section for the full design writeup.
Generation 2's first two real stage-1 attempts both independently restarted
from curric-s5-aggression (reset_retry_checkpoint) with identical flags and
landed at 32% and 27% win rate vs the reference -- a real regression either
way, but too much spread between "identical" runs for repeat fresh restarts
to be a controlled test of anything. The first attempt's own trajectory
(ep_rew_mean climbing from -10.86 toward ~0 by the 240M-step cutoff,
briefly touching positive) looked closer to convergence than the second's,
so retries now continue that attempt's own checkpoint for another full
timesteps budget instead of resetting to foundation again.
Drops retry1 and retry2 (checkpoints, logs, exported bots, eval_history
entries) -- retry2 never trained meaningfully before crashing on the
GoalRateCallback bug just fixed, and retry1 was the inferior of the two
real samples. curriculum_state.json rewinds to attempt 1, in_progress, so
the next run resumes 20260726-1904-curric-s1-unmask/final.zip directly.
Josh Creek