air_approach_weight alone didn't move productive_air_touch_fraction after a
further 180M steps (360M cumulative across all six Stage-5 attempts): an
unredirected air-intercept ball falls short of the goal from gravity and
just lands on the floor, so the already-solved ground game collects the
same episode reward whether or not anything touched the ball in the air.
air_touch_bonus_weight adds a conjunctive event bonus on top of
ball_touch_reward for a touch that's both genuinely aerial and
goal-directed, targeting the actual measured behaviour instead of only the
approach to it.
Stage 5 blocked all three attempts on productive_air_touch_fraction
stuck exactly at 0.0 across a continuous 180M-step lineage, while
goal_rate/upright_fraction/forward_motion_fraction kept improving on
the same budget. forward_velocity_to_ball_weight (the term that solved
Stage 4's ground pursuit) is hard-gated below GROUND_HANDLING_HEIGHT
and does nothing in the air, so Stage 5's air_intercept_chance had no
matching aerial incentive to learn from. air_approach_weight adds the
airborne mirror (nose-first 3D closing speed, no uprightness
multiplier) and folds into HANDLING_REWARD_FLAGS so Stage 6 inherits
it too. Deleted the three blocked attempts and reset state to resume
Stage 5 from the Stage-4 checkpoint with the new term.
20260816-2126-gen5-s4-handling-retry2 exhausted its three attempts and
missed only the 0.80 training goal-rate floor, at 0.7731. Every
evaluation gate passed: 65-22-13 versus promoted/easy.json, 87% non-draw
against an 80% floor, 12.6% physical-side imbalance against a 20%
ceiling, and both handling telemetry floors clear. The round improved the
goal rate monotonically across attempts (0.537 -> 0.683 -> 0.773) and the
checkpoint plays well by hand, so close Stage 4 by human override.
Promote it to Game/bots/promoted/medium.json. Medium and Hard both point
at the new policy: Hard stays a label-only duplicate until a stronger one
earns hard.json, which keeps the tiers monotonic rather than leaving Hard
weaker than Medium.
generation5_state.json flips that log entry to "pass" with a
decision_override block preserving the original verdict and reasoning,
and advances to Stage 5 attempt 1. This is what passing_entry() needs to
resolve Stage 5's resume checkpoint and evaluation reference, and what
league_pool() will need at Stage 6; --skip-to-next-stage would advance
the stage without marking anything as passing and die immediately.
generation5.sh now pulls before launching. Each stage ends in
commit_progress()'s push, which fails and kills the run hours in if the
box is behind origin.
Six rounds of reward shaping (~700M steps) failed to produce upright ground
driving. A critical review of the simulation rather than the reward found
why:
1. The hull was a 1x1x4 box with inertia (1,1,1) and no restoring torque
anywhere, so belly-down and rolled-90 were geometrically identical
resting states. "Upright" was not a physically distinguished state at
all - the reward was paying for a property the simulation did not have.
2. ~65% of episodes spawned ships via _random_position, which samples Y
uniformly over the full 18m volume (mean ~8.7m). The measured
airborne_fraction ~0.44 was largely that spawn distribution, and every
ground-handling term fades out above 3m, so the shaping being tuned
barely ever applied.
3. air_drill_chance 0.20 spawned deliberately unreachable-without-climbing
states in the stage meant to teach ground driving, and its own
air_touch_fraction (0.0002) shows the drills were never solved.
Fixes land in the physics and the task distribution, not the reward:
- ship.tscn: hull 1x1x4 -> 1.6x0.6x4 so it has one stable resting face;
inertia (1,1,1) -> (7,1,7), physically correct for the hull, making
tumbling reluctant while keeping yaw snappy.
- ship.gd: new altitude-faded righting torque (spring-damper toward
belly-down, faded out by 3m so aerials keep full attitude freedom).
This is the grav-plating analogue of Rocket League's auto-righting and
helps human pilots land cleanly too.
- training_mode.gd: new ground_start_chance branch spawning ships level and
resting on the floor with a floor-level ball - the state the handling
stage's rewards are actually written for.
- generation5.py: ground-start-chance 0.50, air-drill-chance 0.20 -> 0.0.
Reward terms are left exactly as they were; they should finally pull in a
direction the ship can go.
Round 4 changed three things at once and two of them cut upright pressure:
grounded_upright_reward went to 0 and ground_tilt_penalty was cut 2.5x,
while the new uprightness multiplier only pays below GROUND_HANDLING_HEIGHT
*and* while moving forward *and* facing the ball - a far narrower slice of
ticks than the penalty it was meant to replace. Net pressure fell and
upright_fraction fell with it (0.268 -> 0.239 -> 0.238, the lowest of any
round). Restore ground_tilt_penalty to 0.05 and change nothing else, so
this is a genuine single-variable test of multiplier plus full tilt
pressure.
The conjunctive mechanism itself held up: forward_motion_fraction reached
its best sustained value (0.242) without goal_rate sagging, ep_rew_mean
turned positive for the first time (+0.28), and eval win rate hit 49% with
no reward hacking.
Also adds grounded_upright_fraction: a diagnostic, deliberately ungated
metric measuring uprightness over real floor-contact ticks instead of
sub-3m ticks. upright_fraction has never exceeded 0.331 across four rounds
and ~560M steps without cheating, and its denominator is dominated by
ballistic transit (airborne_fraction ~0.45, mean_altitude ~4.4m) where
attitude is not meaningfully controllable - so it likely cannot measure
what the 0.45 floor was meant to capture. Re-baseline that floor from what
this reports rather than from another round of reshaping.
Rounds 2 and 3 showed that tuning grounded_upright_reward's magnitude only
slides along a tradeoff instead of resolving it: at 0.015 upright_fraction
climbed to 0.331 while goal_rate sagged to 0.542 (then farmed outright at
0.696/0.366), and at 0.004 goal_rate climbed 0.569->0.604 while
upright_fraction went flat at ~0.26. An additive uprightness bonus is an
alternative to playing well, so the policy just picks whichever is cheaper
and no magnitude buys both behaviours.
Change the mechanism rather than the number: grounded_upright_reward drops
to 0, and uprightness becomes a multiplier inside the nose-led approach
term, which already requires moving forward at the ball. Parked-and-upright
and fast-but-sideways now both pay zero; only upright, forward, nose-on to
the ball pays full. forward-velocity-to-ball rises 0.06 -> 0.15 to offset
the ~2-3x expected-value cut from the new factor, and ground-tilt-penalty
drops 0.05 -> 0.02 now that uprightness is paid positively during play.
Delete the three blocked attempts and reset state to restart from the
Stage-3 foundation.
grounded_upright_reward at 0.015 overshot: four force-retries pushed
upright_fraction from 0.265 to a plateauing 0.331, then the fifth jumped it
to 0.696 (55% over the 0.45 floor) while goal_rate collapsed 0.542->0.366
and forward_motion_fraction fell 0.244->0.184 (vertical_thrust_mean went
negative) - the policy learned to sit pinned upright and farm the bonus
instead of chasing the ball. It was sized "comparable to
time_penalty/ball_distance_penalty" but at 0.015/tick it was actually above
ball_distance_penalty's 0.01/tick worst case, so idling near the ball beat
playing. Cut to 0.004/tick (episode ceiling ~7.2, below
ball_distance_penalty's ~18 worst case). Delete the five blocked attempts
and reset generation5_state.json so the next run starts fresh from the
Stage-3 foundation rather than continuing from the farming checkpoint.
Stage 4's upright/forward-motion telemetry plateaued flat across all three
blocked attempts because ground_tilt_penalty (0.003) was too weak to matter
and nothing penalized sideways/reverse motion at all. Raise
ground_tilt_penalty to 0.05 and add a new non_forward_penalty term
(ship_ai_controller.gd) that directly costs non-forward planar velocity near
the floor, independent of the ball. Delete the three blocked attempts'
checkpoints/logs/exports and reset generation5_state.json so the next run
starts fresh from the Stage-3 foundation checkpoint instead of continuing
from the drifted retry2 weights.
All checkpoints/logs/exported policies here are a continuous-Gaussian,
31-input action/observation shape that generation 4's MultiDiscrete
redesign is structurally incompatible with -- nothing to resume from (see
the prior commit and TRAINING.md's "Generation 4" section). Game/bots/promoted/
(easy.json, and the new reference-grounded.json copied from
curric-s5-aggression before this) is untouched -- both remain valid,
playable evaluation opponents forever via PolicyNetwork's format-versioned
JSON despite their own checkpoints/generation being gone.
- training/checkpoints/*, training/logs/* removed (~3.5GB of working tree,
all generation 1-3 experiment runs).
- Game/bots/*.json flat dump removed (superseded exports; main_menu.gd's
Spectate dropdown will just be empty until the first generation-4 export).
- curriculum_state.json -> curriculum_state_gen3.json, archived alongside
the existing _gen1/_gen2 logs (all three are referenced as postmortem
evidence in TRAINING.md/curriculum.py). A fresh curriculum_state.json
will be created on the next curriculum.py run (load_state() already
handles a missing file).
training/eval_history.json is deliberately NOT reset -- it's the one
continuous cross-generation progress record.
NOT YET PUSHED: this needs the remote Linux training box quiesced first
(kill any active tmux session, confirm it's synced to origin) so its own
run_training.sh doesn't race a still-running job's final commit against
this deletion.
Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.
Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.
- Game/scripts/ship_action_codec.gd (new): single source of truth for a
per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
by training and in-game inference, replacing the continuous Gaussian.
thrust_y's bins are deliberately asymmetric so a random policy drifts
through the volume instead of floor-pinning. Legacy continuous decode
(ai_ship_controller.gd's old logic) preserved verbatim so every
pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
lowered 4x (aerial approaches require pitching); flight telemetry
(airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
high, ships low, kept clear of walls) so aerial practice is forced by the
environment instead of relying on reward-driven exploration alone; snapshot
terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
now-meaningless --reset-std; new EntropyFloorCallback (a persistent
per-rollout ent_coef controller replacing the one-shot std-reset shock)
and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
--tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
grounded stage, full action space live from step 1; deletes generation
1-3's checkpoint-lineage machinery (nothing to resume from); final report
evaluates against both promoted/easy.json and the new
promoted/reference-grounded.json (a copy of curric-s5-aggression, the
strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
intermediate checkpoint files a single stage was writing (~500MB ->
~0.2MB per run); requirements.txt pinned (behaviour here now depends on
specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
mismatch before it silently corrupts 24h of training.
Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.
See TRAINING.md's "Generation 4" section for the full design writeup.
Generation 2's first two real stage-1 attempts both independently restarted
from curric-s5-aggression (reset_retry_checkpoint) with identical flags and
landed at 32% and 27% win rate vs the reference -- a real regression either
way, but too much spread between "identical" runs for repeat fresh restarts
to be a controlled test of anything. The first attempt's own trajectory
(ep_rew_mean climbing from -10.86 toward ~0 by the 240M-step cutoff,
briefly touching positive) looked closer to convergence than the second's,
so retries now continue that attempt's own checkpoint for another full
timesteps budget instead of resetting to foundation again.
Drops retry1 and retry2 (checkpoints, logs, exported bots, eval_history
entries) -- retry2 never trained meaningfully before crashing on the
GoalRateCallback bug just fixed, and retry1 was the inferior of the two
real samples. curriculum_state.json rewinds to attempt 1, in_progress, so
the next run resumes 20260726-1904-curric-s1-unmask/final.zip directly.
CosmicClash Training Bot