Stage 5 blocked after nine attempts and ~540M steps, every one on
productive_air_touch_fraction. Instrumenting the environment rather than
retuning the reward again found three separate causes, none of which was the
policy's competence.
The gate could not register the behaviour. productive_air_touch_fraction
divides by TOTAL touches in the episode, so a strong ground game dilutes it for
identical aerial play. Stage 4's entire purpose is improving that ground game
(it took forward_motion_fraction 0.24 -> 0.48), so Stage 4's success drove
Stage 5's gate toward zero and the two stages were working against each other.
It also explains why every non-zero reading in the whole lineage came from
degenerate episodes whose single touch happened to be aerial: per-episode 1.0,
which is exactly 0.0100 once meaned over SB3's 100-episode buffer, and 0.0100
was every run's observed maximum. Replaced with
productive_air_touch_episode_fraction, which asks whether the episode contained
a productive aerial at all and cannot be diluted by ground play.
The bar was never derived from anything. AIR_TOUCH_HEIGHT was 5.0 and four
rounds of aerial mechanisms were built on top of it without anyone measuring
where the ball goes. New ball-altitude telemetry over normal match play: the
ball averages ~1.6m, the average episode's peak is ~2.4m, and it clears 5m for
~5% of ticks. Lowered to 3.0, this project's existing airborne threshold, with
_place_air_intercept's band retuned 8-14m -> 6-10m. Simulated against real
physics the pair strictly dominates the old one: 67.8% reach (was 53.2%), 57.3%
above-bar touches (was 41.2%), 5.2m of climb instead of 8.2m. The band could
not be lowered alone -- at a 5m bar, 8-14m was optimal and 5-8m collapses
above-bar touches to 4.3%. This reverses Round 9's explicit "AIR_TOUCH_HEIGHT
stays 5.0"; that objection was about comparability, and a metric that read 0.0
for nine attempts has no history to protect. Pre-2026-08-24 air-touch figures
are not comparable with later ones.
Note AIR_TOUCH_HEIGHT also gates air_touch_bonus_weight's payout, so unlike
Round 9 this DOES change the reward function and the usual "don't resume a
policy shaped by a different reward balance" rule is engaged rather than exempt.
Resuming retry2 anyway is justified on narrower grounds: the changed term has
never once fired (productive_air_touch_fraction exactly 0.0 across nine
attempts, air_touch_fraction at ~0.0003 noise), so no learned value estimate is
attached to it, while the ground handling and scoring retry2 does know are
untouched. The flip side is that at a 3m bar a fully-aligned aerial touch now
pays 0.7 + 0.5 = 1.2 against a ground touch's 0.7, which is the intended
incentive but is a live reward change -- if attempts show touch farming near 3m
rather than genuine intercepts, cut air_touch_bonus_weight rather than raising
the threshold back.
The policy could not climb, and the entropy controller could not see it. Its
target is a sum over heads, which read 21% of h_max -- on target -- while
thrust_y alone sat at 14% of its own ceiling. The measured consequence was a
policy commanding ~0.03 mean vertical thrust when hovering needs 0.408
(120/5 = 24 m/s^2 against 9.8 gravity), leaving it in free fall ~84% of every
episode. Added --min-head-entropy-frac so one starved head raises ent_coef
regardless of the aggregate, and --ent-coef-max because a probe pinned the old
0.05 ceiling for its entire duration with the head still starved.
A 200k-step probe from retry2 with all three in place moved air_touch_fraction
from 0/74 rollouts non-zero to 5/98, ent_coef 0.0102 -> 0.0416 and
vertical_thrust_mean 0.031 -> 0.089, with goal_rate, upright_fraction and
forward_motion_fraction all holding. The gate metric was still 0.0 at that
scale, so its 0.02 floor is marked provisional in generation5.py and should be
re-derived from attempt 1's tail rather than trusted.
Stage 5 expands to 90M timesteps and MAX_RETRIES 4, its goal_rate floor drops
0.75 -> 0.72 (every attempt landed 0.7217-0.7369 and was failed by ~2-4% while
winning its paired evaluations 54-25, 63-23 and 47-32), and state resumes from
20260823-1734-gen5-s5-intercepts-retry2 via resume_override.
Verified: generation5.py --dry-run resolves the resume to retry2 with the new
flags, 123 unit tests pass, probe artifacts removed.
productive_air_touch_fraction sat at exactly 0.0 across nine Stage-5
attempts and 540M timesteps. Two rounds of reward shaping were aimed at
it (air_approach_weight, then air_touch_bonus_weight); both worked --
airborne_fraction 0.223->0.258, mean_altitude 2.59->3.25,
vertical_thrust_mean 0.004->0.063 -- and the ship now visibly plays the
ball in the air. The metric could not see it because it counts only
touches with the ball above AIR_TOUCH_HEIGHT (5m), and
_place_air_intercept never produced a reachable one.
Simulating the spawn distribution against the ship's flight envelope
(vertical_thrust 120 / mass 5 = 24 m/s^2 less gravity, drag capping
climb near 12 m/s): a ball spawned 6-12m up at 6-11 m/s is above 5m for
a median of 0.80s, while the ship spawned 7-13m behind, 3-10m below, and
at a dead stop. An ideal interceptor -- point mass, instant attitude, no
righting torque, zero reaction delay -- makes that touch in 0.00% of
episodes and reaches the ball at all in 0.5%.
Retune the drill instead of the reward: ball higher (8-14m) and slower
(4-8 m/s), ship closer (4-9m behind), narrower lateral spread, and a
6-14 m/s planar run-up rather than a standing start -- the dead stop was
the largest single factor. Ideal interceptor now reaches the ball in
~98% of episodes and above 5m in ~37%, so the 0.005 floor has headroom.
AIR_TOUCH_HEIGHT stays 5.0 so the metric remains comparable with earlier
generations.
Resume from retry2 rather than restarting from Stage 4: that rule guards
against a changed reward function invalidating the value function, and
the reward function is untouched here -- only the state distribution
moved, so the policy that already learned to fly is what should be
pointed at a reachable target. Adds a one-shot resume_override to
generation5_state.json, consumed on first use.
air_approach_weight alone didn't move productive_air_touch_fraction after a
further 180M steps (360M cumulative across all six Stage-5 attempts): an
unredirected air-intercept ball falls short of the goal from gravity and
just lands on the floor, so the already-solved ground game collects the
same episode reward whether or not anything touched the ball in the air.
air_touch_bonus_weight adds a conjunctive event bonus on top of
ball_touch_reward for a touch that's both genuinely aerial and
goal-directed, targeting the actual measured behaviour instead of only the
approach to it.
Lands the non-networked Phase 0 tasks from multiplayer-todo.md (ship/camera/
arena refactors, sim constants, background FPS handling) plus a first pass
at exposing graphics/performance settings (presets, resolution scaling,
vsync, FPS cap, perf overlay) and a GPU profiling harness for the
real-hardware follow-up in task 0.15b.
Stage 5 blocked all three attempts on productive_air_touch_fraction
stuck exactly at 0.0 across a continuous 180M-step lineage, while
goal_rate/upright_fraction/forward_motion_fraction kept improving on
the same budget. forward_velocity_to_ball_weight (the term that solved
Stage 4's ground pursuit) is hard-gated below GROUND_HANDLING_HEIGHT
and does nothing in the air, so Stage 5's air_intercept_chance had no
matching aerial incentive to learn from. air_approach_weight adds the
airborne mirror (nose-first 3D closing speed, no uprightness
multiplier) and folds into HANDLING_REWARD_FLAGS so Stage 6 inherits
it too. Deleted the three blocked attempts and reset state to resume
Stage 5 from the Stage-4 checkpoint with the new term.
Six rounds of reward shaping (~700M steps) failed to produce upright ground
driving. A critical review of the simulation rather than the reward found
why:
1. The hull was a 1x1x4 box with inertia (1,1,1) and no restoring torque
anywhere, so belly-down and rolled-90 were geometrically identical
resting states. "Upright" was not a physically distinguished state at
all - the reward was paying for a property the simulation did not have.
2. ~65% of episodes spawned ships via _random_position, which samples Y
uniformly over the full 18m volume (mean ~8.7m). The measured
airborne_fraction ~0.44 was largely that spawn distribution, and every
ground-handling term fades out above 3m, so the shaping being tuned
barely ever applied.
3. air_drill_chance 0.20 spawned deliberately unreachable-without-climbing
states in the stage meant to teach ground driving, and its own
air_touch_fraction (0.0002) shows the drills were never solved.
Fixes land in the physics and the task distribution, not the reward:
- ship.tscn: hull 1x1x4 -> 1.6x0.6x4 so it has one stable resting face;
inertia (1,1,1) -> (7,1,7), physically correct for the hull, making
tumbling reluctant while keeping yaw snappy.
- ship.gd: new altitude-faded righting torque (spring-damper toward
belly-down, faded out by 3m so aerials keep full attitude freedom).
This is the grav-plating analogue of Rocket League's auto-righting and
helps human pilots land cleanly too.
- training_mode.gd: new ground_start_chance branch spawning ships level and
resting on the floor with a floor-level ball - the state the handling
stage's rewards are actually written for.
- generation5.py: ground-start-chance 0.50, air-drill-chance 0.20 -> 0.0.
Reward terms are left exactly as they were; they should finally pull in a
direction the ship can go.
Adversarial review of the previous stage-4 retune found two problems:
non_forward_speed used planar_speed - forward_component, which under-charges
diagonal motion relative to true lateral speed (e.g. ~29% penalty at 45
degrees off the nose instead of the correct ~71%); fixed to the Pythagorean
magnitude for forward-facing angles, full speed for backward-facing ones.
Also, ground_tilt_penalty and non_forward_penalty only ever cost reward near
the floor with nothing offsetting them above it, which could teach a policy
that's still bad at ground handling to just avoid the floor rather than get
better at it. Added grounded_upright_reward (ship_ai_controller.gd) plus a
new ShipObservations.is_floor_contact helper for genuine belly-on-floor
contact detection, so grounding well while upright is the locally profitable
choice, not just the least-punished one.
Stage 4's upright/forward-motion telemetry plateaued flat across all three
blocked attempts because ground_tilt_penalty (0.003) was too weak to matter
and nothing penalized sideways/reverse motion at all. Raise
ground_tilt_penalty to 0.05 and add a new non_forward_penalty term
(ship_ai_controller.gd) that directly costs non-forward planar velocity near
the floor, independent of the ball. Delete the three blocked attempts'
checkpoints/logs/exports and reset generation5_state.json so the next run
starts fresh from the Stage-3 foundation checkpoint instead of continuing
from the drifted retry2 weights.
Extends ShipObservations beyond the old self+1-opponent layout to padded
teammate/opponent arrays (MAX_TEAMMATES=4, MAX_OPPONENTS=5, SIZE=83),
zero-filling slots past the real roster size the same way the old single-
opponent slot was zero-filled when absent.
Slot stability across ticks requires a persistent identity: Ship gains
spawn_index (set once by GameMode.spawn_ship, never reassigned — there's no
despawn path anywhere in this codebase, so a roster is fixed for the whole
episode/match). ai_ship_controller.gd's opponent discovery is rewritten from
"first non-self ship" to classify every other ship by team and sort by
spawn_index; training_mode.gd/ship_ai_controller.gd carry the equivalent
sorted lists through the training path so both agree on slot assignment for
the same roster.
training_mode.gd and match_mode.gd both gain a team_size export (default 1,
so every existing curriculum script and match keeps today's 1v1 behaviour
unchanged). This is plumbing only: no 2v2+ curriculum or reward design, and
no match-mode UI to pick team size, has been done yet. The two checkpoints
in Game/bots/promoted/ are fitted to the old 35-float layout and are not
migrated — expected to go stale until the next training run.
_place_ships_random/_place_air_drill sampled each ship's randomized episode-
start position independently, so a team_size > 1 roster could spawn
interpenetrating (ships are ~1x1x4). Both now resample (up to 20 attempts,
matching the existing corner/fillet rejection-sampling pattern) against
every ship already placed that reset, rejecting anything within
MIN_SHIP_SEPARATION (4.5m, matching the arena spawn-marker spacing) of one.
The goal mouths are now a real navigable hole in the end walls, sized to the
ball rather than the ship — a ship's 1x1 cross-section fits through it, and
there's nothing behind the net to stop it. The escape failsafe previously
only existed in TrainingMode (where a physics regression just wastes
training time); now that any ship can genuinely fly out through an open
goal, every mode needs it or a stray ship/ball falls into the void with no
way back short of quitting. Moved up to GameMode as the shared default
_physics_process, removing TrainingMode's now-duplicate copy.
Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.
Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.
- Game/scripts/ship_action_codec.gd (new): single source of truth for a
per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
by training and in-game inference, replacing the continuous Gaussian.
thrust_y's bins are deliberately asymmetric so a random policy drifts
through the volume instead of floor-pinning. Legacy continuous decode
(ai_ship_controller.gd's old logic) preserved verbatim so every
pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
lowered 4x (aerial approaches require pitching); flight telemetry
(airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
high, ships low, kept clear of walls) so aerial practice is forced by the
environment instead of relying on reward-driven exploration alone; snapshot
terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
now-meaningless --reset-std; new EntropyFloorCallback (a persistent
per-rollout ent_coef controller replacing the one-shot std-reset shock)
and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
--tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
grounded stage, full action space live from step 1; deletes generation
1-3's checkpoint-lineage machinery (nothing to resume from); final report
evaluates against both promoted/easy.json and the new
promoted/reference-grounded.json (a copy of curric-s5-aggression, the
strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
intermediate checkpoint files a single stage was writing (~500MB ->
~0.2MB per run); requirements.txt pinned (behaviour here now depends on
specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
mismatch before it silently corrupts 24h of training.
Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.
See TRAINING.md's "Generation 4" section for the full design writeup.
Each of the three arenas gains an ELEVATED sibling scene that inherits the
base arena and overrides the boundary's goal_mode, the two goal transforms
and the ship spawns, so the goal sits at mid-wall height instead of flush
with the deck. Registered in ArenaRegistry alongside the floor-level arenas,
plus a training_elevated scene for self-play on them.
TrainingMode reads the arena's goal_mode once in _start() and widens the
ball-placement height range to match the goal's real position; on FLOOR
arenas the bound is a no-op, so floor-level training is unchanged. It is read
in _start() rather than _ready() because TrainingMode has no _ready()
override and GameMode._ready() is what discovers the arena first.
Policies trained against floor-level goals are not expected to score on an
elevated one, so the two are kept as separate arenas rather than a variant of
the same entry.
Generation 2's single "unmask" stage (flip vertical/pitch-roll locomotion
from grounded-only to full 3D in one step) failed 3 independent 240M-step
attempts, landing at a stable 32% / 28% / 31% win rate vs curric-s5-aggression
each time -- not noise, and not fixable by more training time (attempts 2-3
each continued the same checkpoint lineage for another full 240M steps with
zero improvement). Every attempt shows train/std collapsing from ~0.30 to
~0.13-0.15 within the first ~10% of steps and never recovering: the policy
locks the newly-opened axes back down before ever meaningfully exploring
them.
Replaces the boolean allow_vertical/allow_pitch_roll mask on ShipAIController
with float vertical_ramp/pitch_roll_ramp multipliers (0.0-1.0), scaling axis
effect in set_action() instead of gating it outright -- the action space
never changes shape, so checkpoints stay resumable across ramp values. The
single unmask stage in curriculum.py becomes 4: three ungated warmup stages
(25%/50%/75% authority, airborne_penalty ramping in step) that train,
checkpoint, and always advance with no eval gate, then the measured stage at
full authority -- same reference, opponent mode, and 240M budget as the 3
failed attempts, for a direct comparison. Adds a "gated" flag/branch to
main()'s loop for the ungated stages.
This is generation 3 of the curriculum; generation 2's state is archived to
curriculum_state_gen2.json (mirroring the earlier gen1 -> gen2 archival) and
curriculum_state.json resets fresh, since its stage 0 no longer means what it
used to. See TRAINING.md's "Generation 3" section for the full postmortem,
stage table, and the open question about whether scaling action effect in
Godot (which PPO's own entropy/exploration math never sees) actually
addresses the collapse.
Stage 5 (aggression) passed (41-47 vs grounded curric-s2-defend, within
the lenient gate but not yet a clear win). Rather than keep the locomotion
mask on indefinitely, stage 6 reopens full 3D controls on top of the
aggression retune and pairs it with a new dense airborne_penalty (scaled
by height above the floor) so the policy learns to prefer staying grounded
through incentives instead of a hard mask — same regime shift that
regressed stage 3, but this time with a mitigation and ~12x the training
time (~240M timesteps / ~24h vs ~20M / ~2h) to actually re-converge
instead of stalling mid-shift.
airborne_penalty follows the existing SHIP_AI_OVERRIDES pattern: default
0 (off) on ship_ai_controller.gd, exposed via train.py's new
--airborne-penalty flag, added to training_mode.gd's allow-list. Also adds
a per-stage timesteps override in curriculum.py (STAGES[n]["timesteps"])
since this is the first stage to need a different budget than the rest.
AIShipController (eval + real gameplay) ran the raw policy output unmasked
regardless of allow_vertical/allow_pitch_roll, while ShipAIController
(training) correctly discarded those axes for grounded curriculum stages.
A grounded-trained model's untrained vertical/pitch-roll output reached the
ship as noise during eval, understating it against models that were never
handicapped this way.
Josh Creek