feat(training): reopen stage 5 with a gate that can see the behaviour

Stage 5 blocked after nine attempts and ~540M steps, every one on
productive_air_touch_fraction. Instrumenting the environment rather than
retuning the reward again found three separate causes, none of which was the
policy's competence.

The gate could not register the behaviour. productive_air_touch_fraction
divides by TOTAL touches in the episode, so a strong ground game dilutes it for
identical aerial play. Stage 4's entire purpose is improving that ground game
(it took forward_motion_fraction 0.24 -> 0.48), so Stage 4's success drove
Stage 5's gate toward zero and the two stages were working against each other.
It also explains why every non-zero reading in the whole lineage came from
degenerate episodes whose single touch happened to be aerial: per-episode 1.0,
which is exactly 0.0100 once meaned over SB3's 100-episode buffer, and 0.0100
was every run's observed maximum. Replaced with
productive_air_touch_episode_fraction, which asks whether the episode contained
a productive aerial at all and cannot be diluted by ground play.

The bar was never derived from anything. AIR_TOUCH_HEIGHT was 5.0 and four
rounds of aerial mechanisms were built on top of it without anyone measuring
where the ball goes. New ball-altitude telemetry over normal match play: the
ball averages ~1.6m, the average episode's peak is ~2.4m, and it clears 5m for
~5% of ticks. Lowered to 3.0, this project's existing airborne threshold, with
_place_air_intercept's band retuned 8-14m -> 6-10m. Simulated against real
physics the pair strictly dominates the old one: 67.8% reach (was 53.2%), 57.3%
above-bar touches (was 41.2%), 5.2m of climb instead of 8.2m. The band could
not be lowered alone -- at a 5m bar, 8-14m was optimal and 5-8m collapses
above-bar touches to 4.3%. This reverses Round 9's explicit "AIR_TOUCH_HEIGHT
stays 5.0"; that objection was about comparability, and a metric that read 0.0
for nine attempts has no history to protect. Pre-2026-08-24 air-touch figures
are not comparable with later ones.

Note AIR_TOUCH_HEIGHT also gates air_touch_bonus_weight's payout, so unlike
Round 9 this DOES change the reward function and the usual "don't resume a
policy shaped by a different reward balance" rule is engaged rather than exempt.
Resuming retry2 anyway is justified on narrower grounds: the changed term has
never once fired (productive_air_touch_fraction exactly 0.0 across nine
attempts, air_touch_fraction at ~0.0003 noise), so no learned value estimate is
attached to it, while the ground handling and scoring retry2 does know are
untouched. The flip side is that at a 3m bar a fully-aligned aerial touch now
pays 0.7 + 0.5 = 1.2 against a ground touch's 0.7, which is the intended
incentive but is a live reward change -- if attempts show touch farming near 3m
rather than genuine intercepts, cut air_touch_bonus_weight rather than raising
the threshold back.

The policy could not climb, and the entropy controller could not see it. Its
target is a sum over heads, which read 21% of h_max -- on target -- while
thrust_y alone sat at 14% of its own ceiling. The measured consequence was a
policy commanding ~0.03 mean vertical thrust when hovering needs 0.408
(120/5 = 24 m/s^2 against 9.8 gravity), leaving it in free fall ~84% of every
episode. Added --min-head-entropy-frac so one starved head raises ent_coef
regardless of the aggregate, and --ent-coef-max because a probe pinned the old
0.05 ceiling for its entire duration with the head still starved.

A 200k-step probe from retry2 with all three in place moved air_touch_fraction
from 0/74 rollouts non-zero to 5/98, ent_coef 0.0102 -> 0.0416 and
vertical_thrust_mean 0.031 -> 0.089, with goal_rate, upright_fraction and
forward_motion_fraction all holding. The gate metric was still 0.0 at that
scale, so its 0.02 floor is marked provisional in generation5.py and should be
re-derived from attempt 1's tail rather than trusted.

Stage 5 expands to 90M timesteps and MAX_RETRIES 4, its goal_rate floor drops
0.75 -> 0.72 (every attempt landed 0.7217-0.7369 and was failed by ~2-4% while
winning its paired evaluations 54-25, 63-23 and 47-32), and state resumes from
20260823-1734-gen5-s5-intercepts-retry2 via resume_override.

Verified: generation5.py --dry-run resolves the resume to retry2 with the new
flags, 123 unit tests pass, probe artifacts removed.
This commit is contained in:
Josh Creek
2026-08-24 10:01:31 +01:00
parent 08eb9f5842
commit cb06300685
6 changed files with 291 additions and 17 deletions
+33
View File
@@ -519,6 +519,39 @@ continuous, otherwise identical, ~20M steps each) before committing to the
full curriculum — every past generation bet a full day on an unfalsifiable
hypothesis, which is what made each failure expensive to diagnose.
### Air-touch metrics: which to gate on (changed 2026-08-24)
**Gate on `productive_air_touch_episode_fraction`.** It is 1.0 for an episode
containing at least one productive aerial and 0.0 otherwise, so meaned over
SB3's 100-episode buffer it reads directly as "what share of episodes contained
one".
**Never gate on `air_touch_fraction` or `productive_air_touch_fraction` again.**
Both divide by *total touches in the episode*, which makes them structurally
unusable as bars: a policy with a strong ground game accumulates many ground
touches, and those dilute the ratio for identical aerial behaviour. Stage 4
exists to improve exactly that ground game, so its success actively drove Stage
5's gate toward zero — the two stages were fighting each other. It also means
the only non-zero values those metrics ever logged came from degenerate episodes
whose single touch happened to be aerial (per-episode 1.0, hence the exactly
`0.0100` that was every run's maximum). They are kept only as continuity with
nine attempts of history.
`AIR_TOUCH_HEIGHT` also moved 5.0 → 3.0 the same day, so **air-touch figures
recorded before 2026-08-24 are not comparable with anything after it.** 5.0 was
never derived from anything; 3.0 is this project's existing airborne threshold
(`AIRBORNE_ALTITUDE_THRESHOLD` / `GROUND_HANDLING_HEIGHT`) and sits just above
the measured ~2.4m mean episode peak ball height. `_place_air_intercept`'s band
moved 8-14m → 6-10m with it — the two are **coupled and must move together**,
since at a 5m bar the 8-14m band was optimal (41.2% above-bar touches) and
lowering the band alone collapses it to 4.3%.
Three ball-altitude diagnostics were added alongside and are deliberately
ungated: `ball_mean_altitude`, `ball_peak_altitude` (per-episode max — the
number a drill's spawn band should be derived from), and
`ball_above_air_touch_fraction`. Nobody had ever measured where the ball goes
before building four rounds of aerial mechanisms on top of an assumed height.
1. `training/test_action_space.py` — offline, seconds. Catches a head-order
mismatch, the single most likely silent killer (trains "fine" for 24h,
produces garbage — e.g. pitch commands driving strafe thrusters — with no