fix(training): make the stage-5 air-intercept drill physically solvable

productive_air_touch_fraction sat at exactly 0.0 across nine Stage-5
attempts and 540M timesteps. Two rounds of reward shaping were aimed at
it (air_approach_weight, then air_touch_bonus_weight); both worked --
airborne_fraction 0.223->0.258, mean_altitude 2.59->3.25,
vertical_thrust_mean 0.004->0.063 -- and the ship now visibly plays the
ball in the air. The metric could not see it because it counts only
touches with the ball above AIR_TOUCH_HEIGHT (5m), and
_place_air_intercept never produced a reachable one.

Simulating the spawn distribution against the ship's flight envelope
(vertical_thrust 120 / mass 5 = 24 m/s^2 less gravity, drag capping
climb near 12 m/s): a ball spawned 6-12m up at 6-11 m/s is above 5m for
a median of 0.80s, while the ship spawned 7-13m behind, 3-10m below, and
at a dead stop. An ideal interceptor -- point mass, instant attitude, no
righting torque, zero reaction delay -- makes that touch in 0.00% of
episodes and reaches the ball at all in 0.5%.

Retune the drill instead of the reward: ball higher (8-14m) and slower
(4-8 m/s), ship closer (4-9m behind), narrower lateral spread, and a
6-14 m/s planar run-up rather than a standing start -- the dead stop was
the largest single factor. Ideal interceptor now reaches the ball in
~98% of episodes and above 5m in ~37%, so the 0.005 floor has headroom.
AIR_TOUCH_HEIGHT stays 5.0 so the metric remains comparable with earlier
generations.

Resume from retry2 rather than restarting from Stage 4: that rule guards
against a changed reward function invalidating the value function, and
the reward function is untouched here -- only the state distribution
moved, so the policy that already learned to fly is what should be
pointed at a reachable target. Adds a one-shot resume_override to
generation5_state.json, consumed on first use.
This commit is contained in:
Josh Creek
2026-08-21 15:14:36 +01:00
parent 23e3dd18f9
commit 818f8e89cd
4 changed files with 146 additions and 13 deletions
+46
View File
@@ -690,6 +690,52 @@ own magnitude) and folded into `HANDLING_REWARD_FLAGS`. The three blocked
attempts were deleted and Stage 5 restarts from Stage 4's checkpoint again
with both terms active.
**Neither reward term was the problem — the drill was unsolvable.** The
third set of three attempts blocked on `productive_air_touch_fraction=0.0`
yet again, but this time the surrounding telemetry told a different story
from Rounds 7-8: the ship had measurably left the floor
(`airborne_fraction` 0.223 → 0.258, `mean_altitude` 2.59 → 3.25,
`vertical_thrust_mean` 0.004 → 0.063, `grounded_upright_fraction` 0.352 →
0.182), and watching a game confirmed it now chases and strikes the ball in
the air. The reward work had worked; the metric could not see it, because
`productive_air_touch_fraction` counts only touches with the *ball* above
`AIR_TOUCH_HEIGHT` (5 m), and `_place_air_intercept`'s spawn geometry never
produced a reachable one.
Simulating that spawn distribution against the ship's real flight envelope
(`vertical_thrust` 120 / `mass` 5 = 24 m/s², less 9.8 gravity, with
`drag_coefficient` 0.98/tick capping climb near 12 m/s) settles it
arithmetically. A ball spawned 6-12 m up moving 6-11 m/s is above 5 m for a
median of **0.80 s**, while the ship spawned 7-13 m behind it, 3-10 m below
it, and **at a dead stop**. An *ideal* interceptor — point mass, instant
attitude, no righting torque, isotropic thrust, zero reaction delay — makes
that touch in **0.00%** of episodes and reaches the ball at all before it
lands in 0.5%. Six attempts and 360M steps were spent optimising against an
event the environment could not produce.
The fix is in `_place_air_intercept` (see its `AIR_INTERCEPT_*` constants):
ball higher and slower, ship closer and already carrying planar speed toward
it. The dead-stop spawn was the single largest factor — a ship in real play
is already moving, and starting from rest spent most of the window just
building speed. The same simulation now puts an ideal interceptor at ~98%
reach and ~37% above 5 m, so the 0.005 floor has real headroom.
`AIR_TOUCH_HEIGHT` deliberately stays at 5.0: lowering the bar to meet a
broken drill would make the metric incomparable with earlier generations.
Unlike every earlier round this does **not** restart from Stage 4's
checkpoint. That rule exists because a changed reward function invalidates
the learned value function; here the reward function is untouched and only
the environment's state distribution moves, so the existing policy — which
already learned to fly — is exactly what should be pointed at a now-reachable
target. `generation5_state.json` carries a one-shot `resume_override` for
this, consumed the first time `resume_checkpoint()` uses it.
The general lesson, and the one worth carrying into later generations: when
a telemetry floor reads *exactly* zero while the behaviour it is meant to
measure is visibly happening, check that the environment can produce the
event at all before touching the reward function again. Cost of not checking
here: three rounds and 360M timesteps.
Stage 6's `league` opponent mode samples a historical exported policy at each
episode reset. Each later stage preserves the preceding shaping and adds one
new difficulty.