chore(training): add air_touch_bonus_weight and restart stage-5 intercepts

air_approach_weight alone didn't move productive_air_touch_fraction after a
further 180M steps (360M cumulative across all six Stage-5 attempts): an
unredirected air-intercept ball falls short of the goal from gravity and
just lands on the floor, so the already-solved ground game collects the
same episode reward whether or not anything touched the ball in the air.
air_touch_bonus_weight adds a conjunctive event bonus on top of
ball_touch_reward for a touch that's both genuinely aerial and
goal-directed, targeting the actual measured behaviour instead of only the
approach to it.
This commit is contained in:
Josh Creek
2026-08-19 22:46:04 +01:00
parent 03f49e59c8
commit 602fa297d0
15 changed files with 100 additions and 260 deletions
+28
View File
@@ -662,6 +662,34 @@ were deleted and Stage 5 restarts from Stage 4's checkpoint with the new term,
same reasoning as every previous mechanism change: don't resume a policy
shaped by an absent term into one where it now exists.
**`air_approach_weight` alone did not fix it.** A further three attempts
(180M more steps, 360M cumulative across all six Stage-5 attempts, 2026-08-18
to 2026-08-19) closed with `productive_air_touch_fraction` still exactly 0.0
and `air_touch_fraction` still at noise level, while `goal_rate` kept clearing
its (lower) floor. Working out the actual physics instead of retuning another
number found the real gap: an unredirected `_place_air_intercept` ball
(spawned 6-12m up, aimed at a goal whose collision box sits at ~0-1.5m) sags
well short of the goal from gravity alone over the required flight
distance — it does not auto-score — so it just falls to the floor, where the
already-solved ground game collects the exact same `goal_reward`/
`ball_touch_reward` regardless of whether anything touched it in the air.
Nothing in the reward ever made a genuinely aerial touch worth more than
waiting the second or two for the ball to land, so `air_approach_weight`'s
dense closing-speed shaping had nothing to reinforce — a second,
independent missing-incentive gap in the same stage, not a training-duration
problem. `air_touch_bonus_weight` (`ship_ai_controller.gd`) closes it
directly: an event bonus on top of `ball_touch_reward`, paid only for a
touch that is both above `AIR_TOUCH_HEIGHT` and goal-directed, scaled by the
exact same alignment factor already gating the base touch reward —
conjunctive, not standalone, so it targets exactly the behaviour
`productive_air_touch_fraction` measures without reopening the RLGym
wall-bounce exploit the original "no standalone air-touch reward" decision
(above) was written to avoid: air-intercept/air-drill spawns are kept away
from every wall by construction. Set to 0.5 (roughly `ball_touch_reward`'s
own magnitude) and folded into `HANDLING_REWARD_FLAGS`. The three blocked
attempts were deleted and Stage 5 restarts from Stage 4's checkpoint again
with both terms active.
Stage 6's `league` opponent mode samples a historical exported policy at each
episode reset. Each later stage preserves the preceding shaping and adds one
new difficulty.