chore(training): add air_touch_bonus_weight and restart stage-5 intercepts

air_approach_weight alone didn't move productive_air_touch_fraction after a
further 180M steps (360M cumulative across all six Stage-5 attempts): an
unredirected air-intercept ball falls short of the goal from gravity and
just lands on the floor, so the already-solved ground game collects the
same episode reward whether or not anything touched the ball in the air.
air_touch_bonus_weight adds a conjunctive event bonus on top of
ball_touch_reward for a touch that's both genuinely aerial and
goal-directed, targeting the actual measured behaviour instead of only the
approach to it.
This commit is contained in:
Josh Creek
2026-08-19 22:46:04 +01:00
parent 03f49e59c8
commit 602fa297d0
15 changed files with 100 additions and 260 deletions
+30
View File
@@ -181,10 +181,40 @@ STANDING_ARGS = ["--ent-coef", "0.01", "--entropy-floor"]
# than continuing retry2's, same reasoning as every previous mechanism
# change in this file: don't resume a policy shaped by an absent term into
# one where it now exists.
#
# Round 8 (2026-08-19): air_approach_weight alone did not move the needle
# either -- another full 180M-step chain (3 more attempts, 360M cumulative
# across all six Stage-5 attempts) closed with productive_air_touch_fraction
# still exactly 0.0 and air_touch_fraction at noise level, while goal_rate
# kept passing its (lower) floor. Working out the physics instead of just
# re-tuning a number found why: an unredirected air-intercept ball (spawned
# 6-12m up, aimed at a goal whose collision box sits at ~0-1.5m) sags well
# short of the goal from gravity alone over the required flight distance --
# it does not auto-score -- so it simply falls to the floor, and the
# already-solved ground game (forward_velocity_to_ball_weight, ball_touch_
# reward, goal_reward) collects the exact same total episode reward either
# way. Nothing ever made touching the ball while it was still genuinely
# airborne worth more than waiting the second or two for it to land, so
# air_approach_weight's dense closing-speed shaping had nothing to reinforce
# -- nowhere near a training-duration problem, a second missing-incentive
# gap in the same stage.
#
# air_touch_bonus_weight (ship_ai_controller.gd) closes it directly: an
# event bonus on top of ball_touch_reward, paid only for a touch that is
# both above AIR_TOUCH_HEIGHT and goal-directed, scaled by the exact same
# alignment factor already gating the base touch reward -- conjunctive, not
# standalone, so it can't be farmed by batting the ball in a useless
# direction, and it targets exactly the behaviour productive_air_touch_
# fraction measures instead of only the approach to it. Set to 0.5 (roughly
# ball_touch_reward's own magnitude, so a fully-aligned aerial touch pays
# ~1.7x a fully-aligned ground one). Also folded into HANDLING_REWARD_FLAGS
# so Stage 6 inherits it. Restarts Stage 5 from Stage 4's checkpoint again,
# same reasoning as every prior mechanism change here.
HANDLING_REWARD_FLAGS = [
"--velocity-to-ball-weight", "0.04",
"--forward-velocity-to-ball-weight", "0.15",
"--air-approach-weight", "0.15",
"--air-touch-bonus-weight", "0.5",
"--ball-distance-penalty", "0.01",
"--ball-touch-reward", "0.7",
"--ball-velocity-to-goal-weight", "0.06",