fix(training): make uprightness a multiplier on the forward-approach reward

Rounds 2 and 3 showed that tuning grounded_upright_reward's magnitude only
slides along a tradeoff instead of resolving it: at 0.015 upright_fraction
climbed to 0.331 while goal_rate sagged to 0.542 (then farmed outright at
0.696/0.366), and at 0.004 goal_rate climbed 0.569->0.604 while
upright_fraction went flat at ~0.26. An additive uprightness bonus is an
alternative to playing well, so the policy just picks whichever is cheaper
and no magnitude buys both behaviours.

Change the mechanism rather than the number: grounded_upright_reward drops
to 0, and uprightness becomes a multiplier inside the nose-led approach
term, which already requires moving forward at the ball. Parked-and-upright
and fast-but-sideways now both pay zero; only upright, forward, nose-on to
the ball pays full. forward-velocity-to-ball rises 0.06 -> 0.15 to offset
the ~2-3x expected-value cut from the new factor, and ground-tilt-penalty
drops 0.05 -> 0.02 now that uprightness is paid positively during play.
Delete the three blocked attempts and reset state to restart from the
Stage-3 foundation.
This commit is contained in:
Josh Creek
2026-08-12 13:04:20 +01:00
parent 5260124274
commit 4f3cf56e28
12 changed files with 62 additions and 267 deletions
+31 -4
View File
@@ -95,6 +95,18 @@ extends AIController3D
# the ball, cratering goal_rate. Keep this term's episode-long ceiling
# (value * ~1800 ticks) below ball_distance_penalty's worst-case episode
# cost, not just below ball_touch_reward/goal_reward.
#
# SUPERSEDED (2026-08-12), kept at 0 for older curricula that set it: the
# magnitude was never the real problem. Retuning it 0.015 -> 0.004 only
# moved along a tradeoff — at 0.015 upright_fraction climbed while
# goal_rate sagged, at 0.004 goal_rate climbed while upright_fraction went
# flat — because an *additive* uprightness reward is an alternative to
# playing well, so the policy just picks whichever is cheaper. Uprightness
# is now a multiplier inside the forward-approach term below instead, which
# makes it conjunctive with (not competing against) moving forward at the
# ball. Prefer that pattern for any future posture shaping; only reach for
# a standalone additive posture bonus if there is genuinely nothing to
# condition it on.
@export var grounded_upright_reward := 0.0
# Per-tick bonus for own speed: 0 stationary, full value (+0.24/s) at
# max_speed. Run07 lesson: after the kickoff flurry both ships parked next to
@@ -296,9 +308,22 @@ func _physics_process(delta):
var closing_speed := ship.linear_velocity.dot(to_ball.normalized())
reward += velocity_to_ball_weight * closing_speed / ship.max_speed
# Ground-handling shaping: forward planar motion while the nose faces the
# ball. It fades out with altitude so an aerial remains free to approach a
# ball using whatever body attitude is effective.
# Ground-handling shaping: upright, forward planar motion while the nose
# faces the ball. It fades out with altitude so an aerial remains free to
# approach a ball using whatever body attitude is effective.
#
# Uprightness is a *multiplier* here rather than a separate additive term,
# and that is the whole point. Stage 4's earlier rounds paid uprightness
# additively (grounded_upright_reward): because additive terms let a
# policy collect whichever one is cheapest, it could either play well
# (tilted, scoring) or sit parked upright (still, not scoring) — and it
# picked one or the other depending purely on that term's magnitude, so
# upright_fraction and goal_rate moved in opposite directions at every
# value tried. As a multiplier, uprightness pays only while the ship is
# also moving forward and nose-on to the ball, so no subset of the three
# behaviours can be farmed in isolation: parked pays zero (forward_speed
# is zero), on-its-side pays zero (uprightness is zero), and only doing
# all three at once pays full.
if forward_velocity_to_ball_weight > 0.0 and ship.global_position.y < GROUND_HANDLING_HEIGHT:
var planar_forward := Vector3(-ship.global_transform.basis.z.x, 0.0, -ship.global_transform.basis.z.z)
var planar_velocity := Vector3(ship.linear_velocity.x, 0.0, ship.linear_velocity.z)
@@ -307,8 +332,10 @@ func _physics_process(delta):
planar_forward = planar_forward.normalized()
var facing_ball: float = maxf(planar_forward.dot(planar_to_ball.normalized()), 0.0)
var forward_speed: float = maxf(planar_velocity.dot(planar_forward), 0.0) / ship.max_speed
var approach_uprightness: float = maxf(ship.global_transform.basis.y.dot(Vector3.UP), 0.0)
var handling_ground_factor: float = 1.0 - clampf(ship.global_position.y / GROUND_HANDLING_HEIGHT, 0.0, 1.0)
reward += forward_velocity_to_ball_weight * forward_speed * facing_ball * handling_ground_factor
reward += forward_velocity_to_ball_weight * forward_speed * facing_ball \
* approach_uprightness * handling_ground_factor
# Dense penalty: distance to the ball, so idling far away bleeds reward
# instead of scoring a safe zero (see ball_distance_penalty).