fix(training): make uprightness a multiplier on the forward-approach reward

Rounds 2 and 3 showed that tuning grounded_upright_reward's magnitude only
slides along a tradeoff instead of resolving it: at 0.015 upright_fraction
climbed to 0.331 while goal_rate sagged to 0.542 (then farmed outright at
0.696/0.366), and at 0.004 goal_rate climbed 0.569->0.604 while
upright_fraction went flat at ~0.26. An additive uprightness bonus is an
alternative to playing well, so the policy just picks whichever is cheaper
and no magnitude buys both behaviours.

Change the mechanism rather than the number: grounded_upright_reward drops
to 0, and uprightness becomes a multiplier inside the nose-led approach
term, which already requires moving forward at the ball. Parked-and-upright
and fast-but-sideways now both pay zero; only upright, forward, nose-on to
the ball pays full. forward-velocity-to-ball rises 0.06 -> 0.15 to offset
the ~2-3x expected-value cut from the new factor, and ground-tilt-penalty
drops 0.05 -> 0.02 now that uprightness is paid positively during play.
Delete the three blocked attempts and reset state to restart from the
Stage-3 foundation.
This commit is contained in:
Josh Creek
2026-08-12 13:04:20 +01:00
parent 5260124274
commit 4f3cf56e28
12 changed files with 62 additions and 267 deletions
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
+31 -4
View File
@@ -95,6 +95,18 @@ extends AIController3D
# the ball, cratering goal_rate. Keep this term's episode-long ceiling
# (value * ~1800 ticks) below ball_distance_penalty's worst-case episode
# cost, not just below ball_touch_reward/goal_reward.
#
# SUPERSEDED (2026-08-12), kept at 0 for older curricula that set it: the
# magnitude was never the real problem. Retuning it 0.015 -> 0.004 only
# moved along a tradeoff — at 0.015 upright_fraction climbed while
# goal_rate sagged, at 0.004 goal_rate climbed while upright_fraction went
# flat — because an *additive* uprightness reward is an alternative to
# playing well, so the policy just picks whichever is cheaper. Uprightness
# is now a multiplier inside the forward-approach term below instead, which
# makes it conjunctive with (not competing against) moving forward at the
# ball. Prefer that pattern for any future posture shaping; only reach for
# a standalone additive posture bonus if there is genuinely nothing to
# condition it on.
@export var grounded_upright_reward := 0.0
# Per-tick bonus for own speed: 0 stationary, full value (+0.24/s) at
# max_speed. Run07 lesson: after the kickoff flurry both ships parked next to
@@ -296,9 +308,22 @@ func _physics_process(delta):
var closing_speed := ship.linear_velocity.dot(to_ball.normalized())
reward += velocity_to_ball_weight * closing_speed / ship.max_speed
# Ground-handling shaping: forward planar motion while the nose faces the
# ball. It fades out with altitude so an aerial remains free to approach a
# ball using whatever body attitude is effective.
# Ground-handling shaping: upright, forward planar motion while the nose
# faces the ball. It fades out with altitude so an aerial remains free to
# approach a ball using whatever body attitude is effective.
#
# Uprightness is a *multiplier* here rather than a separate additive term,
# and that is the whole point. Stage 4's earlier rounds paid uprightness
# additively (grounded_upright_reward): because additive terms let a
# policy collect whichever one is cheapest, it could either play well
# (tilted, scoring) or sit parked upright (still, not scoring) — and it
# picked one or the other depending purely on that term's magnitude, so
# upright_fraction and goal_rate moved in opposite directions at every
# value tried. As a multiplier, uprightness pays only while the ship is
# also moving forward and nose-on to the ball, so no subset of the three
# behaviours can be farmed in isolation: parked pays zero (forward_speed
# is zero), on-its-side pays zero (uprightness is zero), and only doing
# all three at once pays full.
if forward_velocity_to_ball_weight > 0.0 and ship.global_position.y < GROUND_HANDLING_HEIGHT:
var planar_forward := Vector3(-ship.global_transform.basis.z.x, 0.0, -ship.global_transform.basis.z.z)
var planar_velocity := Vector3(ship.linear_velocity.x, 0.0, ship.linear_velocity.z)
@@ -307,8 +332,10 @@ func _physics_process(delta):
planar_forward = planar_forward.normalized()
var facing_ball: float = maxf(planar_forward.dot(planar_to_ball.normalized()), 0.0)
var forward_speed: float = maxf(planar_velocity.dot(planar_forward), 0.0) / ship.max_speed
var approach_uprightness: float = maxf(ship.global_transform.basis.y.dot(Vector3.UP), 0.0)
var handling_ground_factor: float = 1.0 - clampf(ship.global_position.y / GROUND_HANDLING_HEIGHT, 0.0, 1.0)
reward += forward_velocity_to_ball_weight * forward_speed * facing_ball * handling_ground_factor
reward += forward_velocity_to_ball_weight * forward_speed * facing_ball \
* approach_uprightness * handling_ground_factor
# Dense penalty: distance to the ball, so idling far away bleeds reward
# instead of scoring a safe zero (see ball_distance_penalty).