mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-14 02:32:04 +00:00
fix(training): make the stage-5 air-intercept drill physically solvable
productive_air_touch_fraction sat at exactly 0.0 across nine Stage-5 attempts and 540M timesteps. Two rounds of reward shaping were aimed at it (air_approach_weight, then air_touch_bonus_weight); both worked -- airborne_fraction 0.223->0.258, mean_altitude 2.59->3.25, vertical_thrust_mean 0.004->0.063 -- and the ship now visibly plays the ball in the air. The metric could not see it because it counts only touches with the ball above AIR_TOUCH_HEIGHT (5m), and _place_air_intercept never produced a reachable one. Simulating the spawn distribution against the ship's flight envelope (vertical_thrust 120 / mass 5 = 24 m/s^2 less gravity, drag capping climb near 12 m/s): a ball spawned 6-12m up at 6-11 m/s is above 5m for a median of 0.80s, while the ship spawned 7-13m behind, 3-10m below, and at a dead stop. An ideal interceptor -- point mass, instant attitude, no righting torque, zero reaction delay -- makes that touch in 0.00% of episodes and reaches the ball at all in 0.5%. Retune the drill instead of the reward: ball higher (8-14m) and slower (4-8 m/s), ship closer (4-9m behind), narrower lateral spread, and a 6-14 m/s planar run-up rather than a standing start -- the dead stop was the largest single factor. Ideal interceptor now reaches the ball in ~98% of episodes and above 5m in ~37%, so the 0.005 floor has headroom. AIR_TOUCH_HEIGHT stays 5.0 so the metric remains comparable with earlier generations. Resume from retry2 rather than restarting from Stage 4: that rule guards against a changed reward function invalidating the value function, and the reward function is untouched here -- only the state distribution moved, so the policy that already learned to fly is what should be pointed at a reachable target. Adds a one-shot resume_override to generation5_state.json, consumed on first use.
This commit is contained in:
+46
@@ -690,6 +690,52 @@ own magnitude) and folded into `HANDLING_REWARD_FLAGS`. The three blocked
|
||||
attempts were deleted and Stage 5 restarts from Stage 4's checkpoint again
|
||||
with both terms active.
|
||||
|
||||
**Neither reward term was the problem — the drill was unsolvable.** The
|
||||
third set of three attempts blocked on `productive_air_touch_fraction=0.0`
|
||||
yet again, but this time the surrounding telemetry told a different story
|
||||
from Rounds 7-8: the ship had measurably left the floor
|
||||
(`airborne_fraction` 0.223 → 0.258, `mean_altitude` 2.59 → 3.25,
|
||||
`vertical_thrust_mean` 0.004 → 0.063, `grounded_upright_fraction` 0.352 →
|
||||
0.182), and watching a game confirmed it now chases and strikes the ball in
|
||||
the air. The reward work had worked; the metric could not see it, because
|
||||
`productive_air_touch_fraction` counts only touches with the *ball* above
|
||||
`AIR_TOUCH_HEIGHT` (5 m), and `_place_air_intercept`'s spawn geometry never
|
||||
produced a reachable one.
|
||||
|
||||
Simulating that spawn distribution against the ship's real flight envelope
|
||||
(`vertical_thrust` 120 / `mass` 5 = 24 m/s², less 9.8 gravity, with
|
||||
`drag_coefficient` 0.98/tick capping climb near 12 m/s) settles it
|
||||
arithmetically. A ball spawned 6-12 m up moving 6-11 m/s is above 5 m for a
|
||||
median of **0.80 s**, while the ship spawned 7-13 m behind it, 3-10 m below
|
||||
it, and **at a dead stop**. An *ideal* interceptor — point mass, instant
|
||||
attitude, no righting torque, isotropic thrust, zero reaction delay — makes
|
||||
that touch in **0.00%** of episodes and reaches the ball at all before it
|
||||
lands in 0.5%. Six attempts and 360M steps were spent optimising against an
|
||||
event the environment could not produce.
|
||||
|
||||
The fix is in `_place_air_intercept` (see its `AIR_INTERCEPT_*` constants):
|
||||
ball higher and slower, ship closer and already carrying planar speed toward
|
||||
it. The dead-stop spawn was the single largest factor — a ship in real play
|
||||
is already moving, and starting from rest spent most of the window just
|
||||
building speed. The same simulation now puts an ideal interceptor at ~98%
|
||||
reach and ~37% above 5 m, so the 0.005 floor has real headroom.
|
||||
`AIR_TOUCH_HEIGHT` deliberately stays at 5.0: lowering the bar to meet a
|
||||
broken drill would make the metric incomparable with earlier generations.
|
||||
|
||||
Unlike every earlier round this does **not** restart from Stage 4's
|
||||
checkpoint. That rule exists because a changed reward function invalidates
|
||||
the learned value function; here the reward function is untouched and only
|
||||
the environment's state distribution moves, so the existing policy — which
|
||||
already learned to fly — is exactly what should be pointed at a now-reachable
|
||||
target. `generation5_state.json` carries a one-shot `resume_override` for
|
||||
this, consumed the first time `resume_checkpoint()` uses it.
|
||||
|
||||
The general lesson, and the one worth carrying into later generations: when
|
||||
a telemetry floor reads *exactly* zero while the behaviour it is meant to
|
||||
measure is visibly happening, check that the environment can produce the
|
||||
event at all before touching the reward function again. Cost of not checking
|
||||
here: three rounds and 360M timesteps.
|
||||
|
||||
Stage 6's `league` opponent mode samples a historical exported policy at each
|
||||
episode reset. Each later stage preserves the preceding shaping and adds one
|
||||
new difficulty.
|
||||
|
||||
Reference in New Issue
Block a user