jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-11 19:03:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(training): reopen stage 5 with a gate that can see the behaviour

Browse Source 
Stage 5 blocked after nine attempts and ~540M steps, every one on
productive_air_touch_fraction. Instrumenting the environment rather than
retuning the reward again found three separate causes, none of which was the
policy's competence.

The gate could not register the behaviour. productive_air_touch_fraction
divides by TOTAL touches in the episode, so a strong ground game dilutes it for
identical aerial play. Stage 4's entire purpose is improving that ground game
(it took forward_motion_fraction 0.24 -> 0.48), so Stage 4's success drove
Stage 5's gate toward zero and the two stages were working against each other.
It also explains why every non-zero reading in the whole lineage came from
degenerate episodes whose single touch happened to be aerial: per-episode 1.0,
which is exactly 0.0100 once meaned over SB3's 100-episode buffer, and 0.0100
was every run's observed maximum. Replaced with
productive_air_touch_episode_fraction, which asks whether the episode contained
a productive aerial at all and cannot be diluted by ground play.

The bar was never derived from anything. AIR_TOUCH_HEIGHT was 5.0 and four
rounds of aerial mechanisms were built on top of it without anyone measuring
where the ball goes. New ball-altitude telemetry over normal match play: the
ball averages ~1.6m, the average episode's peak is ~2.4m, and it clears 5m for
~5% of ticks. Lowered to 3.0, this project's existing airborne threshold, with
_place_air_intercept's band retuned 8-14m -> 6-10m. Simulated against real
physics the pair strictly dominates the old one: 67.8% reach (was 53.2%), 57.3%
above-bar touches (was 41.2%), 5.2m of climb instead of 8.2m. The band could
not be lowered alone -- at a 5m bar, 8-14m was optimal and 5-8m collapses
above-bar touches to 4.3%. This reverses Round 9's explicit "AIR_TOUCH_HEIGHT
stays 5.0"; that objection was about comparability, and a metric that read 0.0
for nine attempts has no history to protect. Pre-2026-08-24 air-touch figures
are not comparable with later ones.

Note AIR_TOUCH_HEIGHT also gates air_touch_bonus_weight's payout, so unlike
Round 9 this DOES change the reward function and the usual "don't resume a
policy shaped by a different reward balance" rule is engaged rather than exempt.
Resuming retry2 anyway is justified on narrower grounds: the changed term has
never once fired (productive_air_touch_fraction exactly 0.0 across nine
attempts, air_touch_fraction at ~0.0003 noise), so no learned value estimate is
attached to it, while the ground handling and scoring retry2 does know are
untouched. The flip side is that at a 3m bar a fully-aligned aerial touch now
pays 0.7 + 0.5 = 1.2 against a ground touch's 0.7, which is the intended
incentive but is a live reward change -- if attempts show touch farming near 3m
rather than genuine intercepts, cut air_touch_bonus_weight rather than raising
the threshold back.

The policy could not climb, and the entropy controller could not see it. Its
target is a sum over heads, which read 21% of h_max -- on target -- while
thrust_y alone sat at 14% of its own ceiling. The measured consequence was a
policy commanding ~0.03 mean vertical thrust when hovering needs 0.408
(120/5 = 24 m/s^2 against 9.8 gravity), leaving it in free fall ~84% of every
episode. Added --min-head-entropy-frac so one starved head raises ent_coef
regardless of the aggregate, and --ent-coef-max because a probe pinned the old
0.05 ceiling for its entire duration with the head still starved.

A 200k-step probe from retry2 with all three in place moved air_touch_fraction
from 0/74 rollouts non-zero to 5/98, ent_coef 0.0102 -> 0.0416 and
vertical_thrust_mean 0.031 -> 0.089, with goal_rate, upright_fraction and
forward_motion_fraction all holding. The gate metric was still 0.0 at that
scale, so its 0.02 floor is marked provisional in generation5.py and should be
re-derived from attempt 1's tail rather than trusted.

Stage 5 expands to 90M timesteps and MAX_RETRIES 4, its goal_rate floor drops
0.75 -> 0.72 (every attempt landed 0.7217-0.7369 and was failed by ~2-4% while
winning its paired evaluations 54-25, 63-23 and 47-32), and state resumes from
20260823-1734-gen5-s5-intercepts-retry2 via resume_override.

Verified: generation5.py --dry-run resolves the resume to retry2 with the new
flags, 123 unit tests pass, probe artifacts removed.
This commit is contained in:
Josh Creek
2026-08-24 10:01:31 +01:00
parent 08eb9f5842
commit cb06300685
6 changed files with 291 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
training/train.py
+56 -2
View File
@@ -12,6 +12,7 @@ See TRAINING.md at the repo root for the full workflow.
"""
import argparse
import math
import os
import pathlib
@@ -84,6 +85,10 @@ class FlightTelemetryCallback(BaseCallback):
"air_touch_fraction",
"vertical_thrust_mean",
"productive_air_touch_fraction",
"productive_air_touch_episode_fraction",
"ball_above_air_touch_fraction",
"ball_mean_altitude",
"ball_peak_altitude",
"upright_fraction",
"forward_motion_fraction",
"grounded_upright_fraction",
@@ -135,6 +140,7 @@ class EntropyFloorCallback(BaseCallback):
target_end_frac: float = 0.20,
adjust_rate: float = 1.02,
ent_coef_bounds: tuple[float, float] = (1e-4, 0.05),
min_head_frac: float = 0.0,
):
super().__init__()
self.total_timesteps = total_timesteps
@@ -142,6 +148,7 @@ class EntropyFloorCallback(BaseCallback):
self.target_end_frac = target_end_frac
self.adjust_rate = adjust_rate
self.ent_coef_bounds = ent_coef_bounds
self.min_head_frac = min_head_frac
self._is_multi_discrete = False
self._h_max = 0.0
self._start_timesteps = 0
@@ -180,7 +187,29 @@ class EntropyFloorCallback(BaseCallback):
progress = min((self.model.num_timesteps - self._start_timesteps) / self.total_timesteps, 1.0)
target_frac = self.target_start_frac + (self.target_end_frac - self.target_start_frac) * progress
target = target_frac * self._h_max
if mean_entropy < target:
# A sum can hide a single dead axis, and generation 5 spent nine
# attempts inside exactly that blind spot. Stage 5's checkpoints sat at
# a head-entropy sum of ~2.20 against h_max 10.35 — 21%, i.e. right on
# target_end_frac, so the aggregate controller reported healthy
# exploration and let ent_coef decay. Meanwhile thrust_y alone was at
# 0.226 against its own ln(5)=1.609 ceiling (14%), and the measured
# consequence was a policy commanding ~0.03 mean vertical thrust when
# merely hovering needs 0.408 (thrust 120 / mass 5 = 24 m/s^2 against
# 9.8 gravity). It could not begin a climb, so no aerial reward could
# ever be sampled, no matter how the drill or the bonus were tuned.
#
# min_head_frac makes any ONE collapsed head raise ent_coef on its own.
# Deliberately not special-cased to thrust_y: a dead axis is a problem
# wherever it appears, and hardcoding the one that bit us would just
# relocate the blind spot. Default 0.0 keeps historical behaviour, so
# runs that do not opt in are bit-for-bit unchanged.
head_fracs = [e / math.log(n) for e, n in zip(entropies, self.model.action_space.nvec)]
min_frac = min(head_fracs)
self.logger.record("train/entropy_head_min_frac", min_frac)
starved_head = min_frac < self.min_head_frac
if mean_entropy < target or starved_head:
self.model.ent_coef = min(self.model.ent_coef * self.adjust_rate, self.ent_coef_bounds[1])
else:
self.model.ent_coef = max(self.model.ent_coef / self.adjust_rate, self.ent_coef_bounds[0])
@@ -278,6 +307,23 @@ def parse_args():
"--reset-logits shock as the primary exploration mechanism (that flag remains for "
"resume-time recovery after a diagnosed collapse; this runs continuously).",
)
parser.add_argument(
"--ent-coef-max", type=float, default=0.05,
help="Upper bound EntropyFloorCallback may raise ent_coef to. The 0.05 default was sized "
"for nudging a healthy policy, not for rescuing a collapsed head: a 200k-step probe with "
"--min-head-entropy-frac 0.35 pinned ent_coef at 0.05 for the whole run while the starved "
"head still sat at 0.146 of its ceiling, i.e. the controller was saturated and asking for "
"more. Raise this when deliberately breaking a policy out of a local optimum.",
)
parser.add_argument(
"--min-head-entropy-frac", type=float, default=0.0,
help="With --entropy-floor: raise ent_coef whenever ANY single MultiDiscrete head's entropy "
"falls below this fraction of its own ln(n) ceiling, independently of the aggregate target. "
"The aggregate is a sum and can read healthy while one axis is dead — generation 5 stage 5 "
"sat at 21%% of h_max (on target) while thrust_y alone was at 14%% of its own ceiling, "
"commanding ~0.03 mean vertical thrust against the 0.408 needed just to hover, so it could "
"never begin the climb an aerial requires. 0.0 (default) disables, preserving prior behaviour.",
)
parser.add_argument(
"--checkpoint-every", type=int, default=10_000_000,
help="Timesteps between checkpoints. Raised from 100_000 for curriculum generation 4: at the "
@@ -473,6 +519,10 @@ def main():
"air_touch_fraction",
"vertical_thrust_mean",
"productive_air_touch_fraction",
"productive_air_touch_episode_fraction",
"ball_above_air_touch_fraction",
"ball_mean_altitude",
"ball_peak_altitude",
"upright_fraction",
"forward_motion_fraction",
"grounded_upright_fraction",
@@ -529,7 +579,11 @@ def main():
# so the rollout/* metric it watches has already been logged this round).
callbacks = [checkpoint_callback, GoalRateCallback(), FlightTelemetryCallback()]
if args.entropy_floor:
callbacks.append(EntropyFloorCallback(total_timesteps=args.timesteps))
callbacks.append(EntropyFloorCallback(
total_timesteps=args.timesteps,
min_head_frac=args.min_head_entropy_frac,
ent_coef_bounds=(1e-4, args.ent_coef_max),
))
if args.abort_metric is not None and args.abort_below is not None and args.abort_at_steps is not None:
callbacks.append(AbortIfCallback(args.abort_metric, args.abort_below, args.abort_at_steps))