mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-15 21:02:03 +00:00
feat(training): curriculum generation 4 — MultiDiscrete action space redesign
Three curriculum generations (2026-07-21 through 2026-08-04) all tried gating *when* the policy could use vertical thrust/pitch-roll on top of a continuous Gaussian action space, and all three failed the same way: PPO's action-distribution std collapsed within ~10% of steps and never recovered, landing at a 15-32% win rate vs the grounded reference regardless of mechanism (hard mask, then a gradual ramp). Generation 3's final attempt just landed at 24% — the worst of the three. Root cause, verified against this project's own physics: hovering this ship requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust 120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value but never sustain it long enough to earn the reward gradient that would move the mean — no amount of gating *when* the axis acts fixes a problem in *how* the policy represents a decision on it. This also independently found and fixes a real bug: godot_rl never marks an episode timeout as a truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every generation to date. - Game/scripts/ship_action_codec.gd (new): single source of truth for a per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared by training and in-game inference, replacing the continuous Gaussian. thrust_y's bins are deliberately asymmetric so a random policy drifts through the volume instead of floor-pinning. Legacy continuous decode (ai_ship_controller.gd's old logic) preserved verbatim so every pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working unchanged via an optional "action_space" JSON field. - ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only) so the value function can see what wall_contact_penalty fires on. - ship_ai_controller.gd: action space/decode via the codec; drop the vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default lowered 4x (aerial approaches require pitching); flight telemetry (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean) and truncation-snapshot fields on get_info(). - training_mode.gd: new air_drill_chance state-setter branch (ball spawned high, ships low, kept clear of walls) so aerial practice is forced by the environment instead of relying on reward-driven exploration alone; snapshot terminal observations before a timeout reset for the truncation fix. - cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into SB3's TimeLimit.truncated/terminal_observation keys. - train.py: --reset-logits (+ --reset-logits-heads) replaces the now-meaningless --reset-std; new EntropyFloorCallback (a persistent per-rollout ent_coef controller replacing the one-shot std-reset shock) and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/ --tilt-penalty flags; optional AbortIfCallback kill-criterion. - export_policy.py: writes the action_space block for MultiDiscrete models; index-level parity check (argmax per head) instead of comparing floats. - curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no grounded stage, full action space live from step 1; deletes generation 1-3's checkpoint-lineage machinery (nothing to resume from); final report evaluates against both promoted/easy.json and the new promoted/reference-grounded.json (a copy of curric-s5-aggression, the strongest grounded-era artifact, kept as a fixed yardstick). - run_training.sh/.gitignore: commit only final.zip, not the ~2400 intermediate checkpoint files a single stage was writing (~500MB -> ~0.2MB per run); requirements.txt pinned (behaviour here now depends on specific library internals, not just public APIs). - test_action_space.py (new): offline rung-0 check catching a head-order mismatch before it silently corrupts 24h of training. Validated: GDScript compiles clean (Godot --headless --import + script validation), free_play.tscn and training.tscn both boot headless without errors, offline action-space assertions pass. Not yet run: the actual smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4" section, before committing to the full ~32h curriculum. See TRAINING.md's "Generation 4" section for the full design writeup.
This commit is contained in:
Josh Creek
+196
-201
@@ -18,32 +18,49 @@ forever for the wrong reason. When a stage does fail MAX_RETRIES times in a
|
||||
row, the script stops and asks for a human look rather than retrying
|
||||
indefinitely or silently advancing past a bad stage.
|
||||
|
||||
This is generation 3 of the curriculum. Generation 1 (6 stages: score,
|
||||
defend, no_draws, mechanics, aggression, unmask) ran 2026-07-21 through
|
||||
2026-07-26 and is archived in curriculum_state_gen1.json — its final stage
|
||||
("unmask", full 3D flight on top of the aggression retune) failed 3 straight
|
||||
attempts, monotonically worsening (25% -> 20% -> 15% win rate vs
|
||||
curric-s5-aggression) because every retry resumed the same drifting
|
||||
checkpoint under identical flags instead of actually changing anything.
|
||||
Generation 2 (archived in curriculum_state_gen2.json) started a fresh
|
||||
single "unmask" stage seeded directly from curric-s5-aggression's own
|
||||
checkpoint (FOUNDATION_EXPERIMENT below) with retuned reward weights — it
|
||||
also failed 3 attempts, landing at a stable 32% / 28% / 31% win rate each
|
||||
time, ruling out both "retune the reward weights" and "just give it more
|
||||
time" as fixes. Generation 3 replaces the single all-or-nothing unmask
|
||||
flip with a gradual ramp (4 stages: unmask-ramp25/50/75, then unmask at
|
||||
full authority) — see TRAINING.md's "Generation 3" section for the full
|
||||
postmortem and design.
|
||||
This is generation 4 of the curriculum — a full redesign, not a patch.
|
||||
Generations 1-3 (archived in curriculum_state_gen1.json/_gen2.json/_gen3.json)
|
||||
all tried teaching full 3D flight by training grounded first and then
|
||||
opening up vertical/pitch-roll authority (a hard 0/1 mask in gen 1/2, a
|
||||
gradual float ramp in gen 3) on top of a continuous Gaussian action space.
|
||||
All three failed: gen 1's hard mask went 25% -> 20% -> 15% win rate across 3
|
||||
attempts; gen 2's single-flip retune landed at a stable 32%/28%/31%; gen 3's
|
||||
gradual ramp landed at 29%/30%/24% — actually the worst of the three by its
|
||||
final attempt. Every attempt showed the same signature regardless of
|
||||
mechanism: PPO's Gaussian action-distribution std collapsed from ~0.30 to
|
||||
~0.13-0.15 within the first ~10% of steps and never recovered. The root
|
||||
cause: hovering this ship (mass 5.0, vertical_thrust 120, default gravity
|
||||
9.8 — see ship.gd) requires *holding* thrust.y ~= 0.408 continuously; a
|
||||
collapsed near-zero-mean Gaussian can brush that value but never sustain it
|
||||
long enough to earn the reward gradient that would move the mean. No amount
|
||||
of gating *when* the axis is allowed to act fixes a problem in *how* the
|
||||
policy represents a decision on it.
|
||||
|
||||
Generation 4 (see TRAINING.md and Game/scripts/ship_action_codec.gd)
|
||||
replaces the action space itself with per-axis MultiDiscrete bins instead of
|
||||
a continuous Gaussian, trains the full action space from step 1 with no
|
||||
grounded stage at all (no successful self-play RL bot in this problem class
|
||||
gates control authority — see the RLGym/RLBot research cited in
|
||||
TRAINING.md), and adds a state-setter "air drill" episode-start branch
|
||||
(training_mode.gd's air_drill_chance) to force aerial practice instead of
|
||||
relying on reward-driven exploration alone. All of generation 1-3's
|
||||
checkpoint-lineage machinery (FOUNDATION_EXPERIMENT, locomotion-groundedness
|
||||
tracking, resume/reference overrides for skipping a regressed branch) is
|
||||
gone because there is nothing to resume from: every prior checkpoint is a
|
||||
different, incompatible action/observation shape. The two strongest prior
|
||||
artifacts are kept as fixed evaluation references instead (see
|
||||
PROMOTED_EASY/PROMOTED_REFERENCE_GROUNDED below) — they remain playable
|
||||
opponents forever via PolicyNetwork's format-versioned JSON even though
|
||||
their own checkpoints and generation are gone.
|
||||
|
||||
Every experiment name this script generates is timestamped
|
||||
(YYYYMMDD-HHMM-<name>, applied once in run_stage_attempt) so runs stay
|
||||
unique across restarts/generations and sort chronologically in TensorBoard
|
||||
and checkpoints/ — plain names like "curric-s1-score" from generation 1
|
||||
would otherwise collide with generation 2's own stage 1.
|
||||
and checkpoints/.
|
||||
|
||||
Usage:
|
||||
.venv/bin/python curriculum.py # run/resume the curriculum
|
||||
.venv/bin/python curriculum.py --seed-checkpoint checkpoints/run11/final.zip
|
||||
.venv/bin/python curriculum.py --seed-checkpoint checkpoints/some/final.zip
|
||||
.venv/bin/python curriculum.py --force-retry # after fixing something, retry the blocked stage
|
||||
.venv/bin/python curriculum.py --skip-to-next-stage # human judgment call: good enough, move on anyway
|
||||
|
||||
@@ -61,20 +78,14 @@ from datetime import datetime
|
||||
TRAINING_DIR = pathlib.Path(__file__).resolve().parent
|
||||
STATE_PATH = TRAINING_DIR / "curriculum_state.json"
|
||||
EVAL_HISTORY_PATH = TRAINING_DIR / "eval_history.json"
|
||||
ROOKIE_REFERENCE = TRAINING_DIR.parent / "Game" / "bots" / "rookie.json"
|
||||
|
||||
# Generation 1's last cleanly-passing checkpoint (see curriculum_state_gen1.json)
|
||||
# — generations 2 and 3 both build on this directly instead of re-running
|
||||
# stages 1-5.
|
||||
FOUNDATION_EXPERIMENT = "curric-s5-aggression"
|
||||
|
||||
# Groundedness (locomotion-mask state) for experiments that predate this
|
||||
# generation's own log, so _grounded_for_experiment can still answer for
|
||||
# them — see that function.
|
||||
LEGACY_GROUNDED = {
|
||||
"rookie": False,
|
||||
FOUNDATION_EXPERIMENT: True,
|
||||
}
|
||||
# Fixed evaluation references — never touched by training scripts (see
|
||||
# TRAINING.md's "Promoted bots" section) — kept forever as playable
|
||||
# opponents via PolicyNetwork's format-versioned JSON even after their own
|
||||
# checkpoints/generation are gone. The final report (not a gate) evaluates
|
||||
# generation 4's result against both.
|
||||
PROMOTED_EASY = TRAINING_DIR.parent / "Game" / "bots" / "promoted" / "easy.json"
|
||||
PROMOTED_REFERENCE_GROUNDED = TRAINING_DIR.parent / "Game" / "bots" / "promoted" / "reference-grounded.json"
|
||||
|
||||
MAX_RETRIES = 2
|
||||
EVAL_EPISODES = 100
|
||||
@@ -84,118 +95,108 @@ EVAL_EPISODES = 100
|
||||
# module docstring) — advance rather than retry.
|
||||
REGRESSION_MARGIN = 0.15
|
||||
|
||||
# Standing flags applied to every attempt, mirroring next_run.sh: reset-std
|
||||
# reopens exploration every attempt (harmless on fresh starts — train.py
|
||||
# only applies it on --resume), ent-coef keeps it from re-collapsing.
|
||||
STANDING_ARGS = ["--reset-std", "0.3", "--ent-coef", "0.001"]
|
||||
# Standing flags applied to every attempt. Generation 3's "--reset-std 0.3"
|
||||
# (a one-shot shock, and meaningless anyway under MultiDiscrete — there is
|
||||
# no log_std) is gone; EntropyFloorCallback (see train.py) is a continuous
|
||||
# controller instead, which every generation's TensorBoard data argues is
|
||||
# what was actually needed (a single reset at attempt start reliably decayed
|
||||
# away within ~10% of steps, every time). --ent-coef raised an order of
|
||||
# magnitude from generation 3's 0.001: that value was tuned for a Gaussian's
|
||||
# unbounded differential entropy, not MultiDiscrete's bounded (~10-nat)
|
||||
# entropy.
|
||||
STANDING_ARGS = ["--ent-coef", "0.01", "--entropy-floor"]
|
||||
|
||||
# Ball-chasing/scoring reward flags shared by every unmask-ramp stage
|
||||
# (generation 3 — see below): identical across all 4 stages so the ramp
|
||||
# itself is the only studied variable. Lifted from generation 2's single
|
||||
# "unmask" attempt (raised from stage 5's 0.05/0.006/0.5/0.02/60 defaults).
|
||||
_UNMASK_RAMP_SHARED_FLAGS = [
|
||||
"--opponent-mode", "self_play",
|
||||
# Reward-shaping flags shared by every stage so the studied variables (state
|
||||
# mix, opponent mode) stay isolated — carried forward unchanged from
|
||||
# generation 2/3, which the reward-farmability analysis in TRAINING.md
|
||||
# confirmed were never the actual problem. draw_penalty and airborne_penalty
|
||||
# are deliberately NOT overridden here (both default to 0.0 in
|
||||
# training_mode.gd/ship_ai_controller.gd): generation 3's draw_penalty=5 and
|
||||
# airborne_penalty ramping up in lockstep with the unmask ramp were both
|
||||
# grounded-era, anti-flight pressures that have no place in a curriculum
|
||||
# whose entire point is teaching flight.
|
||||
_SHARED_REWARD_FLAGS = [
|
||||
"--velocity-to-ball-weight", "0.08",
|
||||
"--ball-distance-penalty", "0.01",
|
||||
"--ball-touch-reward", "0.7",
|
||||
"--ball-velocity-to-goal-weight", "0.06",
|
||||
"--goal-reward", "80",
|
||||
"--draw-penalty", "5",
|
||||
]
|
||||
|
||||
# A stage dict may additionally set "abort_if": {"metric": "rollout/airborne_
|
||||
# fraction", "below": 0.05, "at_steps": N} to end that attempt early if a
|
||||
# flight-telemetry metric (see train.py's FlightTelemetryCallback) hasn't
|
||||
# cleared a bar by N *absolute* PPO timesteps (model.num_timesteps keeps
|
||||
# accumulating across --resume, so N must account for whatever this stage
|
||||
# inherits from its predecessor, not just this stage's own budget).
|
||||
# Deliberately unset on every stage below for now — rung 5 of TRAINING.md's
|
||||
# validation ladder (a short controlled A/B) should establish what a
|
||||
# sensible threshold actually looks like before any stage bets a real 12h+
|
||||
# budget on a guessed one.
|
||||
STAGES = [
|
||||
{
|
||||
"name": "unmask-ramp25",
|
||||
# Generation 3, step 1/4 of a gradual locomotion-unmask ramp — see
|
||||
# TRAINING.md's "Generation 3" section for the full postmortem.
|
||||
# Generation 2's single all-or-nothing "unmask" stage (flip
|
||||
# vertical_ramp/pitch_roll_ramp 0 -> 1 in one step) failed 3
|
||||
# independent 240M-step attempts in a row, landing at a stable
|
||||
# 32% / 28% / 31% win rate vs curric-s5-aggression each time — not
|
||||
# noise (attempts 2-3 each gave the *same* checkpoint lineage
|
||||
# another full 240M steps with zero improvement) and not fixable by
|
||||
# more time. Every attempt shows train/std collapsing from ~0.30 to
|
||||
# ~0.13-0.15 within the first ~10% of steps and never recovering —
|
||||
# the policy locks the newly-opened axes back down before ever
|
||||
# meaningfully exploring them.
|
||||
#
|
||||
# This stage instead scales vertical_ramp/pitch_roll_ramp to 25%
|
||||
# authority. Ungated (see "gated" below and main()'s loop): this is
|
||||
# a waypoint, not a measured transition — no eval runs, no
|
||||
# regression gate applies, it always advances after training.
|
||||
# airborne_penalty is off (0.0) here: at 25% authority the axis
|
||||
# barely does anything yet, so there's nothing to discourage.
|
||||
"name": "bootstrap",
|
||||
# Stage 1/3: empty-net finishing practice from a random policy — no
|
||||
# live opponent, so the full action space's first behaviour to
|
||||
# emerge is "fly to ball, push it toward the net" without a moving
|
||||
# target complicating credit assignment. Generation 1's own stage 1
|
||||
# (also inert-opponent, also empty-net) was the one stage across all
|
||||
# 3 prior generations that unambiguously passed on its first
|
||||
# attempt — reusing that shape here, just with the full action space
|
||||
# live instead of yaw-only.
|
||||
"flags": [
|
||||
*_UNMASK_RAMP_SHARED_FLAGS,
|
||||
"--vertical-ramp", "0.25",
|
||||
"--pitch-roll-ramp", "0.25",
|
||||
"--airborne-penalty", "0.0",
|
||||
"--opponent-mode", "inert",
|
||||
"--attack-goal-bias", "1.0",
|
||||
"--kickoff-chance", "0.10",
|
||||
"--near-goal-chance", "0.50",
|
||||
"--air-drill-chance", "0.20",
|
||||
*_SHARED_REWARD_FLAGS,
|
||||
],
|
||||
"gated": False,
|
||||
"timesteps": 40_000_000, # ~4h at the standing n-parallel/speedup (20M took ~2h)
|
||||
# Stage 0 MUST set this explicitly — resume_checkpoint()'s stage-0
|
||||
# branch returns None (train from scratch) without it.
|
||||
"resume_from_experiment": FOUNDATION_EXPERIMENT,
|
||||
"gated": False, # ungated waypoint: trains, checkpoints, always advances — no eval
|
||||
"timesteps": 40_000_000, # ~4h at the standing n-parallel/speedup
|
||||
},
|
||||
{
|
||||
"name": "unmask-ramp50",
|
||||
# Step 2/4: 50% authority. airborne_penalty at 1/3 of its final
|
||||
# value — enough to start discouraging unproductive altitude, not
|
||||
# enough to fight the still-partial vertical axis outright.
|
||||
# resume_from_experiment deliberately omitted: chains from
|
||||
# ramp25's pass via _resume_source_experiment's default
|
||||
# (_passing_experiment_for_stage) — do not add an override here.
|
||||
"name": "selfplay",
|
||||
# Stage 2/3: this is where essentially all of the actual learning
|
||||
# happens. Self-play (not frozen) as the main regime — it's what
|
||||
# scales and what Necto/Nexto-class bots actually use; a frozen
|
||||
# target this early would cap skill at "exploits one specific bot"
|
||||
# instead of a moving, improving target. air_drill_chance stays on
|
||||
# at a constant rate throughout (not introduced as a later stage) —
|
||||
# gating *when* a skill is drilled reproduces the exact "gate what
|
||||
# the policy is allowed to do" pattern that failed 3 generations in
|
||||
# a row; only the state mix should vary between stages, never what
|
||||
# the policy can act on.
|
||||
"flags": [
|
||||
*_UNMASK_RAMP_SHARED_FLAGS,
|
||||
"--vertical-ramp", "0.5",
|
||||
"--pitch-roll-ramp", "0.5",
|
||||
"--airborne-penalty", "0.001",
|
||||
"--opponent-mode", "self_play",
|
||||
"--kickoff-chance", "0.15",
|
||||
"--near-goal-chance", "0.25",
|
||||
"--air-drill-chance", "0.25",
|
||||
*_SHARED_REWARD_FLAGS,
|
||||
],
|
||||
"gated": False,
|
||||
"timesteps": 40_000_000,
|
||||
"gated": True,
|
||||
"timesteps": 160_000_000, # ~16h
|
||||
},
|
||||
{
|
||||
"name": "unmask-ramp75",
|
||||
# Step 3/4: 75% authority, airborne_penalty at 2/3 of its final
|
||||
# value. Also chains automatically — no resume_from_experiment.
|
||||
"name": "gauntlet",
|
||||
# Stage 3/3: a stationary opponent (this stage's own predecessor's
|
||||
# export) gives a low-variance measurement — important when the gate
|
||||
# is a 100-episode sample with a lenient 15-point margin — and
|
||||
# catches a self-play fixed point: a policy that only learned to
|
||||
# beat itself will look fine in stage 2 and stall here.
|
||||
# opponent_model_from_previous_stage resolves --opponent-model at
|
||||
# run time to whatever stage 2's own passing export turns out to be
|
||||
# (see run_stage_attempt) rather than a hardcoded name.
|
||||
"flags": [
|
||||
*_UNMASK_RAMP_SHARED_FLAGS,
|
||||
"--vertical-ramp", "0.75",
|
||||
"--pitch-roll-ramp", "0.75",
|
||||
"--airborne-penalty", "0.002",
|
||||
"--opponent-mode", "frozen",
|
||||
"--kickoff-chance", "0.15",
|
||||
"--near-goal-chance", "0.25",
|
||||
"--air-drill-chance", "0.25",
|
||||
*_SHARED_REWARD_FLAGS,
|
||||
],
|
||||
"gated": False,
|
||||
"timesteps": 40_000_000,
|
||||
},
|
||||
{
|
||||
"name": "unmask",
|
||||
# Step 4/4, the measured transition: full ramp (100% authority),
|
||||
# airborne_penalty at its full value — behaviourally and eval-wise
|
||||
# identical to generation 2's "unmask" stage's flags/config, so
|
||||
# this stage's result is a direct, apples-to-apples comparison
|
||||
# against the 3 failed all-or-nothing attempts (same reference,
|
||||
# same opponent mode, same budget). Gated (default True): evaluated
|
||||
# against FOUNDATION_EXPERIMENT exactly like every prior attempt.
|
||||
#
|
||||
# No resume_from_experiment here (deliberately, unlike generation
|
||||
# 2's single-stage version) — this stage chains from ramp75's own
|
||||
# checkpoint via the default resume path, not back to
|
||||
# FOUNDATION_EXPERIMENT; only reference_experiment (the *eval*
|
||||
# opponent) stays FOUNDATION_EXPERIMENT.
|
||||
"flags": [
|
||||
*_UNMASK_RAMP_SHARED_FLAGS,
|
||||
"--vertical-ramp", "1.0",
|
||||
"--pitch-roll-ramp", "1.0",
|
||||
"--airborne-penalty", "0.003",
|
||||
],
|
||||
"grounded": False,
|
||||
"timesteps": 240_000_000, # unchanged from the 3 failed attempts — same budget for a controlled comparison
|
||||
"reference_experiment": FOUNDATION_EXPERIMENT,
|
||||
# On retry, reset to the clean ramp75 checkpoint rather than
|
||||
# compounding a failed full-ramp attempt's own drift — mirrors the
|
||||
# generation 1 -> 2 postmortem (blind same-checkpoint retries only
|
||||
# made things worse).
|
||||
"reset_retry_checkpoint": True,
|
||||
"gated": True,
|
||||
"timesteps": 120_000_000, # ~12h
|
||||
"opponent_model_from_previous_stage": True,
|
||||
},
|
||||
]
|
||||
|
||||
@@ -227,50 +228,6 @@ def _logged_experiment_name(stage_index: int, attempt: int) -> str:
|
||||
raise RuntimeError(f"No logged experiment for stage {stage_index} attempt {attempt}")
|
||||
|
||||
|
||||
def resume_checkpoint(stage_index: int, attempt: int, seed_checkpoint: str | None) -> str | None:
|
||||
if attempt > 0 and not STAGES[stage_index].get("reset_retry_checkpoint"):
|
||||
# Retry: keep training the same stage's own last attempt.
|
||||
prev = _logged_experiment_name(stage_index, attempt - 1)
|
||||
return str(TRAINING_DIR / "checkpoints" / prev / "final.zip")
|
||||
if stage_index == 0 and seed_checkpoint:
|
||||
return seed_checkpoint
|
||||
if stage_index == 0 and not STAGES[0].get("resume_from_experiment"):
|
||||
# Deliberately fresh by default: the curriculum exists because
|
||||
# resuming self-play across a regime change (run10, run11) didn't
|
||||
# work, so a from-scratch stage 1 starts from a random policy under
|
||||
# its own regime unless --seed-checkpoint or resume_from_experiment
|
||||
# says otherwise.
|
||||
return None
|
||||
# Either a later stage chaining off its predecessor, or
|
||||
# reset_retry_checkpoint: this stage's own retries have been drifting
|
||||
# rather than converging (see the "unmask" stage's comment) — resume
|
||||
# from the stage's normal resume source instead of compounding the last
|
||||
# failed attempt's drift.
|
||||
prev_experiment = _resume_source_experiment(stage_index)
|
||||
return str(TRAINING_DIR / "checkpoints" / prev_experiment / "final.zip")
|
||||
|
||||
|
||||
def reference_bot(stage_index: int) -> str:
|
||||
if stage_index == 0 and not STAGES[0].get("reference_experiment"):
|
||||
return str(ROOKIE_REFERENCE)
|
||||
prev_experiment = _reference_source_experiment(stage_index)
|
||||
return str(TRAINING_DIR.parent / "Game" / "bots" / f"{prev_experiment}.json")
|
||||
|
||||
|
||||
# A stage normally chains off "whatever passed at the previous index," but a
|
||||
# stage can instead name an explicit resume_from_experiment/reference_experiment
|
||||
# to skip a since-regressed branch, or (stage 0) to seed from a fixed
|
||||
# foundation checkpoint instead of a from-scratch policy.
|
||||
def _resume_source_experiment(stage_index: int) -> str:
|
||||
override = STAGES[stage_index].get("resume_from_experiment")
|
||||
return override if override else _passing_experiment_for_stage(stage_index - 1)
|
||||
|
||||
|
||||
def _reference_source_experiment(stage_index: int) -> str:
|
||||
override = STAGES[stage_index].get("reference_experiment")
|
||||
return override if override else _passing_experiment_for_stage(stage_index - 1)
|
||||
|
||||
|
||||
def _passing_experiment_for_stage(stage_index: int) -> str:
|
||||
state = load_state()
|
||||
for entry in state["log"]:
|
||||
@@ -279,14 +236,31 @@ def _passing_experiment_for_stage(stage_index: int) -> str:
|
||||
raise RuntimeError(f"No passing attempt recorded for stage {stage_index} ({STAGES[stage_index]['name']})")
|
||||
|
||||
|
||||
def _grounded_for_experiment(experiment: str) -> bool:
|
||||
if experiment in LEGACY_GROUNDED:
|
||||
return LEGACY_GROUNDED[experiment]
|
||||
state = load_state()
|
||||
for entry in state["log"]:
|
||||
if entry["experiment"] == experiment:
|
||||
return STAGES[entry["stage_index"]]["grounded"]
|
||||
raise ValueError(f"Unknown experiment for groundedness lookup: {experiment}")
|
||||
def resume_checkpoint(stage_index: int, attempt: int, seed_checkpoint: str | None) -> str | None:
|
||||
if attempt > 0:
|
||||
# Retry: keep training the same stage's own last attempt. No
|
||||
# per-stage "reset to a clean upstream checkpoint" override in
|
||||
# generation 4 (unlike generation 3's "unmask" stage) — nothing yet
|
||||
# suggests a generation-4 retry needs that; add one if a stage's
|
||||
# retries turn out to be drifting rather than converging.
|
||||
prev = _logged_experiment_name(stage_index, attempt - 1)
|
||||
return str(TRAINING_DIR / "checkpoints" / prev / "final.zip")
|
||||
if stage_index == 0:
|
||||
# Deliberately fresh unless --seed-checkpoint says otherwise: full
|
||||
# action space live from step 1, nothing to inherit — every prior
|
||||
# generation's checkpoints are a different, incompatible
|
||||
# action/observation shape (see module docstring).
|
||||
return seed_checkpoint
|
||||
prev_experiment = _passing_experiment_for_stage(stage_index - 1)
|
||||
return str(TRAINING_DIR / "checkpoints" / prev_experiment / "final.zip")
|
||||
|
||||
|
||||
def reference_bot(stage_index: int) -> str:
|
||||
"""Only called for gated stages (stage 0 is ungated) — the previous
|
||||
stage's own passing export, exactly like every prior generation's
|
||||
default chaining."""
|
||||
prev_experiment = _passing_experiment_for_stage(stage_index - 1)
|
||||
return str(TRAINING_DIR.parent / "Game" / "bots" / f"{prev_experiment}.json")
|
||||
|
||||
|
||||
def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
|
||||
@@ -294,9 +268,6 @@ def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
|
||||
# chronologically in TensorBoard/checkpoints — see module docstring.
|
||||
exp = f"{datetime.now().strftime('%Y%m%d-%H%M')}-{experiment_name(stage_index, attempt)}"
|
||||
resume = resume_checkpoint(stage_index, attempt, args.seed_checkpoint)
|
||||
# A stage can override the run's timesteps budget (see "floor-lock",
|
||||
# which deliberately runs much longer than the ~20M/~2h every stage so
|
||||
# far has used); otherwise it falls back to curriculum.py's own --timesteps.
|
||||
timesteps = STAGES[stage_index].get("timesteps", args.timesteps)
|
||||
cmd = [
|
||||
"./run_training.sh", exp,
|
||||
@@ -308,6 +279,17 @@ def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
|
||||
]
|
||||
if resume:
|
||||
cmd += ["--resume", resume]
|
||||
if STAGES[stage_index].get("opponent_model_from_previous_stage"):
|
||||
prev_experiment = _passing_experiment_for_stage(stage_index - 1)
|
||||
opponent_model = TRAINING_DIR.parent / "Game" / "bots" / f"{prev_experiment}.json"
|
||||
cmd += ["--opponent-model", str(opponent_model)]
|
||||
abort_if = STAGES[stage_index].get("abort_if")
|
||||
if abort_if:
|
||||
cmd += [
|
||||
"--abort-metric", abort_if["metric"],
|
||||
"--abort-below", str(abort_if["below"]),
|
||||
"--abort-at-steps", str(abort_if["at_steps"]),
|
||||
]
|
||||
print(f"\n=== Stage {stage_index + 1}/{len(STAGES)} ({STAGES[stage_index]['name']}), "
|
||||
f"attempt {attempt + 1}/{MAX_RETRIES + 1}: {exp} ===")
|
||||
print(" ".join(cmd))
|
||||
@@ -315,22 +297,9 @@ def run_stage_attempt(stage_index: int, attempt: int, args) -> str:
|
||||
return exp
|
||||
|
||||
|
||||
def reference_grounded(stage_index: int) -> bool:
|
||||
if stage_index == 0 and not STAGES[0].get("reference_experiment"):
|
||||
# rookie.json predates the locomotion mask entirely — always full 3D.
|
||||
return False
|
||||
return _grounded_for_experiment(_reference_source_experiment(stage_index))
|
||||
|
||||
|
||||
def evaluate_attempt(experiment: str, reference: str, episodes: int, stage_index: int) -> dict:
|
||||
def evaluate_attempt(experiment: str, reference: str, episodes: int) -> dict:
|
||||
candidate = TRAINING_DIR.parent / "Game" / "bots" / f"{experiment}.json"
|
||||
cmd = [".venv/bin/python", "evaluate.py", str(candidate), reference, "--episodes", str(episodes)]
|
||||
# Must match how each side was actually trained — see ai_ship_controller.gd's
|
||||
# allow_vertical/allow_pitch_roll and evaluate.py's --grounded-a/-b.
|
||||
if STAGES[stage_index]["grounded"]:
|
||||
cmd.append("--grounded-a")
|
||||
if reference_grounded(stage_index):
|
||||
cmd.append("--grounded-b")
|
||||
print(" ".join(cmd))
|
||||
subprocess.run(cmd, cwd=TRAINING_DIR, check=True)
|
||||
history = json.loads(EVAL_HISTORY_PATH.read_text())
|
||||
@@ -345,6 +314,33 @@ def decide(record: dict) -> str:
|
||||
return "pass"
|
||||
|
||||
|
||||
def final_report(experiment: str) -> None:
|
||||
"""Not a gate — the two numbers that actually answer "did generation 4
|
||||
work?" (see TRAINING.md). promoted/easy.json is the shipped bot;
|
||||
promoted/reference-grounded.json (a copy of generation 3's
|
||||
curric-s5-aggression, made before the flat Game/bots/ dump was scrapped)
|
||||
is the strongest grounded-era artifact and the yardstick generations 1-3
|
||||
were all measured against. reference-grounded.json was trained with the
|
||||
locomotion mask on, so needs --grounded-b; easy.json was itself promoted
|
||||
from a *failed* unmask stage (curric-s6-unmask) and is full 3D like
|
||||
every generation-4 candidate, so needs no flag."""
|
||||
candidate = TRAINING_DIR.parent / "Game" / "bots" / f"{experiment}.json"
|
||||
print("\n=== Curriculum complete — final report (informational, not a gate) ===")
|
||||
for label, reference, extra_flags in [
|
||||
("promoted/easy.json (shipped bot)", PROMOTED_EASY, []),
|
||||
("promoted/reference-grounded.json (strongest grounded-era bot)", PROMOTED_REFERENCE_GROUNDED, ["--grounded-b"]),
|
||||
]:
|
||||
if not reference.exists():
|
||||
print(f" vs {label}: skipped, file not found")
|
||||
continue
|
||||
cmd = [".venv/bin/python", "evaluate.py", str(candidate), str(reference), "--episodes", str(EVAL_EPISODES), *extra_flags]
|
||||
print(" ".join(cmd))
|
||||
subprocess.run(cmd, cwd=TRAINING_DIR, check=True)
|
||||
record = json.loads(EVAL_HISTORY_PATH.read_text())[-1]
|
||||
print(f" vs {label}: {record['wins_a']}-{record['wins_b']} ({record['draws']} draws), "
|
||||
f"win rate {record['win_rate_a']:.0%}")
|
||||
|
||||
|
||||
def commit_progress(experiment: str) -> None:
|
||||
subprocess.run(["git", "add", "curriculum_state.json", "eval_history.json"], cwd=TRAINING_DIR, check=True)
|
||||
result = subprocess.run(["git", "diff", "--cached", "--quiet"], cwd=TRAINING_DIR)
|
||||
@@ -364,8 +360,7 @@ def main():
|
||||
parser.add_argument("--speedup", type=int, default=16)
|
||||
parser.add_argument(
|
||||
"--seed-checkpoint", default=None,
|
||||
help="Resume stage 1 from this checkpoint instead of its default resume source "
|
||||
"(FOUNDATION_EXPERIMENT's checkpoint)",
|
||||
help="Resume stage 1 from this checkpoint instead of training from scratch",
|
||||
)
|
||||
parser.add_argument("--force-retry", action="store_true", help="Retry a blocked stage after human review")
|
||||
parser.add_argument("--skip-to-next-stage", action="store_true", help="Human judgment call: treat the blocked stage as good enough, advance anyway")
|
||||
@@ -413,14 +408,13 @@ def main():
|
||||
last_experiment = experiment
|
||||
|
||||
if not STAGES[stage_index].get("gated", True):
|
||||
# Ungated ramp waypoint (see the unmask-ramp2X stages): trains,
|
||||
# Ungated waypoint (see the bootstrap stage): trains,
|
||||
# checkpoints, and always advances — no eval, no regression
|
||||
# gate, nothing to retry against. See TRAINING.md's
|
||||
# "Generation 3" section.
|
||||
print(f"{experiment}: ungated ramp waypoint — skipping eval, advancing unconditionally")
|
||||
# gate, nothing to retry against.
|
||||
print(f"{experiment}: ungated waypoint — skipping eval, advancing unconditionally")
|
||||
state["log"].append({
|
||||
"stage_index": stage_index, "experiment": experiment, "attempt": attempt,
|
||||
"decision": "pass", "note": "ungated ramp waypoint (no eval)",
|
||||
"decision": "pass", "note": "ungated waypoint (no eval)",
|
||||
})
|
||||
state["stage_index"] += 1
|
||||
state["attempt"] = 0
|
||||
@@ -430,7 +424,7 @@ def main():
|
||||
continue
|
||||
|
||||
reference = reference_bot(stage_index)
|
||||
record = evaluate_attempt(experiment, reference, EVAL_EPISODES, stage_index)
|
||||
record = evaluate_attempt(experiment, reference, EVAL_EPISODES)
|
||||
decision = decide(record)
|
||||
|
||||
print(f"{experiment}: candidate {record['wins_a']}-{record['wins_b']} reference "
|
||||
@@ -469,6 +463,7 @@ def main():
|
||||
# None only if the loop above never ran at all (e.g. re-invoking
|
||||
# after the curriculum was already "done") — nothing new to commit
|
||||
# in that case.
|
||||
final_report(last_experiment)
|
||||
commit_progress(last_experiment)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user