jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-14 09:42:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(training): Replace all-or-nothing unmask with a gradual ramp

Browse Source 
Generation 2's single "unmask" stage (flip vertical/pitch-roll locomotion
from grounded-only to full 3D in one step) failed 3 independent 240M-step
attempts, landing at a stable 32% / 28% / 31% win rate vs curric-s5-aggression
each time -- not noise, and not fixable by more training time (attempts 2-3
each continued the same checkpoint lineage for another full 240M steps with
zero improvement). Every attempt shows train/std collapsing from ~0.30 to
~0.13-0.15 within the first ~10% of steps and never recovering: the policy
locks the newly-opened axes back down before ever meaningfully exploring
them.

Replaces the boolean allow_vertical/allow_pitch_roll mask on ShipAIController
with float vertical_ramp/pitch_roll_ramp multipliers (0.0-1.0), scaling axis
effect in set_action() instead of gating it outright -- the action space
never changes shape, so checkpoints stay resumable across ramp values. The
single unmask stage in curriculum.py becomes 4: three ungated warmup stages
(25%/50%/75% authority, airborne_penalty ramping in step) that train,
checkpoint, and always advance with no eval gate, then the measured stage at
full authority -- same reference, opponent mode, and 240M budget as the 3
failed attempts, for a direct comparison. Adds a "gated" flag/branch to
main()'s loop for the ungated stages.

This is generation 3 of the curriculum; generation 2's state is archived to
curriculum_state_gen2.json (mirroring the earlier gen1 -> gen2 archival) and
curriculum_state.json resets fresh, since its stage 0 no longer means what it
used to. See TRAINING.md's "Generation 3" section for the full postmortem,
stage table, and the open question about whether scaling action effect in
Godot (which PPO's own entropy/exploration math never sees) actually
addresses the collapse.
This commit is contained in:
Josh Creek
2026-07-31 21:47:53 +01:00
parent 0759e1514b
commit 3fd1c00895
8 changed files with 287 additions and 147 deletions
Download Patch File Download Diff File Expand all files Collapse all files
training/curriculum.py
+129 -65
View File
@@ -18,19 +18,22 @@ forever for the wrong reason. When a stage does fail MAX_RETRIES times in a
row, the script stops and asks for a human look rather than retrying
indefinitely or silently advancing past a bad stage.
This is generation 2 of the curriculum. Generation 1 (6 stages: score,
This is generation 3 of the curriculum. Generation 1 (6 stages: score,
defend, no_draws, mechanics, aggression, unmask) ran 2026-07-21 through
2026-07-26 and is archived in curriculum_state_gen1.json — its final stage
("unmask", full 3D flight on top of the aggression retune) failed 3 straight
attempts, monotonically worsening (25% -> 20% -> 15% win rate vs
curric-s5-aggression) because every retry resumed the same drifting
checkpoint under identical flags instead of actually changing anything.
Rather than let generation 1's stage numbering grow indefinitely
(unmask-retry4, retry5, ...), generation 2 starts a fresh stage 1 seeded
directly from curric-s5-aggression's own checkpoint (FOUNDATION_EXPERIMENT
below) — the last stage that actually passed cleanly — carrying over its
trained progress without re-running stages 1-5. See TRAINING.md for the
full generation 1 history and generation 2's design.
Generation 2 (archived in curriculum_state_gen2.json) started a fresh
single "unmask" stage seeded directly from curric-s5-aggression's own
checkpoint (FOUNDATION_EXPERIMENT below) with retuned reward weights — it
also failed 3 attempts, landing at a stable 32% / 28% / 31% win rate each
time, ruling out both "retune the reward weights" and "just give it more
time" as fixes. Generation 3 replaces the single all-or-nothing unmask
flip with a gradual ramp (4 stages: unmask-ramp25/50/75, then unmask at
full authority) — see TRAINING.md's "Generation 3" section for the full
postmortem and design.
Every experiment name this script generates is timestamped
(YYYYMMDD-HHMM-<name>, applied once in run_stage_attempt) so runs stay
@@ -61,7 +64,7 @@ EVAL_HISTORY_PATH = TRAINING_DIR / "eval_history.json"
ROOKIE_REFERENCE = TRAINING_DIR.parent / "Game" / "bots" / "rookie.json"
# Generation 1's last cleanly-passing checkpoint (see curriculum_state_gen1.json)
# — generation 2's stage 1 builds on this directly instead of re-running
# — generations 2 and 3 both build on this directly instead of re-running
# stages 1-5.
FOUNDATION_EXPERIMENT = "curric-s5-aggression"
@@ -86,70 +89,113 @@ REGRESSION_MARGIN = 0.15
# only applies it on --resume), ent-coef keeps it from re-collapsing.
STANDING_ARGS = ["--reset-std", "0.3", "--ent-coef", "0.001"]
# Ball-chasing/scoring reward flags shared by every unmask-ramp stage
# (generation 3 — see below): identical across all 4 stages so the ramp
# itself is the only studied variable. Lifted from generation 2's single
# "unmask" attempt (raised from stage 5's 0.05/0.006/0.5/0.02/60 defaults).
_UNMASK_RAMP_SHARED_FLAGS = [
"--opponent-mode", "self_play",
"--velocity-to-ball-weight", "0.08",
"--ball-distance-penalty", "0.01",
"--ball-touch-reward", "0.7",
"--ball-velocity-to-goal-weight", "0.06",
"--goal-reward", "80",
"--draw-penalty", "5",
]
STAGES = [
{
"name": "unmask",
# Re-opens full 3D controls (no more --no-allow-vertical/
# --no-allow-pitch-roll) on top of the aggression retune, instead of
# keeping locomotion masked indefinitely. The mask blocked *thrust*-
# driven flight outright; airborne_penalty (dense, scaled by height
# above the floor — see ship_ai_controller.gd) is meant to teach the
# policy to prefer staying grounded through incentives rather than a
# hard constraint, so it can start learning when the other axes are
# actually useful (aerial saves, wall recoveries) instead of never
# touching them.
"name": "unmask-ramp25",
# Generation 3, step 1/4 of a gradual locomotion-unmask ramp — see
# TRAINING.md's "Generation 3" section for the full postmortem.
# Generation 2's single all-or-nothing "unmask" stage (flip
# vertical_ramp/pitch_roll_ramp 0 -> 1 in one step) failed 3
# independent 240M-step attempts in a row, landing at a stable
# 32% / 28% / 31% win rate vs curric-s5-aggression each time — not
# noise (attempts 2-3 each gave the *same* checkpoint lineage
# another full 240M steps with zero improvement) and not fixable by
# more time. Every attempt shows train/std collapsing from ~0.30 to
# ~0.13-0.15 within the first ~10% of steps and never recovering —
# the policy locks the newly-opened axes back down before ever
# meaningfully exploring them.
#
# Generation 1 ran this exact transition 3 times (unmask, retry1,
# retry2) with identical flags and got monotonically worse each time
# (25% -> 20% -> 15% win rate vs curric-s5-aggression) — a blind
# retry just continues training the same drifting policy for
# longer, it was never going to converge differently. An adversarial
# review of a first patch (two modest new flags, still resuming the
# drifted retry2 checkpoint) found that insufficient too: the resume
# target was the worst of the three already-degraded checkpoints,
# and the new weights were too small to compete with the unchanged
# ball-pursuit terms. Generation 2's stage 1 instead:
# - resumes from FOUNDATION_EXPERIMENT (curric-s5-aggression)
# directly (resume_from_experiment below) for the first attempt.
# - raises velocity_to_ball_weight and ball_distance_penalty
# further (the actual ball-chasing terms, unchanged since stage
# 5 despite three failed attempts) and ball_touch_reward
# alongside them.
# - raises ball_velocity_to_goal_weight (reward for moving the
# ball toward the goal, not just touching it) and goal_reward
# (the terminal reward for scoring) — both newly exposed via
# train.py, previously only reachable as raw Godot cmdline args.
# - adds draw_penalty (proven effective in generation 1's stage 3
# against passivity), which this transition had never set:
# previously all carrot for scoring, no stick for never scoring.
# This stage instead scales vertical_ramp/pitch_roll_ramp to 25%
# authority. Ungated (see "gated" below and main()'s loop): this is
# a waypoint, not a measured transition — no eval runs, no
# regression gate applies, it always advances after training.
# airborne_penalty is off (0.0) here: at 25% authority the axis
# barely does anything yet, so there's nothing to discourage.
"flags": [
"--opponent-mode", "self_play",
"--velocity-to-ball-weight", "0.08", # up from 0.05
"--ball-distance-penalty", "0.01", # up from 0.006
"--ball-touch-reward", "0.7", # up from 0.5
"--airborne-penalty", "0.003",
"--ball-velocity-to-goal-weight", "0.06", # up from 0.02 (0.004 default)
"--goal-reward", "80", # up from 60 (40 default)
"--draw-penalty", "5",
*_UNMASK_RAMP_SHARED_FLAGS,
"--vertical-ramp", "0.25",
"--pitch-roll-ramp", "0.25",
"--airborne-penalty", "0.0",
],
# 2026-07-29: generation 2's own first two attempts (both independently
# resumed from FOUNDATION_EXPERIMENT under reset_retry_checkpoint,
# identical flags/budget) landed at 32% and 27% win rate — a real
# regression either way, but with enough run-to-run spread that
# "identical fresh restart" isn't a controlled test of anything. The
# first attempt's own trajectory (ep_rew_mean climbing from -10.86
# toward ~0 by the 240M-step cutoff, briefly touching positive) looked
# closer to convergence than the second's, so rather than another
# independent restart from foundation, retries now continue *that*
# attempt's own checkpoint for another full timesteps budget — an
# actual test of "did it just need more time," not another coin flip.
# A third, unrelated attempt crashed immediately (see train.py's
# GoalRateCallback KeyError fix) before contributing any real signal
# and was discarded rather than counted.
"grounded": False,
"timesteps": 240_000_000, # ~24h at the standing n-parallel/speedup (20M took ~2h)
"gated": False,
"timesteps": 40_000_000, # ~4h at the standing n-parallel/speedup (20M took ~2h)
# Stage 0 MUST set this explicitly — resume_checkpoint()'s stage-0
# branch returns None (train from scratch) without it.
"resume_from_experiment": FOUNDATION_EXPERIMENT,
},
{
"name": "unmask-ramp50",
# Step 2/4: 50% authority. airborne_penalty at 1/3 of its final
# value — enough to start discouraging unproductive altitude, not
# enough to fight the still-partial vertical axis outright.
# resume_from_experiment deliberately omitted: chains from
# ramp25's pass via _resume_source_experiment's default
# (_passing_experiment_for_stage) — do not add an override here.
"flags": [
*_UNMASK_RAMP_SHARED_FLAGS,
"--vertical-ramp", "0.5",
"--pitch-roll-ramp", "0.5",
"--airborne-penalty", "0.001",
],
"gated": False,
"timesteps": 40_000_000,
},
{
"name": "unmask-ramp75",
# Step 3/4: 75% authority, airborne_penalty at 2/3 of its final
# value. Also chains automatically — no resume_from_experiment.
"flags": [
*_UNMASK_RAMP_SHARED_FLAGS,
"--vertical-ramp", "0.75",
"--pitch-roll-ramp", "0.75",
"--airborne-penalty", "0.002",
],
"gated": False,
"timesteps": 40_000_000,
},
{
"name": "unmask",
# Step 4/4, the measured transition: full ramp (100% authority),
# airborne_penalty at its full value — behaviourally and eval-wise
# identical to generation 2's "unmask" stage's flags/config, so
# this stage's result is a direct, apples-to-apples comparison
# against the 3 failed all-or-nothing attempts (same reference,
# same opponent mode, same budget). Gated (default True): evaluated
# against FOUNDATION_EXPERIMENT exactly like every prior attempt.
#
# No resume_from_experiment here (deliberately, unlike generation
# 2's single-stage version) — this stage chains from ramp75's own
# checkpoint via the default resume path, not back to
# FOUNDATION_EXPERIMENT; only reference_experiment (the *eval*
# opponent) stays FOUNDATION_EXPERIMENT.
"flags": [
*_UNMASK_RAMP_SHARED_FLAGS,
"--vertical-ramp", "1.0",
"--pitch-roll-ramp", "1.0",
"--airborne-penalty", "0.003",
],
"grounded": False,
"timesteps": 240_000_000, # unchanged from the 3 failed attempts — same budget for a controlled comparison
"reference_experiment": FOUNDATION_EXPERIMENT,
# On retry, reset to the clean ramp75 checkpoint rather than
# compounding a failed full-ramp attempt's own drift — mirrors the
# generation 1 -> 2 postmortem (blind same-checkpoint retries only
# made things worse).
"reset_retry_checkpoint": True,
},
]
@@ -365,6 +411,24 @@ def main():
experiment = run_stage_attempt(stage_index, attempt, args)
last_experiment = experiment
if not STAGES[stage_index].get("gated", True):
# Ungated ramp waypoint (see the unmask-ramp2X stages): trains,
# checkpoints, and always advances — no eval, no regression
# gate, nothing to retry against. See TRAINING.md's
# "Generation 3" section.
print(f"{experiment}: ungated ramp waypoint — skipping eval, advancing unconditionally")
state["log"].append({
"stage_index": stage_index, "experiment": experiment, "attempt": attempt,
"decision": "pass", "note": "ungated ramp waypoint (no eval)",
})
state["stage_index"] += 1
state["attempt"] = 0
state["status"] = "in_progress"
save_state(state)
commit_progress(experiment)
continue
reference = reference_bot(stage_index)
record = evaluate_attempt(experiment, reference, EVAL_EPISODES, stage_index)
decision = decide(record)
training/curriculum_state.json
+3 -52
View File
@@ -1,55 +1,6 @@
{
"stage_index": 0,
"attempt": 2,
"status": "blocked",
"log": [
{
"stage_index": 0,
"experiment": "20260726-1904-curric-s1-unmask",
"attempt": 0,
"eval": {
"timestamp": "2026-07-27T23:30:59+00:00",
"model_a": "/home/jcreek/ai-training/CosmicClash/Game/bots/20260726-1904-curric-s1-unmask.json",
"model_b": "/home/jcreek/ai-training/CosmicClash/Game/bots/curric-s5-aggression.json",
"episodes": 100,
"wins_a": 32,
"wins_b": 50,
"draws": 18,
"win_rate_a": 0.32
},
"decision": "fail"
},
{
"stage_index": 0,
"experiment": "20260729-0837-curric-s1-unmask-retry1",
"attempt": 1,
"eval": {
"timestamp": "2026-07-30T11:24:45+00:00",
"model_a": "/home/jcreek/ai-training/CosmicClash/Game/bots/20260729-0837-curric-s1-unmask-retry1.json",
"model_b": "/home/jcreek/ai-training/CosmicClash/Game/bots/curric-s5-aggression.json",
"episodes": 100,
"wins_a": 28,
"wins_b": 57,
"draws": 15,
"win_rate_a": 0.28
},
"decision": "fail"
},
{
"stage_index": 0,
"experiment": "20260730-1224-curric-s1-unmask-retry2",
"attempt": 2,
"eval": {
"timestamp": "2026-07-31T15:07:16+00:00",
"model_a": "/home/jcreek/ai-training/CosmicClash/Game/bots/20260730-1224-curric-s1-unmask-retry2.json",
"model_b": "/home/jcreek/ai-training/CosmicClash/Game/bots/curric-s5-aggression.json",
"episodes": 100,
"wins_a": 31,
"wins_b": 56,
"draws": 13,
"win_rate_a": 0.31
},
"decision": "fail"
}
]
"attempt": 0,
"status": "in_progress",
"log": []
}
training/curriculum_state_gen2.json
+55
View File
@@ -0,0 +1,55 @@
{
"stage_index": 0,
"attempt": 2,
"status": "blocked",
"log": [
{
"stage_index": 0,
"experiment": "20260726-1904-curric-s1-unmask",
"attempt": 0,
"eval": {
"timestamp": "2026-07-27T23:30:59+00:00",
"model_a": "/home/jcreek/ai-training/CosmicClash/Game/bots/20260726-1904-curric-s1-unmask.json",
"model_b": "/home/jcreek/ai-training/CosmicClash/Game/bots/curric-s5-aggression.json",
"episodes": 100,
"wins_a": 32,
"wins_b": 50,
"draws": 18,
"win_rate_a": 0.32
},
"decision": "fail"
},
{
"stage_index": 0,
"experiment": "20260729-0837-curric-s1-unmask-retry1",
"attempt": 1,
"eval": {
"timestamp": "2026-07-30T11:24:45+00:00",
"model_a": "/home/jcreek/ai-training/CosmicClash/Game/bots/20260729-0837-curric-s1-unmask-retry1.json",
"model_b": "/home/jcreek/ai-training/CosmicClash/Game/bots/curric-s5-aggression.json",
"episodes": 100,
"wins_a": 28,
"wins_b": 57,
"draws": 15,
"win_rate_a": 0.28
},
"decision": "fail"
},
{
"stage_index": 0,
"experiment": "20260730-1224-curric-s1-unmask-retry2",
"attempt": 2,
"eval": {
"timestamp": "2026-07-31T15:07:16+00:00",
"model_a": "/home/jcreek/ai-training/CosmicClash/Game/bots/20260730-1224-curric-s1-unmask-retry2.json",
"model_b": "/home/jcreek/ai-training/CosmicClash/Game/bots/curric-s5-aggression.json",
"episodes": 100,
"wins_a": 31,
"wins_b": 56,
"draws": 13,
"win_rate_a": 0.31
},
"decision": "fail"
}
]
}
training/train.py
+6 -7
View File
@@ -112,13 +112,12 @@ def parse_args():
curriculum.add_argument("--kickoff-chance", type=float, default=None, help="Overrides kickoff_state_chance")
curriculum.add_argument("--near-goal-chance", type=float, default=None, help="Overrides ball_near_goal_chance")
curriculum.add_argument(
"--allow-vertical", action=argparse.BooleanOptionalAction, default=None, help="Allow vertical thrust (default true)"
"--vertical-ramp", type=float, default=None,
help="0.0-1.0: fraction of vertical thrust that reaches the ship (locomotion-unmask ramp; default 1.0)",
)
curriculum.add_argument(
"--allow-pitch-roll",
action=argparse.BooleanOptionalAction,
default=None,
help="Allow pitch/roll rotation (default true)",
"--pitch-roll-ramp", type=float, default=None,
help="0.0-1.0: fraction of pitch/roll rotation that reaches the ship (locomotion-unmask ramp; default 1.0)",
)
curriculum.add_argument(
"--velocity-to-ball-weight", type=float, default=None,
@@ -159,8 +158,8 @@ def _curriculum_kwargs(args) -> dict:
"attack_goal_bias": args.attack_goal_bias,
"kickoff_state_chance": args.kickoff_chance,
"ball_near_goal_chance": args.near_goal_chance,
"ai_allow_vertical": args.allow_vertical,
"ai_allow_pitch_roll": args.allow_pitch_roll,
"ai_vertical_ramp": args.vertical_ramp,
"ai_pitch_roll_ramp": args.pitch_roll_ramp,
"ai_velocity_to_ball_weight": args.velocity_to_ball_weight,
"ai_ball_distance_penalty": args.ball_distance_penalty,
"ai_ball_touch_reward": args.ball_touch_reward,