jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-11 08:23:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(training): curriculum generation 4 — MultiDiscrete action space redesign

Browse Source 
Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.

Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.

- Game/scripts/ship_action_codec.gd (new): single source of truth for a
  per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
  by training and in-game inference, replacing the continuous Gaussian.
  thrust_y's bins are deliberately asymmetric so a random policy drifts
  through the volume instead of floor-pinning. Legacy continuous decode
  (ai_ship_controller.gd's old logic) preserved verbatim so every
  pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
  unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
  so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
  vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
  lowered 4x (aerial approaches require pitching); flight telemetry
  (airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
  and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
  high, ships low, kept clear of walls) so aerial practice is forced by the
  environment instead of relying on reward-driven exploration alone; snapshot
  terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
  SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
  now-meaningless --reset-std; new EntropyFloorCallback (a persistent
  per-rollout ent_coef controller replacing the one-shot std-reset shock)
  and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
  --tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
  index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
  grounded stage, full action space live from step 1; deletes generation
  1-3's checkpoint-lineage machinery (nothing to resume from); final report
  evaluates against both promoted/easy.json and the new
  promoted/reference-grounded.json (a copy of curric-s5-aggression, the
  strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
  intermediate checkpoint files a single stage was writing (~500MB ->
  ~0.2MB per run); requirements.txt pinned (behaviour here now depends on
  specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
  mismatch before it silently corrupts 24h of training.

Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.

See TRAINING.md's "Generation 4" section for the full design writeup.
This commit is contained in:
Josh Creek
2026-08-04 23:27:57 +01:00
parent 8551d9e835
commit 1811e9333e
19 changed files with 1259 additions and 369 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Game/scripts/ship_ai_controller.gd
+92 -58
View File
@@ -7,10 +7,11 @@ extends AIController3D
# the trainer are written into an RLShipController, which the ship pulls like
# any other controller.
#
# Action space is ShipAction verbatim: 6 continuous axes (thrust xyz,
# rotation xyz, each -1..1) + binary turbo. ShipAction axes are ship-local
# (body frame), so they need no team mirroring — only observations do
# (see ShipObservations.canon).
# Action space/layout is owned by ShipActionCodec (get_action_space/
# set_action just delegate to it) — see that file for the per-axis
# MultiDiscrete design and why. ShipAction axes are ship-local (body frame),
# so they need no team mirroring — only observations do (see
# ShipObservations.canon).
# Reward shaping weights. Dense terms accrue per physics tick (60 sim-ticks
# per sim-second); event terms fire once. Exported so tuning needs no code
@@ -54,9 +55,13 @@ extends AIController3D
# relative to the (then far weaker) ball-seeking shaping.
@export var wall_contact_penalty := 0.0025
# Per-tick penalty for not being upright, scaled by tilt: 0 when flat, full
# value (-0.12/s) when inverted. A penalty rather than an upright bonus so a
# flat, idle ship farms nothing.
@export var tilt_penalty := 0.002
# value when inverted. A penalty rather than an upright bonus so a flat, idle
# ship farms nothing. Lowered 4x for curriculum generation 4 (was 0.002,
# -0.12/s): a genuine aerial approach to a high ball requires pitching, and
# the old value quietly opposed the exact behaviour generation 4 is trying
# to teach. Not removed outright — an always-inverted bot still looks bad in
# a shipped game.
@export var tilt_penalty := 0.0005
# Per-tick bonus for own speed: 0 stationary, full value (+0.24/s) at
# max_speed. Run07 lesson: after the kickoff flurry both ships parked next to
# a cornered ball — with every other dense term near zero there, standing
@@ -79,27 +84,15 @@ extends AIController3D
# unaffected; the floor-lock curriculum stage turns it on.
@export var airborne_penalty := 0.0
# Locomotion curriculum: scales how much of the corresponding action axes
# actually reaches the ship, from 0.0 (fully discarded, grounded-only) to
# 1.0 (full effect) — this scales the *effect* of thrust.y/rotation.x/
# rotation.z in set_action, not the action space's shape: the policy always
# outputs values for these axes (always contributing to PPO's entropy/log-
# prob), they're just attenuated here, so checkpoints stay resumable across
# ramp values.
#
# A hard 0/1 flip (the original bool mask) let PPO's action-distribution
# std collapse to ~0.13-0.15 within the first ~10% of steps, before the
# policy ever meaningfully explored the newly-unmasked axes — 3 independent
# 240M-step attempts at the all-or-nothing flip all landed at a stable
# ~28-32% win rate vs curric-s5-aggression (see TRAINING.md's generation 3
# section). A gradual ramp across several short curriculum stages, each
# resuming from the previous ramp value's checkpoint, lets the policy adopt
# each axis incrementally instead of all at once.
@export_range(0.0, 1.0) var vertical_ramp := 1.0
@export_range(0.0, 1.0) var pitch_roll_ramp := 1.0
# Height above which a touch counts toward air_touch_fraction telemetry
# (see get_info) — not a reward term itself, see set_action/get_info's
# comments on why generation 4 deliberately does not add a standalone
# air-touch reward.
const AIR_TOUCH_HEIGHT := 5.0
# Contact normals with y above this are floor contact (exempt from the wall
# penalty); below it they read as wall (sideways) or ceiling (downward).
# Mirrors ShipObservations.FLOOR_NORMAL_MIN_Y (see that file's comment).
const FLOOR_NORMAL_MIN_Y := 0.7
# Longest possible ship-to-ball separation: the enclosure's interior diagonal.
@@ -125,8 +118,32 @@ var attack_goal_position: Vector3
# (true on goal, false on timeout) rather than toggle, so no reset is needed.
var goal_scored_this_episode := false
# Set only in the timeout branch (TrainingMode._physics_process), never on a
# goal — a goal is a genuine terminal (V(s)=0 is correct there); a timeout
# is an artificial episode boundary the value function should be bootstrapped
# through instead (see get_info). Same "always overwritten by both call
# sites, never cleared in reset()" pattern as goal_scored_this_episode above,
# for the same reason.
var truncated_this_episode := false
var terminal_obs: Array = []
var _ticks_since_ball_touch := 1 << 30 # large so the first touch always pays
# Flight telemetry (see get_info) — leading indicators for whether the
# policy is actually using its vertical/pitch-roll authority, visible from
# the first rollout instead of only in a win-rate number measured a full
# training run later. Accumulated per-tick, reset() zeroes them each episode;
# get_info() reports the running fraction/mean so the *final* tick of an
# episode (the one VecMonitor's info_keywords captures) holds the whole
# episode's aggregate.
const AIRBORNE_ALTITUDE_THRESHOLD := 3.0
var _telemetry_ticks := 0
var _airborne_ticks := 0
var _altitude_sum := 0.0
var _thrust_y_sum := 0.0
var _touches := 0
var _air_touches := 0
# Wire up references after the ship is spawned. `attack_goal` is the goal
# this ship scores into (goal.team == opponent's team).
@@ -138,10 +155,8 @@ func setup(p_ship: Ship, p_rl_controller: RLShipController, p_ball: RigidBody3D,
attack_goal_position = p_attack_goal_position
init(ship)
# Contact monitoring for the ball-touch reward (training-only cost;
# the shipped game leaves contact_monitor off).
ship.contact_monitor = true
ship.max_contacts_reported = 8
# ship.contact_monitor is on unconditionally (see ship.gd) since
# ShipObservations now reads it for every ship, not just training agents.
ship.body_entered.connect(_on_ship_body_entered)
@@ -153,36 +168,48 @@ func get_reward() -> float:
return reward
# Symmetric across both self-play agents: reports whether this episode ended
# in a goal at all, not which team scored — a clean "goal rate" signal
# distinct from rollout/ep_rew_mean, which mixes this with dense shaping
# (ball chasing/touching). See train.py's GoalRateCallback.
# Symmetric across both self-play agents. "goal_scored": whether this
# episode ended in a goal at all (not which team) — a clean "goal rate"
# signal distinct from rollout/ep_rew_mean, which mixes this with dense
# shaping (ball chasing/touching); see train.py's GoalRateCallback.
# "truncated"/"terminal_obs": only present on a timeout tick — remapped by
# cosmic_env.py into SB3's expected "TimeLimit.truncated"/
# "terminal_observation" keys so PPO bootstraps V(s) through episode
# timeouts instead of treating every 30s draw as a true terminal state (a
# real, previously-unnoticed bug independent of the action-space work — see
# TRAINING.md). Flight telemetry fields are leading indicators for
# generation 4's core hypothesis (see train.py's FlightTelemetryCallback).
func get_info() -> Dictionary:
return {"goal_scored": goal_scored_this_episode}
var info := {"goal_scored": goal_scored_this_episode}
if truncated_this_episode:
info["truncated"] = true
info["terminal_obs"] = terminal_obs
if _telemetry_ticks > 0:
info["airborne_fraction"] = float(_airborne_ticks) / _telemetry_ticks
info["mean_altitude"] = _altitude_sum / _telemetry_ticks
info["vertical_thrust_mean"] = _thrust_y_sum / _telemetry_ticks
if _touches > 0:
info["air_touch_fraction"] = float(_air_touches) / _touches
return info
func get_action_space() -> Dictionary:
return {
"thrust": {"size": 3, "action_type": "continuous"},
"rotation": {"size": 3, "action_type": "continuous"},
"turbo": {"size": 2, "action_type": "discrete"},
}
return ShipActionCodec.action_space_dict()
func set_action(action) -> void:
var thrust: Array = action["thrust"]
var rot: Array = action["rotation"]
var thrust_y: float = thrust[1] * vertical_ramp
var pitch: float = rot[0] * pitch_roll_ramp
var roll: float = rot[2] * pitch_roll_ramp
rl_controller.action.thrust = Vector3(thrust[0], thrust_y, thrust[2])
rl_controller.action.rotation = Vector3(pitch, rot[1], roll)
rl_controller.action.turbo = int(action["turbo"]) == 1
rl_controller.action = ShipActionCodec.from_indices(action)
func reset():
super()
_ticks_since_ball_touch = 1 << 30
_telemetry_ticks = 0
_airborne_ticks = 0
_altitude_sum = 0.0
_thrust_y_sum = 0.0
_touches = 0
_air_touches = 0
func _physics_process(delta):
@@ -238,19 +265,17 @@ func _physics_process(delta):
var height := maxf(ship.global_position.y, 0.0)
reward -= airborne_penalty * height / ArenaBoundary.INNER_HEIGHT
# Flight telemetry accumulation (see get_info) — not reward, just
# observation of what the policy is actually doing this tick.
_telemetry_ticks += 1
_altitude_sum += ship.global_position.y
if ship.global_position.y > AIRBORNE_ALTITUDE_THRESHOLD:
_airborne_ticks += 1
_thrust_y_sum += rl_controller.action.thrust.y
func _wall_or_ceiling_contact() -> bool:
var state := PhysicsServer3D.body_get_direct_state(ship.get_rid())
if state == null:
return false
for i in state.get_contact_count():
if not state.get_contact_collider_object(i) is ArenaBoundary:
continue
# Normal points from the surface into the ship: floor ≈ +Y (exempt),
# anything flatter or downward is a wall or the ceiling.
if state.get_contact_local_normal(i).y < FLOOR_NORMAL_MIN_Y:
return true
return false
return ShipObservations.contact_normal(ship) != Vector3.ZERO
func _on_ship_body_entered(body: Node) -> void:
@@ -264,3 +289,12 @@ func _on_ship_body_entered(body: Node) -> void:
alignment = clampf(ball.linear_velocity.normalized().dot(to_goal.normalized()), 0.0, 1.0)
reward += ball_touch_reward * lerpf(ball_touch_direction_floor, 1.0, alignment)
_ticks_since_ball_touch = 0
# Telemetry only (see get_info's air_touch_fraction) — not a reward term.
# Generation 4 deliberately doesn't reward high touches directly (see
# TRAINING.md's "why no air-touch reward" note); this just measures
# whether the air-drill state setter is producing genuine aerial
# contests, so a future decision to add one is data-driven.
_touches += 1
if ball.global_position.y > AIR_TOUCH_HEIGHT:
_air_touches += 1