Playing with a gamepad did not work: all six move_* actions had no joypad
event at all, so a pad could yaw/pitch/roll/turbo but could not translate.
Nothing caught it because every action existed and the game booted fine —
no assertion checked that an action is reachable on *both* devices.
Controller layout, on the 6DOF convention (left stick aims, right stick
translates), using all six of the pad's analog axes for the ship's six
degrees of freedom:
left stick yaw + pitch right stick strafe + vertical
LB / RB roll RT / LT forward / back
L3 turbo R3 ball camera
Input is now read with Input.get_axis instead of is_action_pressed, so
triggers and sticks are proportional. Keyboard values are unchanged.
Three rotation bugs found by measuring a real Ship rather than reading the
code:
- apply_torque() is world-space and the torque was never rotated into the
hull's frame (unlike thrust, which uses -ship_basis.z). Roll input became
pitch after a 90 degree turn and inverted at 180, so the controls were
correct flying up-field and backwards flying back.
- ship.tscn's inertia is Vector3(7, 1, 7) but a flat torque was applied to
every axis, giving yaw 7x the angular acceleration of pitch and roll
(172 deg/s vs 52). Torque is now scaled per-axis by inertia, so
rotation_acceleration means rad/s^2 and all three axes match. Yaw is
unchanged.
- pitch_down pitched the nose UP: get_axis's arguments were reversed, so
the I/K keys and the stick each did the opposite of their label.
Menus were unusable on a pad for a separate reason: Godot 4.7 gives
ui_up/down/left/right joypad events by default but leaves ui_accept and
ui_cancel with none (verified against a pristine project), so a controller
could move the highlight and never press anything. A confirms and B goes
back. Gameplay exits on a new leave_gameplay action (Escape / Start) rather
than ui_cancel, so carrying B for menus cannot abandon a live match.
Bindings for both devices are rebindable in Settings -> Controls, persisted
to user://input.cfg — a separate file from settings.cfg because
VideoSettings.save() rewrites that file wholesale and would drop any
section it does not know about. project.godot stays the source of truth for
defaults; overrides are only ever a delta on top of a boot-time snapshot.
Verified: 268 unit tests, the ENet integration gate, and a 16-sample
before/after comparison of networked prediction residuals showing the
physics change does not regress them (median 0.083m -> 0.065m).
Note for follow-up: every policy in Game/bots/ was trained against the old
sluggish, world-axis rotation and will over-rotate until retrained.
multiplayer-next.md was a 1662-line mix of standing architecture spec
and task-completion tracking, most of which was dense per-task DONE
evidence for finished Phases 0-6. Split it:
- MULTIPLAYER_SPEC.md (new): the locked architecture decisions, wire
format, server-side input handling, prediction/reconciliation,
latency/frame-rate budget, and match lifecycle state machine -
standing design reference, not task-tracked.
- multiplayer-next.md (trimmed 1662 -> ~370 lines): only outstanding
work remains - §0 status, §7 Phase 7/8 task tables condensed to
"what's left" per task, §8-11 reference material (refactoring notes,
gotchas, testing, flagged items). Phases 0-6 collapsed to a pointer
at git history instead of ~500 lines of DONE evidence.
Also:
- Repointed every `multiplayer-next.md §N` code comment (N 1-6) across
Game/scripts, Game/tools and Game/tests to MULTIPLAYER_SPEC.md, since
those sections moved. Task-number references (`task N.N`, §7-11)
correctly still point at multiplayer-next.md.
- Updated CLAUDE.md's doc index and docs/TECH_STACK.md's spec-section
citations to match.
- TODO.md: added a "what's left to actually finish multiplayer
(human-actionable)" checklist pulled from multiplayer-next.md §0 and
docs/MATCHMAKING.md - things that need a person (hardware, a design
decision, a Steam App ID, hands on a controller), not more agent code.
multiplayer-todo.md and multiplayer-next.md tracked overlapping
information in two places. Fold everything into multiplayer-next.md
(architecture decisions, wire format, task breakdown with checkboxes,
gotchas list, testing notes) and delete multiplayer-todo.md. Section
numbers are unchanged, so existing code comments citing them by
section/task number still resolve; update every such reference to
point at the new filename.
Implements the rest of the §6.2 lifecycle on top of 5.1's state machine.
5.3 kickoff: the server resets every body and broadcasts the RESULTING
transforms, never a seed - §1's locked decision, because shared-seed
determinism needs both sides to consume the RNG stream in identical
order forever and the first randf() added to the reset path desyncs
silently. Countdown is derived from server_tick on both peers, and a
kickoff that lands after its own resume tick applies immediately and
skips the countdown rather than scheduling into the past.
5.4 goals: goal_scored(scoring_team, score, goal_tick, resume_tick).
Score is authoritative at sensor time, before any presentation. The
reset moved OUT of the sensor path and into the kickoff at resume_tick,
which is what stops the server resetting while clients are still
mid-celebration. Engine.time_scale is never touched.
5.2 clock: tick-derived, no Timer and no _process polling. The goal
pause shifts the absolute end_tick by (resume_tick - goal_tick) rather
than pausing anything, so no float drift accumulates across goals.
5.5 full time: clock expiry -> FULL_TIME -> sudden death on a draw or
RESULTS, golden goal in overtime, then LOBBY on both peers - clients
return to the lobby, not the main menu. get_tree().paused is never used.
Four bugs found and fixed while building this, each by a failing run
rather than by inspection:
- Tick order was load-bearing: _update_kickoff_countdown() clears the
same _kickoff_resume_tick that _update_match_state() reads to leave
WARMUP, so running the countdown first wiped the transition condition
and the match sat frozen in WARMUP forever.
- _apply_match_state resets _state_deadline_tick on every transition, so
a GOAL_PAUSE deadline assigned before _set_match_state was wiped and
the match never resumed. Deadlines are now owned by _apply_match_state.
- Freezing "all bodies" is wrong on a client. Remote ships and the ball
are permanently FREEZE_MODE_KINEMATIC and transform-driven; freezing
them all unfroze the remote ones on the way back out, so they fell
under gravity while the interpolator fought them - 210 hard snaps and
an infinite p99. A client now freezes only the one body it simulates.
- A frozen body never runs _integrate_forces, so the queued kickoff
teleport was stranded by an immediate set_deferred("freeze", true).
Freeze now happens on a strictly later tick, the same pattern Phase 2
used for _pending_reset_gen_bump_tick.
Prediction and reconciliation are suspended while the match is not live:
during a countdown or goal pause the local ship is frozen on both peers,
and running delta transport over those frozen states produced a p95
position error of 2.4e10 m. Input keeps flowing so the server's jitter
buffer does not starve into `stalled`.
Also fixed: a kickoff can arrive before match_config, and body order is
slot order - applying it early placed the BALL at positions[0], on top
of the first ship, which the ball-cam reported as "target vector can't
be zero" 95 times. It is now held until the roster exists.
Test changes: the ball-contact scenario steered by a hand-tuned fixed
heading, which 5.3 broke because kickoff applies KICKOFF_YAW_JITTER - it
flew past the ball in 3/3 runs. It now closes the loop on the actual
bearing using real input actions. Assertions that read a frozen ship
(freeze, thrust) are gated on the match being live, and the hooks now
survive the scene teardown at RESULTS instead of hanging on freed
objects for the full timeout.
Regression: 81 unit tests; free-flight LAN p99 0.143m and 80±20ms, both
0 hard snaps; transition gate 0.00%; ball contact 3/3; two-bot CI.
Closes Phase 4's outstanding action-sequence-correctness invariant, then
fixes two server-side bugs an adversarial review of that work uncovered.
Server simulation, bot observations, collision resources and tick rate are
unchanged: the server_physics_parity trace is byte-for-byte identical to
HEAD across 360 ticks including both ships' full observation vectors.
4.11 - prediction history filed under the ISSUING sequence
_send_local_input filed each post-step predicted state under the timeline's
estimate of the sequence the server would consume this tick, trailing
issuance by input_lead. The body had integrated the intent issued under
_input_seq, so predicted[S] held "state after the intent from now" while
the server's authority for S is "state after action(S)". They agree only
while the stick is still. Filing under _input_seq costs nothing: which
action the ship uses is decided in LocalNetShipController.get_action() and
is untouched.
Every prior Phase 4 gate held its input steady, and a steady input cannot
falsify a sequence label - the 60s runs honestly reported marker=0/3784.
New --exercise-input-transitions role toggles thrust every 6 ticks; it is
the only gate that can catch a label regression. Verified non-vacuous: the
old label fails it at 50%.
4.12 - issued-but-unsimulated sequences, and the release path
An attack (delta > 1) issues and sends several sequences for one local
physics step. Those gap sequences had no recorded prediction, so a server
ack of one reported missing_not_recorded - indistinguishable from ring
loss, costing a teleport and resync suppression several times a minute.
They are now recorded stateless via record_unsimulated() and answered with
a new "skip" decision mode. Free-flight hard snaps: 25/8/4 -> 0/0/0.
A release (delta == 0) re-recorded at the unchanged _input_seq, filing the
current intent under a sequence that went out carrying a different action;
LocalInputTimeline deliberately refuses to mutate an issued sequence, so
the ring contradicted the wire. Recording is now skipped on release ticks.
4.13 - two Phase 3 bugs silently killing player input
(a) InputJitterBuffer.consume() advanced last_applied_seq on every tick
including a starve. Since ingest() discards seq <= last_applied_seq, one
starve on a sequence the client had not sent yet stranded the stream one
ahead of arrivals permanently - both sides advancing in lockstep, every
honest packet discarded on arrival. The client's own input_lead release is
enough to trigger it, so input died for ~30 ticks roughly every 6.5s on a
clean LAN. Now only gives up on a sequence once strictly newer data proves
it lost. Silent-client stall and ring-overflow resync are unchanged.
(b) The seq-range guard bounded incoming seq against highest_ingested_seq,
which only advances inside ingest(), which that guard gates. After a ~2s
host hitch every packet was rejected forever with no diagnostic (600+
consecutive rejections reproduced via SIGSTOP). Third iteration of this
guard; each previous version bounded against a value only the accepted
path could advance. Adds an escape after 10 consecutive rejections, which
grants an attacker nothing the rate limiter does not already bound.
(c) The transitions gate reported PASS at 3.76% while input was completely
dead, because suppression stops _record_metrics - a worse outage yields
fewer samples and a LOWER rate. Now scales the required sample count with
run length and asserts the wire's server_stalled bit. Reverting both fixes
makes it fail at samples 292/600, server_stalled=true, input_lead=12.
Fixing (a) also explained a residual the review had already traced: 151 of
151 action-marker mismatches were the server repeating a stale action on a
starve, not a prediction defect. Marker is now 0.00% in all three
conditions (was 1.7-2.5%), and free-flight p99 improved to
0.141/0.168/0.154m from 0.170/0.176/0.184m.
Two pre-existing test defects fixed alongside: the ball gate asserted
RTT-masking on a link with no RTT (flaked 2 in 5; now asserted only at
rtt >= 20ms, 5/5 under latency), and the two-bot CI compared scores across
a 3-5s window (now polls the scores the server actually held; note
score_changed is emitted only on the client path).
QA: 72 unit tests; 60s free-flight at LAN/80+-20ms/5% loss; transition
gate in all three; 2.0s and 3.5s host-freeze recovery; ball contact x5;
two-bot CI x3; all three abuse roles; net/match_net/clock/lobby smokes.
Phase 4 sign-off still pending a human playtest at ~100ms RTT - the
milestone asks how it feels, which no gate here answers.
Lands the non-networked Phase 0 tasks from multiplayer-todo.md (ship/camera/
arena refactors, sim constants, background FPS handling) plus a first pass
at exposing graphics/performance settings (presets, resolution scaling,
vsync, FPS cap, perf overlay) and a GPU profiling harness for the
real-hardware follow-up in task 0.15b.
Six rounds of reward shaping (~700M steps) failed to produce upright ground
driving. A critical review of the simulation rather than the reward found
why:
1. The hull was a 1x1x4 box with inertia (1,1,1) and no restoring torque
anywhere, so belly-down and rolled-90 were geometrically identical
resting states. "Upright" was not a physically distinguished state at
all - the reward was paying for a property the simulation did not have.
2. ~65% of episodes spawned ships via _random_position, which samples Y
uniformly over the full 18m volume (mean ~8.7m). The measured
airborne_fraction ~0.44 was largely that spawn distribution, and every
ground-handling term fades out above 3m, so the shaping being tuned
barely ever applied.
3. air_drill_chance 0.20 spawned deliberately unreachable-without-climbing
states in the stage meant to teach ground driving, and its own
air_touch_fraction (0.0002) shows the drills were never solved.
Fixes land in the physics and the task distribution, not the reward:
- ship.tscn: hull 1x1x4 -> 1.6x0.6x4 so it has one stable resting face;
inertia (1,1,1) -> (7,1,7), physically correct for the hull, making
tumbling reluctant while keeping yaw snappy.
- ship.gd: new altitude-faded righting torque (spring-damper toward
belly-down, faded out by 3m so aerials keep full attitude freedom).
This is the grav-plating analogue of Rocket League's auto-righting and
helps human pilots land cleanly too.
- training_mode.gd: new ground_start_chance branch spawning ships level and
resting on the floor with a floor-level ball - the state the handling
stage's rewards are actually written for.
- generation5.py: ground-start-chance 0.50, air-drill-chance 0.20 -> 0.0.
Reward terms are left exactly as they were; they should finally pull in a
direction the ship can go.
Extends ShipObservations beyond the old self+1-opponent layout to padded
teammate/opponent arrays (MAX_TEAMMATES=4, MAX_OPPONENTS=5, SIZE=83),
zero-filling slots past the real roster size the same way the old single-
opponent slot was zero-filled when absent.
Slot stability across ticks requires a persistent identity: Ship gains
spawn_index (set once by GameMode.spawn_ship, never reassigned — there's no
despawn path anywhere in this codebase, so a roster is fixed for the whole
episode/match). ai_ship_controller.gd's opponent discovery is rewritten from
"first non-self ship" to classify every other ship by team and sort by
spawn_index; training_mode.gd/ship_ai_controller.gd carry the equivalent
sorted lists through the training path so both agree on slot assignment for
the same roster.
training_mode.gd and match_mode.gd both gain a team_size export (default 1,
so every existing curriculum script and match keeps today's 1v1 behaviour
unchanged). This is plumbing only: no 2v2+ curriculum or reward design, and
no match-mode UI to pick team size, has been done yet. The two checkpoints
in Game/bots/promoted/ are fitted to the old 35-float layout and are not
migrated — expected to go stale until the next training run.
drag_coefficient/angular_drag/the idle angular-drag multiplier were applied
once per physics tick with no delta scaling, correct only because
project.godot never pins physics/common/physics_ticks_per_second and
Godot's default happens to be 60. _tick_scaled(k, state.step) makes the
decay rate invariant to tick rate instead. Also promotes the previously
hardcoded 0.9 idle angular-drag literal to an export, matching its sibling.
Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.
Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.
- Game/scripts/ship_action_codec.gd (new): single source of truth for a
per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
by training and in-game inference, replacing the continuous Gaussian.
thrust_y's bins are deliberately asymmetric so a random policy drifts
through the volume instead of floor-pinning. Legacy continuous decode
(ai_ship_controller.gd's old logic) preserved verbatim so every
pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
lowered 4x (aerial approaches require pitching); flight telemetry
(airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
high, ships low, kept clear of walls) so aerial practice is forced by the
environment instead of relying on reward-driven exploration alone; snapshot
terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
now-meaningless --reset-std; new EntropyFloorCallback (a persistent
per-rollout ent_coef controller replacing the one-shot std-reset shock)
and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
--tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
grounded stage, full action space live from step 1; deletes generation
1-3's checkpoint-lineage machinery (nothing to resume from); final report
evaluates against both promoted/easy.json and the new
promoted/reference-grounded.json (a copy of curric-s5-aggression, the
strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
intermediate checkpoint files a single stage was writing (~500MB ->
~0.2MB per run); requirements.txt pinned (behaviour here now depends on
specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
mismatch before it silently corrupts 24h of training.
Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.
See TRAINING.md's "Generation 4" section for the full design writeup.
ship.gd's controllerless path and player_ship_controller.gd each
allocated a fresh ShipAction every physics tick; ai_ship_controller.gd
and rl_ship_controller.gd already avoid this via a persistent member.
Convert both to reuse a member instance, matching the existing
full-field-overwrite convention (rather than +=/-= off a fresh zero).
Also drop the completed items from TODO.md.
ship.gd, HUDController.gd, and goal.gd each declared their own TEAM_COLORS,
arena_boundary.gd/arena_deck.gdshader had a third pair, and HUD.tscn baked
in a fourth (hardcoded "BLUE"/"ORANGE" labels) — nose, goal rim, end zone,
and scoreboard all rendered different blues. New scripts/team_colors.gd
(class_name TeamColors) is now the single source every one of those reads
from, and team identity moves to purple/green.
Hull/Canopy/EngineGlowL/EngineGlowR are runtime-baked into one ArrayMesh
in _ready via SurfaceTool.append_from, dropping 4 MeshInstance3D children
to 1 (Nose/TailFin stay separate, they're retinted per-team). Skipped in
headless mode like the goal/arena_boundary visual builds, since physics
only cares about CollisionShape3D.
All 6 source surfaces (hull, canopy, engine_l x2, engine_r x2) carry
distinct materials, so this doesn't literally cut draw calls 6 to 3 as
TODO.md assumed — Godot still issues one draw call per surface regardless
of node count. The real win is scene-tree/transform overhead, not batching.
_apply_team_color() allocated a fresh StandardMaterial3D on every call, and
ran at least twice per ship (once from _ready at the default team, once from
the team setter when the game mode assigns the real team). Cache one
StandardMaterial3D per team in a static dict on Ship and reuse it across
every ship on that team.
Ship._emit_telemetry_data() ran get_euler()+trig every physics tick
for every ship regardless of whether a HUD was watching, wasting work
on AI ships and every headless training instance. Disable
_physics_process outright when headless, and skip emission the rest
of the time unless a signal actually has a listener.
Josh Creek