An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:
CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.
HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.
MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.
MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.
MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.
MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).
LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.
LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.
LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.
New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
networked_match.gd's client can now swap its input sampler for a real
AIShipController (--test-bot, optionally --test-bot-model=<path>,
defaulting to bots/promoted/medium.json) instead of PlayerShipController.
Unlike the human sampler, AIShipController needs real scene context
(get_parent() as Ship, plus ball/teammate/opponent discovery via groups),
so it's parented onto the client's own ship via Ship.set_controller()
rather than left floating - and the field's static type widened from
PlayerShipController to the shared ShipController base to allow either.
Known, documented limitation: this client's ships are all
FREEZE_MODE_KINEMATIC and driven purely by transform writes, so nothing
ever writes linear_velocity/angular_velocity onto them - the bot's
observations always see every ship as stationary. It still produces
well-formed, bounded actions from that degraded input (the policy
network's output layer is bounded regardless of input quality), which is
sufficient for this task's actual job: generating realistic sustained
network traffic for CI, not winning matches.
New CI driver (tests/networked_match_ci.gd/.tscn): a headless server plus
two headless --test-bot clients playing a real match. task 3.6's original
acceptance text also named "p95/p99 prediction error" and "snap count" -
both Phase 4 concepts that don't exist until client-side prediction and
its hard-snap threshold are built, so asserting on them now would be
fabricated. What's checked instead: snapshot throughput (500+ received
over an 8s run, comfortably above a 60Hz-scaled floor), and genuine
cross-peer score agreement - forced via a deterministic server-side goal
(bot-vs-bot scoring isn't reliable enough within a short run to gate on),
with each client independently writing its own final score to a peer-id-
keyed file for the host to compare against the other bot's, not just
trusting the server's own view. "Clean stderr" is left as the external
invocation's job, same as every other smoke test in this project.
Verified with real 3-process runs (host + two bots): both clients
independently confirmed identical scores after a forced goal, both saw
500+ snapshots, and all three processes exited 0 with clean stderr on a
representative run (one run separately hit the same known, already-
documented single-benign-error disconnect-timing race task 3.4's own
abuse tests hit - not a new issue). Full regression suite, including the
net-sim-latency milestone gate and the abuse-detection tests, re-run
clean.
Extends net_debug_overlay.gd (Phase 1's RTT/offset display) with the
rest of task 3.7's list: jitter (new RFC3550-style EWMA in
NetworkManager, computed from raw per-sample RTT before Phase 1's own
min-filtering, since that filter is deliberately jitter-insensitive by
design), input buffer depth and input_lead (both already tracked
client-side for task 3.3), snapshot loss (a new EWMA in networked_match.gd
over each received snapshot's own server_tick gap - snapshots go out at a
steady one-tick cadence, so a gap is direct evidence of a drop or
reorder), snapshot age (computed on demand from the same bias-corrected
tick estimate the interpolator itself uses), and bandwidth (new rolling
per-second byte counters in MatchSim, on the two 60Hz hot-path channels
only). Prediction error is deliberately omitted with a comment explaining
why: there's no client-side prediction to measure until Phase 4.
Verified values are live and plausible, not just present, by calling
get_net_debug_stats() directly in a real two-process test and checking
the numbers make sense: bandwidth matched the wire format's own byte
math almost exactly (measured ~2400 B/s sent against a computed 40B x
60Hz, ~3540 B/s received against 59B x 60Hz), and buffer depth/lead/loss
all moved in the correct direction between a clean LAN run and one under
simulated 60ms latency + 10% loss. Full regression suite re-run clean.
MatchSim._recv_input now validates before decoding (§3.1 steps 2-3):
per-peer rolling-1s rate limiting (packet count AND byte budget, dropping
over-budget packets and disconnecting after 3 consecutive over-budget
seconds), and framing validation (redundancy count and payload size
checked against NetCodec's own layout before unpack_input ever runs,
disconnecting after 20 malformed packets). Framing has to be validated
explicitly rather than relying on decode failure: StreamPeerBuffer
silently zero-fills past EOF instead of erroring, a finding from Phase
2's adversarial review.
networked_match.gd's _on_input_received now rejects any seq claiming to
be more than 20 ticks ahead of the current server tick (§3.1 step 4) and
counts (rather than silently ignoring) input from a peer with no slot,
for observability.
Verified with two new permanent regression tests (networked_match_smoke.gd
--role=client-abuse-malformed / client-abuse-flood) that call
MatchSim._recv_input directly with garbage bytes and a legitimate-but-
too-frequent flood, respectively, bypassing the honest client encoder
entirely - the same thing a hostile custom client sending raw ENet
packets would look like. Both confirm real disconnection, not just that
the server tolerates the abuse.
Two bugs surfaced by getting these tests to actually pass cleanly: a
GDScript lambda-capture-by-value mistake in the tests themselves (a
plain `var disconnected := false` mutated inside a signal-handler lambda
never became visible to the enclosing function - fixed by capturing a
single-element Array instead, which is captured by reference); and a
narrow real race where NetworkManager's own ping/pong reply could target
a peer that a concurrent abuse-triggered disconnect had just removed
from the same poll() batch, now guarded. (Passing disconnect_peer's
`force` parameter as an attempted fix for a related one-off benign error
was tried and reverted - it made Godot's own peer-list bookkeeping
inconsistent, producing hundreds of errors instead of one; verified
empirically rather than assumed.)
Full regression suite, including the net-sim-latency milestone gate,
re-run clean.
New InputLeadController (scripts/input_lead_controller.gd, standalone and
unit-tested like input_jitter_buffer.gd): fast attack (+3 immediately,
debounced to once per 30 ticks) on any server-reported starve, slow
release (-1 per 60 ticks, gated behind a one-time 2s clean-surplus bar)
otherwise, clamped [1, 12]. Deliberately the only thing that adapts
buffer depth - the server (InputJitterBuffer) stays a pure reporter, per
§3.3's explicit warning that multiple control loops acting on one plant
(buffer occupancy) oscillate and present as unattributable sticky
controls.
Wired into the client's per-tick input send: a lead change is realized as
extra distance between the client's outgoing sequence numbers and what
the server has consumed - an attack skips extra sequence numbers, a
release duplicates the current one (sent again, unincremented). The
server's ring buffer needs no special handling for either: a skipped seq
is an ordinary drop, a duplicated one is a same-seq resend already
discarded by the existing "already consumed" check.
Verified with real two-process runs: on a clean LAN, one early attack
(a momentary hiccup during connection setup) recovers via two releases
within the test's own ~4s window, settling back near minimum. Under
sustained 30% simulated loss, lead climbs to 7 via repeated attacks and
never releases while genuine loss continues - confirming the debounce,
attack, and release gates all fire on real conditions, not just in
isolated unit tests. Full regression suite, including the net-sim-latency
milestone gate, re-run clean.
Client now sends the last 4 ticks' actions per packet (newest-first,
already-supported by net_codec's wire format from Phase 1) instead of a
single action with no redundancy. Server gains a real per-slot ring
buffer (new InputJitterBuffer class, scripts/input_jitter_buffer.gd) that
consumes exactly one sequence number per physics tick: repeats the last
action on a starve, zeroes only after a sustained 500ms stall, and
reports real input_buffer_depth/last_input_seq/echo_client_send_ms in
every snapshot instead of the hardcoded zeros Phase 2 shipped with.
InputJitterBuffer is a standalone, scene-free RefCounted (same pattern as
net_codec.gd/net_interpolator.gd) specifically so it's unit-testable
against scripted arrival traces (tests/cases/test_input_jitter_buffer.gd):
sequential consumption, redundancy surviving a 3-packet burst loss (3.1's
own acceptance criterion), starvation repeat-then-zero timing, stale/
reordered packet handling, buffered-depth reporting, and ring-wraparound
slot-tagging safety.
One real bug found wiring this into a live match: the server's ring
buffer started counting its own "expected sequence" from 0 the instant a
player's slot was created - well before that player's first real packet
could possibly have arrived (connection handshake, arena/ship spawn all
take real time first). Since both sides only ever advance monotonically
with no resync mechanism, that gap between the server's arbitrary local
counter and the client's actual from-1 sequence numbers never closed,
so the ship simply never received the client's input (0m movement in a
two-process test). Fixed by seeding the buffer's expected-sequence
counter from the client's own numbering on first real ingest, rather
than assuming a shared from-zero baseline.
Verified with real two-process runs: clean baseline movement restored,
zero starvation observed under 25% random simulated input loss (well
above what redundancy-4 needs to fully absorb), and correct starve-then-
stall behaviour confirmed under 100% loss as a sanity check that the
mechanism isn't a silent no-op. Full regression suite, including the
net-sim-latency milestone gate, re-run clean.
An Opus subagent's adversarial review of Phase 2 found real bugs the
smoke tests couldn't catch, since constant-velocity dead reckoning still
moves a ship far enough to pass a "moved > 1.0" check:
- The interpolator never actually interpolated. NetInterpolator.to_tick()
assumes physics_frame * TICK_MS == Time.get_ticks_msec() on the server,
which is off by a steady ~45-55ms in practice (real startup work before
the first physics step, widened by any dropped tick). Every sample_at()
call took the extrapolation branch, 100% of the time, defeating the
interpolation buffer entirely. Fixed with a shared, min-filtered rolling
bias estimate in networked_match.gd, applied before every to_tick() call.
- Goals caused a ~27m visual slide: _reset_gen was bumped before the
queued teleport actually landed, so the client's buffer-clear kept
exactly the stale in-goal sample and lerped a slide to the next, real
one. Fixed by tracking the tick the goal was detected on and only
bumping the generation once strictly later ticks confirm the teleport
has landed - a naive "next _physics_process" boolean flag doesn't
work, since a goal Area's body_entered fires before that same tick's
_physics_process runs, not on the next one.
- _local_input_sampler (a Node, never added to the tree) was never freed
- this was the unexplained "3 resources still in use at exit" warning
on every Phase 2 test run.
- Ball angular velocity decoded 8x too small (rescale_avel was never
called); get_server_time_estimate_ms() was used before the clock had
synced; net_sim.gd's delayed-send timer stopped ticking while the tree
was paused and didn't check connection status before firing;
_broadcast_snapshot's ball index could silently break if a ship were
ever despawned; declared-but-unemitted HUD lifecycle signals showed a
permanently frozen timer widget.
Also confirmed, empirically, several things the review checked and found
fine: a hostile client sending malformed input cannot crash the server,
skipping GameMode's super() drops nothing load-bearing, deterministic
slot assignment is correct with 2 real simultaneous clients, and RPC
authority enforcement genuinely rejects a forging client.
All fixes verified with real two-process runs (including forcing an
actual goal and reading the server's own broadcast stream) and temporary
instrumentation, removed once each fix was confirmed. Full Phase 1 +
Phase 2 regression suite, including the net-sim-latency milestone gate,
re-run clean after every fix.
New NetSim autoload: seeded, CLI-driven (--net-sim-latency/-jitter/-loss/-dup)
latency/jitter/loss/duplicate decorator, a true no-op passthrough unless a
flag is set. Wraps MatchSim.send_input/send_snapshot per the design doc's
scope, plus NetworkManager's ping/pong so the already-tested RTT/clock
measurement becomes the acceptance signal for "raises observed RTT" without
waiting on Phase 3's per-peer snapshot echo.
Two real bugs found while building and verifying this against Phase 2's own
milestone gate (a real match under --net-sim-latency 80 --net-sim-jitter
20, not just LAN): a timestamp captured inside a delayed RPC closure
silently ate that side's own added delay out of the round-trip
measurement instead of adding to it; and a delayed send whose target
disconnected (or whose own process had already shut down) during the hold
threw RPC errors, since the existing get_peers() filtering only checked
validity at schedule time. Fixed by capturing timestamps before handing
off to NetSim, and by having NetSim re-validate the target at fire time.
Phase 2's milestone gate now passes for real: a full 1v1 under simulated
80ms latency / 20ms jitter still shows clean server-authoritative
movement and zero RPC errors. Full Phase 1 + Phase 2 regression suite
re-verified clean with NetSim present but inactive.
Implements tasks 2.1-2.7: NetworkedMatch spawns a deterministic slot
layout from the lobby roster, the server drives each connected peer's
ship via RLShipController fed by decoded client input and broadcasts
60Hz snapshots, and the client renders everything (including its own
ship) from a per-body NetInterpolator with no local prediction yet.
Dual-time remote entities split collider updates (present-time, for
correct contacts) from $Visual updates (interp-delayed, for smoothness).
Camera/HUD wiring and remote engine-flame VFX fell out of the existing
Ship API for free once snapshots were flowing.
Three real bugs found and fixed while getting a two-process test
green: an RPC method named _input collided with Node's built-in
_input virtual and broke the whole MatchSim autoload from loading;
networked_match.gd never called NetworkManager.poll(), so nothing
sent via RPC in this scene reached the wire despite Phase 1's manual
polling being wired up everywhere else; and a match_config
request/response fallback (added to close a startup race) could
double-deliver once polling was fixed, requiring an idempotency guard.
Verified with tests/networked_match_smoke: a real headless two-process
host+client run shows the client rendering 31m of server-authoritative
movement from a held forward-thrust input, with thrust_z=1.0 confirmed
on the interpolated snapshot mid-drive and camera/HUD both wired.
Full Phase 1 regression suite re-run clean alongside it.
Task 2.8 (net_sim.gd latency/jitter/loss decorator) is not yet done;
Phase 2's own gate needs it before it's fully met.
Lands tasks 1.0-1.8 of multiplayer-todo.md: the pure-function test runner,
net_codec (wire format quantizers/pack-unpack), NetworkManager (ENet
transport, manual polling, min-RTT clock sync), MatchNet (handshake,
protocol/tick-rate gating, roster with team+ready state), lobby.tscn (team
columns, switch team, ready toggle), server_boot.tscn (headless dedicated
server with structured logging and an overrun watchdog), and main_menu.gd's
Host/Join-by-IP UI (connecting overlay, cancel, bounded failure path).
Followed by an adversarial review (Opus subagent) that found and fixed two
real bugs - an unvalidated player_name broadcast that let one client's
oversized name head-of-line-block the reliable channel for everyone, and a
server-side roster leak across a host/re-host cycle - plus three gaps in
the test suite itself where a claim of "verified" wasn't actually backed
by what the test checked. All five two-process smoke tests plus the
pure-function suite are green with the strengthened assertions in place.
Lands the non-networked Phase 0 tasks from multiplayer-todo.md (ship/camera/
arena refactors, sim constants, background FPS handling) plus a first pass
at exposing graphics/performance settings (presets, resolution scaling,
vsync, FPS cap, perf overlay) and a GPU profiling harness for the
real-hardware follow-up in task 0.15b.
Stage 5 blocked all three attempts on productive_air_touch_fraction
stuck exactly at 0.0 across a continuous 180M-step lineage, while
goal_rate/upright_fraction/forward_motion_fraction kept improving on
the same budget. forward_velocity_to_ball_weight (the term that solved
Stage 4's ground pursuit) is hard-gated below GROUND_HANDLING_HEIGHT
and does nothing in the air, so Stage 5's air_intercept_chance had no
matching aerial incentive to learn from. air_approach_weight adds the
airborne mirror (nose-first 3D closing speed, no uprightness
multiplier) and folds into HANDLING_REWARD_FLAGS so Stage 6 inherits
it too. Deleted the three blocked attempts and reset state to resume
Stage 5 from the Stage-4 checkpoint with the new term.
20260816-2126-gen5-s4-handling-retry2 exhausted its three attempts and
missed only the 0.80 training goal-rate floor, at 0.7731. Every
evaluation gate passed: 65-22-13 versus promoted/easy.json, 87% non-draw
against an 80% floor, 12.6% physical-side imbalance against a 20%
ceiling, and both handling telemetry floors clear. The round improved the
goal rate monotonically across attempts (0.537 -> 0.683 -> 0.773) and the
checkpoint plays well by hand, so close Stage 4 by human override.
Promote it to Game/bots/promoted/medium.json. Medium and Hard both point
at the new policy: Hard stays a label-only duplicate until a stronger one
earns hard.json, which keeps the tiers monotonic rather than leaving Hard
weaker than Medium.
generation5_state.json flips that log entry to "pass" with a
decision_override block preserving the original verdict and reasoning,
and advances to Stage 5 attempt 1. This is what passing_entry() needs to
resolve Stage 5's resume checkpoint and evaluation reference, and what
league_pool() will need at Stage 6; --skip-to-next-stage would advance
the stage without marking anything as passing and die immediately.
generation5.sh now pulls before launching. Each stage ends in
commit_progress()'s push, which fails and kills the run hours in if the
box is behind origin.
Six rounds of reward shaping (~700M steps) failed to produce upright ground
driving. A critical review of the simulation rather than the reward found
why:
1. The hull was a 1x1x4 box with inertia (1,1,1) and no restoring torque
anywhere, so belly-down and rolled-90 were geometrically identical
resting states. "Upright" was not a physically distinguished state at
all - the reward was paying for a property the simulation did not have.
2. ~65% of episodes spawned ships via _random_position, which samples Y
uniformly over the full 18m volume (mean ~8.7m). The measured
airborne_fraction ~0.44 was largely that spawn distribution, and every
ground-handling term fades out above 3m, so the shaping being tuned
barely ever applied.
3. air_drill_chance 0.20 spawned deliberately unreachable-without-climbing
states in the stage meant to teach ground driving, and its own
air_touch_fraction (0.0002) shows the drills were never solved.
Fixes land in the physics and the task distribution, not the reward:
- ship.tscn: hull 1x1x4 -> 1.6x0.6x4 so it has one stable resting face;
inertia (1,1,1) -> (7,1,7), physically correct for the hull, making
tumbling reluctant while keeping yaw snappy.
- ship.gd: new altitude-faded righting torque (spring-damper toward
belly-down, faded out by 3m so aerials keep full attitude freedom).
This is the grav-plating analogue of Rocket League's auto-righting and
helps human pilots land cleanly too.
- training_mode.gd: new ground_start_chance branch spawning ships level and
resting on the floor with a floor-level ball - the state the handling
stage's rewards are actually written for.
- generation5.py: ground-start-chance 0.50, air-drill-chance 0.20 -> 0.0.
Reward terms are left exactly as they were; they should finally pull in a
direction the ship can go.
Round 4 changed three things at once and two of them cut upright pressure:
grounded_upright_reward went to 0 and ground_tilt_penalty was cut 2.5x,
while the new uprightness multiplier only pays below GROUND_HANDLING_HEIGHT
*and* while moving forward *and* facing the ball - a far narrower slice of
ticks than the penalty it was meant to replace. Net pressure fell and
upright_fraction fell with it (0.268 -> 0.239 -> 0.238, the lowest of any
round). Restore ground_tilt_penalty to 0.05 and change nothing else, so
this is a genuine single-variable test of multiplier plus full tilt
pressure.
The conjunctive mechanism itself held up: forward_motion_fraction reached
its best sustained value (0.242) without goal_rate sagging, ep_rew_mean
turned positive for the first time (+0.28), and eval win rate hit 49% with
no reward hacking.
Also adds grounded_upright_fraction: a diagnostic, deliberately ungated
metric measuring uprightness over real floor-contact ticks instead of
sub-3m ticks. upright_fraction has never exceeded 0.331 across four rounds
and ~560M steps without cheating, and its denominator is dominated by
ballistic transit (airborne_fraction ~0.45, mean_altitude ~4.4m) where
attitude is not meaningfully controllable - so it likely cannot measure
what the 0.45 floor was meant to capture. Re-baseline that floor from what
this reports rather than from another round of reshaping.
Rounds 2 and 3 showed that tuning grounded_upright_reward's magnitude only
slides along a tradeoff instead of resolving it: at 0.015 upright_fraction
climbed to 0.331 while goal_rate sagged to 0.542 (then farmed outright at
0.696/0.366), and at 0.004 goal_rate climbed 0.569->0.604 while
upright_fraction went flat at ~0.26. An additive uprightness bonus is an
alternative to playing well, so the policy just picks whichever is cheaper
and no magnitude buys both behaviours.
Change the mechanism rather than the number: grounded_upright_reward drops
to 0, and uprightness becomes a multiplier inside the nose-led approach
term, which already requires moving forward at the ball. Parked-and-upright
and fast-but-sideways now both pay zero; only upright, forward, nose-on to
the ball pays full. forward-velocity-to-ball rises 0.06 -> 0.15 to offset
the ~2-3x expected-value cut from the new factor, and ground-tilt-penalty
drops 0.05 -> 0.02 now that uprightness is paid positively during play.
Delete the three blocked attempts and reset state to restart from the
Stage-3 foundation.
grounded_upright_reward at 0.015 overshot: four force-retries pushed
upright_fraction from 0.265 to a plateauing 0.331, then the fifth jumped it
to 0.696 (55% over the 0.45 floor) while goal_rate collapsed 0.542->0.366
and forward_motion_fraction fell 0.244->0.184 (vertical_thrust_mean went
negative) - the policy learned to sit pinned upright and farm the bonus
instead of chasing the ball. It was sized "comparable to
time_penalty/ball_distance_penalty" but at 0.015/tick it was actually above
ball_distance_penalty's 0.01/tick worst case, so idling near the ball beat
playing. Cut to 0.004/tick (episode ceiling ~7.2, below
ball_distance_penalty's ~18 worst case). Delete the five blocked attempts
and reset generation5_state.json so the next run starts fresh from the
Stage-3 foundation rather than continuing from the farming checkpoint.
Adversarial review of the previous stage-4 retune found two problems:
non_forward_speed used planar_speed - forward_component, which under-charges
diagonal motion relative to true lateral speed (e.g. ~29% penalty at 45
degrees off the nose instead of the correct ~71%); fixed to the Pythagorean
magnitude for forward-facing angles, full speed for backward-facing ones.
Also, ground_tilt_penalty and non_forward_penalty only ever cost reward near
the floor with nothing offsetting them above it, which could teach a policy
that's still bad at ground handling to just avoid the floor rather than get
better at it. Added grounded_upright_reward (ship_ai_controller.gd) plus a
new ShipObservations.is_floor_contact helper for genuine belly-on-floor
contact detection, so grounding well while upright is the locally profitable
choice, not just the least-punished one.
Stage 4's upright/forward-motion telemetry plateaued flat across all three
blocked attempts because ground_tilt_penalty (0.003) was too weak to matter
and nothing penalized sideways/reverse motion at all. Raise
ground_tilt_penalty to 0.05 and add a new non_forward_penalty term
(ship_ai_controller.gd) that directly costs non-forward planar velocity near
the floor, independent of the ball. Delete the three blocked attempts'
checkpoints/logs/exports and reset generation5_state.json so the next run
starts fresh from the Stage-3 foundation checkpoint instead of continuing
from the drifted retry2 weights.
Extends ShipObservations beyond the old self+1-opponent layout to padded
teammate/opponent arrays (MAX_TEAMMATES=4, MAX_OPPONENTS=5, SIZE=83),
zero-filling slots past the real roster size the same way the old single-
opponent slot was zero-filled when absent.
Slot stability across ticks requires a persistent identity: Ship gains
spawn_index (set once by GameMode.spawn_ship, never reassigned — there's no
despawn path anywhere in this codebase, so a roster is fixed for the whole
episode/match). ai_ship_controller.gd's opponent discovery is rewritten from
"first non-self ship" to classify every other ship by team and sort by
spawn_index; training_mode.gd/ship_ai_controller.gd carry the equivalent
sorted lists through the training path so both agree on slot assignment for
the same roster.
training_mode.gd and match_mode.gd both gain a team_size export (default 1,
so every existing curriculum script and match keeps today's 1v1 behaviour
unchanged). This is plumbing only: no 2v2+ curriculum or reward design, and
no match-mode UI to pick team size, has been done yet. The two checkpoints
in Game/bots/promoted/ are fitted to the old 35-float layout and are not
migrated — expected to go stale until the next training run.
Retraining from scratch removes the constraint that blocked these: the
existing checkpoints in Game/bots/promoted/ no longer need byte-identical
geometry.
- Fix the goal aperture constants (were 1.85/1.65, now match
objects/goal.tscn's actual sensor exactly at 1.75/1.5) so a ball crossing
the visible edge can't fail to score.
- Cut a real navigable hole in each end wall's collision
(_build_end_wall_colliders), replacing the previous solid box — the ball
now genuinely enters the net instead of triggering the sensor a hair
before hitting a solid wall. Correct for both FLOOR and ELEVATED goal
modes via _goal_surround_bounds. A separate, slightly wider
GOAL_VISUAL_APERTURE_* pair keeps the collision hole exact while still
giving goal.gd's bezel/rim frame clearance to be seen against the hull cut.
- Resize the play volume 1.5x (INNER_HALF_X/Z/HEIGHT 12/18/12 -> 18/27/18) to
comfortably fit a 5v5 roster: updates every hand-authored literal in
arena_boundary.tscn/arena_base.tscn/the elevated arena variants that
doesn't derive from those constants, adds 5 spawn markers per team, and
rescales ship_observations.gd's normalization scales and arena_02's
cosmetic decoration/nebula-dust shader uniforms to match.
- Bake the ~170 runtime-generated CollisionShape3D nodes into the scene via
a new bake_colliders()/tools/bake_arena_boundary.gd instead of rebuilding
them on every load — a real load-time cost repeated in every parallel
headless training env. Colliders must be direct children of the
StaticBody3D to register at all, so bake_colliders() parents them onto
self and tags them with a group for idempotent re-baking, rather than
grouping them under an intermediate container node. _ready() self-heals:
it skips regenerating only when an existing bake's goal_mode metadata
matches the current one, so the FLOOR-mode bake in the shared scene is
never silently reused by an ELEVATED arena variant.
_place_ships_random/_place_air_drill sampled each ship's randomized episode-
start position independently, so a team_size > 1 roster could spawn
interpenetrating (ships are ~1x1x4). Both now resample (up to 20 attempts,
matching the existing corner/fillet rejection-sampling pattern) against
every ship already placed that reset, rejecting anything within
MIN_SHIP_SEPARATION (4.5m, matching the arena spawn-marker spacing) of one.
The goal mouths are now a real navigable hole in the end walls, sized to the
ball rather than the ship — a ship's 1x1 cross-section fits through it, and
there's nothing behind the net to stop it. The escape failsafe previously
only existed in TrainingMode (where a physics regression just wastes
training time); now that any ship can genuinely fly out through an open
goal, every mode needs it or a stray ship/ball falls into the void with no
way back short of quitting. Moved up to GameMode as the shared default
_physics_process, removing TrainingMode's now-duplicate copy.
drag_coefficient/angular_drag/the idle angular-drag multiplier were applied
once per physics tick with no delta scaling, correct only because
project.godot never pins physics/common/physics_ticks_per_second and
Godot's default happens to be 60. _tick_scaled(k, state.step) makes the
decay rate invariant to tick rate instead. Also promotes the previously
hardcoded 0.9 idle angular-drag literal to an export, matching its sibling.
VecMonitor's info_keywords does a bare info[key] lookup on a completed
episode's terminal info dict and raises KeyError -- crashing the whole
training run -- if a key is ever absent. get_info() was only including
airborne_fraction/mean_altitude/vertical_thrust_mean when _telemetry_ticks
was nonzero and air_touch_fraction when _touches was nonzero; an episode
with zero ball touches (common, especially early in training) crashed on
the very first rollout in a smoke-test run. All four now always default to
0.0 rather than being conditionally present.
Found via TRAINING.md's Generation 4 validation ladder (rung 2, a 60k-step
smoke run) -- confirmed fixed by rerunning the same smoke run clean, then
export/evaluate parity (rungs 2-3) against Game/bots/promoted/easy.json.
Three curriculum generations (2026-07-21 through 2026-08-04) all tried
gating *when* the policy could use vertical thrust/pitch-roll on top of a
continuous Gaussian action space, and all three failed the same way: PPO's
action-distribution std collapsed within ~10% of steps and never recovered,
landing at a 15-32% win rate vs the grounded reference regardless of
mechanism (hard mask, then a gradual ramp). Generation 3's final attempt
just landed at 24% — the worst of the three.
Root cause, verified against this project's own physics: hovering this ship
requires *holding* thrust.y ~= 0.408 continuously (mass 5.0, vertical_thrust
120, gravity 9.8). A collapsed near-zero-mean Gaussian can brush that value
but never sustain it long enough to earn the reward gradient that would
move the mean — no amount of gating *when* the axis acts fixes a problem in
*how* the policy represents a decision on it. This also independently found
and fixes a real bug: godot_rl never marks an episode timeout as a
truncation, so PPO was bootstrapping V(s)=0 on every 30s draw in every
generation to date.
- Game/scripts/ship_action_codec.gd (new): single source of truth for a
per-axis MultiDiscrete action space (7 heads, nvec [5,5,5,5,5,5,2]) shared
by training and in-game inference, replacing the continuous Gaussian.
thrust_y's bins are deliberately asymmetric so a random policy drifts
through the volume instead of floor-pinning. Legacy continuous decode
(ai_ship_controller.gd's old logic) preserved verbatim so every
pre-generation-4 export (e.g. Game/bots/promoted/easy.json) keeps working
unchanged via an optional "action_space" JSON field.
- ship_observations.gd: append own contact state (SIZE 31 -> 35, append-only)
so the value function can see what wall_contact_penalty fires on.
- ship_ai_controller.gd: action space/decode via the codec; drop the
vertical_ramp/pitch_roll_ramp mechanism entirely; tilt_penalty default
lowered 4x (aerial approaches require pitching); flight telemetry
(airborne_fraction, mean_altitude, air_touch_fraction, vertical_thrust_mean)
and truncation-snapshot fields on get_info().
- training_mode.gd: new air_drill_chance state-setter branch (ball spawned
high, ships low, kept clear of walls) so aerial practice is forced by the
environment instead of relying on reward-driven exploration alone; snapshot
terminal observations before a timeout reset for the truncation fix.
- cosmic_env.py: remap ShipAIController's truncated/terminal_obs info into
SB3's TimeLimit.truncated/terminal_observation keys.
- train.py: --reset-logits (+ --reset-logits-heads) replaces the
now-meaningless --reset-std; new EntropyFloorCallback (a persistent
per-rollout ent_coef controller replacing the one-shot std-reset shock)
and per-head entropy logging; FlightTelemetryCallback; --air-drill-chance/
--tilt-penalty flags; optional AbortIfCallback kill-criterion.
- export_policy.py: writes the action_space block for MultiDiscrete models;
index-level parity check (argmax per head) instead of comparing floats.
- curriculum.py: full rewrite — 3 stages (bootstrap/selfplay/gauntlet), no
grounded stage, full action space live from step 1; deletes generation
1-3's checkpoint-lineage machinery (nothing to resume from); final report
evaluates against both promoted/easy.json and the new
promoted/reference-grounded.json (a copy of curric-s5-aggression, the
strongest grounded-era artifact, kept as a fixed yardstick).
- run_training.sh/.gitignore: commit only final.zip, not the ~2400
intermediate checkpoint files a single stage was writing (~500MB ->
~0.2MB per run); requirements.txt pinned (behaviour here now depends on
specific library internals, not just public APIs).
- test_action_space.py (new): offline rung-0 check catching a head-order
mismatch before it silently corrupts 24h of training.
Validated: GDScript compiles clean (Godot --headless --import + script
validation), free_play.tscn and training.tscn both boot headless without
errors, offline action-space assertions pass. Not yet run: the actual
smoke-training/A-B validation ladder steps in TRAINING.md's "Generation 4"
section, before committing to the full ~32h curriculum.
See TRAINING.md's "Generation 4" section for the full design writeup.
ship.gd's controllerless path and player_ship_controller.gd each
allocated a fresh ShipAction every physics tick; ai_ship_controller.gd
and rl_ship_controller.gd already avoid this via a persistent member.
Convert both to reuse a member instance, matching the existing
full-field-overwrite convention (rather than +=/-= off a fresh zero).
Also drop the completed items from TODO.md.
HUDController found its ship via get_first_node_in_group("ship"), a group
that has 2+ members once a match has an AI opponent — it only worked
because the player ship happened to spawn first. spawn_camera_rig now
wires the HUD's ship the same way it already wires the camera rig's
target.
match_mode.gd fired the signal every frame while only the once-a-second
value is displayed, forcing the HUD to re-format and re-shape the label
each frame. Gate emission on the whole-second value changing, matching
ship.gd's threshold-gated telemetry discipline.
The coroutine stalls mid-countdown while the tree is paused for the
results screen, but _end_match unpauses before the deferred scene
change actually tears things down — letting it resume for a frame and
re-emit kickoff_countdown / unfreeze bodies in the dying scene.
ship.gd, HUDController.gd, and goal.gd each declared their own TEAM_COLORS,
arena_boundary.gd/arena_deck.gdshader had a third pair, and HUD.tscn baked
in a fourth (hardcoded "BLUE"/"ORANGE" labels) — nose, goal rim, end zone,
and scoreboard all rendered different blues. New scripts/team_colors.gd
(class_name TeamColors) is now the single source every one of those reads
from, and team identity moves to purple/green.
WorldEnvironment's Environment sub-resource was shared across every
instantiate() of a cached arena PackedScene, so glow/brightness/sky
tweaks in Arena._ready() compounded further each time a player
re-entered an arena instead of applying fresh.
Hull/Canopy/EngineGlowL/EngineGlowR are runtime-baked into one ArrayMesh
in _ready via SurfaceTool.append_from, dropping 4 MeshInstance3D children
to 1 (Nose/TailFin stay separate, they're retinted per-team). Skipped in
headless mode like the goal/arena_boundary visual builds, since physics
only cares about CollisionShape3D.
All 6 source surfaces (hull, canopy, engine_l x2, engine_r x2) carry
distinct materials, so this doesn't literally cut draw calls 6 to 3 as
TODO.md assumed — Godot still issues one draw call per surface regardless
of node count. The real win is scene-tree/transform overhead, not batching.
_apply_team_color() allocated a fresh StandardMaterial3D on every call, and
ran at least twice per ship (once from _ready at the default team, once from
the team setter when the game mode assigns the real team). Cache one
StandardMaterial3D per team in a static dict on Ship and reuse it across
every ship on that team.
Goal._build_visuals() built 10 MeshInstance3D nodes across 4 materials
(1 pocket + 1 net + 4 bezel-ring + 4 rim-ring boxes) per goal. Replaced
with a single MeshInstance3D wrapping one ArrayMesh with 4 SurfaceTool-
committed surfaces (pocket, net, bezel, rim), one material per surface
via surface_set_material — 10 nodes down to 1, same 4 materials.
Kept 4 surfaces rather than collapsing further: the rim is a tuned
team-tinted emitter, the net carries its own discard shader, and the
pocket/bezel differ in albedo/metallic/roughness. Merging those into a
shared material would be a visible regression, not a free win.
Added a local box-to-SurfaceTool helper (6 quads via arena_boundary.gd's
_add_quad/_add_tri winding-correction trick, copied in rather than
shared since that file's geometry is collision-adjacent). The pocket's
old cull_mode = CULL_FRONT trick is replaced by emitting its geometry
with inverted winding; the net's cull_front stays material-driven since
goal_net.gdshader's own render_mode depends on that winding convention.
Verified in the editor: both team-tinted goals render an intact pocket,
net, bezel and glowing rim with no backface/winding artifacts, and the
headless free_play smoke test still runs clean.
_process() called get_viewport().get_camera_3d() every frame to drive
the containment field's camera-side fade. Cache it the same way
ship_camera.gd caches the ball, revalidating with is_instance_valid
since exactly one camera rig is spawned per game-mode run today.
hud_gauge, hud_attitude_indicator, and hud_heading_tape now skip
queue_redraw() when the newly-lerped value hasn't moved past a small
epsilon, instead of redrawing every frame forever. Angle-wrapping
values (heading, attitude roll) use a new shared angle_delta_deg
helper on HudInstrument so the wrap boundary doesn't read as a false
jump.
HUD.tscn's Instruments node now sets process_mode = 1 (PAUSABLE),
overriding the inherited ALWAYS mode from the HUD root so instruments
stop processing during the post-match pause freeze, while sibling
ResultOverlay keeps running its win-screen tween.
Ship._emit_telemetry_data() ran get_euler()+trig every physics tick
for every ship regardless of whether a HUD was watching, wasting work
on AI ships and every headless training instance. Disable
_physics_process outright when headless, and skip emission the rest
of the time unless a signal actually has a listener.
Josh Creek