A second adversarial review of the previous fix commit found two of its
nine fixes silently defeated each other: the seq-range guard (fix for a
MEDIUM epoch-mismatch finding) capped the exact variable the ring-overflow
resync (fix for the original CRITICAL finding) depends on, making the
resync unreachable in production and recreating permanent input death at
a lower failure threshold, reachable via ordinary server tick loss alone.
- CRITICAL: rebind the seq-range guard to InputJitterBuffer's own
highest_ingested_seq (now public) instead of the consumer-side
last_applied_seq, so it tracks the client's send epoch rather than a
value that can lag arbitrarily far behind during a stall.
- HIGH: InputLeadController's release logic still ANDed the old
`lead > LEAD_MIN` gate onto the new depth-driven condition, so a
backlog the controller never caused still couldn't drain. Split into
two independent decisions: the seq-duplicate action follows real
depth alone; lead's own bookkeeping separately never drops below its
floor.
- MEDIUM: widen the CI driver's movement/stalled sampling margin
(run_seconds - 2.0, was - 0.5) and assert the peer is still in
multiplayer.get_peers() at sample time, since the old margin let the
check pass on residual starvation grace after a bot had already
disconnected.
- LOW: measure horizontal-only displacement in the human smoke test's
movement check — the old 3D-distance bar was beatable by pure
gravity settling with fully dead input.
- LOW: fix a real "clean stderr" violation (match_net.gd broadcasting
a departure notice to a peer whose ENet channels are already torn
down, including a second peer disconnecting in the same poll batch)
by deferring the notification to the next idle frame.
- Wire the server's per-slot stalled bit into the client debug overlay
for real — a prior commit message claimed this already reached the
overlay when only the CI gate actually read it.
Re-verified end-to-end against the real production RPC path (not just
unit tests in isolation, which is how the composition bug got past the
first round): a 2-bot CI match with a 1.5s host SIGSTOP freeze injected
mid-run, well past the 0.6s threshold the review reproduced the bug at,
now recovers cleanly on repeated runs with zero stderr noise.
An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:
CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.
HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.
MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.
MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.
MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.
MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).
LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.
LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.
LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.
New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
Documents all seven Phase 3 tasks (3.1-3.7) with DONE status and
verification evidence, updates the top-level status summary, and records
the phase gate as met - re-verified today under the gate's own exact
condition (--net-sim-latency 80 --net-sim-loss 0.05) on both the human
smoke test and the two-bot CI driver, not just the looser conditions
used during individual task development.
Adds one new gotcha (#38): GDScript lambdas capture enclosing locals by
value, not by reference, which silently broke two separate Phase 3 test
scripts' own disconnect-detection assertions this session (the
production disconnect logic was correct both times; only the test's own
flag-capture pattern was wrong).
Also records a deliberate scope decision for task 3.4: server-side
input_lead enforcement from arrival times was scoped down to
observability rather than built as active enforcement, since the
concrete security requirements (rate limiting, malformed-packet
counting, seq-range rejection, disconnect policy) already close the
load-bearing gaps and the doc's own text calls the remaining edge
"small" - flagged to revisit once Phase 4's prediction work exists to
judge against.
An Opus subagent's adversarial review of Phase 2 found real bugs the
smoke tests couldn't catch, since constant-velocity dead reckoning still
moves a ship far enough to pass a "moved > 1.0" check:
- The interpolator never actually interpolated. NetInterpolator.to_tick()
assumes physics_frame * TICK_MS == Time.get_ticks_msec() on the server,
which is off by a steady ~45-55ms in practice (real startup work before
the first physics step, widened by any dropped tick). Every sample_at()
call took the extrapolation branch, 100% of the time, defeating the
interpolation buffer entirely. Fixed with a shared, min-filtered rolling
bias estimate in networked_match.gd, applied before every to_tick() call.
- Goals caused a ~27m visual slide: _reset_gen was bumped before the
queued teleport actually landed, so the client's buffer-clear kept
exactly the stale in-goal sample and lerped a slide to the next, real
one. Fixed by tracking the tick the goal was detected on and only
bumping the generation once strictly later ticks confirm the teleport
has landed - a naive "next _physics_process" boolean flag doesn't
work, since a goal Area's body_entered fires before that same tick's
_physics_process runs, not on the next one.
- _local_input_sampler (a Node, never added to the tree) was never freed
- this was the unexplained "3 resources still in use at exit" warning
on every Phase 2 test run.
- Ball angular velocity decoded 8x too small (rescale_avel was never
called); get_server_time_estimate_ms() was used before the clock had
synced; net_sim.gd's delayed-send timer stopped ticking while the tree
was paused and didn't check connection status before firing;
_broadcast_snapshot's ball index could silently break if a ship were
ever despawned; declared-but-unemitted HUD lifecycle signals showed a
permanently frozen timer widget.
Also confirmed, empirically, several things the review checked and found
fine: a hostile client sending malformed input cannot crash the server,
skipping GameMode's super() drops nothing load-bearing, deterministic
slot assignment is correct with 2 real simultaneous clients, and RPC
authority enforcement genuinely rejects a forging client.
All fixes verified with real two-process runs (including forcing an
actual goal and reading the server's own broadcast stream) and temporary
instrumentation, removed once each fix was confirmed. Full Phase 1 +
Phase 2 regression suite, including the net-sim-latency milestone gate,
re-run clean after every fix.
New NetSim autoload: seeded, CLI-driven (--net-sim-latency/-jitter/-loss/-dup)
latency/jitter/loss/duplicate decorator, a true no-op passthrough unless a
flag is set. Wraps MatchSim.send_input/send_snapshot per the design doc's
scope, plus NetworkManager's ping/pong so the already-tested RTT/clock
measurement becomes the acceptance signal for "raises observed RTT" without
waiting on Phase 3's per-peer snapshot echo.
Two real bugs found while building and verifying this against Phase 2's own
milestone gate (a real match under --net-sim-latency 80 --net-sim-jitter
20, not just LAN): a timestamp captured inside a delayed RPC closure
silently ate that side's own added delay out of the round-trip
measurement instead of adding to it; and a delayed send whose target
disconnected (or whose own process had already shut down) during the hold
threw RPC errors, since the existing get_peers() filtering only checked
validity at schedule time. Fixed by capturing timestamps before handing
off to NetSim, and by having NetSim re-validate the target at fire time.
Phase 2's milestone gate now passes for real: a full 1v1 under simulated
80ms latency / 20ms jitter still shows clean server-authoritative
movement and zero RPC errors. Full Phase 1 + Phase 2 regression suite
re-verified clean with NetSim present but inactive.
Implements tasks 2.1-2.7: NetworkedMatch spawns a deterministic slot
layout from the lobby roster, the server drives each connected peer's
ship via RLShipController fed by decoded client input and broadcasts
60Hz snapshots, and the client renders everything (including its own
ship) from a per-body NetInterpolator with no local prediction yet.
Dual-time remote entities split collider updates (present-time, for
correct contacts) from $Visual updates (interp-delayed, for smoothness).
Camera/HUD wiring and remote engine-flame VFX fell out of the existing
Ship API for free once snapshots were flowing.
Three real bugs found and fixed while getting a two-process test
green: an RPC method named _input collided with Node's built-in
_input virtual and broke the whole MatchSim autoload from loading;
networked_match.gd never called NetworkManager.poll(), so nothing
sent via RPC in this scene reached the wire despite Phase 1's manual
polling being wired up everywhere else; and a match_config
request/response fallback (added to close a startup race) could
double-deliver once polling was fixed, requiring an idempotency guard.
Verified with tests/networked_match_smoke: a real headless two-process
host+client run shows the client rendering 31m of server-authoritative
movement from a held forward-thrust input, with thrust_z=1.0 confirmed
on the interpolated snapshot mid-drive and camera/HUD both wired.
Full Phase 1 regression suite re-run clean alongside it.
Task 2.8 (net_sim.gd latency/jitter/loss decorator) is not yet done;
Phase 2's own gate needs it before it's fully met.
Lands tasks 1.0-1.8 of multiplayer-todo.md: the pure-function test runner,
net_codec (wire format quantizers/pack-unpack), NetworkManager (ENet
transport, manual polling, min-RTT clock sync), MatchNet (handshake,
protocol/tick-rate gating, roster with team+ready state), lobby.tscn (team
columns, switch team, ready toggle), server_boot.tscn (headless dedicated
server with structured logging and an overrun watchdog), and main_menu.gd's
Host/Join-by-IP UI (connecting overlay, cancel, bounded failure path).
Followed by an adversarial review (Opus subagent) that found and fixed two
real bugs - an unvalidated player_name broadcast that let one client's
oversized name head-of-line-block the reliable channel for everyone, and a
server-side roster leak across a host/re-host cycle - plus three gaps in
the test suite itself where a claim of "verified" wasn't actually backed
by what the test checked. All five two-process smoke tests plus the
pure-function suite are green with the strengthened assertions in place.
match.tscn had picked up team_size=3 from an earlier diagnostic dry run,
which would have made every normal Match spawn 3v3 instead of 1v1 -
reverted to the scene's intended default.
multiplayer-todo.md: task 0.15b's real blocker turned out to be measuring
on a Mac (Apple Silicon's tile-based GPU architecture gave a misleading,
undifferentiated cost profile). Re-ran the same 6-ship-match profiling
harness on reference hardware (RTX 3090) via a real GPU-bound X session -
results in §5.5.2 show the game comfortably clears 500+fps with every
effect on, and SDFGI/SSIL dominate the (now tiny) effects budget as
originally expected. This closes 0.28 (physics threading) as unnecessary
- there's no frame-time variance problem on reference hardware to fix -
and reframes 0.26 (bake GI) as a real but smaller win than assumed, worth
revisiting on lower-end hardware. Also corrected two stale/inaccurate
task rows (0.13, 0.17) found while reconciling the doc against what
actually landed.
Lands the non-networked Phase 0 tasks from multiplayer-todo.md (ship/camera/
arena refactors, sim constants, background FPS handling) plus a first pass
at exposing graphics/performance settings (presets, resolution scaling,
vsync, FPS cap, perf overlay) and a GPU profiling harness for the
real-hardware follow-up in task 0.15b.
Josh Creek