Adds LocalPredictionHistory, a client-owned seq-tagged ring recording
predicted ship state per input sequence, plus wiring in
NetworkedMatch to record predictions on send and compare them against
authoritative snapshots on arrival. Ships stay frozen/interpolated
until 4.3 lands actual correction logic; this round only builds the
comparison machinery and its data.
Includes fixes from two review rounds: resync_required now
self-clears once acknowledgements catch back up (mirrors
InputJitterBuffer's stalled flag), NetBodyState gained a copy()
method to stop diagnostic accessors aliasing ring-owned state, and
corrected comments that had described the local ship as being
force-simulated pre-4.3 when it is still driven by interpolated
transform writes.
A second adversarial review of the previous fix commit found two of its
nine fixes silently defeated each other: the seq-range guard (fix for a
MEDIUM epoch-mismatch finding) capped the exact variable the ring-overflow
resync (fix for the original CRITICAL finding) depends on, making the
resync unreachable in production and recreating permanent input death at
a lower failure threshold, reachable via ordinary server tick loss alone.
- CRITICAL: rebind the seq-range guard to InputJitterBuffer's own
highest_ingested_seq (now public) instead of the consumer-side
last_applied_seq, so it tracks the client's send epoch rather than a
value that can lag arbitrarily far behind during a stall.
- HIGH: InputLeadController's release logic still ANDed the old
`lead > LEAD_MIN` gate onto the new depth-driven condition, so a
backlog the controller never caused still couldn't drain. Split into
two independent decisions: the seq-duplicate action follows real
depth alone; lead's own bookkeeping separately never drops below its
floor.
- MEDIUM: widen the CI driver's movement/stalled sampling margin
(run_seconds - 2.0, was - 0.5) and assert the peer is still in
multiplayer.get_peers() at sample time, since the old margin let the
check pass on residual starvation grace after a bot had already
disconnected.
- LOW: measure horizontal-only displacement in the human smoke test's
movement check — the old 3D-distance bar was beatable by pure
gravity settling with fully dead input.
- LOW: fix a real "clean stderr" violation (match_net.gd broadcasting
a departure notice to a peer whose ENet channels are already torn
down, including a second peer disconnecting in the same poll batch)
by deferring the notification to the next idle frame.
- Wire the server's per-slot stalled bit into the client debug overlay
for real — a prior commit message claimed this already reached the
overlay when only the CI gate actually read it.
Re-verified end-to-end against the real production RPC path (not just
unit tests in isolation, which is how the composition bug got past the
first round): a 2-bot CI match with a 1.5s host SIGSTOP freeze injected
mid-run, well past the 0.6s threshold the review reproduced the bug at,
now recovers cleanly on repeated runs with zero stderr noise.
An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:
CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.
HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.
MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.
MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.
MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.
MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).
LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.
LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.
LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.
New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
New InputLeadController (scripts/input_lead_controller.gd, standalone and
unit-tested like input_jitter_buffer.gd): fast attack (+3 immediately,
debounced to once per 30 ticks) on any server-reported starve, slow
release (-1 per 60 ticks, gated behind a one-time 2s clean-surplus bar)
otherwise, clamped [1, 12]. Deliberately the only thing that adapts
buffer depth - the server (InputJitterBuffer) stays a pure reporter, per
§3.3's explicit warning that multiple control loops acting on one plant
(buffer occupancy) oscillate and present as unattributable sticky
controls.
Wired into the client's per-tick input send: a lead change is realized as
extra distance between the client's outgoing sequence numbers and what
the server has consumed - an attack skips extra sequence numbers, a
release duplicates the current one (sent again, unincremented). The
server's ring buffer needs no special handling for either: a skipped seq
is an ordinary drop, a duplicated one is a same-seq resend already
discarded by the existing "already consumed" check.
Verified with real two-process runs: on a clean LAN, one early attack
(a momentary hiccup during connection setup) recovers via two releases
within the test's own ~4s window, settling back near minimum. Under
sustained 30% simulated loss, lead climbs to 7 via repeated attacks and
never releases while genuine loss continues - confirming the debounce,
attack, and release gates all fire on real conditions, not just in
isolated unit tests. Full regression suite, including the net-sim-latency
milestone gate, re-run clean.
Client now sends the last 4 ticks' actions per packet (newest-first,
already-supported by net_codec's wire format from Phase 1) instead of a
single action with no redundancy. Server gains a real per-slot ring
buffer (new InputJitterBuffer class, scripts/input_jitter_buffer.gd) that
consumes exactly one sequence number per physics tick: repeats the last
action on a starve, zeroes only after a sustained 500ms stall, and
reports real input_buffer_depth/last_input_seq/echo_client_send_ms in
every snapshot instead of the hardcoded zeros Phase 2 shipped with.
InputJitterBuffer is a standalone, scene-free RefCounted (same pattern as
net_codec.gd/net_interpolator.gd) specifically so it's unit-testable
against scripted arrival traces (tests/cases/test_input_jitter_buffer.gd):
sequential consumption, redundancy surviving a 3-packet burst loss (3.1's
own acceptance criterion), starvation repeat-then-zero timing, stale/
reordered packet handling, buffered-depth reporting, and ring-wraparound
slot-tagging safety.
One real bug found wiring this into a live match: the server's ring
buffer started counting its own "expected sequence" from 0 the instant a
player's slot was created - well before that player's first real packet
could possibly have arrived (connection handshake, arena/ship spawn all
take real time first). Since both sides only ever advance monotonically
with no resync mechanism, that gap between the server's arbitrary local
counter and the client's actual from-1 sequence numbers never closed,
so the ship simply never received the client's input (0m movement in a
two-process test). Fixed by seeding the buffer's expected-sequence
counter from the client's own numbering on first real ingest, rather
than assuming a shared from-zero baseline.
Verified with real two-process runs: clean baseline movement restored,
zero starvation observed under 25% random simulated input loss (well
above what redundancy-4 needs to fully absorb), and correct starve-then-
stall behaviour confirmed under 100% loss as a sanity check that the
mechanism isn't a silent no-op. Full regression suite, including the
net-sim-latency milestone gate, re-run clean.
Lands tasks 1.0-1.8 of multiplayer-todo.md: the pure-function test runner,
net_codec (wire format quantizers/pack-unpack), NetworkManager (ENet
transport, manual polling, min-RTT clock sync), MatchNet (handshake,
protocol/tick-rate gating, roster with team+ready state), lobby.tscn (team
columns, switch team, ready toggle), server_boot.tscn (headless dedicated
server with structured logging and an overrun watchdog), and main_menu.gd's
Host/Join-by-IP UI (connecting overlay, cancel, bounded failure path).
Followed by an adversarial review (Opus subagent) that found and fixed two
real bugs - an unvalidated player_name broadcast that let one client's
oversized name head-of-line-block the reliable channel for everyone, and a
server-side roster leak across a host/re-host cycle - plus three gaps in
the test suite itself where a claim of "verified" wasn't actually backed
by what the test checked. All five two-process smoke tests plus the
pure-function suite are green with the strengthened assertions in place.
Josh Creek