8 Commits

Author	SHA1	Message	Date
Josh Creek	2325313ad2	fix(multiplayer): adversarial review fixes for Phase 3 ... An Opus subagent's adversarial review of Phase 3 found a critical, silent, permanent bug plus eight smaller real issues, all empirically verified with real two- and three-process runs: CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a player's input once the un-consumed backlog exceeded the ring's capacity - a fresh arrival would land in the exact slot consume() was still waiting on, and since both counters only ever advance, the gap never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze: client movement dropped from ~26m to 0.00m at ~0.7s, worse under real loss (a lossy link lowered the fatal threshold to ~400ms), and reachable via ordinary clock drift with no external trigger at all. Fixed by tracking the highest seq ever ingested and having consume() jump directly to what the ring can still provide once the gap exceeds capacity, instead of starving through an unrecoverable span. Re-verified with a 3s freeze (well past the original threshold): full recovery. HIGH: InputLeadController's release logic was gated on its own past attacks (lead > LEAD_MIN) rather than the real server-reported depth, so a backlog it didn't itself cause was never drained. Fixed to gate on actual depth vs target. MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds" streak hard-reset to 0 on any clean window, letting a duty-cycled flood (burst, one clean window, repeat) sustain ~33x budget indefinitely with zero warnings. Replaced with a leaky-bucket accumulator immune to the same evasion by construction. MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock epochs (server process uptime vs. client's own from-zero seq numbering), so it never actually protected anything on a long-running server and could silently drop an honest client's input forever. Bound against the buffer's own last_applied_seq instead. MEDIUM: InputJitterBuffer.stalled was computed but never reached the wire - the one signal that would have made the ring-overflow bug visible anywhere. Now wired through _ship_to_net_body_state. MEDIUM: task 3.6's CI driver's assertions didn't depend on client input reaching the server at all, so it kept passing with the ring-overflow bug actively triggered. Added real ship-movement and non-stalled checks, sampled while bots are still connected (an initial attempt sampled after their own legitimate disconnect, which starves identically to the bug). LOW-MEDIUM: a lead change silently mislabelled _input_history's older entries, since the wire format has no per-entry seq field. Fixed by handling each delta case (ordinary/release/attack) on its own terms. LOW: bandwidth and snapshot-loss overlay metrics froze at their last value during a total outage instead of decaying - exactly when they matter most. Both now report honest post-outage values. LOW: a guard comment on NetworkManager._ping misdescribed the actual disconnect_peer() arguments in use. Corrected. New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data _instead_of_starving_forever, test_release_drains_a_backlog_it_never_ caused_itself, and client-abuse-flood-dutycycle (reproduces the exact duty-cycle evasion). Full regression suite, including the net-sim-latency milestone gate, all abuse roles, and the CI driver, re-run clean after every fix.	2026-08-20 15:28:44 +01:00
Josh Creek	caa9f44ab6	feat(multiplayer): Phase 3 task 3.6 - --test-bot client mode + CI driver ... networked_match.gd's client can now swap its input sampler for a real AIShipController (--test-bot, optionally --test-bot-model=<path>, defaulting to bots/promoted/medium.json) instead of PlayerShipController. Unlike the human sampler, AIShipController needs real scene context (get_parent() as Ship, plus ball/teammate/opponent discovery via groups), so it's parented onto the client's own ship via Ship.set_controller() rather than left floating - and the field's static type widened from PlayerShipController to the shared ShipController base to allow either. Known, documented limitation: this client's ships are all FREEZE_MODE_KINEMATIC and driven purely by transform writes, so nothing ever writes linear_velocity/angular_velocity onto them - the bot's observations always see every ship as stationary. It still produces well-formed, bounded actions from that degraded input (the policy network's output layer is bounded regardless of input quality), which is sufficient for this task's actual job: generating realistic sustained network traffic for CI, not winning matches. New CI driver (tests/networked_match_ci.gd/.tscn): a headless server plus two headless --test-bot clients playing a real match. task 3.6's original acceptance text also named "p95/p99 prediction error" and "snap count" - both Phase 4 concepts that don't exist until client-side prediction and its hard-snap threshold are built, so asserting on them now would be fabricated. What's checked instead: snapshot throughput (500+ received over an 8s run, comfortably above a 60Hz-scaled floor), and genuine cross-peer score agreement - forced via a deterministic server-side goal (bot-vs-bot scoring isn't reliable enough within a short run to gate on), with each client independently writing its own final score to a peer-id- keyed file for the host to compare against the other bot's, not just trusting the server's own view. "Clean stderr" is left as the external invocation's job, same as every other smoke test in this project. Verified with real 3-process runs (host + two bots): both clients independently confirmed identical scores after a forced goal, both saw 500+ snapshots, and all three processes exited 0 with clean stderr on a representative run (one run separately hit the same known, already- documented single-benign-error disconnect-timing race task 3.4's own abuse tests hit - not a new issue). Full regression suite, including the net-sim-latency milestone gate and the abuse-detection tests, re-run clean.	2026-08-20 13:40:48 +01:00
Josh Creek	9d8a8080ba	feat(multiplayer): Phase 3 task 3.7 - debug net overlay extension ... Extends net_debug_overlay.gd (Phase 1's RTT/offset display) with the rest of task 3.7's list: jitter (new RFC3550-style EWMA in NetworkManager, computed from raw per-sample RTT before Phase 1's own min-filtering, since that filter is deliberately jitter-insensitive by design), input buffer depth and input_lead (both already tracked client-side for task 3.3), snapshot loss (a new EWMA in networked_match.gd over each received snapshot's own server_tick gap - snapshots go out at a steady one-tick cadence, so a gap is direct evidence of a drop or reorder), snapshot age (computed on demand from the same bias-corrected tick estimate the interpolator itself uses), and bandwidth (new rolling per-second byte counters in MatchSim, on the two 60Hz hot-path channels only). Prediction error is deliberately omitted with a comment explaining why: there's no client-side prediction to measure until Phase 4. Verified values are live and plausible, not just present, by calling get_net_debug_stats() directly in a real two-process test and checking the numbers make sense: bandwidth matched the wire format's own byte math almost exactly (measured ~2400 B/s sent against a computed 40B x 60Hz, ~3540 B/s received against 59B x 60Hz), and buffer depth/lead/loss all moved in the correct direction between a clean LAN run and one under simulated 60ms latency + 10% loss. Full regression suite re-run clean.	2026-08-20 13:32:16 +01:00
Josh Creek	b290f49143	feat(multiplayer): Phase 3 task 3.4 - input validation, rate limiting, disconnect policy ... MatchSim._recv_input now validates before decoding (§3.1 steps 2-3): per-peer rolling-1s rate limiting (packet count AND byte budget, dropping over-budget packets and disconnecting after 3 consecutive over-budget seconds), and framing validation (redundancy count and payload size checked against NetCodec's own layout before unpack_input ever runs, disconnecting after 20 malformed packets). Framing has to be validated explicitly rather than relying on decode failure: StreamPeerBuffer silently zero-fills past EOF instead of erroring, a finding from Phase 2's adversarial review. networked_match.gd's _on_input_received now rejects any seq claiming to be more than 20 ticks ahead of the current server tick (§3.1 step 4) and counts (rather than silently ignoring) input from a peer with no slot, for observability. Verified with two new permanent regression tests (networked_match_smoke.gd --role=client-abuse-malformed / client-abuse-flood) that call MatchSim._recv_input directly with garbage bytes and a legitimate-but- too-frequent flood, respectively, bypassing the honest client encoder entirely - the same thing a hostile custom client sending raw ENet packets would look like. Both confirm real disconnection, not just that the server tolerates the abuse. Two bugs surfaced by getting these tests to actually pass cleanly: a GDScript lambda-capture-by-value mistake in the tests themselves (a plain `var disconnected := false` mutated inside a signal-handler lambda never became visible to the enclosing function - fixed by capturing a single-element Array instead, which is captured by reference); and a narrow real race where NetworkManager's own ping/pong reply could target a peer that a concurrent abuse-triggered disconnect had just removed from the same poll() batch, now guarded. (Passing disconnect_peer's `force` parameter as an attempted fix for a related one-off benign error was tried and reverted - it made Godot's own peer-list bookkeeping inconsistent, producing hundreds of errors instead of one; verified empirically rather than assumed.) Full regression suite, including the net-sim-latency milestone gate, re-run clean.	2026-08-20 13:27:03 +01:00
Josh Creek	5bbb319161	feat(multiplayer): Phase 3 task 3.3 - client-owned input_lead control loop ... New InputLeadController (scripts/input_lead_controller.gd, standalone and unit-tested like input_jitter_buffer.gd): fast attack (+3 immediately, debounced to once per 30 ticks) on any server-reported starve, slow release (-1 per 60 ticks, gated behind a one-time 2s clean-surplus bar) otherwise, clamped [1, 12]. Deliberately the only thing that adapts buffer depth - the server (InputJitterBuffer) stays a pure reporter, per §3.3's explicit warning that multiple control loops acting on one plant (buffer occupancy) oscillate and present as unattributable sticky controls. Wired into the client's per-tick input send: a lead change is realized as extra distance between the client's outgoing sequence numbers and what the server has consumed - an attack skips extra sequence numbers, a release duplicates the current one (sent again, unincremented). The server's ring buffer needs no special handling for either: a skipped seq is an ordinary drop, a duplicated one is a same-seq resend already discarded by the existing "already consumed" check. Verified with real two-process runs: on a clean LAN, one early attack (a momentary hiccup during connection setup) recovers via two releases within the test's own ~4s window, settling back near minimum. Under sustained 30% simulated loss, lead climbs to 7 via repeated attacks and never releases while genuine loss continues - confirming the debounce, attack, and release gates all fire on real conditions, not just in isolated unit tests. Full regression suite, including the net-sim-latency milestone gate, re-run clean.	2026-08-20 13:14:40 +01:00
Josh Creek	86a597f0f5	feat(multiplayer): Phase 3 tasks 3.1/3.2/3.5 - input redundancy + server jitter buffer ... Client now sends the last 4 ticks' actions per packet (newest-first, already-supported by net_codec's wire format from Phase 1) instead of a single action with no redundancy. Server gains a real per-slot ring buffer (new InputJitterBuffer class, scripts/input_jitter_buffer.gd) that consumes exactly one sequence number per physics tick: repeats the last action on a starve, zeroes only after a sustained 500ms stall, and reports real input_buffer_depth/last_input_seq/echo_client_send_ms in every snapshot instead of the hardcoded zeros Phase 2 shipped with. InputJitterBuffer is a standalone, scene-free RefCounted (same pattern as net_codec.gd/net_interpolator.gd) specifically so it's unit-testable against scripted arrival traces (tests/cases/test_input_jitter_buffer.gd): sequential consumption, redundancy surviving a 3-packet burst loss (3.1's own acceptance criterion), starvation repeat-then-zero timing, stale/ reordered packet handling, buffered-depth reporting, and ring-wraparound slot-tagging safety. One real bug found wiring this into a live match: the server's ring buffer started counting its own "expected sequence" from 0 the instant a player's slot was created - well before that player's first real packet could possibly have arrived (connection handshake, arena/ship spawn all take real time first). Since both sides only ever advance monotonically with no resync mechanism, that gap between the server's arbitrary local counter and the client's actual from-1 sequence numbers never closed, so the ship simply never received the client's input (0m movement in a two-process test). Fixed by seeding the buffer's expected-sequence counter from the client's own numbering on first real ingest, rather than assuming a shared from-zero baseline. Verified with real two-process runs: clean baseline movement restored, zero starvation observed under 25% random simulated input loss (well above what redundancy-4 needs to fully absorb), and correct starve-then- stall behaviour confirmed under 100% loss as a sanity check that the mechanism isn't a silent no-op. Full regression suite, including the net-sim-latency milestone gate, re-run clean.	2026-08-20 13:07:33 +01:00
Josh Creek	14698d4ccb	fix(multiplayer): adversarial review fixes for Phase 2 ... An Opus subagent's adversarial review of Phase 2 found real bugs the smoke tests couldn't catch, since constant-velocity dead reckoning still moves a ship far enough to pass a "moved > 1.0" check: - The interpolator never actually interpolated. NetInterpolator.to_tick() assumes physics_frame * TICK_MS == Time.get_ticks_msec() on the server, which is off by a steady ~45-55ms in practice (real startup work before the first physics step, widened by any dropped tick). Every sample_at() call took the extrapolation branch, 100% of the time, defeating the interpolation buffer entirely. Fixed with a shared, min-filtered rolling bias estimate in networked_match.gd, applied before every to_tick() call. - Goals caused a ~27m visual slide: _reset_gen was bumped before the queued teleport actually landed, so the client's buffer-clear kept exactly the stale in-goal sample and lerped a slide to the next, real one. Fixed by tracking the tick the goal was detected on and only bumping the generation once strictly later ticks confirm the teleport has landed - a naive "next _physics_process" boolean flag doesn't work, since a goal Area's body_entered fires before that same tick's _physics_process runs, not on the next one. - _local_input_sampler (a Node, never added to the tree) was never freed - this was the unexplained "3 resources still in use at exit" warning on every Phase 2 test run. - Ball angular velocity decoded 8x too small (rescale_avel was never called); get_server_time_estimate_ms() was used before the clock had synced; net_sim.gd's delayed-send timer stopped ticking while the tree was paused and didn't check connection status before firing; _broadcast_snapshot's ball index could silently break if a ship were ever despawned; declared-but-unemitted HUD lifecycle signals showed a permanently frozen timer widget. Also confirmed, empirically, several things the review checked and found fine: a hostile client sending malformed input cannot crash the server, skipping GameMode's super() drops nothing load-bearing, deterministic slot assignment is correct with 2 real simultaneous clients, and RPC authority enforcement genuinely rejects a forging client. All fixes verified with real two-process runs (including forcing an actual goal and reading the server's own broadcast stream) and temporary instrumentation, removed once each fix was confirmed. Full Phase 1 + Phase 2 regression suite, including the net-sim-latency milestone gate, re-run clean after every fix.	2026-08-20 12:43:33 +01:00
Josh Creek	39a41c016c	feat(multiplayer): Phase 2 server-authoritative simulation, dumb client ... Implements tasks 2.1-2.7: NetworkedMatch spawns a deterministic slot layout from the lobby roster, the server drives each connected peer's ship via RLShipController fed by decoded client input and broadcasts 60Hz snapshots, and the client renders everything (including its own ship) from a per-body NetInterpolator with no local prediction yet. Dual-time remote entities split collider updates (present-time, for correct contacts) from $Visual updates (interp-delayed, for smoothness). Camera/HUD wiring and remote engine-flame VFX fell out of the existing Ship API for free once snapshots were flowing. Three real bugs found and fixed while getting a two-process test green: an RPC method named _input collided with Node's built-in _input virtual and broke the whole MatchSim autoload from loading; networked_match.gd never called NetworkManager.poll(), so nothing sent via RPC in this scene reached the wire despite Phase 1's manual polling being wired up everywhere else; and a match_config request/response fallback (added to close a startup race) could double-deliver once polling was fixed, requiring an idempotency guard. Verified with tests/networked_match_smoke: a real headless two-process host+client run shows the client rendering 31m of server-authoritative movement from a held forward-thrust input, with thrust_z=1.0 confirmed on the interpolated snapshot mid-drive and camera/HUD both wired. Full Phase 1 regression suite re-run clean alongside it. Task 2.8 (net_sim.gd latency/jitter/loss decorator) is not yet done; Phase 2's own gate needs it before it's fully met.	2026-08-20 08:42:13 +01:00