18 Commits

Author	SHA1	Message	Date
Josh Creek	aac00f148e	fix(tests): stop the ENet residual gate failing on host scheduling noise ... The client-bot smoke asserted remote_residual_position_p99 < 0.3. Commit 52ee1810 both passed and failed on that assertion, in runs three seconds apart on identical code: two clients in one run reported 0.324 and 0.187 with matching snapshot counts, scores and slot checks. A p99 over a few hundred samples is its worst handful, so on a shared CI host it measures how often the process was descheduled as much as how well the interpolator tracks. Locally the same test reports p95 0.025m and p99 0.081-0.149m; CI's p99 runs 2-4x higher on a healthy build, which is the entire margin the 0.3 bar had. Assert two bars instead of one. The tight numeric bar moves to p95, which is stable run to run, and p99 is bounded by the product's own REMOTE_VISUAL_MAX_OFFSET/REMOTE_VISUAL_MAX_ROTATION_DEGREES: past those the visual smoother stops absorbing a correction in a single step, so exceeding them is a real defect rather than a slow runner. Referencing the constants also means the test follows the product if that tolerance is ever retuned. This is a genuine trade, not a free win: a regression that pushed p99 from 0.2 to 0.35 while leaving p95 healthy now passes where it once failed. That band is exactly where the noise lives -- 0.324 was observed on a healthy build -- so the old bar could not tell that regression from a busy runner either, and paid for the ambiguity with false reds. Both percentiles are printed with their bars so a future failure shows which one moved. Checked first whether goal-driven kickoff teleports were polluting the metric; they are not. Both the ship and ball paths already skip accumulation across a reset_gen change.	2026-09-05 22:22:30 +01:00
Josh Creek	46fe696a58	fix(tests): measure cumulative travel, not displacement, in the ENet host check ... run_ci_host_check asserted input reached the server by comparing each bot ship's position against one recorded before the check forced a goal. But a goal's kickoff teleports every ship back to spawn (_begin_kickoff -> reset_ships), so that comparison measured only the distance covered since the last reset — a window whose length depends on when the sample lands relative to the kickoff rather than on whether input was flowing at all. It failed on master with peers at 0.51m and 0.23m against a 0.5m threshold: one passed by a centimetre, the other failed, with both connected, neither stalled, and every other assertion in the run green. The commit it failed on touches only training JSON, and the push two minutes earlier passed on identical game code. Accumulate per-tick path length in _await_recording_score instead, discarding any single-frame step over 2.0m as a teleport — Ship.max_speed (35 m/s) is hard-clamped each tick in _integrate_forces, so 60Hz caps legitimate travel at ~0.58m. Same 0.5m threshold now reads 28-75m across runs, and it is strictly stronger than before: it asserts input kept arriving for the whole wait rather than that the ship merely ended up somewhere else.	2026-08-24 08:40:06 +01:00
Josh Creek	24d6a547d6	fix(ci): isolate imported client test image	2026-08-21 20:16:01 +01:00
Josh Creek	ff725e1ffa	feat(multiplayer): §6.3 late joiners take a vacated slot at the next kickoff ... "Spectate now, take the slot at the next kickoff" was a print statement. The server logged it and never acted; on the client, _is_spectator was assigned once in _on_match_config_received and never revisited - and that handler returns early whenever _slots is non-empty, so no rebroadcast could promote an in-match spectator. The reconnect path only worked because a returning player is a fresh process. Server: late joiners are queued in arrival order and the queue is drained from _begin_kickoff, before the reset transforms are read, so a promoted player's ship is placed by that same kickoff and the controller swap lands on an already-frozen body. A slot is available only once its player has gone AND their 30s reservation has lapsed - §6.4 outranks §6.3, since taking a reserved slot would quietly break the reconnect promise. _abort_if_abandoned now counts a waiting spectator as somebody present, or the one person queued for the slot that just opened is dumped to the lobby at the moment they were about to get it. Client: new broadcast slot_assigned (reliable, channel 0). Broadcast because every client holds its own slot list and one naming the wrong peer keeps flying somebody else's ship as a remote body; reliable because no per-snapshot field would re-converge a client that missed it. The promoted client undoes what made the body remote - fresh interpolator, physics interpolation back on, offsets cleared - and deliberately does not unfreeze, clearing _local_prediction_ready so the next snapshot teleports it to a real authoritative pose first. The controller-attach block moved to _take_local_ownership rather than being copied. New --role=host-latejoin/--role=client-latejoin and --slot-reservation-seconds=. Verified 4/4 both sides: queued, NOT promoted merely because the reservation lapsed, takes the slot at the kickoff, same ship instance, and both peers independently measure ~45.7m under its input. Control with a 90s reservation: kickoff fires, nothing is promoted, the slot still reads the departed player's name.	2026-08-21 16:47:35 +01:00
Josh Creek	5714829c13	test(multiplayer): grade §6.4's reconnect from the returning player's side ... The disconnect scenario only ever asserted the server's bookkeeping, and the client's half was failing every run. run_disconnect_host_check ticked 60 physics frames past the reclaim and then shut the server down, so the reconnecting client - whose wiring check waits a 2.0s settle before it looks at anything - had its peer torn out from under it and reported "current_scene is not NetworkedMatch after 2.0s". The host printed PASS throughout, and the host was the side anyone read. The hold is now a real window (8s), and the host also asserts that the reconnected player's input reaches the server and moves the ship the server owns - every other assertion there is slot bookkeeping that would hold identically for a client whose input pipeline came back dead. Both position and connection state are sampled while the peer is still connected: the client leaves on its own schedule, and an end-of-hold sample reported still_connected=false for a good run. New --role=client-reconnect asserts the returning player is not a spectator, owns a slot with its own peer_id, has a real ship, rejoined a live match with the clock already known (§6.2 step 2's bootstrap), and can still drive. That set is chosen because a stale _last_match_config once made a reconnecting player a spectator, and that bug was visible in this scenario's own logs while it reported PASS. Verified 3/3 both sides. Control: rejoining while the slot is still occupied fails on is_player=false - and since the first control run reported it as the generic "lost its ship mid-drive", the spectator case is now diagnosed before the drive rather than after.	2026-08-21 16:25:25 +01:00
Josh Creek	866efa0d9b	fix(multiplayer): server no longer rate-limits a backlog it caused itself ... Closes task 5.10's three recording gaps, and the gap-closing found a real input-loss bug. Replay log: a failed write now ends the log permanently instead of desyncing every later record's framing; close() is called from _exit_tree with a summary, since the RefCounted destructor closes it implicitly but never says whether the log is complete; rejected packets are recorded with their reason in the kind byte (framing unchanged, FORMAT_VERSION 2 so "no rejects" differs from "this build never recorded them"). Recording is capped at 8 per peer per window - uncapped, the diagnostic is a remote disk-fill amplifier, since the attacker picks the packet rate. Uncapped totals live on MatchSim and survive the peer's disconnect. The bug: a 2s host stall has the client sending at 60Hz throughout, and ENet delivers that whole backlog in the first window after resume - 70 of an honest client's packets rejected as "rate limit exceeded". Redundancy does not cover it, because the dropped packets are contiguous: 0 of 70 rescued, and 82 of 923 sequences (8.88%, ~1.4s of input) never reached the server, against 0.00% with no stall. Every prediction gate passed. Fixed by granting each already-tracked peer a capped, two-window packet grace when the server detects its own wall-clock stall. Rate-limit rejects 70 -> 0, sequences missing 8.88% -> 0.00%, seq-guard rejects 9 -> 0. Controls on the unfixed build lost 4.34/7.52/7.86%. All three abuse roles still disconnect and no flood induced a stall, so the grace cannot be farmed. Also corrects an earlier wrong conclusion: the reviewer's free-flight p95 0.688 is real and reproduces on two processes with 0.0% snapshot loss. The plain --role=client drive fails the 0.5 free-flight bound in 3 of 8 runs because that drive is mostly a contact test - the harness comment already said so - leaving a cohort as small as 12 samples. Near-surface error is genuinely several times open-air error, so the calibrated bound now belongs to --exercise-free-flight alone and the plain role asserts the always-well-sampled all-cohort percentiles at 1.2/2.0, printing the free-flight numbers as reported-not-asserted. 6/6 plain runs pass where 3/7 failed; tightening to 0.3 still fails. tools/replay_dump.gd reads a log back: counts by kind, plus how much of the input sequence stream reached the server once redundancy is counted.	2026-08-21 16:10:43 +01:00
Josh Creek	e51dc765a2	test(multiplayer): report transport health on prediction-quality failures ... A percentile alone cannot tell "the predictor regressed" from "the client never received the data". The client gate now prints snapshot_loss / snapshot_age / rtt on every run, and on a quality failure with >20% loss says explicitly that the run was transport-starved. It deliberately does not convert the failure into a pass: a client that cannot receive snapshots is still a failed run, just a differently-diagnosed one. Both directions of the new branch verified non-vacuously (forced true so it fires and formats; restored so it stays quiet on a healthy run while the INFO line still prints). Records the investigation behind it in multiplayer-todo.md: the reviewer's 3-process p95 0.688 did not reproduce. An idle third process costs nothing (p99 0.094), a spectator costs a small but real amount (p99 0.094-0.146), and snapshot loss held at 0.0% even under 2x CPU oversubscription - all an order of magnitude inside the 0.5/2.0 gates. Also notes that a previously working class_name can silently drop out of the .godot class cache, which surfaces as a bogus parse error with nothing in git status to explain it.	2026-08-21 15:24:58 +01:00
Josh Creek	7a1668c902	fix(multiplayer): second adversarial review - Esc, stranded clients, clock ... A second adversarial review (this one able to RUN things, unlike the first) reproduced five defects. Fixing the critical and high ones. CRITICAL - Esc no longer left a networked match, and a client whose server vanished was stranded forever. Two independent bugs composing: _unhandled_input (added for spectator target cycling) overrode GameMode._unhandled_input and returned early for every non-spectator without ever calling super(), silently killing ui_cancel -> main menu; and NetworkedMatch never connected NetworkManager.disconnected_from_ server the way lobby.gd does. Measured: a client whose host exited emitted 7,235 engine errors in ~18s and only left because a test timer fired. Now 1 benign teardown error, and it returns to the main menu. HIGH - the match clock lost up to 3 seconds of regulation per goal. _on_goal_registered extended end_tick by the celebration only (resume_tick - goal_tick) and never by the 180-tick kickoff countdown that follows it, while _update_clock derived remaining time from the current tick regardless of _clock_running - so regulation drained during every stoppage. Measured 660 PLAYING ticks for a 14s match against 840 expected: exactly one WARMUP lost. The HUD also opened at 0:17 for a 14s match because the initial arm folded WARMUP into end_tick. Replaced the per-goal arithmetic with bank-and-rebase: entering any non-live state banks the remaining ticks, leaving it rebases end_tick off the banked value. That covers celebration and countdown together and cannot drift, since nothing has to predict how long a stoppage will be. clock_state and match_bootstrap now carry remaining_ticks, which is authoritative whenever the clock is stopped. Verified with the reviewer's own metric: 840 PLAYING ticks for a 14s match, exactly. MEDIUM - clients never froze at FULL_TIME/RESULTS. The freeze handling sat inside `if multiplayer.is_server()`, so a local player flew around for the whole 8s results screen while every other peer saw their ship parked. Not fixed, and now demonstrated rather than merely suspected: - The 30s slot reservation is keyed on display NAME, so a stranger can take a departed player's ship and the real player is then locked out (reproduced). Worse than first thought: MatchNet.local_player_name defaults to "Player" and uniqueness is never enforced, so collisions are the common case, not an attack setup. Needs a real identity token; §6.2 step 1 reserves auth_ticket for Phase 7. - §6.3's "late joiner takes the slot at the next kickoff" is unimplemented - _is_spectator is assigned once and never revisited - while the server logs that it happened. - Replay log still ignores store_* return values, never records malformed/rejected inputs, and close() has no caller. - --role=host-disconnect grades the reconnecting client on ~1s of life before the host quits, and never asserts the client owns _my_slot. Regression: 87 unit tests; free-flight LAN; transition gate 0.00%; goal cycle; spectator; disconnect and reconnect; full match to RESULTS.	2026-08-21 12:17:06 +01:00
Josh Creek	b5e9dff33c	fix(multiplayer): Phase 5 adversarial review fixes - reconnect, spectators ... An adversarial review found five real defects in the Phase 5 lifecycle work. Two were critical and both were verified against controls. CRITICAL - a reconnecting client silently became a spectator. _try_reclaim_slot() swapped slot.peer_id, but MatchSim caches the last match_config and replays THAT to whoever asks. A reconnecting client in a fresh process requested config, received the pre-disconnect peer-id array, could not find itself, left _my_slot null and fell through to the spectator path - no ship, no input, for the rest of the match. The evidence was already in my own disconnect-test logs ("no slot for this peer - spectating", my_slot_ok=false) and I dismissed it: the host-side check only asserted the SERVER reclaimed the slot, never that the returning client owned it. Config is now rebroadcast on reclaim. Verified: my_slot_ok=false -> true. CRITICAL - spectators received no snapshots at all. §6.3 says a spectator "receives identical snapshots (the snapshot is already a broadcast - zero extra server work)". That was only ever true of the body SEGMENT: _broadcast_snapshot unicasts one packet per SLOT, so a peer without a slot got nothing - no poses, no reset_gen, no match_state byte. Spectating was entirely non-functional. The segment is still shared, so this is one extra send per spectator. Verified against a control: 0 snapshots and state stuck at LOADING before, 361 snapshots and PLAYING after. HIGH - cycling the spectator camera to the ball was a type error. ShipCameraRig.target is declared `var target: Ship` and the rig reaches into ship-only API, so it would have fired the moment anyone cycled past the last ship. Cycling is ships-only; the rig already has its own ball-cam mode for watching the ball. MEDIUM - clients never received match_ended or overtime_started. Both emitted only inside server-side logic, so a client froze and returned to the lobby without a result and its timer never switched to overtime. Derived from replicated state instead of adding two more RPCs: the client already has the authoritative score, and the transition is the event. MEDIUM - the goal cinematic ignored its authoritative window. goal_tick and resume_tick arrived and were unused; the client started a fresh fixed-length timer on RPC receipt, so a reliable retransmit could run the celebration past the server's window and into the next kickoff. _goal_pause_seconds() now returns the time actually remaining, clamped so an elapsed window cannot produce a non-positive timer. Also added: a match_bootstrap RPC carrying state, score, clock and reset_gen to one peer. match_config alone carries arena and roster only, so a late joiner or reconnecting player had no score or clock until the next goal happened to fire. It is sent on join AND on every request_match_config retry - the join-time send has exactly the same race match_config already had (the server sends it before the peer has loaded the match scene and connected its listeners), which the control run exposed: state was reaching PLAYING via the snapshot byte, not the bootstrap. New test: --role=client-spectator asserts a slotless peer receives the snapshot stream, follows the lifecycle, agrees with the wire byte, and can cycle targets without ever handing the camera a non-Ship. Verified non-vacuous. The ball-contact steering now closes all the way to 1.2m instead of coasting from 3m, which was missing the ball outright in roughly 1 run in 4. Not fixed, and still open: the 30s slot reservation is keyed on the player's display name, so any peer can claim a departed player's ship by choosing their name. §6.2 step 1 reserves auth_ticket for Phase 7; this needs a real identity token, not a name. Regression: 87 unit tests; free-flight LAN; transition gate 0.00%; ball contact 4/4; goal cycle; full match to RESULTS/LOBBY; disconnect and reconnect; spectator; two-bot CI.	2026-08-21 11:34:48 +01:00
Josh Creek	a5cbc977b5	feat(multiplayer): Phase 5 tasks 5.6-5.10 - disconnects, spectators, replay log ... Completes Phase 5's implementation. Every task is verified at 1v1; the 3v3 phase gate itself has not been run and remains outstanding. 5.6/5.7 disconnects: a ship is never despawned. The slot keeps it and swaps the controller (--fill-bots gives it a bot, the default leaves it inert per §1.4), sets `stalled` immediately so the nameplate greys out rather than waiting ~500ms for the abandoned jitter buffer to starve, and reserves the slot for 30s keyed by player name so a reconnect gets the same ship back. 5.7 was a real bug, found by the test rather than by review: SlotInfo.controller was declared RLShipController, but the takeover swaps in an AIShipController or the base controller - the narrower type makes that assignment fail its type check, leaving the field pointing at the controller set_controller() just queue_free()d. It surfaced as controller_valid=false on the first run. The per-tick action write is now also gated on `is RLShipController`, since a disconnected slot's bot drives itself and overwriting it from a starving buffer would pin it to the departed player's last input. §6.4's two rules conflict: reserve for 30s, but abort when the last human leaves. Applied naively the abort wins instantly in a 1v1 and the reservation can never be redeemed, making reconnect unreachable exactly when it matters. Abort now waits for no connections AND no outstanding reservations. 5.8 spectators: a slotless peer spawns no ship and receives the same snapshot broadcast. HUDController.spectator_mode keeps the clock, score and goal celebration and hides only the ship instrument cluster - it previously push_error'd and bailed, leaving a spectator with a dead HUD. Camera cycles ships in slot order then the ball. --max-spectators caps it, counted from the live peer list so a dropped spectator cannot leak a unit of the cap. 5.9 escape respawn: new GameMode._on_bodies_respawned() virtual; NetworkedMatch bumps reset_gen through Phase 2's deferred path so the bump and the respawned pose land in the same broadcast. Single-player modes are unaffected - the base is a no-op. 5.10 replay log: scripts/replay_log.gd, --replay-log=<path>, storing the wire bytes verbatim in both directions rather than re-serialising - a re-encode would launder away precisely the malformed payload being chased. A live 6s match recorded 1115 records (557 inputs / 558 snapshots) and a stored snapshot decodes back to server_tick=100 match_state=WARMUP bodies=2. Note for future work: --check-only --script is the only thing that catches a parse error in networked_match.gd, because the unit runner never loads it. Two separate breakages passed the full unit suite while breaking every two-process run. A new class_name also needs --import before it resolves. Test surface: --role=host-disconnect (three-process 5.6/5.7 scenario), --match-length=<s>, --replay-log, --fill-bots/--no-fill-bots, --max-spectators. The ball-contact scenario now steers at the ball with closed-loop real input instead of a hand-tuned fixed heading, which 5.3 broke by adding KICKOFF_YAW_JITTER; thrusting while turning took it from 2/3 to 5/5. Regression: 87 unit tests; free-flight LAN p99 0.094m with 0 hard snaps; transition gate 0.00%; ball contact 5/5; lifecycle goal cycle and full match to RESULTS/LOBBY; disconnect+reconnect; two-bot CI.	2026-08-21 10:25:15 +01:00
Josh Creek	3d6906b981	feat(multiplayer): Phase 5 tasks 5.2-5.5 - clock, kickoff, goals, full time ... Implements the rest of the §6.2 lifecycle on top of 5.1's state machine. 5.3 kickoff: the server resets every body and broadcasts the RESULTING transforms, never a seed - §1's locked decision, because shared-seed determinism needs both sides to consume the RNG stream in identical order forever and the first randf() added to the reset path desyncs silently. Countdown is derived from server_tick on both peers, and a kickoff that lands after its own resume tick applies immediately and skips the countdown rather than scheduling into the past. 5.4 goals: goal_scored(scoring_team, score, goal_tick, resume_tick). Score is authoritative at sensor time, before any presentation. The reset moved OUT of the sensor path and into the kickoff at resume_tick, which is what stops the server resetting while clients are still mid-celebration. Engine.time_scale is never touched. 5.2 clock: tick-derived, no Timer and no _process polling. The goal pause shifts the absolute end_tick by (resume_tick - goal_tick) rather than pausing anything, so no float drift accumulates across goals. 5.5 full time: clock expiry -> FULL_TIME -> sudden death on a draw or RESULTS, golden goal in overtime, then LOBBY on both peers - clients return to the lobby, not the main menu. get_tree().paused is never used. Four bugs found and fixed while building this, each by a failing run rather than by inspection: - Tick order was load-bearing: _update_kickoff_countdown() clears the same _kickoff_resume_tick that _update_match_state() reads to leave WARMUP, so running the countdown first wiped the transition condition and the match sat frozen in WARMUP forever. - _apply_match_state resets _state_deadline_tick on every transition, so a GOAL_PAUSE deadline assigned before _set_match_state was wiped and the match never resumed. Deadlines are now owned by _apply_match_state. - Freezing "all bodies" is wrong on a client. Remote ships and the ball are permanently FREEZE_MODE_KINEMATIC and transform-driven; freezing them all unfroze the remote ones on the way back out, so they fell under gravity while the interpolator fought them - 210 hard snaps and an infinite p99. A client now freezes only the one body it simulates. - A frozen body never runs _integrate_forces, so the queued kickoff teleport was stranded by an immediate set_deferred("freeze", true). Freeze now happens on a strictly later tick, the same pattern Phase 2 used for _pending_reset_gen_bump_tick. Prediction and reconciliation are suspended while the match is not live: during a countdown or goal pause the local ship is frozen on both peers, and running delta transport over those frozen states produced a p95 position error of 2.4e10 m. Input keeps flowing so the server's jitter buffer does not starve into `stalled`. Also fixed: a kickoff can arrive before match_config, and body order is slot order - applying it early placed the BALL at positions[0], on top of the first ship, which the ball-cam reported as "target vector can't be zero" 95 times. It is now held until the roster exists. Test changes: the ball-contact scenario steered by a hand-tuned fixed heading, which 5.3 broke because kickoff applies KICKOFF_YAW_JITTER - it flew past the ball in 3/3 runs. It now closes the loop on the actual bearing using real input actions. Assertions that read a frozen ship (freeze, thrust) are gated on the match being live, and the hooks now survive the scene teardown at RESULTS instead of hanging on freed objects for the full timeout. Regression: 81 unit tests; free-flight LAN p99 0.143m and 80±20ms, both 0 hard snaps; transition gate 0.00%; ball contact 3/3; two-bot CI.	2026-08-21 10:01:39 +01:00
Josh Creek	9f28c02488	feat(multiplayer): Phase 5 task 5.1 - match lifecycle state machine ... Adds the §6.1 state machine, its broadcast, and the client side that follows it. Physics, freezing and input are deliberately NOT gated on state yet - 5.3 and 5.4 own freeze/unfreeze at kickoff and goal, and doing it here would change the conditions every Phase 4 prediction gate was measured under. scripts/match_state.gd holds the enum and transition table as pure data with no scene or RPC dependency, so the table is checked exhaustively rather than by example: every state reachable, every state has an exit, no self-transitions, abort-to-LOBBY from anywhere per §6.4, illegal shortcuts rejected, unknown values refused rather than coerced. The enum values are the wire format - match_state has been a u8 in the snapshot header since §2.4 - so a test pins them; only append, never renumber. The server validates every transition and push_errors an illegal one rather than following it. Clients deliberately do NOT enforce the table: authoritative state must be accepted, and a late joiner legitimately jumps straight to PLAYING. Two channels carry the state. state_change (reliable, channel 0) is prompt and carries an absolute at_tick, never a duration. The snapshot's match_state byte is the catch-up path for a client not yet sent a transition - a late joiner, or the window between scene load and the first RPC. The byte needs a tick guard, and this was found the hard way. Snapshots are unreliable_ordered on channel 2 and ordering holds only within a channel, so a state_change for tick N routinely arrives before an in-flight snapshot from tick N-2. Without the guard the client applies the new state then gets dragged back by the older byte, oscillating on every transition - observed directly as LOADING -> WARMUP -> LOBBY -> PLAYING -> LOBBY while running a deliberately-broken-byte control. Only a byte at least as new as match_state_since_tick is accepted. WARMUP_TICKS/GOAL_PAUSE_TICKS are honest placeholders so 5.1 drives real transitions to verify against; 5.3 and 5.4 replace them. The server also leaves LOADING immediately rather than waiting for scene_ready, which does not exist yet. New smoke flag --exercise-match-state, passed to both roles: the host forces a goal to drive a GOAL_PAUSE cycle, the client records the sequence and asserts every consecutive pair is legal, that ticks are monotonic, and that the wire byte agrees with its own state. Observed LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP with tick deltas matching the configured durations exactly. Verified against a control: hardcoding the snapshot byte back to 0 fails both the byte assertion and the transition-legality assertion. The byte is asserted separately from the RPC precisely because everything else in the check is RPC-driven and would pass with a dead byte - the same gap that hid the Phase 4 label bug (gotcha 47). Regression: 81 unit tests; 60s free-flight LAN (p99 0.148m, 0 hard snaps, marker 0/3364); transition gate 0.00%; ball contact; two-bot CI.	2026-08-21 09:31:22 +01:00
Josh Creek	75f485667b	feat(multiplayer): Phase 4 prediction correctness + two input-death fixes ... Closes Phase 4's outstanding action-sequence-correctness invariant, then fixes two server-side bugs an adversarial review of that work uncovered. Server simulation, bot observations, collision resources and tick rate are unchanged: the server_physics_parity trace is byte-for-byte identical to HEAD across 360 ticks including both ships' full observation vectors. 4.11 - prediction history filed under the ISSUING sequence _send_local_input filed each post-step predicted state under the timeline's estimate of the sequence the server would consume this tick, trailing issuance by input_lead. The body had integrated the intent issued under _input_seq, so predicted[S] held "state after the intent from now" while the server's authority for S is "state after action(S)". They agree only while the stick is still. Filing under _input_seq costs nothing: which action the ship uses is decided in LocalNetShipController.get_action() and is untouched. Every prior Phase 4 gate held its input steady, and a steady input cannot falsify a sequence label - the 60s runs honestly reported marker=0/3784. New --exercise-input-transitions role toggles thrust every 6 ticks; it is the only gate that can catch a label regression. Verified non-vacuous: the old label fails it at 50%. 4.12 - issued-but-unsimulated sequences, and the release path An attack (delta > 1) issues and sends several sequences for one local physics step. Those gap sequences had no recorded prediction, so a server ack of one reported missing_not_recorded - indistinguishable from ring loss, costing a teleport and resync suppression several times a minute. They are now recorded stateless via record_unsimulated() and answered with a new "skip" decision mode. Free-flight hard snaps: 25/8/4 -> 0/0/0. A release (delta == 0) re-recorded at the unchanged _input_seq, filing the current intent under a sequence that went out carrying a different action; LocalInputTimeline deliberately refuses to mutate an issued sequence, so the ring contradicted the wire. Recording is now skipped on release ticks. 4.13 - two Phase 3 bugs silently killing player input (a) InputJitterBuffer.consume() advanced last_applied_seq on every tick including a starve. Since ingest() discards seq <= last_applied_seq, one starve on a sequence the client had not sent yet stranded the stream one ahead of arrivals permanently - both sides advancing in lockstep, every honest packet discarded on arrival. The client's own input_lead release is enough to trigger it, so input died for ~30 ticks roughly every 6.5s on a clean LAN. Now only gives up on a sequence once strictly newer data proves it lost. Silent-client stall and ring-overflow resync are unchanged. (b) The seq-range guard bounded incoming seq against highest_ingested_seq, which only advances inside ingest(), which that guard gates. After a ~2s host hitch every packet was rejected forever with no diagnostic (600+ consecutive rejections reproduced via SIGSTOP). Third iteration of this guard; each previous version bounded against a value only the accepted path could advance. Adds an escape after 10 consecutive rejections, which grants an attacker nothing the rate limiter does not already bound. (c) The transitions gate reported PASS at 3.76% while input was completely dead, because suppression stops _record_metrics - a worse outage yields fewer samples and a LOWER rate. Now scales the required sample count with run length and asserts the wire's server_stalled bit. Reverting both fixes makes it fail at samples 292/600, server_stalled=true, input_lead=12. Fixing (a) also explained a residual the review had already traced: 151 of 151 action-marker mismatches were the server repeating a stale action on a starve, not a prediction defect. Marker is now 0.00% in all three conditions (was 1.7-2.5%), and free-flight p99 improved to 0.141/0.168/0.154m from 0.170/0.176/0.184m. Two pre-existing test defects fixed alongside: the ball gate asserted RTT-masking on a link with no RTT (flaked 2 in 5; now asserted only at rtt >= 20ms, 5/5 under latency), and the two-bot CI compared scores across a 3-5s window (now polls the scores the server actually held; note score_changed is emitted only on the client path). QA: 72 unit tests; 60s free-flight at LAN/80+-20ms/5% loss; transition gate in all three; 2.0s and 3.5s host-freeze recovery; ball contact x5; two-bot CI x3; all three abuse roles; net/match_net/clock/lobby smokes. Phase 4 sign-off still pending a human playtest at ~100ms RTT - the milestone asks how it feels, which no gate here answers.	2026-08-21 09:17:19 +01:00
Josh Creek	cf73074e27	fix(multiplayer): resolve composition regression from second adversarial review ... A second adversarial review of the previous fix commit found two of its nine fixes silently defeated each other: the seq-range guard (fix for a MEDIUM epoch-mismatch finding) capped the exact variable the ring-overflow resync (fix for the original CRITICAL finding) depends on, making the resync unreachable in production and recreating permanent input death at a lower failure threshold, reachable via ordinary server tick loss alone. - CRITICAL: rebind the seq-range guard to InputJitterBuffer's own highest_ingested_seq (now public) instead of the consumer-side last_applied_seq, so it tracks the client's send epoch rather than a value that can lag arbitrarily far behind during a stall. - HIGH: InputLeadController's release logic still ANDed the old `lead > LEAD_MIN` gate onto the new depth-driven condition, so a backlog the controller never caused still couldn't drain. Split into two independent decisions: the seq-duplicate action follows real depth alone; lead's own bookkeeping separately never drops below its floor. - MEDIUM: widen the CI driver's movement/stalled sampling margin (run_seconds - 2.0, was - 0.5) and assert the peer is still in multiplayer.get_peers() at sample time, since the old margin let the check pass on residual starvation grace after a bot had already disconnected. - LOW: measure horizontal-only displacement in the human smoke test's movement check — the old 3D-distance bar was beatable by pure gravity settling with fully dead input. - LOW: fix a real "clean stderr" violation (match_net.gd broadcasting a departure notice to a peer whose ENet channels are already torn down, including a second peer disconnecting in the same poll batch) by deferring the notification to the next idle frame. - Wire the server's per-slot stalled bit into the client debug overlay for real — a prior commit message claimed this already reached the overlay when only the CI gate actually read it. Re-verified end-to-end against the real production RPC path (not just unit tests in isolation, which is how the composition bug got past the first round): a 2-bot CI match with a 1.5s host SIGSTOP freeze injected mid-run, well past the 0.6s threshold the review reproduced the bug at, now recovers cleanly on repeated runs with zero stderr noise.	2026-08-20 18:26:12 +01:00
Josh Creek	2325313ad2	fix(multiplayer): adversarial review fixes for Phase 3 ... An Opus subagent's adversarial review of Phase 3 found a critical, silent, permanent bug plus eight smaller real issues, all empirically verified with real two- and three-process runs: CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a player's input once the un-consumed backlog exceeded the ring's capacity - a fresh arrival would land in the exact slot consume() was still waiting on, and since both counters only ever advance, the gap never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze: client movement dropped from ~26m to 0.00m at ~0.7s, worse under real loss (a lossy link lowered the fatal threshold to ~400ms), and reachable via ordinary clock drift with no external trigger at all. Fixed by tracking the highest seq ever ingested and having consume() jump directly to what the ring can still provide once the gap exceeds capacity, instead of starving through an unrecoverable span. Re-verified with a 3s freeze (well past the original threshold): full recovery. HIGH: InputLeadController's release logic was gated on its own past attacks (lead > LEAD_MIN) rather than the real server-reported depth, so a backlog it didn't itself cause was never drained. Fixed to gate on actual depth vs target. MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds" streak hard-reset to 0 on any clean window, letting a duty-cycled flood (burst, one clean window, repeat) sustain ~33x budget indefinitely with zero warnings. Replaced with a leaky-bucket accumulator immune to the same evasion by construction. MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock epochs (server process uptime vs. client's own from-zero seq numbering), so it never actually protected anything on a long-running server and could silently drop an honest client's input forever. Bound against the buffer's own last_applied_seq instead. MEDIUM: InputJitterBuffer.stalled was computed but never reached the wire - the one signal that would have made the ring-overflow bug visible anywhere. Now wired through _ship_to_net_body_state. MEDIUM: task 3.6's CI driver's assertions didn't depend on client input reaching the server at all, so it kept passing with the ring-overflow bug actively triggered. Added real ship-movement and non-stalled checks, sampled while bots are still connected (an initial attempt sampled after their own legitimate disconnect, which starves identically to the bug). LOW-MEDIUM: a lead change silently mislabelled _input_history's older entries, since the wire format has no per-entry seq field. Fixed by handling each delta case (ordinary/release/attack) on its own terms. LOW: bandwidth and snapshot-loss overlay metrics froze at their last value during a total outage instead of decaying - exactly when they matter most. Both now report honest post-outage values. LOW: a guard comment on NetworkManager._ping misdescribed the actual disconnect_peer() arguments in use. Corrected. New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data _instead_of_starving_forever, test_release_drains_a_backlog_it_never_ caused_itself, and client-abuse-flood-dutycycle (reproduces the exact duty-cycle evasion). Full regression suite, including the net-sim-latency milestone gate, all abuse roles, and the CI driver, re-run clean after every fix.	2026-08-20 15:28:44 +01:00
Josh Creek	caa9f44ab6	feat(multiplayer): Phase 3 task 3.6 - --test-bot client mode + CI driver ... networked_match.gd's client can now swap its input sampler for a real AIShipController (--test-bot, optionally --test-bot-model=<path>, defaulting to bots/promoted/medium.json) instead of PlayerShipController. Unlike the human sampler, AIShipController needs real scene context (get_parent() as Ship, plus ball/teammate/opponent discovery via groups), so it's parented onto the client's own ship via Ship.set_controller() rather than left floating - and the field's static type widened from PlayerShipController to the shared ShipController base to allow either. Known, documented limitation: this client's ships are all FREEZE_MODE_KINEMATIC and driven purely by transform writes, so nothing ever writes linear_velocity/angular_velocity onto them - the bot's observations always see every ship as stationary. It still produces well-formed, bounded actions from that degraded input (the policy network's output layer is bounded regardless of input quality), which is sufficient for this task's actual job: generating realistic sustained network traffic for CI, not winning matches. New CI driver (tests/networked_match_ci.gd/.tscn): a headless server plus two headless --test-bot clients playing a real match. task 3.6's original acceptance text also named "p95/p99 prediction error" and "snap count" - both Phase 4 concepts that don't exist until client-side prediction and its hard-snap threshold are built, so asserting on them now would be fabricated. What's checked instead: snapshot throughput (500+ received over an 8s run, comfortably above a 60Hz-scaled floor), and genuine cross-peer score agreement - forced via a deterministic server-side goal (bot-vs-bot scoring isn't reliable enough within a short run to gate on), with each client independently writing its own final score to a peer-id- keyed file for the host to compare against the other bot's, not just trusting the server's own view. "Clean stderr" is left as the external invocation's job, same as every other smoke test in this project. Verified with real 3-process runs (host + two bots): both clients independently confirmed identical scores after a forced goal, both saw 500+ snapshots, and all three processes exited 0 with clean stderr on a representative run (one run separately hit the same known, already- documented single-benign-error disconnect-timing race task 3.4's own abuse tests hit - not a new issue). Full regression suite, including the net-sim-latency milestone gate and the abuse-detection tests, re-run clean.	2026-08-20 13:40:48 +01:00
Josh Creek	b290f49143	feat(multiplayer): Phase 3 task 3.4 - input validation, rate limiting, disconnect policy ... MatchSim._recv_input now validates before decoding (§3.1 steps 2-3): per-peer rolling-1s rate limiting (packet count AND byte budget, dropping over-budget packets and disconnecting after 3 consecutive over-budget seconds), and framing validation (redundancy count and payload size checked against NetCodec's own layout before unpack_input ever runs, disconnecting after 20 malformed packets). Framing has to be validated explicitly rather than relying on decode failure: StreamPeerBuffer silently zero-fills past EOF instead of erroring, a finding from Phase 2's adversarial review. networked_match.gd's _on_input_received now rejects any seq claiming to be more than 20 ticks ahead of the current server tick (§3.1 step 4) and counts (rather than silently ignoring) input from a peer with no slot, for observability. Verified with two new permanent regression tests (networked_match_smoke.gd --role=client-abuse-malformed / client-abuse-flood) that call MatchSim._recv_input directly with garbage bytes and a legitimate-but- too-frequent flood, respectively, bypassing the honest client encoder entirely - the same thing a hostile custom client sending raw ENet packets would look like. Both confirm real disconnection, not just that the server tolerates the abuse. Two bugs surfaced by getting these tests to actually pass cleanly: a GDScript lambda-capture-by-value mistake in the tests themselves (a plain `var disconnected := false` mutated inside a signal-handler lambda never became visible to the enclosing function - fixed by capturing a single-element Array instead, which is captured by reference); and a narrow real race where NetworkManager's own ping/pong reply could target a peer that a concurrent abuse-triggered disconnect had just removed from the same poll() batch, now guarded. (Passing disconnect_peer's `force` parameter as an attempted fix for a related one-off benign error was tried and reverted - it made Godot's own peer-list bookkeeping inconsistent, producing hundreds of errors instead of one; verified empirically rather than assumed.) Full regression suite, including the net-sim-latency milestone gate, re-run clean.	2026-08-20 13:27:03 +01:00
Josh Creek	39a41c016c	feat(multiplayer): Phase 2 server-authoritative simulation, dumb client ... Implements tasks 2.1-2.7: NetworkedMatch spawns a deterministic slot layout from the lobby roster, the server drives each connected peer's ship via RLShipController fed by decoded client input and broadcasts 60Hz snapshots, and the client renders everything (including its own ship) from a per-body NetInterpolator with no local prediction yet. Dual-time remote entities split collider updates (present-time, for correct contacts) from $Visual updates (interp-delayed, for smoothness). Camera/HUD wiring and remote engine-flame VFX fell out of the existing Ship API for free once snapshots were flowing. Three real bugs found and fixed while getting a two-process test green: an RPC method named _input collided with Node's built-in _input virtual and broke the whole MatchSim autoload from loading; networked_match.gd never called NetworkManager.poll(), so nothing sent via RPC in this scene reached the wire despite Phase 1's manual polling being wired up everywhere else; and a match_config request/response fallback (added to close a startup race) could double-deliver once polling was fixed, requiring an idempotency guard. Verified with tests/networked_match_smoke: a real headless two-process host+client run shows the client rendering 31m of server-authoritative movement from a held forward-thrust input, with thrust_z=1.0 confirmed on the interpolated snapshot mid-drive and camera/HUD both wired. Full Phase 1 regression suite re-run clean alongside it. Task 2.8 (net_sim.gd latency/jitter/loss decorator) is not yet done; Phase 2's own gate needs it before it's fully met.	2026-08-20 08:42:13 +01:00