feat(multiplayer): Phase 4 prediction correctness + two input-death fixes

Closes Phase 4's outstanding action-sequence-correctness invariant, then
fixes two server-side bugs an adversarial review of that work uncovered.
Server simulation, bot observations, collision resources and tick rate are
unchanged: the server_physics_parity trace is byte-for-byte identical to
HEAD across 360 ticks including both ships' full observation vectors.

4.11 - prediction history filed under the ISSUING sequence

_send_local_input filed each post-step predicted state under the timeline's
estimate of the sequence the server would consume this tick, trailing
issuance by input_lead. The body had integrated the intent issued under
_input_seq, so predicted[S] held "state after the intent from now" while
the server's authority for S is "state after action(S)". They agree only
while the stick is still. Filing under _input_seq costs nothing: which
action the ship uses is decided in LocalNetShipController.get_action() and
is untouched.

Every prior Phase 4 gate held its input steady, and a steady input cannot
falsify a sequence label - the 60s runs honestly reported marker=0/3784.
New --exercise-input-transitions role toggles thrust every 6 ticks; it is
the only gate that can catch a label regression. Verified non-vacuous: the
old label fails it at 50%.

4.12 - issued-but-unsimulated sequences, and the release path

An attack (delta > 1) issues and sends several sequences for one local
physics step. Those gap sequences had no recorded prediction, so a server
ack of one reported missing_not_recorded - indistinguishable from ring
loss, costing a teleport and resync suppression several times a minute.
They are now recorded stateless via record_unsimulated() and answered with
a new "skip" decision mode. Free-flight hard snaps: 25/8/4 -> 0/0/0.

A release (delta == 0) re-recorded at the unchanged _input_seq, filing the
current intent under a sequence that went out carrying a different action;
LocalInputTimeline deliberately refuses to mutate an issued sequence, so
the ring contradicted the wire. Recording is now skipped on release ticks.

4.13 - two Phase 3 bugs silently killing player input

(a) InputJitterBuffer.consume() advanced last_applied_seq on every tick
including a starve. Since ingest() discards seq <= last_applied_seq, one
starve on a sequence the client had not sent yet stranded the stream one
ahead of arrivals permanently - both sides advancing in lockstep, every
honest packet discarded on arrival. The client's own input_lead release is
enough to trigger it, so input died for ~30 ticks roughly every 6.5s on a
clean LAN. Now only gives up on a sequence once strictly newer data proves
it lost. Silent-client stall and ring-overflow resync are unchanged.

(b) The seq-range guard bounded incoming seq against highest_ingested_seq,
which only advances inside ingest(), which that guard gates. After a ~2s
host hitch every packet was rejected forever with no diagnostic (600+
consecutive rejections reproduced via SIGSTOP). Third iteration of this
guard; each previous version bounded against a value only the accepted
path could advance. Adds an escape after 10 consecutive rejections, which
grants an attacker nothing the rate limiter does not already bound.

(c) The transitions gate reported PASS at 3.76% while input was completely
dead, because suppression stops _record_metrics - a worse outage yields
fewer samples and a LOWER rate. Now scales the required sample count with
run length and asserts the wire's server_stalled bit. Reverting both fixes
makes it fail at samples 292/600, server_stalled=true, input_lead=12.

Fixing (a) also explained a residual the review had already traced: 151 of
151 action-marker mismatches were the server repeating a stale action on a
starve, not a prediction defect. Marker is now 0.00% in all three
conditions (was 1.7-2.5%), and free-flight p99 improved to
0.141/0.168/0.154m from 0.170/0.176/0.184m.

Two pre-existing test defects fixed alongside: the ball gate asserted
RTT-masking on a link with no RTT (flaked 2 in 5; now asserted only at
rtt >= 20ms, 5/5 under latency), and the two-bot CI compared scores across
a 3-5s window (now polls the scores the server actually held; note
score_changed is emitted only on the client path).

QA: 72 unit tests; 60s free-flight at LAN/80+-20ms/5% loss; transition
gate in all three; 2.0s and 3.5s host-freeze recovery; ball contact x5;
two-bot CI x3; all three abuse roles; net/match_net/clock/lobby smokes.

Phase 4 sign-off still pending a human playtest at ~100ms RTT - the
milestone asks how it feels, which no gate here answers.
This commit is contained in:
Josh Creek
2026-08-21 09:17:19 +01:00
parent 3d3024ae8a
commit 75f485667b
70 changed files with 2212 additions and 272 deletions
+86 -17
View File
@@ -4,7 +4,7 @@ Working document for the online multiplayer effort. `TODO.md` points here.
Everything below is written so an agent (or a person) can pick up a single numbered task, do it, verify it against a stated acceptance criterion, and stop. Sections 16 are the decisions those tasks assume; read them before picking up work in Phase 2 or later.
**Status: Phase 0 done, Phase 1 done, Phase 2 done, Phase 3 done — both phase gates passing.** A real two-process 1v1 runs: a headless server hosts, a client joins through the lobby, spawns into a server-picked arena, drives its ship via real held input (now with real redundancy, a server-side jitter buffer, and a client-owned adaptive `input_lead`), and renders server-authoritative movement (verified: 2231 m over a 2 s held-thrust drive, purely from interpolated snapshots) with camera and HUD attached — holding under `--net-sim-latency 80 --net-sim-loss 0.05`, Phase 3's own gate condition, on both the human smoke test and a two-headless-bot CI run (task 3.6) that forces a goal and confirms both bots independently agree on the resulting score. Input is now also validated and abuse-resistant: a hostile client sending malformed or flooded packets gets disconnected, verified with two permanent regression tests that bypass the honest client encoder entirely. No own-ship/ball prediction yet (Phase 4) — everything the client renders, including its own ship, comes from the interpolation buffer. See §7 for per-task status and evidence.
**Status: Phase 4's correctness gates are green; sign-off waits on a human playtest. Phase 3 needed two real fixes to get there (task 4.13).** The client now has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. The action-sequence-correctness gap that blocked Phase 4 was a mislabelled prediction history, now fixed and permanently gated (task 4.11). An adversarial review of that fix then found two Phase 3 bugs that were silently killing a connected player's input — periodically on a clean LAN, and permanently after any ~2 s host hitch — both now fixed with verified controls (task 4.13). What remains is not a measurement: nobody has played it at ~100 ms RTT to judge feel, which is what the milestone actually asks. See §7 for the implemented work, evidence, and the one open architectural question (a contact-cohort-only shadow world).
---
@@ -278,14 +278,19 @@ Comparing server state at tick `A` against **`predicted[A]`** — the client's o
```
- **`MAX_VISUAL_OFFSET = 0.4 m`**, not 2.0. The hull is 4 m long; a 2 m offset means being rendered half a ship-length from your own collider for ~280 ms, so you clip walls you visibly cleared — a felt bug in a game built around wall-riding. Beyond 0.4 m, show the correction. A visible correction is honest; an invisible 2 m lie is not.
**HARD SNAP**
**HARD CORRECT**
- Set body transform and velocities from the server values, **caught up** to the current tick (below), `reset_physics_interpolation()` on the body *and* on `$Visual`, zero the visual offset.
- **Backfill the prediction ring** for ticks `A..current`. Do **not** clear it — "missing `predicted[A]`" is itself a snap condition, so clearing guarantees the next snapshot also snaps, turning isolated snaps into bursts.
- Apply the same sequence-matched authoritative pose and velocity delta to the current local body, reset body and `$Visual` interpolation, and clear the visual offset. It is physically the same correction as soft correction; only its presentation differs.
**Catch-up replays the ship's own force formulas, not ballistic dead-reckoning.** Input-free extrapolation is not unbiased: turbo acceleration is `150 × 2.5 / 5` = **75 m/s²**, so a 6-tick catch-up lands ~0.375 m short *in the direction the player is accelerating*, on every snap, and the ship feels permanently rubbery under sustained thrust. Replaying 612 stored `ShipAction`s through `apply_thruster_forces` / `apply_rotation_forces` / `apply_drag_and_limits` / `apply_righting_torque` (`ship.gd:361-486` — pure float math, no Jolt dependency) is ~30 lines and ~1000 float ops.
**Settled Phase 4 decision — delta transport, not one-body replay.** For every matched snapshot, overwrite `predicted[A]` with authority, transport its pose and linear/angular-velocity delta through each retained state `A+1..current`, and apply that same delta once to the live local Jolt body. This keeps retained history coherent, so a later snapshot does not correct an already-corrected pre-delta trajectory a second time.
> This is **not** world rollback and does not touch locked decision 1. It replays one body against a frozen world and needs no determinism guarantee.
Do **not** analytically replay stored actions. That approximation cannot reproduce Jolt integration or contact manifolds (friction, restitution, walls, ships, and ball), therefore it becomes least trustworthy exactly where reconciliation is most noticeable. This is still neither whole-world rollback nor a change to server physics: it is client-only state transport around a server-authoritative simulation.
For reset generation changes, place exact authority, begin a new history epoch, and do not consume pre-reset actions. For missing or overflowed history, place authority once and suppress stale acknowledgements until a new matched sequence is recorded; never manufacture future history by filling it with one stale authority state.
> Same-sequence **pre-correction** residual remains diagnostic telemetry. With a server input jitter buffer, it is not by itself a presentation-quality gate: the server may have integrated an action at a different physical instant from the client. Acceptance must report it separately by free-flight/contact/reset/resync cohort, while gating post-correction/presentation error and hard-snap behaviour.
>
> That the two sides integrate the **same action** for a given sequence is a separate claim, and a checkable one — it is what the action marker and task 4.11's `--exercise-input-transitions` gate exist for. Keep the two apart: "right action, different instant" is expected here; "wrong action" is a bug, and was one.
### 4.5 Camera and visuals
@@ -881,20 +886,79 @@ No own-ship prediction yet: the client renders everything, including its own shi
| # | Task | Acceptance |
|---|---|---|
| 4.1 `[D:3.1]` | `LocalNetShipController`: single sample per tick, `copy()`, input history ring | Exactly one `get_action()` per tick; no aliasing in the history |
| 4.2 `[D:4.1]` | Prediction state ring (128) and snapshot→`predicted[A]` matching | `predicted[A]` resolves for every snapshot on a clean link |
| 4.3 `[D:4.2, 0.14]` | `net_ship_predictor.gd`: snap-vs-blend decision, full-immediate velocity correction, teleport queue, `reset_gen` handling, **ring backfill after a snap** | A snap is never followed by an immediately-forced second snap |
| 4.4 `[D:4.3, 0.2]` | Visual offset with `_tick_scaled(0.88)` decay, `MAX_VISUAL_OFFSET = 0.4`, `reset_physics_interpolation()` on body **and** `$Visual` | No mesh smear on snap; no visible offset beyond 0.4 m |
| 4.5 `[D:4.3]` `[P]` | Catch-up by **replaying stored `ShipAction`s** through the ship's own force formulas | No directional bias under sustained turbo; ship does not feel rubbery |
| 4.6 `[D:4.3]` | **Ball local prediction**: dynamic locally from the tick your predicted ship contacts it for `min(RTT, 250 ms)`; server ball applied to a shadow copy throughout; blend back over 150 ms, hard-snap past 3 m. Triggered by the existing `Ship.ball_contact` (`ship.gd:115`) | Your own touches register visually on contact, not ~RTT later; behind a setting |
| 4.7 `[D:4.4]` `[P]` | Tuning pass with debug-menu sliders: snap thresholds, decay `k`, `MAX_VISUAL_OFFSET`, `INTERP_DELAY` | Values recorded in this document once settled |
| 4.8 `[D:4.4]` `[P]` | Prediction-error telemetry (p50/p95/p99, snap rate) into the overlay and the CI assertions | Snap rate <1/min in normal 1v1 play at 80 ms simulated RTT |
| **4.9** `[D:4.4]` | **L1 — extrapolate remote `$Visual` to present time** (§5.6): render remote ships and the ball at `server_time_est` rather than `server_time_est - INTERP_DELAY`, feeding the residual through 4.4's soft-correct pipeline. **Collapses §4.1's dual clock** — collider and visual share one time, so §5.4b's `_process`/`_physics_process` split for remote bodies is removed. Keep interpolation behind a flag for A/B | **≈30 ms off world response** (174 → ~144 before L4). Measured p99 extrapolation error < 0.3 m and < 5°; correction pops are visible on hard direction changes and nowhere else; A/B against the interpolated path is a deliberate, recorded judgement |
| **4.10** `[D:4.9]` `[P]` | **L3 — adaptive jitter-buffer depth**: target 0 on links with jitter below a threshold, rising to 1+ under jitter, replacing §3.3's fixed `target_depth = 1` | ≈8 ms off world response on clean links with no increase in starve rate; degrades to today's behaviour under `--net-sim-jitter 20` |
| 4.1 `[D:3.1]` | **DONE.** Immediate local input with immutable sequence/redundancy bookkeeping; server/training action semantics unchanged | 60 unit tests and 60s LAN/jitter/loss runs pass |
| 4.2 `[D:4.1]` | **DONE.** 128-entry sequence-tagged prediction history and snapshot matching | Same-sequence free-flight samples resolve in all 60s runs |
| 4.3 `[D:4.2, 0.14]` | **DONE.** Atomic staged reconciliation, delta rebase, epoch/reset and missing-history recovery | No free-flight hard snaps across LAN, 80±20ms, or 5% loss 60s runs |
| 4.4 `[D:4.3, 0.2]` | **DONE.** Client-only bounded position and rotation visual offsets/decay; interpolation reset | Free-flight p99 raw residual ≤0.207m; exposed visual p99 0m in final matrix |
| 4.5 `[D:4.3]` `[P]` | **REJECTED / SUPERSEDED.** Analytic one-body action replay was removed in favour of same-sequence delta transport | Jolt/contact nondeterminism makes replay unsuitable; see §4.4 |
| 4.6 `[D:4.3]` | **DONE.** Client-only dynamic proxy, authoritative shadow, RTT-limited reveal, 150 ms blend and 3 m handoff | Final contact run: same-frame reveal, 4 blends, max 152ms, no hard handoff |
| 4.7 `[D:4.4]` `[P]` | **DONE.** Client-only debug keys tune thresholds, decay, visual offset and remote-present A/B | Defaults remain 2m/60°/0.4m and render-only tuning never reaches server/training |
| 4.8 `[D:4.4]` `[P]` | **DONE.** p50/p95/p99 residual telemetry, elapsed-time snap rate, reason/cohort counters | Final free-flight p99: LAN .150m; 80±20ms .149m; 5% loss .207m; zero hard snaps |
| **4.9** `[D:4.4]` | **DONE.** Present-time remote visual extrapolation, angular integration, and render-only residual correction; delayed interpolation remains an A/B debug mode | Final two-bot present-time p99 ≤.208m / 3.146°, below .3m / 5° gate |
| **4.10** `[D:4.9]` `[P]` | **DONE.** Signed starvation sentinel and client hysteresis/cooldown; headless `--test-bot` remains target depth 1 | Jitter run observed starvation fallback; stable runs preserve safe target behavior |
> **Ball prediction is not optional and not Phase 8.** With §4.1 in place the touch registers correctly on the server, but the ball still *renders* a third of a beat late — your ship visibly passes through it before it moves. In a game whose entire point is hitting a ball, that is the difference between "networked" and "broken", and it is the same machinery as own-ship prediction applied to one more body. Do it while the prediction code is warm. Buffering server ball state into a *shadow* copy (rather than discarding it) is what lets you measure disagreement continuously instead of discovering a 3 m error at window end.
**Phase gate — MILESTONE:** at simulated 100 ms RTT both the ship and the ball feel local; corrections are invisible in free flight and read as bumps on contact. **World response measures ≤130 ms at 60 ms RTT** (§5.6's L1 + L4 target), on whatever graphics settings the machine is running.
| 4.11 `[D:4.2]` | **DONE.** Prediction history is filed under the **issuing** sequence, and a forced-input-transition trace gates the label | Marker mismatch 0.001.3% (was 9.3% LAN / 24% at 80±20ms); control run at the old label fails the same gate at 50% |
| 4.12 `[D:4.11]` | **DONE.** Issued-but-unsimulated (attack-gap) sequences are recorded and skipped rather than diagnosed as history loss; the release path no longer re-files an already-issued sequence | Free-flight hard snaps 0 across all three 60 s conditions, down from 25/8/4 `missing_not_recorded` |
| **4.13** `[D:4.12]` | **DONE — two server-side input-death bugs found by adversarial review, both reproduced and fixed with controls.** A starve no longer advances past a sequence the client has not sent; the seq-range guard can no longer latch shut permanently | Marker 0.00% in all three conditions (was 1.72.5%); 2.0 s and 3.5 s host freezes now recover; control runs with each fix reverted fail the gate |
**Phase gate — correctness gates MET; the milestone's felt-quality half remains untested.** The action-sequence-correctness gap is closed and permanently gated (4.11), the two seq-delta paths it exposed are fixed (4.12), and an adversarial review's two server-side input-death bugs are fixed with controls (4.13). What has *not* happened is the original milestone's actual subject: nobody has played this with hands on a controller at ~100 ms RTT to judge whether ship and ball feel local and whether contact corrections read as bumps. Numbers cannot answer that, and the contact cohort is where the remaining known weakness lives (see the shadow-world note below). Sign off after a human playtest, not before.
> **Read 4.13 before trusting any earlier Phase 4 evidence.** Until this session the server was silently discarding a connected player's input for ~30 ticks roughly every 6.5 seconds on a clean LAN, and permanently after any ~2 s host hitch. Every Phase 4 number recorded before 4.13 was measured through that, and the gates reported green throughout — for the same reason they missed the label bug in 4.11: a steady input cannot distinguish "the server repeated my last action" from "the server applied my real action".
**The mislabelled prediction history, and why every earlier gate missed it.** `_send_local_input` filed each post-step predicted state under `_local_net_controller.last_applied_seq` — the timeline's *estimate of the sequence the server would consume this tick*, which trails issuance by `input_lead`. The body had actually integrated the current raw intent, issued under `_input_seq`. So `predicted[S]` held "state after integrating the intent from now" while the server's authority for `S` is "state after integrating `action(S)`", sampled `input_lead` ticks earlier. The two agree **only while the commanded action is constant** — and every Phase 4 acceptance trace held its input steady (`move_forward` held, or the free-flight hover alternating on a 0.7 s/0.3 s period). A steady input cannot falsify a sequence label: the marker reads 0/N under a correct and an incorrect label alike. The 60-second free-flight runs genuinely reported `marker=0/3784`; the instrument was fine, the trace was blind.
Filing the state under `_input_seq` fixes it and costs nothing. The code comment that had rejected this ("avoids turning client prediction into an input-delay queue") conflated *which action the ship uses* — decided in `LocalNetShipController.get_action()`, still the raw current intent, still immediate, untouched by this change — with *which sequence its resulting state is filed under*. Measured with `--exercise-input-transitions` (below):
| condition | `input_lead` | old label | filed under `_input_seq` |
|---|---|---|---|
| LAN | 1 | 35/376 (9.3%) | 06/456582 (01.3%) |
| LAN, adversarial toggle phase | 1 | 289/576 (50.2%) | — |
| 80±20 ms | 3 | 97/404 (24%) | 0/424 (0%) |
Mismatch scales with `input_lead`, exactly as the mechanism predicts. It also **cut pre-existing `missing_not_recorded` hard snaps 4×** on the 60 s LAN free-flight run (25 → 6, 24.8/min → 6.0/min): the old label's per-tick +1 cursor was an estimate that could drift off the sequence the server actually acknowledged, while an issued sequence is by construction the thing the server acknowledges.
**Task 4.12 — the two seq-delta paths, and what is left.** Relabelling exposed two further places where the history disagreed with the wire, both now fixed:
- **Attack gaps (`delta > 1`).** The lead controller skips sequence numbers to buy server buffer margin. Those sequences are filled with repeat-last actions and genuinely **sent**, and the server genuinely acknowledges them — but the client took exactly one physics step that tick, so no post-step state exists for them. They were simply absent from the ring, which `compare_authoritative` could only report as `missing_not_recorded`: indistinguishable from real ring loss, and therefore a hard snap, a full authority teleport, and armed resync suppression **several times a minute during ordinary play**. They are now recorded stateless via `record_unsimulated()` and report their own `unsimulated_gap` status, which `NetShipPredictor.decide()` answers with a new `"skip"` mode — no correction, no teleport, no suppression, no snap counted, its own metrics cohort. The next simulated sequence (a tick or two later) reconciles normally. **Result: free-flight hard snaps went from 25 / 8 / 4 to 0 / 0 / 0** across the LAN, 80±20 ms and 5%-loss 60-second runs; the doc's own long-standing target was <1/min and LAN was measuring 24.8/min.
- **Release (`delta == 0`).** `_send_local_input` re-recorded at the unchanged `_input_seq`, filing the *current* intent under a sequence that had already gone out carrying a different action. `LocalInputTimeline.issue()` deliberately refuses to mutate an already-issued sequence ("may be in flight or consumed"), so the ring was contradicting the wire outright. Recording is now skipped entirely on a release tick; the existing `predicted[S]` is already correct, and the extra unlabelled local step is precisely the tick of latency the release exists to recover.
**The residual is solved — it was not a prediction bug at all.** An adversarial review intersected every sequence the server starved on against every sequence the marker flagged, across five two-process runs: **151 of 151 mismatches were the server repeating a stale action on a starve**, zero unexplained. When the server starves on seq `S` it repeats `action(S-k)` but still acks `S`, so the snapshot's `thrust_z` honestly describes a different action than `predicted[S]` — the marker was correctly reporting a real client/server disagreement that prediction did not cause and could not fix. The apparent correlation with `input_lead` was a confound: the conditions that raise the lead are the conditions that produce starves. Fixing the starvation cause (task 4.13 below) took the marker to **0.00% in all three conditions**, including 80±20 ms and 5% loss where it had been 1.72.5%.
Two sub-findings from that investigation, recorded because both are counter-intuitive: `dequantize_thrust_z_bin(quantize_thrust_z_bin(0.0))` returns **0.142857**, not 0.0 (7 bins over [-1,1], `roundi(3.5) == 4`), so a server-reported `thrust_z` of 0.14 literally means "exactly zero" — the 0.26 threshold absorbs it, as designed. And `_pending_local_reconciliation` keeps only the newest snapshot, so acks are dropped whenever two snapshots land in one physics tick: **the marker under-samples, and the true action-disagreement rate is higher than it reports.**
> **The client-only shadow Jolt world is still the open question, but it is now scoped to the contact cohort alone.** Even perfectly labelled, the client predicts contacts against remote ships and the ball sitting at interpolated-*delayed* positions, so a contact-cohort prediction cannot be sequence-correct in the live world — no amount of bookkeeping fixes that, and a shadow world is the only thing that does. It is a large subsystem and effectively the whole-world rollback §1's locked decisions set out to avoid, so **do not build it before a playtest says the contact cohort actually reads badly to a human.** Free flight no longer needs it.
**New smoke role — `--exercise-input-transitions`.** Toggles forward thrust every 6 physics ticks (~100 ms) with alternating yaw, and asserts the action marker stays under 5% mismatch over ≥200 samples. This is the **only** gate here that can catch a sequence-label regression, for the reason above, so it must not be folded into the steady-input free-flight run:
```
godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=host --drive-seconds=8
godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=client --drive-seconds=8 --exercise-input-transitions
```
Verified against a working control: reverting the one-line label makes this gate fail at 50.2% mismatch, so it is not vacuous.
### Task 4.13 — two server-side input-death bugs the Phase 4 gates could not see
Both are **Phase 3 code**, both predate this session, and both were found by an adversarial review of the Phase 4 changes rather than by any gate. Neither was caused by 4.11/4.12; both are squarely in the way of Phase 4's *feel* milestone, so they are fixed here.
**(a) A starve stranded the input stream one sequence ahead of arrivals — permanently.** `InputJitterBuffer.consume()` set `last_applied_seq = expected` on **every** tick, including a starve. Because `ingest()` discards anything `seq <= last_applied_seq`, a single starve on a sequence the client had not sent yet left the server permanently one ahead: both sides then advance one per tick, the gap never closes, and **every honest packet is discarded on arrival**. The client's own `input_lead` RELEASE (`delta == 0`, which deliberately issues no new sequence for one tick) is sufficient to trigger it — so this fired roughly **every 6.5 seconds of ordinary play on a clean LAN**, blacking out input for 30 ticks until the lead controller's `MIN_CHANGE_INTERVAL_TICKS` debounce permitted a +3 attack to jump the client clear. The reviewer measured 2 blackouts in a 23 s run and 2 in a 30 s run, with the host applying the *same repeated action* for 30 consecutive ticks while the wire carried fresh input every one of them. Fixed by only giving up on `expected` when strictly newer data has arrived, which proves it lost rather than merely late. Both escape paths are untouched: a silent client still zeroes and stalls on `STARVE_ZERO_TICKS`, and a far-behind consumer still hits the ring-overflow resync.
**(b) The seq-range guard was a one-way door.** `_on_input_received` bounded incoming `seq` against `jb.highest_ingested_seq + RING_SIZE` — but `highest_ingested_seq` only ever advances *inside* `ingest()`, which that same guard gates. Once a client's live sequence got more than 32 ahead (a host stall drops the intervening packets wholesale, since input is unreliable), every subsequent packet was rejected, the bound could never move again, and **that player's input was dead for the rest of the match with no diagnostic**. Reproduced with a 2 s `SIGSTOP` host freeze: 600+ consecutive rejections, the server applying zero thrust across 1300 sequences while the client's wire carried full thrust throughout. This is the **third** iteration of this guard, and the structural lesson is that each previous version bounded against a value only the accepted path could advance. Fixed by keeping the bound but adding an escape: after `SEQ_REJECT_RESYNC_LIMIT` (10) consecutive rejections, accept and let the existing resync machinery re-establish the baseline. This grants an attacker nothing — walking the epoch forward by sustained rejection costs the same packets as walking it forward by acceptance, and §3.4's rate limiter already bounds that rate.
**(c) The gate printed PASS while input was permanently dead.** The `--exercise-input-transitions` gate reported `SMOKE PASS` at 3.76% mismatch on a run where input was completely dead, because *suppressed reconciliation stops calling `_record_metrics`* — so the worse the outage, the fewer marker samples and the **lower** the reported mismatch rate. Every other assertion in that path (`local_prediction_ok`, `moved > 1.0`) reads the client's own action and position, which a client flying purely on prediction satisfies perfectly. Fixed by scaling the required sample count with run length (`max(200, drive_seconds * 30)`, half of nominal 60 Hz) and asserting the wire's `server_stalled` bit. **Verified non-vacuous:** reverting both fixes and re-running the 3.5 s freeze fails at `samples 292/600` with `server_stalled=true` and `input_lead=12` (LEAD_MAX) — while reporting `marker=1/292 = 0.34%`, which the old gate would have passed.
**QA matrix, re-run in full after 4.11 + 4.12 + 4.13** (all green): **72 unit tests**; 60 s free-flight at LAN / 80±20 ms / 5% loss — p99 raw **0.141 / 0.168 / 0.154 m**, exposed visual p99 0.000 m, **0 hard snaps in every condition**, marker 0/3484, 0/3049, 0/3397; forced-input-transition gate at LAN, 80±20 ms **and** 5% loss, all **0.00%**; 2.0 s and 3.5 s `SIGSTOP` host-freeze recovery; ball contact ×5; two-bot CI ×3; all three abuse roles; `net_smoke`, `match_net_smoke` (incl. `host_recycle`), `clock_smoke`, `lobby_smoke`.
Note the free-flight p99 **improved** (0.170/0.176/0.184 → 0.141/0.168/0.154) and `input_lead` now sits at 1 on LAN instead of oscillating to 34. Both are downstream of 4.13(a): the periodic blackouts were degrading prediction accuracy and driving the lead controller.
**Two test defects fixed alongside, both pre-existing and both surfaced by 4.13(a):**
- **Ball-contact gate flaked 2 in 5.** `ball_proxy_moved_before_authority_count` requires the predicted proxy to have visibly moved *before the next authoritative ball state arrives* — but snapshots land every ~16.7 ms at 60 Hz, so on a loopback LAN the entire pre-authority window is about one physics tick and observing it is a coin flip. It is also the least interesting case: the counter measures RTT-masking, and LAN has no RTT to mask. Measured 5/5 passes (count 23) at `--net-sim-latency=80`. Now asserted only when `NetworkManager.rtt_ms >= 20`, with the same-frame reveal — the test's real claim, correct in every run either way — carrying the gate on LAN. Runs asserting the masking behaviour should pass `--net-sim-latency`.
- **Two-bot CI compared scores across a 35 s window.** The host checked each client's recorded score against its own score at *read* time, but clients write theirs several seconds earlier; any goal in between failed the run with both bots agreeing perfectly with each other. Latent until 4.13(a) made the bots effective enough to reliably score a second goal — then it failed 2 of 3 runs, every failure `server=2` vs `both clients=1`. The host now polls and records every score it actually holds, and asserts both clients agree **with each other** and that what they saw is a state the server genuinely passed through. 3/3 green, including a run ending 11 where the clients had recorded 01. (Polling, not `score_changed`: that signal is emitted only in `_on_score_update_received`, the *client* path — the server mutates `score` directly in `_record_goal` and never emits. Connecting to it recorded nothing but the initial 00.)
> **Follow-up, not done:** `LocalNetShipController.last_applied_seq` is now write-only and `LocalInputTimeline.consume()` is vestigial to the reconciler (still unit-tested, still advancing `_last_applied_action`, but nothing reads the result). Left in place rather than removed as unreviewed scope — but it now looks load-bearing and is not.
### Phase 5 — Match lifecycle
@@ -1030,6 +1094,11 @@ No own-ship prediction yet: the client renders everything, including its own shi
44. **When adding a "is this connection still healthy" check to a test, sample it while the peer is still actively connected — not after its own end-of-run disconnect, which produces symptoms indistinguishable from the bug being checked for.** Fixing gotcha 43 first sampled `InputJitterBuffer.stalled` and ship movement *after* the full test run (plus a buffer for score-file writes), which meant both readings came from ~4s after the bot had already legitimately shut down — a departed peer's input naturally starves and goes `stalled=true` too, and that's correct, expected behaviour, not the bug. Move the check to a moment still comfortably inside the peer's own active connection window.
45. **Two fixes landed in the same commit, each individually correct in isolation, can share a variable and silently cancel each other out — and a fix's own unit test can miss it by testing the mechanism in isolation from the thing that defeats it.** Gotcha 39's resync fix and gotcha 42's guard rebound were reviewed, tested, and verified independently, each against its own scenario, both passing. Combined, the guard caps the exact variable (`highest_ingested_seq`) the resync's own trigger condition depends on, making it permanently unreachable — recreating the original critical bug at a *lower* failure threshold than before either fix existed. The resync's own new unit test called `InputJitterBuffer.ingest()` directly, which is correct in isolation but bypasses the guard entirely, so it could never have caught this regardless of how thorough it was on its own terms. **When two fixes in the same round touch the same subsystem, explicitly re-test the combination end-to-end** (here: a real `SIGSTOP` freeze against the actual production RPC call path, not a direct unit-level call into the class the fix lives in) — passing tests for each fix individually is not evidence the pair composes correctly.
46. **A guard that bounds an incoming value against the *consumer's* position, rather than against the *producer's* own epoch, re-introduces exactly the "consumer can never catch up past a stall" failure it's often added specifically to prevent.** The seq-range guard bounded `seq` against `last_applied_seq` (advanced only by `consume()`, i.e. gated on however fast the physics tick loop is actually running) rather than `highest_ingested_seq` (advanced by `ingest()`, i.e. gated on however fast packets are actually arriving and being processed by `poll()`) — during a stall where ticks fall behind but polling keeps pace (the common case: a single-frame hitch, or `Engine.max_physics_steps_per_frame` capping tick catch-up while `poll()` itself isn't similarly capped), bounding against the lagging consumer rejects the very packets that would let the buffer refill and the resync condition ever trigger. Bound against whichever side of a producer/consumer pair is not the one already known to be falling behind.
47. **A trace that holds its inputs steady cannot falsify anything about *which sequence* a prediction is filed under — and "we hold thrust for 60 seconds" describes almost every prediction test people write.** Phase 4's history was filed under the wrong sequence (the estimated server-consumption seq, `input_lead` ticks behind issuance, instead of the issuing seq), and the dedicated action-marker instrument built to catch exactly that reported a flawless `marker=0/3784` across 60-second LAN, 80±20 ms and 5%-loss runs. It was not broken: while the commanded action is constant, "the intent from this tick" and "the action the server consumes for seq S" hold the same *value*, so a right and a wrong label are indistinguishable. Only an input **edge** separates them, and only for about `input_lead` ticks per edge. The bug then scales with `input_lead` — 9.3% mismatch at lead 1, 24% at lead 3 — meaning it was worst precisely on the impaired links the test matrix existed to cover, and invisible in all of them. **When a test is meant to validate a label, an index, or a phase relationship rather than a magnitude, the trace has to change that quantity frequently**; a steady-state trace validates the magnitude and silently asserts nothing about the label.
48. **A guard whose bound is derived from a value only the ACCEPTED path can advance is a latch, not a guard.** The seq-range check has now been written three times — bounded against server uptime, then `last_applied_seq`, then `highest_ingested_seq` — and all three could permanently reject an honest client's input, because in every version the quantity being compared against could only move forward via a packet that got through. Once enough drift or loss accumulated, nothing could ever move it again. The property to check when writing a guard like this is not "is the bound correct?" but "**if this guard rejects everything from now on, what advances the bound?**" If the answer is "an accepted packet", it needs an independent escape path (here: resync after N consecutive rejections) regardless of how well-chosen the bound is.
49. **Advancing a consumer cursor past data that has not arrived is not a lossy shortcut — it is permanent, because the producer-side filter then rejects the very data being waited for.** `InputJitterBuffer.consume()` advanced `last_applied_seq` on a starve, and `ingest()` discards `seq <= last_applied_seq`. One starve on a sequence the client had not sent yet therefore stranded the stream one ahead of arrivals *forever* — both sides advancing in lockstep, the gap never closing, every packet discarded on arrival. The client's own routine `input_lead` release was enough to trigger it, roughly every 6.5 s on a clean LAN. **Only give up on an expected item once strictly newer data proves it lost**; "it hasn't arrived yet" and "it will never arrive" are different states and must not share a code path.
50. **A metric that stops sampling during a failure will report that failure as healthy.** The action-marker gate printed `SMOKE PASS` at 3.76 % on a run where the player's input was permanently dead — because reconciliation suppression stops `_record_metrics` being called, so the worse the outage, the fewer samples and the *lower* the computed mismatch **rate**. Every rate-shaped assertion needs a companion assertion on the **denominator** (here: a sample count scaled to run length), or an outage silently becomes an absence of evidence and then evidence of absence.
51. **An architectural blocker inherited from a previous session is a claim to verify, not a premise to build on.** Phase 4 was handed over blocked on approval for a client-only shadow Jolt world — a large subsystem, and effectively the whole-world rollback §1's locked decisions rule out. The actual same-sequence defect turned out to be a one-line mislabel, falsifiable in about an hour with instrumentation that already existed; the shadow world remains genuinely necessary for the *contact* cohort but nothing else, which is a far smaller commitment than "Phase 4 is blocked on it." Reconstruct the failing invariant from the code and reproduce it against a control before accepting a scope estimate attached to it — especially when the recommendation arrives without the cheaper alternative recorded as tested.
---