fix(multiplayer): adversarial review fixes for Phase 3

An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:

CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.

HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.

MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.

MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.

MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.

MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).

LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.

LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.

LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.

New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
This commit is contained in:
Josh Creek
2026-08-20 15:28:44 +01:00
parent 10040f7339
commit 2325313ad2
11 changed files with 384 additions and 71 deletions
+29
View File
@@ -35,6 +35,14 @@ var _ring_seq: PackedInt32Array = PackedInt32Array()
# comment for why an un-seeded buffer would otherwise never converge with
# what the client is actually sending.
var _seeded := false
# Highest seq ever seen by ingest(), regardless of whether it's still in the
# ring — consume()'s only way to tell "the data is gone because the ring
# overflowed" apart from "the data just hasn't arrived yet". See consume()'s
# own comment for why this exists: an adversarial review found that without
# it, a backlog bigger than RING_SIZE (a host stall, or persistent client/
# server clock drift) permanently zeroed a connected player's input for the
# rest of the match.
var _highest_ingested_seq := -1
func _init() -> void:
@@ -64,6 +72,8 @@ func ingest(newest_seq: int, actions: Array) -> void:
# "expected" with reality the moment real data first exists.
last_applied_seq = newest_seq - actions.size()
_seeded = true
if newest_seq > _highest_ingested_seq:
_highest_ingested_seq = newest_seq
for i in actions.size():
var seq: int = newest_seq - i
if seq <= last_applied_seq:
@@ -95,6 +105,25 @@ func consume() -> ShipAction:
return last_action
var expected := last_applied_seq + 1
var idx := expected % RING_SIZE
# Ring-overflow resync. A fixed-size ring can only ever hold RING_SIZE
# ticks of not-yet-consumed data at once — if the caller has fallen
# further behind the newest data actually arriving than that (a host
# stall, or persistent client/server clock drift), every tick between
# "expected" and "_highest_ingested_seq - RING_SIZE" has already been
# irrecoverably overwritten by more recent arrivals landing on the same
# ring slots. Waiting for it tick-by-tick would starve — and, past
# STARVE_ZERO_TICKS, zero this player's ship — for the ENTIRE gap even
# though fresh, real input already exists in the ring right now. An
# adversarial review found and reproduced this exact failure (a ~0.7s
# host freeze permanently zeroed a connected player's input for the
# rest of the match, with no self-recovery). Skip the unrecoverable
# span and resync directly to what the ring can still actually provide.
if _highest_ingested_seq - expected >= RING_SIZE:
last_applied_seq = _highest_ingested_seq - RING_SIZE
expected = last_applied_seq + 1
idx = expected % RING_SIZE
if _ring_seq[idx] == expected:
last_action = _ring_action[idx]
starved_ticks = 0