fix(multiplayer): adversarial review fixes for Phase 3

An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:

CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.

HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.

MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.

MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.

MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.

MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).

LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.

LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.

LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.

New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
This commit is contained in:
Josh Creek
2026-08-20 15:28:44 +01:00
parent 10040f7339
commit 2325313ad2
11 changed files with 384 additions and 71 deletions
+44 -7
View File
@@ -38,7 +38,16 @@ const RATE_LIMIT_PACKETS_PER_SEC := 110
# sync by hand.
const RATE_LIMIT_BYTES_PER_SEC := RATE_LIMIT_PACKETS_PER_SEC * (NetCodec.INPUT_HEADER_SIZE + NetCodec.MAX_REDUNDANCY * NetCodec.INPUT_ENTRY_SIZE)
const RATE_LIMIT_WINDOW_MS := 1000
const RATE_LIMIT_OVER_BUDGET_SECONDS_TO_DISCONNECT := 3
# Leaky-bucket excess tolerance, expressed in the same "N seconds' worth of
# budget" terms the original consecutive-streak design used. An adversarial
# review found that design — a streak counter that HARD-RESET to 0 on any
# single clean window — was trivially evaded by a duty-cycled flood (burst,
# then one clean window, repeat): reproduced sustaining ~33x the packet
# budget indefinitely with zero disconnect warnings. A leaky bucket doesn't
# care how the excess is distributed in time — see the window-roll logic
# below for how it accumulates and drains.
const RATE_LIMIT_EXCESS_PACKETS_TO_DISCONNECT := RATE_LIMIT_PACKETS_PER_SEC * 3
const RATE_LIMIT_EXCESS_BYTES_TO_DISCONNECT := RATE_LIMIT_BYTES_PER_SEC * 3
const MALFORMED_LIMIT_TO_DISCONNECT := 20
@@ -46,7 +55,14 @@ class _PeerInputState:
var window_start_ms := 0
var packets_this_window := 0
var bytes_this_window := 0
var over_budget_seconds := 0
# Leaky bucket: grows by this window's actual total, drains by one
# window's worth of budget, every window — regardless of whether that
# window was itself over or under budget. A steady rate at or under
# budget nets to zero forever (never accumulates); any sustained AVERAGE
# above budget accumulates over time no matter how it's shaped into
# bursts, unlike a streak counter a clean gap can reset to 0.
var excess_packets := 0.0
var excess_bytes := 0.0
var malformed_count := 0
@@ -57,7 +73,9 @@ var _peer_input_state: Dictionary = {} # peer_id -> _PeerInputState, server on
# messages, not what §2's byte-budget analysis or a live overlay cares
# about. Rolling per-second counters, recomputed opportunistically on each
# send/receive rather than on a timer — nothing needs the rate outside of
# an on-demand overlay read anyway.
# an on-demand overlay read anyway. Use get_bytes_sent_per_sec() /
# get_bytes_received_per_sec() to READ these, not the raw fields directly
# — see those functions for why.
const BANDWIDTH_WINDOW_MS := 1000
var bytes_sent_per_sec := 0.0
var bytes_received_per_sec := 0.0
@@ -67,6 +85,25 @@ var _received_window_start_ms := 0
var _received_window_bytes := 0
# An adversarial review found bytes_*_per_sec only ever gets recomputed
# INSIDE _track_sent()/_track_received() — i.e. only when traffic actually
# arrives — so if traffic stops entirely (right before a disconnect, or
# during exactly the kind of outage this overlay exists to diagnose), the
# last computed rate displays forever instead of decaying toward zero.
# Report zero once meaningfully more than one window has passed with
# nothing tracked, rather than trusting a stale field.
func get_bytes_sent_per_sec() -> float:
if Time.get_ticks_msec() - _sent_window_start_ms > BANDWIDTH_WINDOW_MS * 2:
return 0.0
return bytes_sent_per_sec
func get_bytes_received_per_sec() -> float:
if Time.get_ticks_msec() - _received_window_start_ms > BANDWIDTH_WINDOW_MS * 2:
return 0.0
return bytes_received_per_sec
func _ready() -> void:
NetworkManager.client_disconnected.connect(func(peer_id: int) -> void: _peer_input_state.erase(peer_id))
@@ -161,13 +198,13 @@ func _recv_input(bytes: PackedByteArray) -> void:
# when nothing is arriving anyway.
var now_ms := Time.get_ticks_msec()
if now_ms - state.window_start_ms >= RATE_LIMIT_WINDOW_MS:
var was_over_budget := state.packets_this_window > RATE_LIMIT_PACKETS_PER_SEC or state.bytes_this_window > RATE_LIMIT_BYTES_PER_SEC
state.over_budget_seconds = (state.over_budget_seconds + 1) if was_over_budget else 0
state.excess_packets = maxf(0.0, state.excess_packets + float(state.packets_this_window) - float(RATE_LIMIT_PACKETS_PER_SEC))
state.excess_bytes = maxf(0.0, state.excess_bytes + float(state.bytes_this_window) - float(RATE_LIMIT_BYTES_PER_SEC))
state.window_start_ms = now_ms
state.packets_this_window = 0
state.bytes_this_window = 0
if state.over_budget_seconds >= RATE_LIMIT_OVER_BUDGET_SECONDS_TO_DISCONNECT:
_disconnect_abusive_peer(peer_id, "input rate limit exceeded for %d consecutive seconds" % state.over_budget_seconds)
if state.excess_packets > RATE_LIMIT_EXCESS_PACKETS_TO_DISCONNECT or state.excess_bytes > RATE_LIMIT_EXCESS_BYTES_TO_DISCONNECT:
_disconnect_abusive_peer(peer_id, "input rate limit exceeded (excess_packets=%.0f excess_bytes=%.0f)" % [state.excess_packets, state.excess_bytes])
return
state.packets_this_window += 1