multiplayer-next.md was a 1662-line mix of standing architecture spec
and task-completion tracking, most of which was dense per-task DONE
evidence for finished Phases 0-6. Split it:
- MULTIPLAYER_SPEC.md (new): the locked architecture decisions, wire
format, server-side input handling, prediction/reconciliation,
latency/frame-rate budget, and match lifecycle state machine -
standing design reference, not task-tracked.
- multiplayer-next.md (trimmed 1662 -> ~370 lines): only outstanding
work remains - §0 status, §7 Phase 7/8 task tables condensed to
"what's left" per task, §8-11 reference material (refactoring notes,
gotchas, testing, flagged items). Phases 0-6 collapsed to a pointer
at git history instead of ~500 lines of DONE evidence.
Also:
- Repointed every `multiplayer-next.md §N` code comment (N 1-6) across
Game/scripts, Game/tools and Game/tests to MULTIPLAYER_SPEC.md, since
those sections moved. Task-number references (`task N.N`, §7-11)
correctly still point at multiplayer-next.md.
- Updated CLAUDE.md's doc index and docs/TECH_STACK.md's spec-section
citations to match.
- TODO.md: added a "what's left to actually finish multiplayer
(human-actionable)" checklist pulled from multiplayer-next.md §0 and
docs/MATCHMAKING.md - things that need a person (hardware, a design
decision, a Steam App ID, hands on a controller), not more agent code.
multiplayer-todo.md and multiplayer-next.md tracked overlapping
information in two places. Fold everything into multiplayer-next.md
(architecture decisions, wire format, task breakdown with checkboxes,
gotchas list, testing notes) and delete multiplayer-todo.md. Section
numbers are unchanged, so existing code comments citing them by
section/task number still resolve; update every such reference to
point at the new filename.
An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:
CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.
HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.
MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.
MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.
MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.
MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).
LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.
LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.
LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.
New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
Extends net_debug_overlay.gd (Phase 1's RTT/offset display) with the
rest of task 3.7's list: jitter (new RFC3550-style EWMA in
NetworkManager, computed from raw per-sample RTT before Phase 1's own
min-filtering, since that filter is deliberately jitter-insensitive by
design), input buffer depth and input_lead (both already tracked
client-side for task 3.3), snapshot loss (a new EWMA in networked_match.gd
over each received snapshot's own server_tick gap - snapshots go out at a
steady one-tick cadence, so a gap is direct evidence of a drop or
reorder), snapshot age (computed on demand from the same bias-corrected
tick estimate the interpolator itself uses), and bandwidth (new rolling
per-second byte counters in MatchSim, on the two 60Hz hot-path channels
only). Prediction error is deliberately omitted with a comment explaining
why: there's no client-side prediction to measure until Phase 4.
Verified values are live and plausible, not just present, by calling
get_net_debug_stats() directly in a real two-process test and checking
the numbers make sense: bandwidth matched the wire format's own byte
math almost exactly (measured ~2400 B/s sent against a computed 40B x
60Hz, ~3540 B/s received against 59B x 60Hz), and buffer depth/lead/loss
all moved in the correct direction between a clean LAN run and one under
simulated 60ms latency + 10% loss. Full regression suite re-run clean.
MatchSim._recv_input now validates before decoding (§3.1 steps 2-3):
per-peer rolling-1s rate limiting (packet count AND byte budget, dropping
over-budget packets and disconnecting after 3 consecutive over-budget
seconds), and framing validation (redundancy count and payload size
checked against NetCodec's own layout before unpack_input ever runs,
disconnecting after 20 malformed packets). Framing has to be validated
explicitly rather than relying on decode failure: StreamPeerBuffer
silently zero-fills past EOF instead of erroring, a finding from Phase
2's adversarial review.
networked_match.gd's _on_input_received now rejects any seq claiming to
be more than 20 ticks ahead of the current server tick (§3.1 step 4) and
counts (rather than silently ignoring) input from a peer with no slot,
for observability.
Verified with two new permanent regression tests (networked_match_smoke.gd
--role=client-abuse-malformed / client-abuse-flood) that call
MatchSim._recv_input directly with garbage bytes and a legitimate-but-
too-frequent flood, respectively, bypassing the honest client encoder
entirely - the same thing a hostile custom client sending raw ENet
packets would look like. Both confirm real disconnection, not just that
the server tolerates the abuse.
Two bugs surfaced by getting these tests to actually pass cleanly: a
GDScript lambda-capture-by-value mistake in the tests themselves (a
plain `var disconnected := false` mutated inside a signal-handler lambda
never became visible to the enclosing function - fixed by capturing a
single-element Array instead, which is captured by reference); and a
narrow real race where NetworkManager's own ping/pong reply could target
a peer that a concurrent abuse-triggered disconnect had just removed
from the same poll() batch, now guarded. (Passing disconnect_peer's
`force` parameter as an attempted fix for a related one-off benign error
was tried and reverted - it made Godot's own peer-list bookkeeping
inconsistent, producing hundreds of errors instead of one; verified
empirically rather than assumed.)
Full regression suite, including the net-sim-latency milestone gate,
re-run clean.
New NetSim autoload: seeded, CLI-driven (--net-sim-latency/-jitter/-loss/-dup)
latency/jitter/loss/duplicate decorator, a true no-op passthrough unless a
flag is set. Wraps MatchSim.send_input/send_snapshot per the design doc's
scope, plus NetworkManager's ping/pong so the already-tested RTT/clock
measurement becomes the acceptance signal for "raises observed RTT" without
waiting on Phase 3's per-peer snapshot echo.
Two real bugs found while building and verifying this against Phase 2's own
milestone gate (a real match under --net-sim-latency 80 --net-sim-jitter
20, not just LAN): a timestamp captured inside a delayed RPC closure
silently ate that side's own added delay out of the round-trip
measurement instead of adding to it; and a delayed send whose target
disconnected (or whose own process had already shut down) during the hold
threw RPC errors, since the existing get_peers() filtering only checked
validity at schedule time. Fixed by capturing timestamps before handing
off to NetSim, and by having NetSim re-validate the target at fire time.
Phase 2's milestone gate now passes for real: a full 1v1 under simulated
80ms latency / 20ms jitter still shows clean server-authoritative
movement and zero RPC errors. Full Phase 1 + Phase 2 regression suite
re-verified clean with NetSim present but inactive.
Lands tasks 1.0-1.8 of multiplayer-todo.md: the pure-function test runner,
net_codec (wire format quantizers/pack-unpack), NetworkManager (ENet
transport, manual polling, min-RTT clock sync), MatchNet (handshake,
protocol/tick-rate gating, roster with team+ready state), lobby.tscn (team
columns, switch team, ready toggle), server_boot.tscn (headless dedicated
server with structured logging and an overrun watchdog), and main_menu.gd's
Host/Join-by-IP UI (connecting overlay, cancel, bounded failure path).
Followed by an adversarial review (Opus subagent) that found and fixed two
real bugs - an unvalidated player_name broadcast that let one client's
oversized name head-of-line-block the reliable channel for everyone, and a
server-side roster leak across a host/re-host cycle - plus three gaps in
the test suite itself where a claim of "verified" wasn't actually backed
by what the test checked. All five two-process smoke tests plus the
pure-function suite are green with the strengthened assertions in place.
Josh Creek