Commit Graph

8 Commits

Author SHA1 Message Date
Josh Creek a5cbc977b5 feat(multiplayer): Phase 5 tasks 5.6-5.10 - disconnects, spectators, replay log
Completes Phase 5's implementation. Every task is verified at 1v1; the
3v3 phase gate itself has not been run and remains outstanding.

5.6/5.7 disconnects: a ship is never despawned. The slot keeps it and
swaps the controller (--fill-bots gives it a bot, the default leaves it
inert per §1.4), sets `stalled` immediately so the nameplate greys out
rather than waiting ~500ms for the abandoned jitter buffer to starve,
and reserves the slot for 30s keyed by player name so a reconnect gets
the same ship back.

5.7 was a real bug, found by the test rather than by review:
SlotInfo.controller was declared RLShipController, but the takeover
swaps in an AIShipController or the base controller - the narrower type
makes that assignment fail its type check, leaving the field pointing at
the controller set_controller() just queue_free()d. It surfaced as
controller_valid=false on the first run. The per-tick action write is
now also gated on `is RLShipController`, since a disconnected slot's bot
drives itself and overwriting it from a starving buffer would pin it to
the departed player's last input.

§6.4's two rules conflict: reserve for 30s, but abort when the last
human leaves. Applied naively the abort wins instantly in a 1v1 and the
reservation can never be redeemed, making reconnect unreachable exactly
when it matters. Abort now waits for no connections AND no outstanding
reservations.

5.8 spectators: a slotless peer spawns no ship and receives the same
snapshot broadcast. HUDController.spectator_mode keeps the clock, score
and goal celebration and hides only the ship instrument cluster - it
previously push_error'd and bailed, leaving a spectator with a dead HUD.
Camera cycles ships in slot order then the ball. --max-spectators caps
it, counted from the live peer list so a dropped spectator cannot leak a
unit of the cap.

5.9 escape respawn: new GameMode._on_bodies_respawned() virtual;
NetworkedMatch bumps reset_gen through Phase 2's deferred path so the
bump and the respawned pose land in the same broadcast. Single-player
modes are unaffected - the base is a no-op.

5.10 replay log: scripts/replay_log.gd, --replay-log=<path>, storing the
wire bytes verbatim in both directions rather than re-serialising - a
re-encode would launder away precisely the malformed payload being
chased. A live 6s match recorded 1115 records (557 inputs / 558
snapshots) and a stored snapshot decodes back to server_tick=100
match_state=WARMUP bodies=2.

Note for future work: --check-only --script is the only thing that
catches a parse error in networked_match.gd, because the unit runner
never loads it. Two separate breakages passed the full unit suite while
breaking every two-process run. A new class_name also needs --import
before it resolves.

Test surface: --role=host-disconnect (three-process 5.6/5.7 scenario),
--match-length=<s>, --replay-log, --fill-bots/--no-fill-bots,
--max-spectators. The ball-contact scenario now steers at the ball with
closed-loop real input instead of a hand-tuned fixed heading, which 5.3
broke by adding KICKOFF_YAW_JITTER; thrusting while turning took it from
2/3 to 5/5.

Regression: 87 unit tests; free-flight LAN p99 0.094m with 0 hard snaps;
transition gate 0.00%; ball contact 5/5; lifecycle goal cycle and full
match to RESULTS/LOBBY; disconnect+reconnect; two-bot CI.
2026-08-21 10:25:15 +01:00
Josh Creek 3d6906b981 feat(multiplayer): Phase 5 tasks 5.2-5.5 - clock, kickoff, goals, full time
Implements the rest of the §6.2 lifecycle on top of 5.1's state machine.

5.3 kickoff: the server resets every body and broadcasts the RESULTING
transforms, never a seed - §1's locked decision, because shared-seed
determinism needs both sides to consume the RNG stream in identical
order forever and the first randf() added to the reset path desyncs
silently. Countdown is derived from server_tick on both peers, and a
kickoff that lands after its own resume tick applies immediately and
skips the countdown rather than scheduling into the past.

5.4 goals: goal_scored(scoring_team, score, goal_tick, resume_tick).
Score is authoritative at sensor time, before any presentation. The
reset moved OUT of the sensor path and into the kickoff at resume_tick,
which is what stops the server resetting while clients are still
mid-celebration. Engine.time_scale is never touched.

5.2 clock: tick-derived, no Timer and no _process polling. The goal
pause shifts the absolute end_tick by (resume_tick - goal_tick) rather
than pausing anything, so no float drift accumulates across goals.

5.5 full time: clock expiry -> FULL_TIME -> sudden death on a draw or
RESULTS, golden goal in overtime, then LOBBY on both peers - clients
return to the lobby, not the main menu. get_tree().paused is never used.

Four bugs found and fixed while building this, each by a failing run
rather than by inspection:

- Tick order was load-bearing: _update_kickoff_countdown() clears the
  same _kickoff_resume_tick that _update_match_state() reads to leave
  WARMUP, so running the countdown first wiped the transition condition
  and the match sat frozen in WARMUP forever.
- _apply_match_state resets _state_deadline_tick on every transition, so
  a GOAL_PAUSE deadline assigned before _set_match_state was wiped and
  the match never resumed. Deadlines are now owned by _apply_match_state.
- Freezing "all bodies" is wrong on a client. Remote ships and the ball
  are permanently FREEZE_MODE_KINEMATIC and transform-driven; freezing
  them all unfroze the remote ones on the way back out, so they fell
  under gravity while the interpolator fought them - 210 hard snaps and
  an infinite p99. A client now freezes only the one body it simulates.
- A frozen body never runs _integrate_forces, so the queued kickoff
  teleport was stranded by an immediate set_deferred("freeze", true).
  Freeze now happens on a strictly later tick, the same pattern Phase 2
  used for _pending_reset_gen_bump_tick.

Prediction and reconciliation are suspended while the match is not live:
during a countdown or goal pause the local ship is frozen on both peers,
and running delta transport over those frozen states produced a p95
position error of 2.4e10 m. Input keeps flowing so the server's jitter
buffer does not starve into `stalled`.

Also fixed: a kickoff can arrive before match_config, and body order is
slot order - applying it early placed the BALL at positions[0], on top
of the first ship, which the ball-cam reported as "target vector can't
be zero" 95 times. It is now held until the roster exists.

Test changes: the ball-contact scenario steered by a hand-tuned fixed
heading, which 5.3 broke because kickoff applies KICKOFF_YAW_JITTER - it
flew past the ball in 3/3 runs. It now closes the loop on the actual
bearing using real input actions. Assertions that read a frozen ship
(freeze, thrust) are gated on the match being live, and the hooks now
survive the scene teardown at RESULTS instead of hanging on freed
objects for the full timeout.

Regression: 81 unit tests; free-flight LAN p99 0.143m and 80±20ms, both
0 hard snaps; transition gate 0.00%; ball contact 3/3; two-bot CI.
2026-08-21 10:01:39 +01:00
Josh Creek 9f28c02488 feat(multiplayer): Phase 5 task 5.1 - match lifecycle state machine
Adds the §6.1 state machine, its broadcast, and the client side that
follows it. Physics, freezing and input are deliberately NOT gated on
state yet - 5.3 and 5.4 own freeze/unfreeze at kickoff and goal, and
doing it here would change the conditions every Phase 4 prediction gate
was measured under.

scripts/match_state.gd holds the enum and transition table as pure data
with no scene or RPC dependency, so the table is checked exhaustively
rather than by example: every state reachable, every state has an exit,
no self-transitions, abort-to-LOBBY from anywhere per §6.4, illegal
shortcuts rejected, unknown values refused rather than coerced. The enum
values are the wire format - match_state has been a u8 in the snapshot
header since §2.4 - so a test pins them; only append, never renumber.

The server validates every transition and push_errors an illegal one
rather than following it. Clients deliberately do NOT enforce the table:
authoritative state must be accepted, and a late joiner legitimately
jumps straight to PLAYING.

Two channels carry the state. state_change (reliable, channel 0) is
prompt and carries an absolute at_tick, never a duration. The snapshot's
match_state byte is the catch-up path for a client not yet sent a
transition - a late joiner, or the window between scene load and the
first RPC.

The byte needs a tick guard, and this was found the hard way. Snapshots
are unreliable_ordered on channel 2 and ordering holds only within a
channel, so a state_change for tick N routinely arrives before an
in-flight snapshot from tick N-2. Without the guard the client applies
the new state then gets dragged back by the older byte, oscillating on
every transition - observed directly as LOADING -> WARMUP -> LOBBY ->
PLAYING -> LOBBY while running a deliberately-broken-byte control. Only
a byte at least as new as match_state_since_tick is accepted.

WARMUP_TICKS/GOAL_PAUSE_TICKS are honest placeholders so 5.1 drives real
transitions to verify against; 5.3 and 5.4 replace them. The server also
leaves LOADING immediately rather than waiting for scene_ready, which
does not exist yet.

New smoke flag --exercise-match-state, passed to both roles: the host
forces a goal to drive a GOAL_PAUSE cycle, the client records the
sequence and asserts every consecutive pair is legal, that ticks are
monotonic, and that the wire byte agrees with its own state. Observed
LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP with tick deltas
matching the configured durations exactly.

Verified against a control: hardcoding the snapshot byte back to 0 fails
both the byte assertion and the transition-legality assertion. The byte
is asserted separately from the RPC precisely because everything else in
the check is RPC-driven and would pass with a dead byte - the same gap
that hid the Phase 4 label bug (gotcha 47).

Regression: 81 unit tests; 60s free-flight LAN (p99 0.148m, 0 hard
snaps, marker 0/3364); transition gate 0.00%; ball contact; two-bot CI.
2026-08-21 09:31:22 +01:00
Josh Creek 2325313ad2 fix(multiplayer): adversarial review fixes for Phase 3
An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:

CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.

HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.

MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.

MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.

MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.

MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).

LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.

LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.

LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.

New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
2026-08-20 15:28:44 +01:00
Josh Creek 9d8a8080ba feat(multiplayer): Phase 3 task 3.7 - debug net overlay extension
Extends net_debug_overlay.gd (Phase 1's RTT/offset display) with the
rest of task 3.7's list: jitter (new RFC3550-style EWMA in
NetworkManager, computed from raw per-sample RTT before Phase 1's own
min-filtering, since that filter is deliberately jitter-insensitive by
design), input buffer depth and input_lead (both already tracked
client-side for task 3.3), snapshot loss (a new EWMA in networked_match.gd
over each received snapshot's own server_tick gap - snapshots go out at a
steady one-tick cadence, so a gap is direct evidence of a drop or
reorder), snapshot age (computed on demand from the same bias-corrected
tick estimate the interpolator itself uses), and bandwidth (new rolling
per-second byte counters in MatchSim, on the two 60Hz hot-path channels
only). Prediction error is deliberately omitted with a comment explaining
why: there's no client-side prediction to measure until Phase 4.

Verified values are live and plausible, not just present, by calling
get_net_debug_stats() directly in a real two-process test and checking
the numbers make sense: bandwidth matched the wire format's own byte
math almost exactly (measured ~2400 B/s sent against a computed 40B x
60Hz, ~3540 B/s received against 59B x 60Hz), and buffer depth/lead/loss
all moved in the correct direction between a clean LAN run and one under
simulated 60ms latency + 10% loss. Full regression suite re-run clean.
2026-08-20 13:32:16 +01:00
Josh Creek b290f49143 feat(multiplayer): Phase 3 task 3.4 - input validation, rate limiting, disconnect policy
MatchSim._recv_input now validates before decoding (§3.1 steps 2-3):
per-peer rolling-1s rate limiting (packet count AND byte budget, dropping
over-budget packets and disconnecting after 3 consecutive over-budget
seconds), and framing validation (redundancy count and payload size
checked against NetCodec's own layout before unpack_input ever runs,
disconnecting after 20 malformed packets). Framing has to be validated
explicitly rather than relying on decode failure: StreamPeerBuffer
silently zero-fills past EOF instead of erroring, a finding from Phase
2's adversarial review.

networked_match.gd's _on_input_received now rejects any seq claiming to
be more than 20 ticks ahead of the current server tick (§3.1 step 4) and
counts (rather than silently ignoring) input from a peer with no slot,
for observability.

Verified with two new permanent regression tests (networked_match_smoke.gd
--role=client-abuse-malformed / client-abuse-flood) that call
MatchSim._recv_input directly with garbage bytes and a legitimate-but-
too-frequent flood, respectively, bypassing the honest client encoder
entirely - the same thing a hostile custom client sending raw ENet
packets would look like. Both confirm real disconnection, not just that
the server tolerates the abuse.

Two bugs surfaced by getting these tests to actually pass cleanly: a
GDScript lambda-capture-by-value mistake in the tests themselves (a
plain `var disconnected := false` mutated inside a signal-handler lambda
never became visible to the enclosing function - fixed by capturing a
single-element Array instead, which is captured by reference); and a
narrow real race where NetworkManager's own ping/pong reply could target
a peer that a concurrent abuse-triggered disconnect had just removed
from the same poll() batch, now guarded. (Passing disconnect_peer's
`force` parameter as an attempted fix for a related one-off benign error
was tried and reverted - it made Godot's own peer-list bookkeeping
inconsistent, producing hundreds of errors instead of one; verified
empirically rather than assumed.)

Full regression suite, including the net-sim-latency milestone gate,
re-run clean.
2026-08-20 13:27:03 +01:00
Josh Creek 7b150ef72e feat(multiplayer): task 2.8 net_sim.gd, close out Phase 2
New NetSim autoload: seeded, CLI-driven (--net-sim-latency/-jitter/-loss/-dup)
latency/jitter/loss/duplicate decorator, a true no-op passthrough unless a
flag is set. Wraps MatchSim.send_input/send_snapshot per the design doc's
scope, plus NetworkManager's ping/pong so the already-tested RTT/clock
measurement becomes the acceptance signal for "raises observed RTT" without
waiting on Phase 3's per-peer snapshot echo.

Two real bugs found while building and verifying this against Phase 2's own
milestone gate (a real match under --net-sim-latency 80 --net-sim-jitter
20, not just LAN): a timestamp captured inside a delayed RPC closure
silently ate that side's own added delay out of the round-trip
measurement instead of adding to it; and a delayed send whose target
disconnected (or whose own process had already shut down) during the hold
threw RPC errors, since the existing get_peers() filtering only checked
validity at schedule time. Fixed by capturing timestamps before handing
off to NetSim, and by having NetSim re-validate the target at fire time.

Phase 2's milestone gate now passes for real: a full 1v1 under simulated
80ms latency / 20ms jitter still shows clean server-authoritative
movement and zero RPC errors. Full Phase 1 + Phase 2 regression suite
re-verified clean with NetSim present but inactive.
2026-08-20 08:50:47 +01:00
Josh Creek 39a41c016c feat(multiplayer): Phase 2 server-authoritative simulation, dumb client
Implements tasks 2.1-2.7: NetworkedMatch spawns a deterministic slot
layout from the lobby roster, the server drives each connected peer's
ship via RLShipController fed by decoded client input and broadcasts
60Hz snapshots, and the client renders everything (including its own
ship) from a per-body NetInterpolator with no local prediction yet.
Dual-time remote entities split collider updates (present-time, for
correct contacts) from $Visual updates (interp-delayed, for smoothness).
Camera/HUD wiring and remote engine-flame VFX fell out of the existing
Ship API for free once snapshots were flowing.

Three real bugs found and fixed while getting a two-process test
green: an RPC method named _input collided with Node's built-in
_input virtual and broke the whole MatchSim autoload from loading;
networked_match.gd never called NetworkManager.poll(), so nothing
sent via RPC in this scene reached the wire despite Phase 1's manual
polling being wired up everywhere else; and a match_config
request/response fallback (added to close a startup race) could
double-deliver once polling was fixed, requiring an idempotency guard.

Verified with tests/networked_match_smoke: a real headless two-process
host+client run shows the client rendering 31m of server-authoritative
movement from a held forward-thrust input, with thrust_z=1.0 confirmed
on the interpolated snapshot mid-drive and camera/HUD both wired.
Full Phase 1 regression suite re-run clean alongside it.

Task 2.8 (net_sim.gd latency/jitter/loss decorator) is not yet done;
Phase 2's own gate needs it before it's fully met.
2026-08-20 08:42:13 +01:00