diff --git a/multiplayer-todo.md b/multiplayer-todo.md index ba303488..f5639b5f 100644 --- a/multiplayer-todo.md +++ b/multiplayer-todo.md @@ -4,7 +4,49 @@ Working document for the online multiplayer effort. `TODO.md` points here. Everything below is written so an agent (or a person) can pick up a single numbered task, do it, verify it against a stated acceptance criterion, and stop. Sections 1–6 are the decisions those tasks assume; read them before picking up work in Phase 2 or later. -**Status: Phase 5's tasks are all implemented and individually verified at 1v1; its 3v3 phase gate has not been run. Phase 4's correctness gates are green and its sign-off waits on a human playtest.** The client now has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. The action-sequence-correctness gap that blocked Phase 4 was a mislabelled prediction history, now fixed and permanently gated (task 4.11). An adversarial review of that fix then found two Phase 3 bugs that were silently killing a connected player's input — periodically on a clean LAN, and permanently after any ~2 s host hitch — both now fixed with verified controls (task 4.13). What remains is not a measurement: nobody has played it at ~100 ms RTT to judge feel, which is what the milestone actually asks. See §7 for the implemented work, evidence, and the one open architectural question (a contact-cohort-only shadow world). +**Status: every task in Phases 0–5 is implemented and verified. Both milestones' remaining work is verification a machine cannot do — a human playtest at ~100 ms RTT (Phase 4) and a real 3v3 session (Phase 5). Phases 6 and 7 are unstarted.** The client has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. See the outstanding list immediately below for what is left and why, §7 for the implemented work and its evidence, and §11 for what is deliberately deferred. + +--- + +## 0. Outstanding work — the short list + +The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 0–5 contain no unfinished tasks. + +### Blocking sign-off — the work exists, the verification does not + +| # | What | Why it is not done | Detail | +|---|---|---|---| +| A | **Phase 4 human playtest at ~100 ms RTT.** Does the ship feel local? Does the ball? Do contact corrections read as bumps or as glitches? | Needs hands on a controller. Every numeric gate is green; feel is the milestone's actual subject and no percentile can answer it. | Phase 4 gate | +| B | **Phase 5 3v3 gate**: a full start-to-finish match with 6 players, a mid-match disconnect, and a late joiner. | Needs a real multi-client session. Every scenario is verified at 1v1 plus a two-bot CI match; nothing has run at 3v3. | Phase 5 gate | + +These two are independent and can be done in either order, but B is the cheaper of the two to arrange and would also exercise A's conditions incidentally. + +### Known defects, not fixed + +| # | What | Severity | Detail | +|---|---|---|---| +| C | **Slot reservation and takeover are keyed on display name alone.** Any peer connecting with a departed player's name inside the 30 s window claims their slot, ship and team. | Real, demonstrated. Bounded by needing a genuine disconnect to race. | §11 | +| D | **Input is still lost at the transport layer during a long server stall**, variably — 7 of 8 runs measured 0.00 % of the sequence stream missing, the eighth 23.54 %. | Low. Distinct from the rate-limiter cause, which is fixed. The seq-guard resync visibly recovers it. | Phase 5 notes | +| E | **A second `Unable to send packet on channel N` stderr race**, in `_broadcast_snapshot` rather than the fixed site in `_remove_player`. | Cosmetic, but it violates the clean-stderr convention the tests rely on. Only reproduced via the adversarial abuse role. | §11 | + +C is the one to plan around: it is fixed for free by task **7.4** (Steam auth tickets in `hello`), which is why it has not been given a bespoke solution. Anything that ships to strangers before Phase 7 needs it addressed first. + +### Open architectural question + +| # | What | Detail | +|---|---|---| +| F | **A contact-cohort-only shadow world.** The remaining known prediction weakness is the contact cohort. Whether it is worth a client-side shadow Jolt world scoped to contacts alone is undecided — and deliberately so until A supplies the felt evidence. | Phase 4 notes | + +### Unstarted phases + +- **Phase 6 — dedicated server productionisation** (7 tasks): export preset, CLI surface, structured logging, arena rotation, systemd/Docker/`SERVER.md`, CI against the *exported binary*. Gate: `docker run` a server, connect from another machine over the internet, play a full match. +- **Phase 7 — Steam transport, browser, identity** (5 tasks): GodotSteam, the `NetTransport` boundary extracted from two working implementations, server browser, auth tickets and ban list, feature-gating so ENet direct-connect never becomes the degraded path. Carries the fix for **C**. + +Phase 6 has no dependency on Phase 7 and is the natural next block of work: it is what turns a thing that runs in two terminals into a thing someone else can host. + +### Deferred by choice, not forgotten + +120 Hz simulation, the latency-gap *measurement* (task 4.9's acceptance criterion), audio hooks, split-screen — all in §11 with what each would buy and cost. --- @@ -903,7 +945,7 @@ No own-ship prediction yet: the client renders everything, including its own shi | 4.12 `[D:4.11]` | **DONE.** Issued-but-unsimulated (attack-gap) sequences are recorded and skipped rather than diagnosed as history loss; the release path no longer re-files an already-issued sequence | Free-flight hard snaps 0 across all three 60 s conditions, down from 25/8/4 `missing_not_recorded` | | **4.13** `[D:4.12]` | **DONE — two server-side input-death bugs found by adversarial review, both reproduced and fixed with controls.** A starve no longer advances past a sequence the client has not sent; the seq-range guard can no longer latch shut permanently | Marker 0.00% in all three conditions (was 1.7–2.5%); 2.0 s and 3.5 s host freezes now recover; control runs with each fix reverted fail the gate | -**Phase gate — correctness gates MET; the milestone's felt-quality half remains untested.** The action-sequence-correctness gap is closed and permanently gated (4.11), the two seq-delta paths it exposed are fixed (4.12), and an adversarial review's two server-side input-death bugs are fixed with controls (4.13). What has *not* happened is the original milestone's actual subject: nobody has played this with hands on a controller at ~100 ms RTT to judge whether ship and ball feel local and whether contact corrections read as bumps. Numbers cannot answer that, and the contact cohort is where the remaining known weakness lives (see the shadow-world note below). Sign off after a human playtest, not before. +**Phase gate — correctness gates MET; the milestone's felt-quality half remains untested.** The action-sequence-correctness gap is closed and permanently gated (4.11), the two seq-delta paths it exposed are fixed (4.12), and an adversarial review's two server-side input-death bugs are fixed with controls (4.13). What has *not* happened is the original milestone's actual subject: nobody has played this with hands on a controller at ~100 ms RTT to judge whether ship and ball feel local and whether contact corrections read as bumps. Numbers cannot answer that, and the contact cohort is where the remaining known weakness lives (see the shadow-world note below). Sign off after a human playtest, not before — item **A** of §0. > **Read 4.13 before trusting any earlier Phase 4 evidence.** Until this session the server was silently discarding a connected player's input for ~30 ticks roughly every 6.5 seconds on a clean LAN, and permanently after any ~2 s host hitch. Every Phase 4 number recorded before 4.13 was measured through that, and the gates reported green throughout — for the same reason they missed the label bug in 4.11: a steady input cannot distinguish "the server repeated my last action" from "the server applied my real action". @@ -928,7 +970,7 @@ Mismatch scales with `input_lead`, exactly as the mechanism predicts. It also ** Two sub-findings from that investigation, recorded because both are counter-intuitive: `dequantize_thrust_z_bin(quantize_thrust_z_bin(0.0))` returns **0.142857**, not 0.0 (7 bins over [-1,1], `roundi(3.5) == 4`), so a server-reported `thrust_z` of 0.14 literally means "exactly zero" — the 0.26 threshold absorbs it, as designed. And `_pending_local_reconciliation` keeps only the newest snapshot, so acks are dropped whenever two snapshots land in one physics tick: **the marker under-samples, and the true action-disagreement rate is higher than it reports.** -> **The client-only shadow Jolt world is still the open question, but it is now scoped to the contact cohort alone.** Even perfectly labelled, the client predicts contacts against remote ships and the ball sitting at interpolated-*delayed* positions, so a contact-cohort prediction cannot be sequence-correct in the live world — no amount of bookkeeping fixes that, and a shadow world is the only thing that does. It is a large subsystem and effectively the whole-world rollback §1's locked decisions set out to avoid, so **do not build it before a playtest says the contact cohort actually reads badly to a human.** Free flight no longer needs it. +> **The client-only shadow Jolt world is still the open question (item F of §0), but it is now scoped to the contact cohort alone.** Even perfectly labelled, the client predicts contacts against remote ships and the ball sitting at interpolated-*delayed* positions, so a contact-cohort prediction cannot be sequence-correct in the live world — no amount of bookkeeping fixes that, and a shadow world is the only thing that does. It is a large subsystem and effectively the whole-world rollback §1's locked decisions set out to avoid, so **do not build it before a playtest says the contact cohort actually reads badly to a human.** Free flight no longer needs it. **New smoke role — `--exercise-input-transitions`.** Toggles forward thrust every 6 physics ticks (~100 ms) with alternating yaw, and asserts the action marker stays under 5% mismatch over ≥200 samples. This is the **only** gate here that can catch a sequence-label regression, for the reason above, so it must not be folded into the steady-input free-flight run: @@ -973,7 +1015,7 @@ Note the free-flight p99 **improved** (0.170/0.176/0.184 → 0.141/0.168/0.154) | 5.7 `[D:5.6]` | **DONE.** `_swap_slot_controller()` rebinds in the same transaction; `slot.controller` retyped to the base `ShipController`; every use `is_instance_valid`-guarded | The disconnect test caught the real bug: the narrower `RLShipController` type made the swap assignment fail, leaving a freed reference | | 5.8 `[D:5.1]` `[P]` | **DONE.** A slotless peer spectates (no ship spawned, same snapshot stream), `HUDController.spectator_mode` keeps clock/score/celebration and hides only the ship cluster, camera cycles ships then ball, `--max-spectators` cap. **§6.3's "take the slot at the next kickoff" is now implemented, not just printed** — the line claiming it was there from the start while `_is_spectator` was assigned once and never revisited (see the Phase 5 note below) | Spectator path exercised by the mid-match joiner; HUD no longer `push_error`s and bails with a dead HUD; promotion verified 4/4 from both sides, with a control proving §6.4's reservation outranks the queue | | 5.9 `[D:5.3]` `[P]` | **DONE.** New `GameMode._on_bodies_respawned()` virtual; `NetworkedMatch` bumps `reset_gen` through Phase 2's deferred path so the bump and the respawned pose land in the same broadcast | Single-player modes unaffected (base is a no-op) | -| 5.10 `[D:5.1]` `[P]` | **DONE.** `scripts/replay_log.gd`, `--replay-log=`, storing wire bytes verbatim in both directions | Live 6 s match recorded 1115 records (557 inputs / 558 snapshots); a stored snapshot decodes back to `server_tick=100 match_state=WARMUP bodies=2`; 6 unit tests incl. truncation and foreign-file rejection | +| 5.10 `[D:5.1]` `[P]` | **DONE.** `scripts/replay_log.gd`, `--replay-log=`, storing wire bytes verbatim in both directions — plus, after a review found three recording gaps, REJECTED packets with their reason in the kind byte (capped per window so the log cannot become a remote disk-fill amplifier), a failed write that ends the log instead of desyncing its framing, an explicit `close()` with a summary, and `tools/replay_dump.gd` to read one back. The reject recording immediately found a real bug: the server was rate-limiting a stall backlog it had caused itself, losing 8.88% of a player's input | Live 6 s match recorded 1115 records (557 inputs / 558 snapshots); a stored snapshot decodes back to `server_tick=100 match_state=WARMUP bodies=2`; 6 unit tests incl. truncation and foreign-file rejection | > `Ship.set_controller` (`ship.gd:213-218`) calls `queue_free()` on the outgoing controller. Task 5.7 exists because the takeover path in 5.6 otherwise leaves `MatchNet` holding a freed reference — the exact class of bug that surfaces as a random server crash weeks later. @@ -1028,7 +1070,7 @@ The other durable improvement from the first investigation still stands: a perce **And the bug the recording immediately found: the server rate-limited a backlog it caused itself.** A 2s host stall (`SIGSTOP`, standing in for a GC/IO/scheduler hitch) has the client sending at 60Hz throughout, and ENet delivers the whole backlog in the first window after resume — **70 of an honest client's input packets rejected as "rate limit exceeded"**, against a limit that client never came close to violating. Redundancy does not cover it, and that was the assumption worth checking rather than asserting: the dropped packets are *contiguous*, so each one's redundancy window falls inside the same dropped run. Measured with the new log: **0 of 70 rescued, and 82 of 923 sequences (8.88%, ~1.4s of that player's input) never reached the server at all**, versus 0.00% on an otherwise identical run with no stall. Every prediction gate still passed — this is the same class as the Phase 3/4 input-death bugs, invisible to every gate that reads only the client's own state. -Fixed by not policing a backlog the server caused: `MatchSim._physics_process` watches for a wall-clock gap over `STALL_DETECT_MS` (a stalled process doesn't run that callback either, so the first frame after the stall sees the whole gap, which is exactly the size of the backlog about to arrive) and grants each *already-tracked* peer a capped, two-window packet grace. The leaky bucket drains against the same graced budget, or a stall would still accumulate excess toward a disconnect for traffic the server just explicitly allowed. Results: 2s stall, rate-limit rejects 70 → **0**, sequences missing 8.88% → **0.00%**, and `REJECTED_SEQ_GUARD` 9 → 0 as a second-order confirmation (the guard was firing partly *because* the dropped backlog let the client's epoch run away). Across eight stall runs on the fixed build, 7 measured 0.00% missing; the eighth measured 23.54% with zero rate-limit rejects and the seq-guard resync visibly doing its job — a separate, occasional transport-level loss during the stall that this change does not address and does not make worse. The three control runs on the unfixed build lost 4.34%, 7.52% and 7.86%, every time. +Fixed by not policing a backlog the server caused: `MatchSim._physics_process` watches for a wall-clock gap over `STALL_DETECT_MS` (a stalled process doesn't run that callback either, so the first frame after the stall sees the whole gap, which is exactly the size of the backlog about to arrive) and grants each *already-tracked* peer a capped, two-window packet grace. The leaky bucket drains against the same graced budget, or a stall would still accumulate excess toward a disconnect for traffic the server just explicitly allowed. Results: 2s stall, rate-limit rejects 70 → **0**, sequences missing 8.88% → **0.00%**, and `REJECTED_SEQ_GUARD` 9 → 0 as a second-order confirmation (the guard was firing partly *because* the dropped backlog let the client's epoch run away). Across eight stall runs on the fixed build, 7 measured 0.00% missing; the eighth measured 23.54% with zero rate-limit rejects and the seq-guard resync visibly doing its job — a separate, occasional transport-level loss during the stall that this change does not address and does not make worse (**item D of §0**). The three control runs on the unfixed build lost 4.34%, 7.52% and 7.86%, every time. Abuse detection is unweakened and this was checked rather than argued: all three abuse roles still disconnect, and **no flood induced a server stall in any run**, so the grace cannot be farmed by flooding. An attacker who *can* induce server stalls to earn budget already has a strictly worse capability than sending extra input packets. @@ -1050,7 +1092,7 @@ New `--role=client-reconnect` grades the returning player: not a spectator, owns **New/changed test surface:** `--exercise-match-state` (both roles; host forces a goal, client validates the whole observed sequence and the wire byte), `--role=host-disconnect` for the 5.6/5.7 three-process scenario (paired with `--role=client-reconnect`, which grades the returning player), `--role=host-latejoin`/`--role=client-latejoin` plus `--slot-reservation-seconds=` for §6.3's kickoff promotion, `--match-length=` to reach `FULL_TIME` in a short run, `--replay-log=`, `--fill-bots`/`--no-fill-bots`, `--max-spectators=`. The ball-contact scenario now **steers at the ball with closed-loop real input** instead of a hand-tuned fixed-heading burst, which 5.3 broke by adding `KICKOFF_YAW_JITTER` (0 contacts in 3/3 runs); it thrusts while turning rather than hovering to aim, which took it from 2/3 to 5/5. -**Phase gate:** a full 3v3 start-to-finish including a mid-match disconnect and a late joiner. **Not yet run** — every scenario above was verified at 1v1 (plus a two-bot CI match). The 3v3 gate needs a real multi-client session and is the outstanding item for this phase, alongside Phase 4's own un-run human playtest. +**Phase gate:** a full 3v3 start-to-finish including a mid-match disconnect and a late joiner. **Not yet run** — every scenario above was verified at 1v1 (plus a two-bot CI match). The 3v3 gate needs a real multi-client session; it is item **B** of §0, alongside Phase 4's un-run human playtest (item **A**). ### Phase 6 — Dedicated server productionisation @@ -1072,7 +1114,7 @@ New `--role=client-reconnect` grades the returning player: not a spectator, owns > **Rcon is deferred past v1.** An authenticated remote command channel is a real security surface, and `--max-matches` plus a supervisor covers most of the need with none of it. -**Phase gate:** `docker run` a server, connect from another machine over the internet, play a full match. +**Phase gate:** `docker run` a server, connect from another machine over the internet, play a full match. **Precondition, not a footnote:** §0 item **C** — slot reservations keyed on display name alone — is fixed by task 7.4, so exposing this build to strangers is gated on that, not on this phase. ### Phase 7 — Steam transport, browser, identity @@ -1197,6 +1239,10 @@ godot --path Game -- --connect 127.0.0.1:27015 --name Alice ## 11. Flagged, not solved +**Slot reservation and takeover are keyed on display name alone — item C of §0, and the only open item here with a security character.** `_try_reclaim_slot` matches a joining peer against a departed slot on `slot.player_name == player_name` and nothing else. There is no secret, no token, and no uniqueness constraint on names anywhere in `MatchNet`, so any peer that connects during the 30 s reservation window using a departed player's display name is handed their slot, their ship (mid-flight, at whatever pose it holds), and their team. Demonstrated with a real three-process run, not reasoned about. §6.3's late-joiner queue inherits the same weakness for the name it records, though the queue itself is ordered by arrival and cannot be jumped, so the reservation reclaim is the exploitable path. + +Bounded, but not by much: the attacker must race a genuine disconnect, and they must know the name — which is displayed to everyone in the lobby. The right fix is the one §6.2 step 1 already specifies and Phase 7 already schedules: `hello` carries an `auth_ticket`, and the reservation is keyed to the resulting verified identity rather than to a string the client chooses. **Building a bespoke token now would be inventing half of task 7.4 and then throwing it away**, so this is deliberately left for that task — with the consequence stated plainly: this build must not be exposed to strangers before 7.4 lands, and it is a listed precondition of Phase 6's "connect from another machine over the internet" gate rather than a footnote to it. + **Low-latency present and graphics presets** — *now specified*, see §5.4, §5.5 and tasks 0.17/0.17b. Left here as a pointer because they are the largest wins in the document per line of code changed, and they are video settings rather than netcode. **120 Hz simulation** — deliberately deferred, not dismissed. §5.4 and §5.6 record what it would buy (≈21 ms of world response once L1 has taken the interpolation buffer out, plus ≈8 ms of own-ship feel — the difference between ≈127 ms and ≈107 ms), what it costs (a full bot retrain, half the server density, double the bandwidth), and the one rule that keeps the door open: `TICK_HZ`, never `60`. @@ -1207,4 +1253,4 @@ godot --path Game -- --connect 127.0.0.1:27015 --name Alice **Split-screen.** Tracked separately in `TODO.md`; unrelated to this effort, though the camera-outside-the-ship structure that enables it is the same structure this plan relies on. -**A second, distinct source of the same "Unable to send packet on channel N, max channels: 0" stderr noise, in `networked_match.gd`'s `_broadcast_snapshot` rather than `match_net.gd`'s `_remove_player`.** Only reproduced via the deliberately-adversarial `client-abuse-malformed` smoke role: `_broadcast_snapshot`'s per-peer send races `match_sim.gd`'s host-forced `disconnect_peer()` (the abuse-disconnect path) against the same tick's `connected_peers.has(slot.peer_id)` snapshot, the same general shape of race as the fixed site but on a different call path (a server-initiated forced disconnect, not a normal client-initiated one) and not currently known to be reachable from ordinary play. Left for a dedicated pass — not fixed under this round's time pressure, since the fixed site (gotcha 46's neighbor, the round-2 addendum above) was the one an adversarial review actually flagged as a "clean stderr" violation in the tests this project's own conventions rely on. +**A second, distinct source of the same "Unable to send packet on channel N, max channels: 0" stderr noise — item E of §0, in `networked_match.gd`'s `_broadcast_snapshot` rather than `match_net.gd`'s `_remove_player`.** Only reproduced via the deliberately-adversarial `client-abuse-malformed` smoke role: `_broadcast_snapshot`'s per-peer send races `match_sim.gd`'s host-forced `disconnect_peer()` (the abuse-disconnect path) against the same tick's `connected_peers.has(slot.peer_id)` snapshot, the same general shape of race as the fixed site but on a different call path (a server-initiated forced disconnect, not a normal client-initiated one) and not currently known to be reachable from ordinary play. Left for a dedicated pass — not fixed under this round's time pressure, since the fixed site (gotcha 46's neighbor, the round-2 addendum above) was the one an adversarial review actually flagged as a "clean stderr" violation in the tests this project's own conventions rely on.