diff --git a/CLAUDE.md b/CLAUDE.md index b5594191..74e3a89d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,7 +14,8 @@ Because the gameplay concept (vehicle soccer) can't be copyrighted but specific The prose docs carry far more design rationale than the code comments, and several are load-bearing: -- `multiplayer-next.md` — **the single multiplayer tracking document**: architecture decisions, the wire format, implementation evidence, a numbered "gotchas" list (§9), and the current task breakdown with checkboxes, all in one file. Start at §0 for "what's left". Code comments cite it constantly by section/task number (`§2.4`, `task 5.10`); when a comment does, that section is the real explanation. Phases 0–6 are done and mostly archival; day-to-day work is Phase 7 (Steam) and Phase 8 (matchmaking), whose numbered task breakdown and acceptance criteria live in §7, because that is the format tasks are picked up from. +- `multiplayer-next.md` — **the multiplayer task-tracking document**: outstanding work (§0), a numbered "gotchas" list (§9), and the current task breakdown with checkboxes (§7), all in one file. Start at §0 for "what's left". Day-to-day work is Phase 7 (Steam) and Phase 8 (matchmaking), whose numbered task breakdown and acceptance criteria live in §7, because that is the format tasks are picked up from. Phases 0–6 are done and archival — their evidence lives in git history, not the current doc. +- `MULTIPLAYER_SPEC.md` — the architecture decisions, wire format, server-side input handling, prediction/reconciliation, latency/frame-rate budget and match lifecycle state machine, as sections 1–6. Code comments across `Game/scripts/` cite it constantly by section number (`§2.4`, `§4.1`); many still say `multiplayer-next.md §N` for `N` 1–6 from before this doc was split out — when a comment does, the content is now here, not there. `multiplayer-next.md`'s own §7+ cites `§N` the same way and disambiguates by number (1–6 → this doc, 7+ → itself). - `TRAINING.md` — the full RL workflow (training, curriculum generations, export, eval, difficulty tiers). - `SERVER.md` — dedicated-server build, config, systemd deploy, sizing. - `STEAM.md` — optional GodotSteam custom-build setup and the transport contract. diff --git a/Game/scripts/input_jitter_buffer.gd b/Game/scripts/input_jitter_buffer.gd index 6f7da428..ae7937d0 100644 --- a/Game/scripts/input_jitter_buffer.gd +++ b/Game/scripts/input_jitter_buffer.gd @@ -1,7 +1,7 @@ class_name InputJitterBuffer extends RefCounted -# Per-player server-side input state (multiplayer-next.md §3, task 3.2). +# Per-player server-side input state (MULTIPLAYER_SPEC.md §3; multiplayer-next.md task 3.2). # Deliberately a standalone RefCounted with no scene/RPC dependency — same # reason net_codec.gd and net_interpolator.gd are pure classes — so task # 3.5's unit tests can drive it with scripted arrival traces with no live @@ -18,7 +18,7 @@ extends RefCounted # class's, since only the caller knows the current server tick. const RING_SIZE := 32 -# 500ms at 60Hz (multiplayer-next.md §3.2's own numbers) — a duration, not a +# 500ms at 60Hz (MULTIPLAYER_SPEC.md §3.2's own numbers) — a duration, not a # tick-rate-derived constant, so left as a literal rather than pulling in # SimConstants for one number. const STARVE_ZERO_TICKS := 30 diff --git a/Game/scripts/input_lead_controller.gd b/Game/scripts/input_lead_controller.gd index 9f2f7902..feeacd95 100644 --- a/Game/scripts/input_lead_controller.gd +++ b/Game/scripts/input_lead_controller.gd @@ -1,7 +1,7 @@ class_name InputLeadController extends RefCounted -# Client-owned input_lead control loop (multiplayer-next.md §3.3, task 3.3). +# Client-owned input_lead control loop (MULTIPLAYER_SPEC.md §3.3; multiplayer-next.md task 3.3). # Standalone RefCounted, same reason as input_jitter_buffer.gd — scene-free # so it's directly unit-testable against scripted depth traces. # diff --git a/Game/scripts/local_prediction_history.gd b/Game/scripts/local_prediction_history.gd index c28aeb8b..b3394982 100644 --- a/Game/scripts/local_prediction_history.gd +++ b/Game/scripts/local_prediction_history.gd @@ -3,7 +3,7 @@ extends RefCounted const NetBodyState = preload("res://scripts/net_body_state.gd") -# Client-owned local-ship prediction history (multiplayer-next.md §4.3). +# Client-owned local-ship prediction history (MULTIPLAYER_SPEC.md §4.3). # This is deliberately independent of NetworkedMatch and the scene tree so # sequence/ring behaviour can be tested from scripted traces. Each entry is # tagged with its full sequence number: an old value in a wrapped slot is diff --git a/Game/scripts/match_net.gd b/Game/scripts/match_net.gd index 779411ac..4e429b26 100644 --- a/Game/scripts/match_net.gd +++ b/Game/scripts/match_net.gd @@ -1,7 +1,7 @@ extends Node # Autoload (project.godot [autoload] MatchNet). Handshake + roster layer on -# top of NetworkManager's raw transport (§2.5, §1.3 of multiplayer-next.md). +# top of NetworkManager's raw transport (§2.5, §1.3 of MULTIPLAYER_SPEC.md). # hello/welcome, strict protocol_version and physics_ticks_per_second # gating, player_joined/player_left, and — since lobby.tscn (task 1.5) needs # somewhere durable to keep it across the lobby→match scene transition — diff --git a/Game/scripts/match_sim.gd b/Game/scripts/match_sim.gd index f80a2bb5..d6a1ec7f 100644 --- a/Game/scripts/match_sim.gd +++ b/Game/scripts/match_sim.gd @@ -41,7 +41,7 @@ signal match_bootstrap_received(state: int, at_tick: int, score: Dictionary, end # §6.3 task 5.8: a spectator has been given a vacated slot at a kickoff. signal slot_assigned_received(peer_id: int, slot_index: int) -# Input validation (multiplayer-next.md §3.1 steps 2-3, task 3.4). Deliberately +# Input validation (MULTIPLAYER_SPEC.md §3.1 steps 2-3; multiplayer-next.md task 3.4). Deliberately # lives here rather than in NetworkedMatch: framing/rate abuse is a protocol- # level concern independent of any particular match's roster/slot state, and # this autoload already owns the RPC that receives the raw bytes. diff --git a/Game/scripts/match_state.gd b/Game/scripts/match_state.gd index 1320e66e..f3431fd6 100644 --- a/Game/scripts/match_state.gd +++ b/Game/scripts/match_state.gd @@ -1,6 +1,6 @@ class_name MatchState -# Match lifecycle states (multiplayer-next.md §6.1, task 5.1). +# Match lifecycle states (MULTIPLAYER_SPEC.md §6.1; multiplayer-next.md task 5.1). # # Pure data + a transition table, deliberately with no scene, RPC or # NetworkedMatch dependency — same reason net_codec.gd and diff --git a/Game/scripts/net_body_state.gd b/Game/scripts/net_body_state.gd index 48e77a9e..40ed5e8c 100644 --- a/Game/scripts/net_body_state.gd +++ b/Game/scripts/net_body_state.gd @@ -1,6 +1,6 @@ extends RefCounted -# Plain data holder for one body's snapshot state (§2.4 of multiplayer-next.md). +# Plain data holder for one body's snapshot state (§2.4 of MULTIPLAYER_SPEC.md). # Deliberately not Ship/Ball themselves, and deliberately not a scene-tree # node — NetCodec's pack/unpack must stay callable from pure-function tests # with no live scene. Phase 2's snapshot writer fills one of these per body diff --git a/Game/scripts/net_codec.gd b/Game/scripts/net_codec.gd index 52411b32..0a0df169 100644 --- a/Game/scripts/net_codec.gd +++ b/Game/scripts/net_codec.gd @@ -1,7 +1,7 @@ class_name NetCodec # Wire-format constants, quantisers, and pack/unpack for the two hot-path -# packets (§2 of multiplayer-next.md). Pure functions only — no networking, +# packets (§2 of MULTIPLAYER_SPEC.md). Pure functions only — no networking, # no autoload state — so they're testable head-on by tests/test_runner.tscn # without a live connection. # @@ -48,7 +48,7 @@ const BODY_FLAG_STALLED := 1 << 5 const BODY_FLAG_QUAT_W_SIGN := 1 << 6 # --- Quantisation ranges (§2.4 — derived from arena/gameplay constants, not -# restated prose; see multiplayer-next.md for the ArenaBoundary/Ship/Ball +# restated prose; see MULTIPLAYER_SPEC.md for the ArenaBoundary/Ship/Ball # constants these are sized against) --- const POS_RANGE := 64.0 # metres, ± const VEL_RANGE := 64.0 # m/s, ± diff --git a/Game/scripts/net_interpolator.gd b/Game/scripts/net_interpolator.gd index 8382c66b..4ab07561 100644 --- a/Game/scripts/net_interpolator.gd +++ b/Game/scripts/net_interpolator.gd @@ -3,7 +3,7 @@ extends RefCounted # Buffers recent snapshot samples for ONE remote body and produces # interpolated states at any requested (possibly fractional) server tick — -# used twice per body (multiplayer-next.md §4.1/§4.6, "dual-time remote +# used twice per body (MULTIPLAYER_SPEC.md §4.1/§4.6, "dual-time remote # entities"): once at the present-time estimate for the collider, once # further back at present-minus-INTERP_DELAY for $Visual. # diff --git a/Game/scripts/net_ship_predictor.gd b/Game/scripts/net_ship_predictor.gd index ae07ed9a..b5729fb4 100644 --- a/Game/scripts/net_ship_predictor.gd +++ b/Game/scripts/net_ship_predictor.gd @@ -1,6 +1,6 @@ extends RefCounted -# Local-ship reconciliation policy (multiplayer-next.md §4.4). Kept out of +# Local-ship reconciliation policy (MULTIPLAYER_SPEC.md §4.4). Kept out of # NetworkedMatch so the decision table is pure-testable; the imperative half # only writes Ship's existing Jolt-safe queued correction hooks. diff --git a/Game/scripts/network_manager.gd b/Game/scripts/network_manager.gd index 9905f637..984a5140 100644 --- a/Game/scripts/network_manager.gd +++ b/Game/scripts/network_manager.gd @@ -3,7 +3,7 @@ extends Node # Autoload (project.godot [autoload] NetworkManager). Owns transport-neutral # hosting, joining, shutdown, and connection-state signals. Lives # at a fixed autoload path so RPC NodePaths never depend on which scene is -# loaded (§1.3 of multiplayer-next.md's derived decisions). +# loaded (§1.3 of MULTIPLAYER_SPEC.md's derived decisions). # # server_relay = false is set the moment a peer exists: the default `true` # lets any client rpc() any other client *through the server*, which this diff --git a/Game/scripts/ship.gd b/Game/scripts/ship.gd index 91124204..a5326483 100644 --- a/Game/scripts/ship.gd +++ b/Game/scripts/ship.gd @@ -145,7 +145,7 @@ func queue_teleport_with_velocity(to: Transform3D, new_linear_velocity: Vector3, _has_pending_teleport = true -# --- Netcode correction hooks (Phase 4; see multiplayer-next.md §4.4) --- +# --- Netcode correction hooks (Phase 4; see MULTIPLAYER_SPEC.md §4.4) --- # Both stay zero until Phase 4 wires a reconciliation pass in, so the guarded # hook in _integrate_forces below is a no-op today. # Velocity delta from a soft correction, consumed once then cleared — diff --git a/Game/scripts/sim_constants.gd b/Game/scripts/sim_constants.gd index fc560d19..3074196b 100644 --- a/Game/scripts/sim_constants.gd +++ b/Game/scripts/sim_constants.gd @@ -4,7 +4,7 @@ class_name SimConstants # constant derived from "60 Hz" (Ship._tick_scaled's decay reference, # reaction_ticks' export range, TrainingMode.TICKS_PER_SIM_SECOND) reads this # instead of restating the literal, so changing it changes every derived -# constant coherently — see multiplayer-next.md §5.6 on why a future 120 Hz +# constant coherently — see MULTIPLAYER_SPEC.md §5.6 on why a future 120 Hz # simulation needs to be a config change plus a retrain, not a protocol # rewrite hunting down bare 60s. # diff --git a/Game/scripts/video_settings.gd b/Game/scripts/video_settings.gd index d77a3862..6f22160a 100644 --- a/Game/scripts/video_settings.gd +++ b/Game/scripts/video_settings.gd @@ -24,7 +24,7 @@ extends Node # independently of stretch mode, since it scales the 3D viewport's own internal # resolution before this blit rather than the window itself. Task 0.15b also # found an unexplained ~6% non-uniform width scaling on this project's one -# tested (Mac/Retina) machine — see multiplayer-next.md §5.5.1 — which needs +# tested (Mac/Retina) machine — see MULTIPLAYER_SPEC.md §5.5.1 — which needs # understanding before stretch mode is touched, not blindly carrying into a # resolution-dependent change. # @@ -49,7 +49,7 @@ const SETTINGS_PATH := "user://settings.cfg" # preset -> bundle applied to the individual fields below. CUSTOM has no # bundle: selecting it just stops future preset changes from overwriting # whatever the individual fields currently hold. Task 0.15b's measured -# per-effect costs (multiplayer-next.md §5.5.1) were too noisy to rank these +# per-effect costs (MULTIPLAYER_SPEC.md §5.5.1) were too noisy to rank these # against each other, so each rung is "meaningfully fewer full-screen passes # than the one above it" rather than a precisely tuned ladder. const PRESET_BUNDLES := { @@ -252,7 +252,7 @@ func apply_fps_cap() -> void: # Called once by each arena's _ready() (and again on settings_changed, so an # already-loaded arena updates live) to fold the user's glow/brightness # preference into that arena's own baked Environment tuning, and to gate the -# preset-controlled full-screen passes (§5.5 of multiplayer-next.md). +# preset-controlled full-screen passes (§5.5 of MULTIPLAYER_SPEC.md). func apply_to_environment(env: Environment) -> void: if env == null: return diff --git a/Game/tests/cases/test_net_codec.gd b/Game/tests/cases/test_net_codec.gd index 016d1717..218f8631 100644 --- a/Game/tests/cases/test_net_codec.gd +++ b/Game/tests/cases/test_net_codec.gd @@ -97,7 +97,7 @@ func test_snapshot_roundtrip_seven_bodies() -> void: var packet := NetCodec.pack_snapshot(555, -2, 1234, segment) assert_eq(packet.size(), NetCodec.SNAPSHOT_CLIENT_HEADER_SIZE + segment.size(), "full packet size") - assert_eq(packet.size(), 169, "matches multiplayer-next.md §2.4's 169 B payload figure for 7 bodies") + assert_eq(packet.size(), 169, "matches MULTIPLAYER_SPEC.md §2.4's 169 B payload figure for 7 bodies") var decoded := NetCodec.unpack_snapshot(packet) assert_eq(decoded["last_input_seq"], 555, "last_input_seq") diff --git a/Game/tools/gpu_profile_harness.gd b/Game/tools/gpu_profile_harness.gd index e4d42b75..9c65592a 100644 --- a/Game/tools/gpu_profile_harness.gd +++ b/Game/tools/gpu_profile_harness.gd @@ -1,7 +1,7 @@ extends Node # One-off GPU frame-time profiling harness for task 0.15b's real-hardware -# follow-up (multiplayer-next.md §5.5.1) — the automated Mac passes gave +# follow-up (MULTIPLAYER_SPEC.md §5.5.1) — the automated Mac passes gave # inconsistent, sometimes implausible numbers (stale-process contention, # and Apple Silicon's tile-based GPU architecture is a poor stand-in for the # target reference hardware). Run this directly on a machine with a real @@ -47,7 +47,7 @@ func _ready() -> void: var match_scene := load("res://scenes/match.tscn") as PackedScene _match = match_scene.instantiate() - # 3v3 = 6 ships, matching the scenario multiplayer-next.md §5.5 measures. + # 3v3 = 6 ships, matching the scenario MULTIPLAYER_SPEC.md §5.5 measures. _match.team_size = 3 # Direct-scene-run fallback path (see match_mode.gd:_make_opponent_controller) # — gives every AI ship a real trained policy so thruster VFX/movement diff --git a/MULTIPLAYER_SPEC.md b/MULTIPLAYER_SPEC.md new file mode 100644 index 00000000..f4c38ebf --- /dev/null +++ b/MULTIPLAYER_SPEC.md @@ -0,0 +1,582 @@ +# Online multiplayer — architecture and wire-format spec + +The standing design reference for the online multiplayer effort: locked +architecture decisions, the wire format, server-side input handling, +prediction/reconciliation, the latency/frame-rate budget, and the match +lifecycle state machine. This describes *how the system works* — it is not +task-tracked and does not distinguish implemented from not-yet-implemented; +for that, and for the outstanding task list, see +[`multiplayer-next.md`](multiplayer-next.md), which cites sections here by +number (`§2.4`, `§4.1`, …) and assumes them as background before picking up +Phase 2 or later work. + +--- + +## 1. Architecture decisions + +### 1.1 Locked decisions + +| # | Decision | Why | +|---|---|---| +| 1 | **Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation.** | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. | +| 2 | **Dedicated servers only.** Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. | +| 3 | **ENet first**, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and **must never become the degraded path**. | +| 4 | **Community discovery uses no custom backend; superseded for queued play by Phase 8.** | Steam's server APIs remain enough for the community browser. Casual/ranked queues, durable ratings, allocation and authoritative results require the project-owned Go control plane specified in `docs/MATCHMAKING.md`; it does not replace the browser or direct-IP path. | + +### 1.2 Rejected alternatives + +- **Peer-authoritative ships** (each client owns its own transform). Easiest to build, feels perfect locally, and is trivially cheatable — it directly contradicts `README.md`'s stated anti-cheat position. Ship-vs-ship collisions also become ambiguous with no arbiter. +- **Deterministic lockstep / rollback.** See decision 1. +- **`MultiplayerSynchronizer` / `MultiplayerSpawner`.** The decisive objection is not bandwidth. It is that `last_processed_input_seq` **must** arrive in the same packet as the state it describes, or reconciliation is off by a snapshot — and a synchroniser gives you nowhere to put it. It also writes replicated properties directly onto the node, which is exactly wrong for a `RigidBody3D` under prediction: incoming state has to enter a compare-against-history pipeline, not be stamped onto `global_transform`. You would end up building the correction pipeline anyway, with the synchroniser as pure overhead. Secondary objections: one packet per body (7 bodies × ~50 B of UDP/IP/ENet framing versus one coalesced snapshot), no per-field quantisation, and no client-side interpolation. + + `MultiplayerSpawner` is unnecessary for a separate reason: the roster is fixed at match start and fully described by the `match_config` message, and **no ship is ever despawned** (§6.4). +- **Seeded RNG for kickoff jitter.** Shared-seed determinism requires both sides to consume the RNG stream in exactly the same order forever. The first `randf()` anyone later adds anywhere in the reset path — a spawn VFX variation, a cosmetic, a commentary line — silently desyncs kickoff positions with no error message. The server broadcasts the resulting transforms instead: 336 bytes, once per kickoff, cannot rot. + +### 1.3 Derived decisions + +**All hot-path RPCs live on autoloads.** `/root/NetworkManager` and `/root/MatchNet` exist at identical paths on every peer regardless of which scene is loaded, which side is headless, or whether a client is mid-scene-transition. This deletes the entire "NodePaths must match across peers" class of bugs, kills a family of late-join races where an RPC arrives before its target node exists, and warms Godot's RPC path cache once at connect so it never re-sends a full path on scene change. + +**Entities are addressed by integer slot, never by path.** The snapshot is `[slot 0..N-1]` in a fixed order established by `match_config`. `MatchNet` holds an `Array[Node] _slots` populated at spawn. + +**One server process hosts exactly one match.** This is forced, not chosen: `ship.gd:162` resolves the arena boundary via `get_tree().get_first_node_in_group("arena_boundary")` and `ai_ship_controller.gd` discovers its roster via `get_tree().get_nodes_in_group("ship")`. Both are tree-global, so two matches in one scene tree would cross-wire instantly. It is recorded here because it determines the RAM figure in §1.4. + +### 1.4 Server sizing — bandwidth and CPU are not the constraint + +Worth establishing up front, because §2 and §3 repeatedly trade bandwidth for latency and somebody will eventually want to trade back. + +`ArenaBoundary.bake_colliders()` generates roughly 168 box colliders (corner fillets, base wrap, ceiling, end walls) plus the scene's own slabs, 2 goal backstops, 2 `Area3D` sensors, and 7 dynamic bodies (the ball with `continuous_cd`). Estimated per-tick cost: + +| Component | ms/tick | +|---|---:| +| Jolt step | 0.15 – 0.4 | +| Godot headless main loop | 0.1 – 0.3 | +| Bot inference, amortised | ~0.3 | +| **Total, of a 16.7 ms budget** | **0.6 – 1.1** | + +→ **~6–10 concurrent matches per modern core**, ~150–250 MB RSS per process. 100 concurrent matches ≈ 12–16 cores and ~20 GB — a single mid-tier VPS. Upstream bandwidth for a full 6-player match is ~630 kbit/s (§2.4). + +**Neither CPU nor bandwidth is scarce. Latency is.** Optimise accordingly. + +--- + +## 2. Wire format + +Two peers must agree byte-for-byte, so this is specified rather than sketched. + +### 2.1 Channels + +| Channel | Transfer mode | Contents | +|---|---|---| +| 0 | reliable | handshake, `match_config`, kickoff, goal, clock, state changes, chat, admin | +| 1 | unreliable-ordered | client → server input | +| 2 | unreliable-ordered | server → client snapshots | + +Unreliable-**ordered** (ENet sequenced-unreliable, drops stale) rather than plain unreliable for both hot paths: we carry explicit sequence numbers, and a reordered late packet is worthless work. Separating them stops a large reliable `match_config` from head-of-line-blocking state on a lossy link. + +**Set `ENetMultiplayerPeer.server_relay = false`.** It defaults to `true`, which lets any client `rpc()` any other client *through your server*. With it off, clients can only talk to peer 1. Single highest-value one-line security change in this document. + +### 2.2 Packet header + +**Every hot-path packet opens with a 1-byte type + version.** A capture then decodes standalone, and a mismatched build fails loudly instead of decoding garbage straight into `state.transform`. + +Hot paths carry a single `PackedByteArray` RPC argument (≈14 B of Godot RPC framing once the path cache is warm). Control messages on channel 0 use normal typed arguments — they are rare and readability beats bytes. + +### 2.3 Input packet — client → server, channel 1, 60 Hz + +``` +u8 type_version +u32 seq server-tick-space sequence of the NEWEST action +u8 count 1..4 (MAX_REDUNDANCY) +u32 ack_snapshot_tick newest snapshot tick this client has processed +u16 client_send_ms wrapping ms clock, echoed back for RTT +--- repeated `count` times, newest first --- +i8 thrust_x, thrust_y, thrust_z value = clamp(round(v*127), -127, 127) +i8 rot_x, rot_y, rot_z +u8 flags bit0 = turbo +``` + +**12 + 7×4 = 40 B payload**, ~90 B on the wire with UDP/IP/ENet framing → **~43 kbit/s up per client**. + +- **Redundancy 4** is what makes an unreliable input channel safe: starvation requires four consecutive losses (~66 ms). +- **`i8` per axis, not 3-bit bins.** Bins matching `ShipActionCodec.HEADS` would cut an action to 3 bytes, but they permanently foreclose analog gamepad sticks, which this game will want. `round(v*127)/127` round-trips `-1/0/+1` exactly, so today's digital input (`player_ship_controller.gd` is `is_action_pressed`-only) is lossless. +- **The encoding is itself a validator.** `i8/127` cannot express NaN, Inf, or a value outside `[-1.008, 1.008]`. Half of "sanitise untrusted client input" is solved by not using Variant encoding. + +### 2.4 Snapshot — server → client, channel 2, 60 Hz default + +Per-client header built per peer; body buffer built once per tick and reused across peers. + +``` +--- per-client header (7 B) --- +u32 last_input_seq newest input from THIS client the server has applied +i8 input_buffer_depth jitter-buffer occupancy; negative = starved +u16 echo_client_send_ms from that input packet, for RTT + +--- shared body header (8 B) --- +u8 type_version +u32 server_tick Engine.get_physics_frames() on the server +u8 match_state see §6.1 +u8 reset_gen increments on every authoritative teleport +u8 body_count + +--- repeated body_count times, slot order fixed by match_config (22 B each) --- +i16 pos_x, pos_y, pos_z range ±64 m -> 1.95 mm +i16 quat_x, quat_y, quat_z w = ±sqrt(1-x²-y²-z²), sign in flags +i16 vel_x, vel_y, vel_z range ±64 m/s -> 1.95 mm/s +i8 avel_x, avel_y, avel_z ships ±4 rad/s; ball ±32 rad/s +u8 flags bit0 frozen, bit1 turbo, bits2-4 thrust_z bin, + bit5 stalled, bit6 quat_w sign +``` + +7 bodies → **8 + 7 + 7×22 = 169 B payload**, ~219 B on the wire. + +| | per client down | server up, 6 clients | + 10 spectators | +|---|---:|---:|---:| +| 60 Hz | 105 kbit/s | 631 kbit/s | 1.68 Mbit/s | + +MTU headroom is ~6× (ENet fragments above ~1400 B); a hypothetical 10v10 at 21 bodies is 477 B and still fits. **This format does not need delta compression.** + +**Plain `i16` quaternion components, not smallest-three.** Smallest-three saves 4 B/body and is the textbook answer. It is also exactly where a hand-rolled codec goes subtly wrong — off-by-one in the 2-bit index, sign of the dropped component, renormalisation drift. Three `i16`s plus a sign bit give ~3e-5 rad with no bit-shifting, for 2 B/body (≈3 kbit/s). Take the bytes. + +**Quantisation ranges derive from constants, not from prose.** `ArenaBoundary.INNER_HALF_X = 18.0`, `INNER_HALF_Z = 27.0`, `INNER_HEIGHT = 18.0` (`arena_boundary.gd:8-10`) plus `GameMode.ESCAPE_MARGIN = 15.0`; `Ship.max_speed = 35.0` (`ship.gd:16`); `Ball.MAX_SPEED = 32.0` (`ball.gd:17`). + +**The flags byte must carry `turbo` and a 3-bit `thrust_z` bin.** `_integrate_forces` is not called on frozen bodies, so remote ships on a client never pull `get_action()`, and `Ship._update_movement_vfx()` (`ship.gd:293`) reads `_current_action.thrust.z` and `turbo`. Without those bits, every remote ship flies with dead engines. + +### 2.5 Reliable control messages, channel 0 + +`hello` · `welcome` · `player_joined` · `player_left` · `ready_state` · `match_config` · `scene_ready` · `kickoff` · `state_change` · `goal_scored` · `clock_state` · `match_ended` · `chat` · `server_shutdown`. + +--- + +## 3. Server-side input handling + +Per-player server state: + +```gdscript +class PlayerSlot: + var peer_id: int + var slot: int # snapshot index + var ring: Array[ShipAction] # FIXED 32 entries, indexed seq % 32 + var ring_seq: PackedInt32Array # 32 entries, seq stored at each index (-1 = empty) + var last_applied_seq: int + var last_action: ShipAction + var starved_ticks: int + var packets_this_second: int + var remote_controller: RLShipController +``` + +### 3.1 Ingestion + +`@rpc("any_peer", "unreliable_ordered", channel = 1)`, in order: + +1. `multiplayer.get_remote_sender_id()` → look up slot. Unknown sender → drop and count. +2. **Rate limit.** `packets_this_second > 110` (60 Hz × 1.5 + 20) → drop. Three consecutive seconds over budget → disconnect with `RATE_LIMIT`. Same for a byte budget. +3. **Framing.** `count > 4` or `payload_size != 12 + count*7` → drop, count malformed. 20 malformed → disconnect. +4. **Sequence range.** `seq > server_tick + 20` → drop. (Not 120: `input_lead` is clamped to 12, so anything above ~20 is broken or hostile.) This is why the ring is fixed-size and indexed `seq % 32` — **a client can never make the server allocate.** +5. For each action, newest first at descending seq: `seq <= last_applied_seq` → discard (already consumed); else write `ring[seq % 32]`. +6. **Decode with per-axis clamp only:** + ```gdscript + action.thrust = Vector3(b[0]/127.0, b[1]/127.0, b[2]/127.0).clampf(-1.0, 1.0) + ``` + +> **Never normalise the thrust vector.** A player holding W+A+E legitimately produces `thrust = (1,1,1)`, length 1.73, and each axis uses a different power constant — `thrust_power 150`, `maneuvering_thrust 75`, `vertical_thrust 120` (`ship.gd:12-14`). Normalising would silently change the flight model for honest players. Per-axis clamp combined with the `i8` encoding is complete validation: the reachable value space is exactly what a legitimate client can produce. + +### 3.2 Consumption — once per server physics tick, before the step + +``` +expected = last_applied_seq + 1 +if ring holds expected: + action = ring[expected % 32]; starved_ticks = 0 +else: + action = last_action # REPEAT — do not zero + starved_ticks += 1 + if starved_ticks > 30: # 500 ms + action = ZERO_ACTION; flags.stalled = true +last_applied_seq = expected +last_action = action +remote_controller.action = action +``` + +**Repeat-last, not zero.** Player inputs are heavily autocorrelated at 60 Hz — the odds that a held thrust was released on exactly the dropped tick are low, and the client predicted with the real input either way, so repeating minimises expected divergence. It is also consistent with `AIShipController`, which already holds its action between decisions. Zeroing after 500 ms stops a disconnecting player's ship flying into a wall at full throttle forever. + +### 3.3 Jitter buffer — one control loop, not three + +An earlier draft had the server adapting `target_depth`, the server fast-forward-dropping queued actions, **and** the client slewing `input_lead`. Three integrators acting on one plant (buffer occupancy) with different time constants is a textbook oscillation; on a jittery link it hunts, and it presents to the player as intermittent sticky controls that are nearly impossible to attribute. + +**The server reports `input_buffer_depth` in every snapshot and does nothing else adaptive. The client owns `input_lead` exclusively.** + +- `target_depth = 1` (16.7 ms), not 2. With redundancy-4 you have already bought the insurance depth 2 provides; depth 2 is 16.7 ms of pure input latency for nothing. +- Client `input_lead` clamp `[1, 12]`, **fast attack / slow release**: on any starve, increase by up to 3 **immediately**; decrease by 1 per 60 ticks only after 2 s of clean surplus. A symmetric ±1-per-500 ms slew takes two seconds to absorb a wifi spike, during which the player steers and the ship does not turn — the most rage-inducing failure mode in any netcode. +- Changing `input_lead` means skipping or duplicating one tick's sequence number. Never change it more than once per 30 ticks. + +**Enforce `input_lead` server-side from observed arrival times.** A client that fakes starvation to drive `input_lead` to 1 gets its inputs applied with less server-side buffering than honest players — a small but real responsiveness edge. The `i8` encoding does nothing about this; only observing actual arrival timing does. + +--- + +## 4. Prediction and reconciliation + +### 4.1 Two clocks for remote entities — the load-bearing correction + +The obvious design runs remote ships and the ball as frozen kinematic proxies at `server_time_est - INTERP_DELAY` while predicting the local ship to *now*. **That is wrong**, and it is wrong in a way that only shows up over real latency: + +- Two ships closing at 50 m/s put the opponent's collider **3.5 m** from truth. The hull is a `BoxShape3D` of `(1.6, 0.6, 4)` (`ship.tscn:12`) — that is most of a ship length of positional lie. +- A fast ball is **2.2 m** off against a 0.5 m radius — four ball diameters. +- `ship.tscn:16` has `collision_mask = 7`: ships collide with ships, the ball, and the arena. Ship-vs-ship contact is *constant* in vehicle soccer, not incidental. + +So prediction would not diverge occasionally due to timing noise. It would diverge **deterministically and in the same direction on essentially every contact**, and the hard-snap threshold would become the steady state rather than a backstop. + +**Fix: separate the collider clock from the render clock.** + +| | runs at | why | +|---|---|---| +| remote body **collider** | `server_time_est`, extrapolated forward from the newest snapshot by ~one-way + half a snapshot interval | Extrapolation error over ~45 ms at real accelerations (`thrust_power 150 / mass 5` = 30 m/s², 75 m/s² on turbo — `ship.gd:12,15`, `ship.tscn:17`) is ~0.03–0.08 m. Two orders of magnitude better than 3.5 m. | +| remote **`$Visual`** | `server_time_est - INTERP_DELAY` | Smooth, jitter-free rendering. | + +This is the same trick applied to the local ship, pointed the other way. It costs one extra transform write per remote body per tick. + +### 4.2 Where each piece lives + +| Concern | Location | +|---|---| +| sample + send input | `LocalNetShipController._physics_process` — runs before the physics step, guarantees exactly one sample/tick | +| record predicted state | same, at top of tick N (state = result of N−1) | +| apply velocity / teleport correction | `Ship._integrate_forces`, ~15 guarded lines — the only Jolt-safe place to write `state.transform` / `state.linear_velocity` | +| visual smoothing | `Ship/$Visual.global_transform`, set in `_physics_process` | +| snap-vs-blend decision | `net_ship_predictor.gd` (child node) | +| remote bodies | `net_interpolator.gd` | + +### 4.3 Per-tick, own ship + +1. `predicted[current_tick - 1] = {transform, linear_velocity, angular_velocity}` — ring of 128. +2. `var a := _player.get_action().copy()` — **must copy.** `player_ship_controller.gd` reuses a single `ShipAction` across ticks (its own header warns about this); buffering it aliases every history entry to the same object. +3. `_action = a`, returned by `get_action()` this tick so `Ship._integrate_forces` samples input exactly once. +4. `input_history[seq] = a`, `seq = predicted_server_tick + input_lead`. +5. Build and send the packet with the last 4 entries. + +`Ship._integrate_forces` then runs completely unchanged. + +### 4.4 On snapshot arrival + +``` +A = last_input_seq +if reset_gen changed OR predicted[A] missing OR flags.frozen != local frozen: + HARD SNAP +else if pos_err > 2.0 m OR rot_err > 60°: + HARD SNAP +else: + SOFT CORRECT +``` + +Comparing server state at tick `A` against **`predicted[A]`** — the client's own state at that same tick — makes the delta latency-free by construction. That is the entire reason for keeping the prediction ring, and it is why this works acceptably without resimulation: **never blend current state toward stale state.** + +**SOFT CORRECT** + +- **Velocity: applied in full, immediately.** `net_vel_correction += (srv.linvel - predicted[A].linvel)`, consumed once in `_integrate_forces`. Velocity error is invisible to the player but is the *cause* of future position error; blending it just prolongs divergence. +- **Position/rotation: physics moves in full, rendering does not.** Queue the body teleport, and simultaneously offset `$Visual` by the negation. Net visual movement at the instant of correction: zero. The body is where the server says; the rendered ship catches up. +- **Decay** each physics tick, reusing the existing convention at `ship.gd:450`: + ```gdscript + var k := _tick_scaled(0.88, delta) # 63% gone in ~130 ms, 95% in ~280 ms + ``` +- **`MAX_VISUAL_OFFSET = 0.4 m`**, not 2.0. The hull is 4 m long; a 2 m offset means being rendered half a ship-length from your own collider for ~280 ms, so you clip walls you visibly cleared — a felt bug in a game built around wall-riding. Beyond 0.4 m, show the correction. A visible correction is honest; an invisible 2 m lie is not. + +**HARD CORRECT** + +- Apply the same sequence-matched authoritative pose and velocity delta to the current local body, reset body and `$Visual` interpolation, and clear the visual offset. It is physically the same correction as soft correction; only its presentation differs. + +**Delta transport, not one-body replay.** For every matched snapshot, overwrite `predicted[A]` with authority, transport its pose and linear/angular-velocity delta through each retained state `A+1..current`, and apply that same delta once to the live local Jolt body. This keeps retained history coherent, so a later snapshot does not correct an already-corrected pre-delta trajectory a second time. + +Do **not** analytically replay stored actions. That approximation cannot reproduce Jolt integration or contact manifolds (friction, restitution, walls, ships, and ball), therefore it becomes least trustworthy exactly where reconciliation is most noticeable. This is still neither whole-world rollback nor a change to server physics: it is client-only state transport around a server-authoritative simulation. + +For reset generation changes, place exact authority, begin a new history epoch, and do not consume pre-reset actions. For missing or overflowed history, place authority once and suppress stale acknowledgements until a new matched sequence is recorded; never manufacture future history by filling it with one stale authority state. + +> Same-sequence **pre-correction** residual remains diagnostic telemetry. With a server input jitter buffer, it is not by itself a presentation-quality gate: the server may have integrated an action at a different physical instant from the client. Acceptance must report it separately by free-flight/contact/reset/resync cohort, while gating post-correction/presentation error and hard-snap behaviour. +> +> That the two sides integrate the **same action** for a given sequence is a separate claim, and a checkable one. Keep the two apart: "right action, different instant" is expected here; "wrong action" is a bug, and was one (`multiplayer-next.md` §9 gotcha 47). + +### 4.5 Camera and visuals + +**The camera must follow `$Visual`, not the body.** `ship_camera.gd:115`, `:149`, `:150` read `target.global_transform` directly. Left as-is, every soft correct makes the *camera* jump the full error while the *mesh* smoothly lags — strictly worse than snapping, because the world lurches around a player whose ship slides inside the frame. + +**And it must read `$Visual.get_global_transform_interpolated()` from `_process`, not `global_transform` from `_physics_process`** (rationale in §5.4). `Node3D.get_global_transform_interpolated()` exists precisely for a camera tracking a physics-interpolated body; `global_transform` returns the last physics tick's pose, so a `_process` camera reading it would chase a 60 Hz staircase at 240 fps. + +> **Ordering hazard**, straight from the engine docs: `get_global_transform_interpolated()` "creates an interpolation pump on the `Node3D` the first time it is called, which can respond to physics interpolation resets… be sure to call it at least once before resetting the `Node3D` physics interpolation." Every hard snap calls `reset_physics_interpolation()` on `$Visual`. **Prime the pump when the camera's `target` is assigned**, not lazily on the first frame, or the first snap of the match streaks the camera. + +`project.godot` has `physics_interpolation=true`, and `$Visual`'s own local transform is interpolated too — so `reset_physics_interpolation()` must be called on `$Visual` as well as the body, or every snap smears the mesh for a frame. + +### 4.6 Remote bodies on the client + +- `freeze = true`, `freeze_mode = FREEZE_MODE_KINEMATIC` — **not `STATIC`**, or Jolt cannot derive contact velocity from the per-tick transform delta and your predicted ship hits a static wall instead of a moving ship. +- `net_interpolator.gd` samples the snapshot buffer (last 8 per body); collider at `server_time_est` (§4.1), `$Visual` at `server_time_est - INTERP_DELAY`. +- **The two samples run on different clocks *and* different callbacks.** The collider is a physics concern: `_physics_process`, 60 Hz. `$Visual` is a render concern: `_process`, sampled at true render time with `physics_interpolation_mode = OFF` so Godot does not interpolate an already-per-frame transform. On a 240 Hz client this is 240 distinct remote-ship positions per second instead of 60, and one fewer tick of lag, for no extra cost — the buffer lerp is happening either way (§5.4). +- `INTERP_DELAY = one_way_ms + snapshot_interval * 1.5 + 2.5 * jitter_ewma`, clamped `[25, 200] ms`. At 60 ms RTT / 60 Hz / 5 ms jitter that is 30 + 25 + 12.5 ≈ **68 ms**. + +> **The `one_way_ms` term is not optional, and omitting it is a silent architectural failure.** `server_time_est` (§4.7) estimates what the server clock reads *right now*. The newest snapshot in hand was stamped `one_way` ago. So rendering `$Visual` at `server_time_est - INTERP_DELAY` only interpolates if `INTERP_DELAY ≥ one_way`. Set it to the buffer alone (~38 ms at 60 Hz) and the render cursor lands *on or past* the newest sample: every remote entity is permanently dead-reckoned. **The 25 ms clamp floor is reachable on LAN only.** +- Past the newest snapshot, extrapolate on last known velocity for at most 150 ms, then hold. **Never extrapolate indefinitely** — a stuck ship reads better than one flying through a wall. +- **Never write `linear_velocity` to a frozen body.** Godot/Jolt zeroes and holds velocity on frozen bodies, so `ball.gd:35`'s `linear_velocity.length()` trail driver will not work that way. `Ball.set_visual_speed(speed)` mirrors the `Ship.set_visual_action(thrust_z, turbo)` pattern. Don't route presentation data through a property the physics server owns. +- Call `reset_physics_interpolation()` on remote bodies at every kickoff. + +### 4.7 Clock + +`server_time_est = local_ms + clock_offset`, `clock_offset` from ping/pong on channel 0 every 1 s using the **minimum-RTT sample in a rolling 5 s window** (the min-RTT sample has the least queueing error). + +**Freeze `tick_offset` at match start.** Seed it exactly from the handshake (`server_tick + round(one_way / tick_ms)`) and absorb all subsequent drift into `input_lead` alone. The prediction ring is indexed in server-tick space, so slewing `tick_offset` during play silently reinterprets every historical entry and produces sporadic, unreproducible false snaps. Re-seed only across a kickoff boundary. + +--- + +## 5. Latency and frame-rate budget + +Three of the largest terms are invisible to a netcode document that only counts network hops. Record the budget so future changes are argued against a number. + +Client at 60 Hz physics, 60 ms RTT, 5 ms jitter, 60 Hz snapshots. **Display at 60 Hz with vsync on** — the Godot default, and the worst case. §5.4 redoes the display-dependent rows for 120/144/165/240/360 Hz. + +### 5.1 Own ship (predicted) — input to pixel + +| Stage | ms | | scales with fps? | +|---|---:|---|---| +| OS input → `Input.is_action_pressed` | 10 | 0.5 × frame interval + device polling | partly | +| wait for next physics tick | 8 | avg of 0–16.7 | no — 60 Hz physics | +| physics step applies force | 0 | | | +| Godot physics interpolation | 8 | mean, worst case 16.7 | no — 60 Hz physics | +| render + vsync present | 25 | 1.5 refresh intervals, vsync defaults on | yes | +| **Total** | **≈52** | | | + +This is the **existing single-player floor**, unchanged by netcode. A low-latency present would take it to ~35 ms (§5.4). Note: **16 of the 52 ms do not move no matter how many frames the client draws.** That is the price of a 60 Hz simulation. + +### 5.2 World response — the number that decides whether this ships + +| Stage | ms | | +|---|---:|---| +| input freshness | 10 | 0.5 × frame interval + ~2 ms device polling | +| wait for next physics tick | 8 | | +| manual multiplayer flush | ~0 | | +| client → server transit | 30 | RTT/2 | +| jitter buffer, `target_depth = 1` | 17 | | +| server tick + flush | 8 | | +| **server → client transit** | **30** | RTT/2 — the return leg | +| interpolation buffer beyond arrival | 38 | `interval × 1.5 + 2.5 × jitter`; the `one_way` half of `INTERP_DELAY` is the row above | +| client physics interpolation | 8 | | +| render + present | 25 | vsync on, 60 Hz display | +| **World response, opponents** | **≈174** | | +| **Ball, with local prediction** | **≈52** | same as own ship | +| Both, at 144 Hz + low-latency present | **148 / 26** | §5.4 | + +For reference, Rocket League runs 120 Hz physics and predicts both car and ball locally; its equivalent at 60 ms RTT is roughly 90–110 ms. + +**≈174 ms as designed here is not competitive, and this document should not pretend otherwise.** §5.6 gets to ≈127 ms with two changes that touch no graphics setting and require no bot retrain, and to ≈103 ms with 120 Hz simulation — inside the reference band. Read §5.6 before treating this table as a verdict. + +What *is* settled is the shape of the design: a locally-predicted ball and own ship at ≈52 ms is the difference between this being playable and not, and a 30 Hz / default-poll / interpolated-ball design would land near ≈250. + +### 5.3 Why 60 Hz snapshots, not 30 + +- Interpolation buffer: the `interval × 1.5` term is **50 ms at 30 Hz vs 25 at 60**, on top of the one-way term both share (§4.6), plus a half-interval of cadence quantisation. +- Interpolation fidelity: at `MAX_SPEED = 32` the ball moves **1.07 m between samples at 30 Hz** — more than its own diameter, so any wall bounce landing between two samples gets lerped as a straight line *through the wall*. At 60 Hz it is 0.53 m. +- Cost: 300 kbit/s. Per §1.4, bandwidth is not the constraint. + +Keep `--snapshot-hz 30` as an explicit degraded mode. + +### 5.4 High-refresh-rate clients — 120 / 144 / 165 / 240 / 360 Hz + +Players on high-refresh displays are the ones most sensitive to everything in this document. Three places in the code must run per rendered frame, not per physics tick, for this to be true — these are now implemented (Phase 0); the reasoning is kept here because it explains why the split exists. + +#### What frame rate actually buys + +| Display | present | own ship / ball (§5.1) | world response (§5.2) | with low-latency present | +|---|---:|---:|---:|---:| +| 60 Hz | 25.0 | **52** | **174** | 35 / 157 | +| 120 Hz | 12.5 | **35** | **158** | 27 / 149 | +| 144 Hz | 10.4 | **33** | **155** | 26 / 148 | +| 165 Hz | 9.1 | **31** | **153** | 25 / 147 | +| 240 Hz | 6.3 | **27** | **149** | 23 / 145 | +| 360 Hz | 4.2 | **24** | **146** | 21 / 144 | + +> **This table's reachability depends on the render budget — see §5.5 for what was actually measured on reference hardware.** + +Three conclusions to design around: + +1. **60 → 144 Hz is worth ~19 ms on own-ship feel. 144 → 360 Hz is worth ~9.** The curve flattens hard, because 16 ms of the remaining budget is the 60 Hz physics tick plus its interpolation and does not move. +2. **A low-latency present is worth more at 60 Hz (−17 ms) than the entire jump from 144 to 360 Hz.** It costs one settings dropdown. +3. **Frame rate barely moves world response** — 174 → 146 across the whole 60–360 range, because that budget is dominated by RTT and the interpolation buffer. Frame rate is an *own-ship feel* lever, not a netcode one. Say this to players plainly; someone who buys a 360 Hz monitor to see opponents sooner has been mis-sold. + +#### Frame-time variance, not mean frame rate, is the real target + +At 240 fps the frame budget is **4.17 ms**, and physics runs at 60 Hz — so **one frame in four carries the entire physics tick** and must still fit in 4.17 ms. On that frame the client pays, in one go: the Jolt step over 7 dynamic bodies against a 172-shape compound; 7 × `Ship._integrate_forces`; 6 × `_update_movement_vfx`; and on decision ticks, bot inference — `policy_network.gd` is a pure-GDScript MLP at **31→64→64→7 ≈ 6.5k multiply-accumulates per bot**, so five bots landing together is ~33k GDScript float ops in one frame. + +**The physics tick sets a floor on 1%-low frame time that no graphics setting can lower.** A game that averages 240 fps but drops one frame in four to 8 ms is not a 240 fps game. Profile p99, not mean. + +#### What frame rate does *not* buy, so nobody optimises the wrong thing + +**Input sampling does not improve.** `player_ship_controller.gd:15-38` reads seven `Input.is_action_pressed` calls — all digital, all held-state — and `Ship._integrate_forces` pulls them once per physics tick. The state read at the tick *is* the freshest state; sampling it 240 times a second returns the same value 4 times in a row. **Do not build a sub-tick input accumulator.** If analog stick support is added later this changes, and the right answer is then a time-weighted average over the tick, not a higher sample rate. + +**Physics interpolation stays on.** It costs ~8 ms (§5.1) and is the single largest fps-independent term after the tick wait, so it will look like a target. It is not: without it a 60 Hz simulation presents 60 distinct world states per second regardless of frame rate, which is precisely the stepping a 240 Hz display was bought to avoid. Leave it on; do not expose a toggle. + +#### Why physics stays at 60 Hz, and what a bump would cost + +The honest answer to "our players want 240 fps responsiveness" is that **simulation rate, not frame rate, is the binding constraint** — 16 ms of own-ship latency and ~33 ms of world response sit behind it, and §5.2 shows frame rate alone cannot get world response under ~146 ms. Doubling to 120 Hz (Rocket League's rate, with snapshots raised alongside) would take world response from ≈174 to **≈141 ms** and own-ship from 52 to **≈44**, at 60 Hz display — or **≈115 ms** combined with a 144 Hz display and a low-latency present. + +That is a bigger win than every tuning parameter in §3 and §4 combined. It is nonetheless **out of scope for v1**, for reasons that are about the project rather than the netcode: + +- **Every policy in `Game/bots/` is invalidated.** `ship.gd:450`'s `_tick_scaled` is defined against a 60 Hz reference and `ai_ship_controller.gd`'s `reaction_ticks` counts ticks. A bump means a full retrain — and per `TODO.md` the generation-5 curriculum is still running. +- **Server density halves**, ~6–10 matches per core to ~3–5 (§1.4). +- **Bandwidth roughly doubles**: input 43 → 86 kbit/s up, snapshots 105 → 210 kbit/s per client, 631 kbit/s → 1.26 Mbit/s per 6-player match. Still not the constraint, but 100 concurrent matches becomes ~126 Mbit/s of server uplink, which is a hosting-plan question rather than a rounding error. + +**The consequence for this plan is a hard rule: 60 is a constant named `NetCodec.TICK_HZ`/`SimConstants.TICK_HZ`, never a literal.** Ring sizes, `INTERP_DELAY`, `input_lead` clamps, seq-window bounds, snapshot cadence and the timeout constants all derive from it — this is already true in code. Done this way, a later bump is a config change plus a retrain — not a protocol rewrite. + +#### Client display settings + +`VideoSettings` implements a `Preset` system (Low/Medium/High/Custom), `vsync_mode` (Adaptive default), refresh-derived `fps_cap_divisor`, and `resolution_scale` — see §5.5 for what these bought when actually measured. The design reasoning that shaped them: + +- **Adaptive vsync** (`FIFO_RELAXED`) is FIFO while the renderer keeps up and tears only on a *missed* vblank — the right default for a game that will sometimes drop below refresh, avoiding FIFO's half-rate cliff. +- **FPS cap options are derived from the display**, not a fixed list — non-divisor caps beat against scanout (cap at 100 on a 144 Hz display and `gcd(100,144) = 4`: visible micro-stutter). +- `Engine.max_fps` is a throttle, not a pacer — it has no knowledge of scanout and never phase-locks to a vblank. +- Godot cannot report the *negotiated* present mode — `DisplayServer.window_get_vsync_mode()` echoes back the mode you stored, not the driver's actual grant. A live fps readout is the honest alternative. + +### 5.5 Can this build produce frames at all? — measured + +§5.4's table describes a device-class question, not a code question, and the +render configuration was a showcase build, not a competitive one by default. +**Measured on real reference hardware (RTX 3090, Linux, via +`Game/tools/gpu_profile_harness.gd`)**, 6-ship 3v3 Match, 1080p: + +| | p50 | p99 | fps (p50) | +|---|---:|---:|---:| +| All effects on (project defaults) | 1.85 ms | 2.98 ms | 540 | +| All effects off | 0.53 ms | 1.53 ms | 1883 | + +At 540 fps p50 with every effect enabled, **this scene is nowhere near +GPU-bound on reference-class desktop hardware** — the "must hit 144 fps" +framing this section originally worried about does not hold at that +hardware tier. SDFGI and SSIL account for over half of the effects' total +cost (0.36 ms and 0.25 ms respectively), matching the original expectation +that voxel cone tracing and a full-res screen-space GI pass would be the +expensive ones. + +An earlier pass on Apple Silicon (M4, Metal) measured a much lower, +undifferentiated ~55 fps ceiling with all effects clustering at 2.9–3.8 ms +each — this was a poor stand-in for the target platform: Apple's +tile-based-deferred GPU architecture forces a full system-memory resolve on +any pass reading neighbouring pixels (SSAO, SSIL, glow, `screen_texture`), +which is a largely constant per-pass tax rather than proportional to each +effect's real cost. Treat that number as informative about relative +ordering only, not as a stand-in for desktop-GPU behaviour. + +**Still open: no low/mid-tier GPU has been profiled.** The 3090 result rules +out "the game is GPU-bound on reasonable hardware" as a near-term concern, +but says nothing about a GTX 1660 or an integrated Iris/Vega part, which is +where a real preset ladder actually earns its keep. Re-run +`gpu_profile_harness.tscn` on weaker hardware before spending more effort on +frame-time optimisation. Baking the arena GI to retire SDFGI is real +but smaller than originally assumed on a 3090-class GPU — it stays worth +doing for low-end/integrated GPUs, unmeasured; a separate-physics-thread +prototype was closed without implementation, since no +frame-time variance problem exists to fix on reference hardware. + +### 5.6 Closing the gap to the reference — without lowering settings + +§5.2 lands at ≈174 ms against a ~90–110 ms reference band. The instinct is that reaching it means trading visual quality for frames. **It does not.** Decompose the 174: + +At 60 ms RTT, 60 ms is transit and irreducible in code. That leaves **114 ms of local overhead**, of which frame rate governs only two terms — input freshness (10) and present (25) — and *quality settings* govern neither directly. Present latency is a function of vsync mode and swapchain depth, not of how many effects are enabled; a 60 fps client with a shallow present queue beats a 240 fps client with a deep one. **The entire 60 → 240 fps range is worth ~12 ms once a low-latency present is in place** (§5.4). The other ~100 ms is netcode time model and simulation rate. + +Four levers, none of which touches a graphics setting: + +| | Lever | Saves | Risk | +|---|---|---:|---| +| **L1** | **Extrapolate remote *visuals* to present time** instead of interpolating the past | **−30** | Mis-prediction pops | +| **L2** | 120 Hz simulation | −21 | Bot retrain, ½ server density, 2× bandwidth | +| **L3** | Adaptive jitter-buffer depth, 0 on clean links | −8 | Starvation on jittery links | +| **L4** | Shallow present queue + Adaptive vsync | −17 | Throughput loss if GPU-bound | + +**L1 is the big one, and it is nearly free.** §4.1 already computes remote entities' **present-time** state — that was the fatal correction that put the collider at `server_time_est`. `$Visual` is then deliberately rendered ~68 ms in the past for smoothness. **Render it at present time too and the whole 37.5 ms interpolation buffer disappears**, leaving only a residual for error smoothing. + +The reason this is safe here is that ships have bounded acceleration and the hull is large. Extrapolating with known velocity, error is `½·a·t²` over the full 68 ms horizon: + +| | max accel | error @ 38 ms | error @ 68 ms | +|---|---:|---:|---:| +| position, cruise | 30 m/s² | 0.022 m | **0.069 m** | +| position, turbo | 75 m/s² | 0.054 m | **0.173 m** | +| yaw | 20 rad/s² | 0.8° | **2.6°** | +| pitch / roll | 2.9 rad/s² | 0.1° | **0.4°** | + +**0.17 m and 2.6° worst case, against a 4 m hull.** Feed the residual through the same soft-correct pipeline already specified for the local ship (§4.4) and remote ships are visually at present time with a sub-decimetre wobble. + +Two bonuses: it **collapses §4.1's dual clock back into one** — collider and visual both at `server_time_est`, so §5.4's `_process`/`_physics_process` split and the two-regimes-for-one-node-name hazard both go away — and it applies to the ball, which is near-ballistic between contacts and therefore extrapolates better than ships do. + +The cost is real but narrow: a remote ship that *reverses input* at the moment you sample it mispredicts by the numbers above and then visibly corrects. Interpolation never mispredicts; it is just always late. This is the genuine trade, and it is the one the reference class makes. **L1–L4 are not yet implemented; this remains netcode work, not measurement — see `multiplayer-next.md` §11 for status.** + +| Term | today | L1 + L4 (v1) | + L2 + L3 | at 144 fps | +|---|---:|---:|---:|---:| +| input freshness | 10 | 10 | 10 | 5.5 | +| wait for next tick | 8.3 | 8.3 | 4.2 | 4.2 | +| client → server | 30 | 30 | 30 | 30 | +| jitter buffer | 16.7 | 16.7 | 4.2 | 4.2 | +| server tick + flush | 8 | 8 | 4 | 4 | +| server → client | 30 | 30 | 30 | 30 | +| interp buffer → extrapolation residual | 37.5 | 8 | 8 | 8 | +| client physics interpolation | 8.3 | 8.3 | 4.2 | 4.2 | +| present | 25 | 8.3 | 8.3 | 3.5 | +| **World response** | **≈174** | **≈127** | **≈103** | **≈94** | + +**≈103 ms at 60 fps with every effect enabled**, and ≈94 at 144 fps — inside the reference band, without disabling SDFGI, SSIL, SSAO or shadows. Sequencing follows ms-per-unit-of-risk: **L4 then L1 first (≈127 ms, no bot retrain, no protocol change)**; L2 and L3 after, when a retrain is affordable. + +> **The largest lever is not on this list.** All of the above assumes 60 ms RTT. Regional server siting that puts most players on a 30 ms RTT takes ≈127 to ≈97 and ≈103 to ≈73 with no code at all. Server siting (Phase 6/Phase 8 in `multiplayer-next.md`) owns it, and it should be argued against these numbers. + +### 5.7 The next tier — and where it stops paying + +#### Frame rate: SDFGI is the wrong tool for this arena + +`arena.gd`/`goal.gd` have **no `_process`, no `_physics_process`, no `AnimationPlayer` and no `Tween`** — the floor, walls, ceiling, goals and every light are static for the entire match. SDFGI exists to light *dynamic* worlds, and it pays for that by re-voxelising cascades as the camera moves — and this camera never stops moving. It is the most expensive optional effect in the frame (§5.5), doing continuous work to solve a problem this project does not have. **Replace `sdfgi_enabled` with baked GI** (`LightmapGI` or `VoxelGI`) — still open, see §5.5's "still open" note; the relative win is real but the absolute win on reference-class hardware is small. + +#### Latency: what is actually left, after L1–L4 and 120 Hz simulation + +At 144 fps the budget would be ≈94 ms — **and 60 of that is RTT.** The remaining ~34 ms of local overhead sits at or near a floor set by physics rate or hardware. Two small code ideas remain, both trading visual stability for a few ms: forward-extrapolating the local `$Visual` instead of interpolating the last two ticks (~4 ms, risk of overshoot on collision), and tightening the extrapolation-error smoothing (~4 ms, more visible correction pops). **That is the whole remaining code budget.** Regional server siting is worth 4× that for free (§5.6). + +Two limits worth keeping in mind before spending a month on the last 5 ms: + +1. **Past ~100 ms, you are optimising 3–4 ms at a time against a 60 ms constant.** Server siting and matchmaking dominate everything else from that point on. +2. **"Lowest lag" and "best feel" diverge at the end.** Every remaining lever buys milliseconds by predicting further ahead and correcting harder. Past a point that makes the game feel *worse*. **The human playtest is the authority; the budget table is not.** + +--- + +## 6. Match lifecycle + +### 6.1 State machine + +``` +LOBBY -> LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP -> ... + -> FULL_TIME -> OVERTIME_WARMUP -> OVERTIME -> ... + -> RESULTS -> LOBBY +``` + +Broadcast as the `match_state` byte in every snapshot, and on transition via `state_change(state, at_tick)`. + +### 6.2 Sequence + +1. **Connect.** Client sends `hello(protocol_version, physics_ticks_per_second, display_name, auth_ticket)`. Server rejects a mismatch on **either** version or tick rate, with a reason string, then `disconnect_peer`. `auth_ticket` is an empty `PackedByteArray` until Phase 7 lands — the field is reserved. +2. **Welcome.** Server assigns `player_id`, balances teams, replies `welcome(player_id, server_info, roster, match_state, server_tick, score, end_tick)`, broadcasts `player_joined`. +3. **Lobby.** `ready_toggle()`; start when all ready, or `--auto-start` after `--min-players` plus a countdown. +4. **Config.** `match_config(match_id, arena_path, team_size, match_length_ticks, roster[], seed)`. `roster[i] = {slot, team, spawn_index, player_id, name, is_bot}` — **slot order here is the snapshot's body order for the whole match.** The client validates `arena_path` against `ArenaRegistry.ARENAS` before `load()`; a malicious or buggy server must not be able to make a client load an arbitrary `res://` path. +5. **Load.** Both sides load `networked_match.tscn`. Each peer loads the arena and spawns the roster in slot order. Client additionally spawns a camera rig on its own ship and adds `HUD.tscn` **in code**. Client sends `scene_ready(match_id)`. +6. **Kickoff.** Server waits for all `scene_ready` (10 s timeout → proceed). Broadcasts `kickoff(reset_transforms[], countdown_start_tick, reset_gen)`. Both sides freeze bodies. HUD counts down from `server_tick`, not a local `Timer`. At `countdown_start_tick + 180` the server unfreezes and broadcasts `state_change(PLAYING)`. +7. **Play.** Inputs up, snapshots down. +8. **Goal.** Server's `Goal` sensor fires → `_handle_goal_scored` debounce → `goal_scored(scoring_team, score, goal_tick, resume_tick)`. Bodies freeze. Clients play the cinematic within `[goal_tick, resume_tick]`. At `resume_tick`: `kickoff(...)`. +9. **Clock.** Tick-derived: `remaining_ticks = end_tick - current_server_tick`. `end_tick` and a `running` flag ship in `match_config` and in `clock_state(running, end_tick, at_tick)`. +10. **Full time / overtime / results.** `RESULTS` holds, then `state_change(LOBBY)` and both sides load `lobby.tscn`. **Clients return to the lobby, not the main menu** — a community server that empties every 2.5 minutes is dead on arrival. + +**Every lifecycle message carries absolute ticks**, never durations. **Specify the late-arrival case explicitly**: a `kickoff` that lands after its own `resume_tick` must apply the reset immediately and skip the countdown, not schedule it into the past. + +### 6.3 Late joiners and spectators + +`welcome` carries full state, so a late joiner reconstructs immediately. + +- Free slot and state is `LOBBY`/`WARMUP` → join as a player now. +- Free slot mid-match → **spectate now, take the slot at the next kickoff.** Swapping a controller at a kickoff boundary is free; mid-play it is not. +- No free slot → spectator. A spectator receives identical snapshots (the snapshot is already a broadcast — zero extra server work), spawns no ship, and points a camera rig at a chosen ship or the ball. Cap with `--max-spectators`. + +### 6.4 Disconnects — no ship is ever despawned + +On `peer_disconnected` the server **keeps the ship and swaps its controller**: + +1. `--fill-bots`: replace with an `AIShipController` on the server's configured model. +2. `--no-fill-bots` (default for public servers, see §1.4): swap to the base `ShipController` — inert but simulated. + +Set `flags.stalled` so clients can grey out the nameplate. Reserve the slot for 30 s keyed by identity so a reconnect gets its ship back. If the last human leaves, abort to `LOBBY`. + +**Justification is wire-format simplicity, not the bot cache.** Fixed slot order means the snapshot needs no add/remove machinery, no `MultiplayerSpawner`, and no re-indexing. That reason stands on its own. diff --git a/TODO.md b/TODO.md index 0c8fecc1..51a6ae8d 100644 --- a/TODO.md +++ b/TODO.md @@ -27,3 +27,20 @@ Phase 7 begins with optional GodotSteam bootstrap and a transport boundary; dire **Tasks 0.1–0.15, 0.18–0.25, 0.27, 0.29 are done** (see the Phase 0 table in `multiplayer-next.md` for what each one actually changed — several deviated from the original plan for concrete GDScript/Godot reasons recorded inline). Remaining, all blocked on **0.15b (profile, on reference hardware, in the live editor — not done)**: 0.16 (camera to `_process`), 0.17/0.17b/0.17c/0.17d (graphics presets, vsync, resolution scaling), **0.26 (bake the arena GI to retire SDFGI — the largest frame-time win available, costs no image quality since the arena is fully static)**, and 0.28 (physics separate-thread prototype, flagged as the riskiest task in the phase). These need a human at the editor with real hardware to profile and eyeball, not further code changes. - [ ] Possible v0.2 split-screen: spawn one `ship_camera_rig` + viewport per local player (camera is already outside the ship scene to allow this). Unrelated to online play. + +### What's left to actually finish multiplayer (human-actionable) + +Everything below needs a person — hardware, a design decision, an external account, or hands on a controller — not more code from an agent working alone. Full detail for each is linked; this list exists so nothing falls through the cracks. Ordered roughly as it blocks. + +- [ ] **Decide the join-signing design for the Phase 8 root blocker.** No real deployment can advance a match past `PROCESS_READY` today because nothing calls the (fully built and tested) assignment-publishing path in production — it needs a join-signing key shared between allocator and game server, roster-digest computation, and per-player authorization construction, deliberately flagged rather than built pending this decision. See `multiplayer-next.md` §0 ("the actual current root blocker") and §8.31. +- [ ] **Phase 4 playtest at ~100 ms RTT** — does the ship/ball feel local, do contact corrections read as bumps or glitches? Every numeric gate is green; this is a feel judgment no metric can answer. `multiplayer-next.md` §0, gate A. +- [ ] **Phase 5 3v3 gate** — a full 6-player match start to finish, with a mid-match disconnect and a late joiner. Only verified so far at 1v1 plus a two-bot CI match. `multiplayer-next.md` §0, gate B. +- [ ] **Phase 6 external gate** — run the exported Docker server and clients from separate real machines over the internet, then play a full match (controlled test only, since defect C below is still open). `multiplayer-next.md` §0. +- [ ] **Acquire a project-owned Steamworks App ID and coordinate with Valve** — hard prerequisite for Phase 7 (browser, verified tickets, bans, production credentials, ticketed Hosted Dedicated Server SDR) and therefore for Phase 8. `multiplayer-next.md` §0, Phase 7; `STEAM.md`. +- [ ] **Supply custom GodotSteam client/server build templates** and pin them in `steam-dependencies.lock.json` (`COSMIC_CLASH_STEAM_CLIENT_GODOT` / `COSMIC_CLASH_STEAM_SERVER_GODOT`) — `make verify-steam-templates` refuses a stock Godot binary until these exist. `STEAM.md`. +- [ ] **Reference-hardware profiling (task 0.15b)** in the live editor on real low/mid-tier hardware — blocks 0.16, 0.17/0.17b/0.17c/0.17d, 0.26 (arena GI bake), and 0.28 (physics separate-thread prototype). Covered above; listed again here because it also gates Phase 5.5's graphics QA gate for multiplayer sign-off. +- [ ] **Stand up the live Kubernetes cluster and Agones deployment** for Phase 8 — provider-portable manifests exist, but nothing has run against a real cluster; needs the provider-specific deployment overlay (network, DNS, secrets) per `docs/MATCHMAKING.md`. +- [ ] **Give Phase 8.48 its own Compose smoke fixture** so the allocated-mode flow stops depending on `compose.phase6-smoke.yml`'s hardcoded port, first-come slots, and `--max-matches=2`. `multiplayer-next.md` §0 task table. +- [ ] **Release-evidence and human sign-off gates for Phase 8 production launch** — once the above are done, someone needs to actually run and sign off the production-shaped checks `multiplayer-next.md` §7 lists as infrastructure/production-dependent. + +Defect **C** (slot reservation keyed on display name alone — real, demonstrated, exploitable during the 30 s disconnect window) is not its own action item: it is fixed for free by the Steam auth tickets in task 7.4 above, so nothing to do until Steam identity lands. diff --git a/docs/TECH_STACK.md b/docs/TECH_STACK.md index 6cf3c885..2d867b26 100644 --- a/docs/TECH_STACK.md +++ b/docs/TECH_STACK.md @@ -56,7 +56,7 @@ snapshot/restore API. That fact is why the multiplayer architecture is server-authoritative with client-side prediction of only the local ship, rather than rollback/resimulation netcode — rollback would require deterministic replay, which no physics engine choice here provides -(`multiplayer-next.md` §1, decision 1). +(`MULTIPLAYER_SPEC.md` §1, decision 1). ## Multiplayer transport: Godot's built-in `MultiplayerAPI` over ENet @@ -71,7 +71,7 @@ Design choices layered on top of the built-in peer, and why: - **`ENetMultiplayerPeer.server_relay` is forced to `false`.** It defaults to `true`, which lets any client `rpc()` any other client *through the server* — incompatible with a server-authoritative model. Called out in - `multiplayer-next.md` §2.1 as "the single highest-value one-line security + `MULTIPLAYER_SPEC.md` §2.1 as "the single highest-value one-line security change in the document." - **Manual multiplayer polling**, not Godot's automatic idle-frame poll. `NetworkManager` calls `set_multiplayer_poll_enabled(false)` because the @@ -84,7 +84,7 @@ Design choices layered on top of the built-in peer, and why: - **A custom binary wire format** (`net_codec.gd`) rather than raw RPC argument marshalling, for compact, quantised input/snapshot packets sent at high frequency — no stated alternative was considered in the docs, but - the packet-size/channel-intent design in `multiplayer-next.md` §2 is + the packet-size/channel-intent design in `MULTIPLAYER_SPEC.md` §2 is extensive and deliberate. ## Optional multiplayer transport: Steam (GodotSteam) @@ -96,14 +96,14 @@ Relay), from a custom GodotSteam-patched Godot build (not stock Godot — use ENet only, and a build without the `steam` feature is fully functional without it. -**Why it's optional and why raw ENet remains primary:** `multiplayer-next.md` -states plainly that "Docker/VPS is the primary v1 deployment path. Raw ENet -self-hosting needs port forwarding, and SDR is Phase 7 — so [the ENet -phases] ship something that works on LAN or a VPS and nowhere else." Steam/SDR -is being added later specifically to remove the port-forwarding requirement -and to supply verified player identity — direct-IP ENet's slot-reclaim logic -is keyed by display name today, which is insecure against a public server -(see `multiplayer-next.md`). +**Why it's optional and why raw ENet remains primary:** per +`multiplayer-next.md`, Docker/VPS is the primary v1 deployment path, and raw +ENet self-hosting needs port forwarding while SDR is Phase 7 — so the ENet +phases ship something that works on LAN or a VPS today, and nowhere else +yet. Steam/SDR is being added later specifically to remove the +port-forwarding requirement and to supply verified player identity — +direct-IP ENet's slot-reclaim logic is keyed by display name today, which is +insecure against a public server (`multiplayer-next.md` §0, known defect C). ## Dedicated server hosting: Docker (primary) or native systemd diff --git a/multiplayer-next.md b/multiplayer-next.md index d5df5941..80dfab14 100644 --- a/multiplayer-next.md +++ b/multiplayer-next.md @@ -1,826 +1,105 @@ -# Online multiplayer — architecture and task breakdown +# Online multiplayer — task breakdown -The single tracking document for the online multiplayer effort: architecture -decisions, the wire format, current progress, and a numbered task breakdown -with checkboxes, all in one place. `TODO.md` points here for anything -multiplayer-related. +The tracking document for the online multiplayer effort's outstanding work: +what's left, why, and the task breakdown. `TODO.md` points here for anything +multiplayer-related. The architecture decisions, wire format, input +handling, prediction, latency budget, and match lifecycle spec this work +assumes now live in **[`MULTIPLAYER_SPEC.md`](MULTIPLAYER_SPEC.md)** as its +own sections 1–6 — read those before picking up work in Phase 2 or later. **How to use this doc:** start at §0 for what's outstanding right now. Pick up a single numbered task, do it, verify it against its stated acceptance -criterion, mark it `[x]` **DONE**, and stop. Sections 1–6 are the decisions -those tasks assume; read them before picking up work in Phase 2 or later. §9 -is a running gotchas list — check it before debugging something that looks -like a Godot/Jolt engine quirk, and add to it when you find a new one. +criterion, and stop. §9 is a running gotchas list — check it before debugging +something that looks like a Godot/Jolt engine quirk, and add to it when you +find a new one. -**Status: every task in Phases 0–6 is implemented and verified locally.** Both multiplayer milestones still need human verification — a playtest at ~100 ms RTT (Phase 4) and a real 3v3 session (Phase 5). Phase 6's public-internet gate is no longer blocked for allocated reconnects, which use signed identity; direct/community reservations retain the documented display-name limitation. The export, Docker, rotation/drain, and CI work remain complete. Phase 7's Steam foundation is in progress. **Phase 8 — matchmaking, ranked, and per-match server autoscaling — remains a 1.0 launch blocker and is partially implemented:** the Go API/domain/store/Redis/allocator paths, migrations, supervisor, authenticated Kubernetes/Agones adapter, hardened Fleet baseline, testkit, and offline end-to-end path are in place. Production Steam identity/SDR, live cluster/public-network execution, release evidence, and human gates remain. **A real deployment cannot complete a match end to end today**: the allocator never actually publishes a player's signed assignment in production (see §0's root-blocker callout, §8.31), so no match can advance past `PROCESS_READY` — this is flagged, not yet fixed. It is the first phase to add a component outside the Godot project, and its design lives in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). The client has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. See the outstanding list immediately below for what is left and why, §7 for the implemented work and its evidence, and §11 for what is deliberately deferred. +**This revision keeps only outstanding work.** Phases 0–6 are fully +implemented and verified locally; their task-by-task implementation evidence +has been trimmed from this document and lives in git history +(`git log -- multiplayer-next.md`) rather than here. Phase 7 (Steam) and +Phase 8 (matchmaking) are in progress — the tables below list only what +remains on each task, not what's already built. **Phase 8 is a 1.0 launch +blocker**, adds a component outside the Godot project (a Go backend +service), and has a critical open blocker: see §0. --- ## 0. Outstanding work — the short list -The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 0–5 contain no unfinished tasks. +The one place to look before planning. Everything here is also written up +where it belongs; this is the index, not the detail. -**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is in progress.** It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.53 are in §7; the design is in [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). Three findings would break a naive implementation: +**Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch +blocker and is in progress.** It is larger than anything below and adds a +backend service outside the Godot project. Tasks are in §7; the design is in +[`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). -| # | Finding | Why it bites | -|---|---|---| -| Task 8.28 | ~~Godot's stdout is block-buffered off a TTY — a detached container logs *nothing*, so `server_started` never appears~~ **Fixed**: `deploy/cosmic-clash-server` now wraps the exec in `stdbuf -oL -eL`. Verified live — a real `docker run -d` container showed zero log output for 20+ seconds, including the startup line, and `docker stop`'s SIGTERM lost it permanently rather than delaying it (Godot has no SIGTERM hook); the wrapped launcher shows the startup line within 3s of the same scenario. This affected the already-shipped community server (Docker *and* native systemd both route through this script), not only the not-yet-built Agones path | Process-ready must be an explicit Agones call after static validation/listen, independent of this fix — the API/registration boundary never depended on log output either way, so this was a real operational bug (silent `docker logs`/`journalctl`), not a correctness gap in the process-ready design | -| Task 8.29 | ~~`--port` defaults to 7777 and the Dockerfile hardcodes `EXPOSE 7777/udp`~~ **Fixed locally**: the allocated supervisor replaces the child port with the Agones-assigned endpoint and exports SDR variables only for Hosted-SDR | Live Agones passthrough/NAT and multi-match validation remain infrastructure gates | -| Task 8.48 | `compose.phase6-smoke.yml` hardcodes the port, first-come slots and `--max-matches=2` | The allocated flow needs its own fixture so Phase 6 behavior and invocations stay unchanged | - -**The actual current root blocker (found 2026-09-04, not yet fixed)**: nothing in production ever publishes a player's signed match assignment. `store.SaveAssignment`/`SaveAssignments`/`SaveVerifiedAssignmentRoster` and `allocator.Service.PublishRoster` are fully built and tested in isolation, but no real code path (`allocator/worker.go`, `cmd/allocator`) ever calls them — only tests do, by seeding the table directly rather than exercising the real write path. Since `AdvanceServerRegistration`'s SQL requires an `assignments` row per participant before a match can reach `ASSIGNMENT_READY`, **a real deployment cannot advance any match past `PROCESS_READY`** — no player can ever receive a real assignment or connect, regardless of how correct every other piece (including §8.41's client-side connect-wiring fix) is. See §8.31 for the full detail. Closing it needs new security-relevant design (a join-signing key shared between allocator and game server, roster-digest computation, per-player authorisation construction) — flagged rather than built, at the user's explicit direction, pending a decision on that design. +**The current root blocker**: nothing in production ever publishes a +player's signed match assignment. `store.SaveAssignment`/`SaveAssignments`/ +`SaveVerifiedAssignmentRoster` and `allocator.Service.PublishRoster` are +fully built and tested in isolation, but no real code path +(`allocator/worker.go`, `cmd/allocator`) ever calls them — only tests do, by +seeding the table directly rather than exercising the real write path. Since +`AdvanceServerRegistration`'s SQL requires an `assignments` row per +participant before a match can reach `ASSIGNMENT_READY`, **a real deployment +cannot advance any match past `PROCESS_READY`** — no player can ever receive +a real assignment or connect, regardless of how correct every other piece +(including the client-side connect-wiring in task 8.41) is. See task 8.31 +for the full detail. Closing it needs new security-relevant design (a +join-signing key shared between allocator and game server, roster-digest +computation, per-player authorisation construction) — flagged rather than +built, at the user's explicit direction, pending a decision on that design. ### Blocking sign-off — the work exists, the verification does not | # | What | Why it is not done | Detail | |---|---|---|---| -| A | **Phase 4 human playtest at ~100 ms RTT.** Does the ship feel local? Does the ball? Do contact corrections read as bumps or as glitches? | Needs hands on a controller. Every numeric gate is green; feel is the milestone's actual subject and no percentile can answer it. | Phase 4 gate | -| B | **Phase 5 3v3 gate**: a full start-to-finish match with 6 players, a mid-match disconnect, and a late joiner. | Needs a real multi-client session. Every scenario is verified at 1v1 plus a two-bot CI match; nothing has run at 3v3. | Phase 5 gate | +| A | **Phase 4 human playtest at ~100 ms RTT.** Does the ship feel local? Does the ball? Do contact corrections read as bumps or as glitches? | Needs hands on a controller. Every numeric gate is green; feel is the milestone's actual subject and no percentile can answer it. | §5.7 | +| B | **Phase 5 3v3 gate**: a full start-to-finish match with 6 players, a mid-match disconnect, and a late joiner. | Needs a real multi-client session. Every scenario is verified at 1v1 plus a two-bot CI match; nothing has run at 3v3. | §6 | -These two are independent and can be done in either order, but B is the cheaper of the two to arrange and would also exercise A's conditions incidentally. +These two are independent and can be done in either order, but B is the +cheaper of the two to arrange and would also exercise A's conditions +incidentally. ### Known defects | # | What | Severity | Detail | |---|---|---|---| -| C | **Slot reservation and takeover are keyed on display name alone.** Any peer connecting with a departed player's name inside the 30 s window claims their slot, ship and team. | Real, demonstrated. Bounded by needing a genuine disconnect to race. | §11 | -| D | **Input is still lost at the transport layer during a long server stall**, variably — 7 of 8 runs measured 0.00 % of the sequence stream missing, the eighth 23.54 %. | Low. Distinct from the rate-limiter cause, which is fixed. The seq-guard resync visibly recovers it. | Phase 5 notes | -| E | **A second `Unable to send packet on channel N` stderr race**, in `_broadcast_snapshot` rather than the fixed site in `_remove_player`. | **Fixed.** Server-side abuse disconnects invalidate the peer before closing it, and snapshot sends re-check that invalidation at the transport boundary. | §11 | +| C | **Slot reservation and takeover are keyed on display name alone, for direct/unauthenticated servers only.** For allocated (signed-roster) matches this is resolved — reconnect reclaim and late-join promotion carry the verified `PlayerID` across peer-id changes. Direct/community servers with no Steam identity still resolve reclaim by display name; a peer connecting with a departed player's name inside the 30 s window claims their slot. | Real, demonstrated, bounded to the unauthenticated direct-server path. | §11, Phase 7 task 7.4 | +| D | **Input is still lost at the transport layer during a long server stall**, variably — 7 of 8 runs measured 0.00 % of the sequence stream missing, the eighth 23.54 %. | Low. Distinct from the rate-limiter cause, which is fixed. The seq-guard resync visibly recovers it. | §9 gotchas 39, 48, 49 | -C is the one to plan around: it is fixed for free by task **7.4** (Steam auth tickets in `hello`), which is why it has not been given a bespoke solution. Anything that ships to strangers before Phase 7 needs it addressed first. +The residual half of C (direct/community servers) is fixed for free by task +**7.4** (Steam auth tickets in `hello`) once Phase 7 lands; it has not been +given a bespoke solution for that reason. ### Open architectural question | # | What | Detail | |---|---|---| -| F | **A contact-cohort-only shadow world.** The remaining known prediction weakness is the contact cohort. Whether it is worth a client-side shadow Jolt world scoped to contacts alone is undecided — and deliberately so until A supplies the felt evidence. | Phase 4 notes | +| F | **A contact-cohort-only shadow world.** The remaining known prediction weakness is the contact cohort. Whether it is worth a client-side shadow Jolt world scoped to contacts alone is undecided — and deliberately so until A supplies the felt evidence. | §5.7 | ### Unstarted phases -- **Phase 6 external gate:** run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until **C** is fixed. -- **Phase 7 — Steam transport, browser, identity and production SDR** (8 tasks): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, verified tickets, bans, production credentials and ticketed Hosted Dedicated Server SDR await a project-owned Steamworks App ID and Valve coordination. Carries the fix for **C** and is the hard prerequisite for Phase 8. +- **Phase 6 external gate:** run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until **C** is fully closed (i.e. Phase 7 lands). +- **Phase 7 — Steam transport, browser, identity and production SDR** (8 tasks, in progress): the optional bootstrap and `NetTransport` foundation now exist, but custom Steam client/server export templates have not yet been supplied. Browser, verified tickets, bans, production credentials and ticketed Hosted Dedicated Server SDR await a project-owned Steamworks App ID and Valve coordination. Carries the fix for the direct-server half of **C** and is the hard prerequisite for Phase 8's production Steam identity. -Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into something another person can host. Phase 7 is the next block because Steam identity is required before public exposure. +Phase 6's external gate has no dependency on Phase 7 for a controlled test, +but Phase 7 is next in priority because Steam identity is required before +public exposure. ### Deferred by choice, not forgotten -120 Hz simulation, the latency-gap *measurement* (task 4.9's acceptance criterion), authored audio, split-screen — all in §11 with what each would buy and cost. The procedural audio hooks are implemented; authored assets and production mixing remain open in `TODO.md`. +120 Hz simulation, the latency-gap *measurement* (§5.7's acceptance +criterion), authored audio, split-screen — all in §11 with what each would +buy and cost. The procedural audio hooks are implemented; authored assets +and production mixing remain open in `TODO.md`. --- -## 1. Architecture decisions - -### 1.1 Locked decisions - -| # | Decision | Why | -|---|---|---| -| 1 | **Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation.** | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. | -| 2 | **Dedicated servers only.** Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. | -| 3 | **ENet first**, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and **must never become the degraded path**. | -| 4 | **Community discovery uses no custom backend; superseded for queued play by Phase 8.** | Steam's server APIs remain enough for the community browser. Casual/ranked queues, durable ratings, allocation and authoritative results require the project-owned Go control plane specified in `docs/MATCHMAKING.md`; it does not replace the browser or direct-IP path. | - -### 1.2 Rejected alternatives - -- **Peer-authoritative ships** (each client owns its own transform). Easiest to build, feels perfect locally, and is trivially cheatable — it directly contradicts `README.md`'s stated anti-cheat position. Ship-vs-ship collisions also become ambiguous with no arbiter. -- **Deterministic lockstep / rollback.** See decision 1. -- **`MultiplayerSynchronizer` / `MultiplayerSpawner`.** The decisive objection is not bandwidth. It is that `last_processed_input_seq` **must** arrive in the same packet as the state it describes, or reconciliation is off by a snapshot — and a synchroniser gives you nowhere to put it. It also writes replicated properties directly onto the node, which is exactly wrong for a `RigidBody3D` under prediction: incoming state has to enter a compare-against-history pipeline, not be stamped onto `global_transform`. You would end up building the correction pipeline anyway, with the synchroniser as pure overhead. Secondary objections: one packet per body (7 bodies × ~50 B of UDP/IP/ENet framing versus one coalesced snapshot), no per-field quantisation, and no client-side interpolation. - - `MultiplayerSpawner` is unnecessary for a separate reason: the roster is fixed at match start and fully described by the `match_config` message, and **no ship is ever despawned** (§6.4). -- **Seeded RNG for kickoff jitter.** Shared-seed determinism requires both sides to consume the RNG stream in exactly the same order forever. The first `randf()` anyone later adds anywhere in the reset path — a spawn VFX variation, a cosmetic, a commentary line — silently desyncs kickoff positions with no error message. The server broadcasts the resulting transforms instead: 336 bytes, once per kickoff, cannot rot. - -### 1.3 Derived decisions - -**All hot-path RPCs live on autoloads.** `/root/NetworkManager` and `/root/MatchNet` exist at identical paths on every peer regardless of which scene is loaded, which side is headless, or whether a client is mid-scene-transition. This deletes the entire "NodePaths must match across peers" class of bugs, kills a family of late-join races where an RPC arrives before its target node exists, and warms Godot's RPC path cache once at connect so it never re-sends a full path on scene change. - -**Entities are addressed by integer slot, never by path.** The snapshot is `[slot 0..N-1]` in a fixed order established by `match_config`. `MatchNet` holds an `Array[Node] _slots` populated at spawn. - -**One server process hosts exactly one match.** This is forced, not chosen: `ship.gd:162` resolves the arena boundary via `get_tree().get_first_node_in_group("arena_boundary")` and `ai_ship_controller.gd` discovers its roster via `get_tree().get_nodes_in_group("ship")`. Both are tree-global, so two matches in one scene tree would cross-wire instantly. It is recorded here because it determines the RAM figure in §1.4. - -### 1.4 Server sizing — bandwidth and CPU are not the constraint - -Worth establishing up front, because §2 and §3 repeatedly trade bandwidth for latency and somebody will eventually want to trade back. - -`ArenaBoundary.bake_colliders()` generates roughly 168 box colliders (corner fillets, base wrap, ceiling, end walls) plus the scene's own slabs, 2 goal backstops, 2 `Area3D` sensors, and 7 dynamic bodies (the ball with `continuous_cd`). Estimated per-tick cost: - -| Component | ms/tick | -|---|---:| -| Jolt step | 0.15 – 0.4 | -| Godot headless main loop | 0.1 – 0.3 | -| Bot inference, amortised (see task 0.8) | ~0.3 | -| **Total, of a 16.7 ms budget** | **0.6 – 1.1** | - -→ **~6–10 concurrent matches per modern core**, ~150–250 MB RSS per process. 100 concurrent matches ≈ 12–16 cores and ~20 GB — a single mid-tier VPS. Upstream bandwidth for a full 6-player match is ~630 kbit/s (§2.4). - -**Neither CPU nor bandwidth is scarce. Latency is.** Optimise accordingly. - ---- - -## 2. Wire format - -Two peers must agree byte-for-byte, so this is specified rather than sketched. - -### 2.1 Channels - -| Channel | Transfer mode | Contents | -|---|---|---| -| 0 | reliable | handshake, `match_config`, kickoff, goal, clock, state changes, chat, admin | -| 1 | unreliable-ordered | client → server input | -| 2 | unreliable-ordered | server → client snapshots | - -Unreliable-**ordered** (ENet sequenced-unreliable, drops stale) rather than plain unreliable for both hot paths: we carry explicit sequence numbers, and a reordered late packet is worthless work. Separating them stops a large reliable `match_config` from head-of-line-blocking state on a lossy link. - -> **Verify at implementation time.** Godot's `ENetMultiplayerPeer` reserves low ENet channels for its own system messages and offsets `transfer_channel` on top. The intent above is "three logically distinct channels"; the concrete indices may need an offset. Confirm empirically, don't assume. - -**Set `ENetMultiplayerPeer.server_relay = false`.** It defaults to `true`, which lets any client `rpc()` any other client *through your server*. With it off, clients can only talk to peer 1. Single highest-value one-line security change in this document. - -### 2.2 Packet header - -**Every hot-path packet opens with a 1-byte type + version.** A capture then decodes standalone, and a mismatched build fails loudly instead of decoding garbage straight into `state.transform`. - -Hot paths carry a single `PackedByteArray` RPC argument (≈14 B of Godot RPC framing once the path cache is warm). Control messages on channel 0 use normal typed arguments — they are rare and readability beats bytes. - -### 2.3 Input packet — client → server, channel 1, 60 Hz - -``` -u8 type_version -u32 seq server-tick-space sequence of the NEWEST action -u8 count 1..4 (MAX_REDUNDANCY) -u32 ack_snapshot_tick newest snapshot tick this client has processed -u16 client_send_ms wrapping ms clock, echoed back for RTT ---- repeated `count` times, newest first --- -i8 thrust_x, thrust_y, thrust_z value = clamp(round(v*127), -127, 127) -i8 rot_x, rot_y, rot_z -u8 flags bit0 = turbo -``` - -**12 + 7×4 = 40 B payload**, ~90 B on the wire with UDP/IP/ENet framing → **~43 kbit/s up per client**. - -- **Redundancy 4** is what makes an unreliable input channel safe: starvation requires four consecutive losses (~66 ms). -- **`i8` per axis, not 3-bit bins.** Bins matching `ShipActionCodec.HEADS` would cut an action to 3 bytes, but they permanently foreclose analog gamepad sticks, which this game will want. `round(v*127)/127` round-trips `-1/0/+1` exactly, so today's digital input (`player_ship_controller.gd` is `is_action_pressed`-only) is lossless. -- **The encoding is itself a validator.** `i8/127` cannot express NaN, Inf, or a value outside `[-1.008, 1.008]`. Half of "sanitise untrusted client input" is solved by not using Variant encoding. - -### 2.4 Snapshot — server → client, channel 2, 60 Hz default - -Per-client header built per peer; body buffer built once per tick and reused across peers. - -``` ---- per-client header (7 B) --- -u32 last_input_seq newest input from THIS client the server has applied -i8 input_buffer_depth jitter-buffer occupancy; negative = starved -u16 echo_client_send_ms from that input packet, for RTT - ---- shared body header (8 B) --- -u8 type_version -u32 server_tick Engine.get_physics_frames() on the server -u8 match_state see §6.1 -u8 reset_gen increments on every authoritative teleport -u8 body_count - ---- repeated body_count times, slot order fixed by match_config (22 B each) --- -i16 pos_x, pos_y, pos_z range ±64 m -> 1.95 mm -i16 quat_x, quat_y, quat_z w = ±sqrt(1-x²-y²-z²), sign in flags -i16 vel_x, vel_y, vel_z range ±64 m/s -> 1.95 mm/s -i8 avel_x, avel_y, avel_z ships ±4 rad/s; ball ±32 rad/s -u8 flags bit0 frozen, bit1 turbo, bits2-4 thrust_z bin, - bit5 stalled, bit6 quat_w sign -``` - -7 bodies → **8 + 7 + 7×22 = 169 B payload**, ~219 B on the wire. - -| | per client down | server up, 6 clients | + 10 spectators | -|---|---:|---:|---:| -| 60 Hz | 105 kbit/s | 631 kbit/s | 1.68 Mbit/s | - -MTU headroom is ~6× (ENet fragments above ~1400 B); a hypothetical 10v10 at 21 bodies is 477 B and still fits. **This format does not need delta compression.** - -**Plain `i16` quaternion components, not smallest-three.** Smallest-three saves 4 B/body and is the textbook answer. It is also exactly where a hand-rolled codec goes subtly wrong — off-by-one in the 2-bit index, sign of the dropped component, renormalisation drift — in a project that has no test framework yet. Three `i16`s plus a sign bit give ~3e-5 rad with no bit-shifting, for 2 B/body (≈3 kbit/s). Take the bytes. - -**Quantisation ranges derive from constants, not from prose.** `ArenaBoundary.INNER_HALF_X = 18.0`, `INNER_HALF_Z = 27.0`, `INNER_HEIGHT = 18.0` (`arena_boundary.gd:8-10`) plus `GameMode.ESCAPE_MARGIN = 15.0`; `Ship.max_speed = 35.0` (`ship.gd:16`); `Ball.MAX_SPEED = 32.0` (`ball.gd:17`). - -> `CLAUDE.md`'s Architecture section states the play volume as "inner x ±12, z ±18, height 12, goal lines z ±17". **That is stale** — see the real constants above. Task 0.13 fixes the doc. - -**The flags byte must carry `turbo` and a 3-bit `thrust_z` bin.** `_integrate_forces` is not called on frozen bodies, so remote ships on a client never pull `get_action()`, and `Ship._update_movement_vfx()` (`ship.gd:293`) reads `_current_action.thrust.z` and `turbo`. Without those bits, every remote ship flies with dead engines. - -### 2.5 Reliable control messages, channel 0 - -`hello` · `welcome` · `player_joined` · `player_left` · `ready_state` · `match_config` · `scene_ready` · `kickoff` · `state_change` · `goal_scored` · `clock_state` · `match_ended` · `chat` · `server_shutdown`. - ---- - -## 3. Server-side input handling - -Per-player server state: - -```gdscript -class PlayerSlot: - var peer_id: int - var slot: int # snapshot index - var ring: Array[ShipAction] # FIXED 32 entries, indexed seq % 32 - var ring_seq: PackedInt32Array # 32 entries, seq stored at each index (-1 = empty) - var last_applied_seq: int - var last_action: ShipAction - var starved_ticks: int - var packets_this_second: int - var remote_controller: RLShipController # see §7 task 5.7 — null on takeover -``` - -### 3.1 Ingestion - -`@rpc("any_peer", "unreliable_ordered", channel = 1)`, in order: - -1. `multiplayer.get_remote_sender_id()` → look up slot. Unknown sender → drop and count. -2. **Rate limit.** `packets_this_second > 110` (60 Hz × 1.5 + 20) → drop. Three consecutive seconds over budget → disconnect with `RATE_LIMIT`. Same for a byte budget. -3. **Framing.** `count > 4` or `payload_size != 12 + count*7` → drop, count malformed. 20 malformed → disconnect. -4. **Sequence range.** `seq > server_tick + 20` → drop. (Not 120: `input_lead` is clamped to 12, so anything above ~20 is broken or hostile.) This is why the ring is fixed-size and indexed `seq % 32` — **a client can never make the server allocate.** -5. For each action, newest first at descending seq: `seq <= last_applied_seq` → discard (already consumed); else write `ring[seq % 32]`. -6. **Decode with per-axis clamp only:** - ```gdscript - action.thrust = Vector3(b[0]/127.0, b[1]/127.0, b[2]/127.0).clampf(-1.0, 1.0) - ``` - -> **Never normalise the thrust vector.** A player holding W+A+E legitimately produces `thrust = (1,1,1)`, length 1.73, and each axis uses a different power constant — `thrust_power 150`, `maneuvering_thrust 75`, `vertical_thrust 120` (`ship.gd:12-14`). Normalising would silently change the flight model for honest players. Per-axis clamp combined with the `i8` encoding is complete validation: the reachable value space is exactly what a legitimate client can produce. - -### 3.2 Consumption — once per server physics tick, before the step - -``` -expected = last_applied_seq + 1 -if ring holds expected: - action = ring[expected % 32]; starved_ticks = 0 -else: - action = last_action # REPEAT — do not zero - starved_ticks += 1 - if starved_ticks > 30: # 500 ms - action = ZERO_ACTION; flags.stalled = true -last_applied_seq = expected -last_action = action -remote_controller.action = action -``` - -**Repeat-last, not zero.** Player inputs are heavily autocorrelated at 60 Hz — the odds that a held thrust was released on exactly the dropped tick are low, and the client predicted with the real input either way, so repeating minimises expected divergence. It is also consistent with `AIShipController`, which already holds its action between decisions. Zeroing after 500 ms stops a disconnecting player's ship flying into a wall at full throttle forever. - -### 3.3 Jitter buffer — one control loop, not three - -An earlier draft had the server adapting `target_depth`, the server fast-forward-dropping queued actions, **and** the client slewing `input_lead`. Three integrators acting on one plant (buffer occupancy) with different time constants is a textbook oscillation; on a jittery link it hunts, and it presents to the player as intermittent sticky controls that are nearly impossible to attribute. - -**The server reports `input_buffer_depth` in every snapshot and does nothing else adaptive. The client owns `input_lead` exclusively.** - -- `target_depth = 1` (16.7 ms), not 2. With redundancy-4 you have already bought the insurance depth 2 provides; depth 2 is 16.7 ms of pure input latency for nothing. -- Client `input_lead` clamp `[1, 12]`, **fast attack / slow release**: on any starve, increase by up to 3 **immediately**; decrease by 1 per 60 ticks only after 2 s of clean surplus. A symmetric ±1-per-500 ms slew takes two seconds to absorb a wifi spike, during which the player steers and the ship does not turn — the most rage-inducing failure mode in any netcode. -- Changing `input_lead` means skipping or duplicating one tick's sequence number. Never change it more than once per 30 ticks. - -**Enforce `input_lead` server-side from observed arrival times.** A client that fakes starvation to drive `input_lead` to 1 gets its inputs applied with less server-side buffering than honest players — a small but real responsiveness edge. The `i8` encoding does nothing about this; only observing actual arrival timing does. - ---- - -## 4. Prediction and reconciliation - -### 4.1 Two clocks for remote entities — the load-bearing correction - -The obvious design runs remote ships and the ball as frozen kinematic proxies at `server_time_est - INTERP_DELAY` while predicting the local ship to *now*. **That is wrong**, and it is wrong in a way that only shows up over real latency: - -- Two ships closing at 50 m/s put the opponent's collider **3.5 m** from truth. The hull is a `BoxShape3D` of `(1.6, 0.6, 4)` (`ship.tscn:12`) — that is most of a ship length of positional lie. -- A fast ball is **2.2 m** off against a 0.5 m radius — four ball diameters. -- `ship.tscn:16` has `collision_mask = 7`: ships collide with ships, the ball, and the arena. Ship-vs-ship contact is *constant* in vehicle soccer, not incidental. - -So prediction would not diverge occasionally due to timing noise. It would diverge **deterministically and in the same direction on essentially every contact**, and the hard-snap threshold would become the steady state rather than a backstop. - -**Fix: separate the collider clock from the render clock.** - -| | runs at | why | -|---|---|---| -| remote body **collider** | `server_time_est`, extrapolated forward from the newest snapshot by ~one-way + half a snapshot interval | Extrapolation error over ~45 ms at real accelerations (`thrust_power 150 / mass 5` = 30 m/s², 75 m/s² on turbo — `ship.gd:12,15`, `ship.tscn:17`) is ~0.03–0.08 m. Two orders of magnitude better than 3.5 m. | -| remote **`$Visual`** | `server_time_est - INTERP_DELAY` | Smooth, jitter-free rendering. | - -This is the same trick applied to the local ship, pointed the other way. It costs one extra transform write per remote body per tick. - -### 4.2 Where each piece lives - -| Concern | Location | -|---|---| -| sample + send input | `LocalNetShipController._physics_process` — runs before the physics step, guarantees exactly one sample/tick | -| record predicted state | same, at top of tick N (state = result of N−1) | -| apply velocity / teleport correction | `Ship._integrate_forces`, ~15 guarded lines — the only Jolt-safe place to write `state.transform` / `state.linear_velocity` | -| visual smoothing | `Ship/$Visual.global_transform`, set in `_physics_process` | -| snap-vs-blend decision | `net_ship_predictor.gd` (child node) | -| remote bodies | `net_interpolator.gd` | - -### 4.3 Per-tick, own ship - -1. `predicted[current_tick - 1] = {transform, linear_velocity, angular_velocity}` — ring of 128. -2. `var a := _player.get_action().copy()` — **must copy.** `player_ship_controller.gd` reuses a single `ShipAction` across ticks (its own header warns about this); buffering it aliases every history entry to the same object. See task 0.1. -3. `_action = a`, returned by `get_action()` this tick so `Ship._integrate_forces` samples input exactly once. -4. `input_history[seq] = a`, `seq = predicted_server_tick + input_lead`. -5. Build and send the packet with the last 4 entries. - -`Ship._integrate_forces` then runs completely unchanged. - -### 4.4 On snapshot arrival - -``` -A = last_input_seq -if reset_gen changed OR predicted[A] missing OR flags.frozen != local frozen: - HARD SNAP -else if pos_err > 2.0 m OR rot_err > 60°: - HARD SNAP -else: - SOFT CORRECT -``` - -Comparing server state at tick `A` against **`predicted[A]`** — the client's own state at that same tick — makes the delta latency-free by construction. That is the entire reason for keeping the prediction ring, and it is why this works acceptably without resimulation: **never blend current state toward stale state.** - -**SOFT CORRECT** - -- **Velocity: applied in full, immediately.** `net_vel_correction += (srv.linvel - predicted[A].linvel)`, consumed once in `_integrate_forces`. Velocity error is invisible to the player but is the *cause* of future position error; blending it just prolongs divergence. -- **Position/rotation: physics moves in full, rendering does not.** Queue the body teleport, and simultaneously offset `$Visual` by the negation. Net visual movement at the instant of correction: zero. The body is where the server says; the rendered ship catches up. -- **Decay** each physics tick, reusing the existing convention at `ship.gd:450`: - ```gdscript - var k := _tick_scaled(0.88, delta) # 63% gone in ~130 ms, 95% in ~280 ms - ``` -- **`MAX_VISUAL_OFFSET = 0.4 m`**, not 2.0. The hull is 4 m long; a 2 m offset means being rendered half a ship-length from your own collider for ~280 ms, so you clip walls you visibly cleared — a felt bug in a game built around wall-riding. Beyond 0.4 m, show the correction. A visible correction is honest; an invisible 2 m lie is not. - -**HARD CORRECT** - -- Apply the same sequence-matched authoritative pose and velocity delta to the current local body, reset body and `$Visual` interpolation, and clear the visual offset. It is physically the same correction as soft correction; only its presentation differs. - -**Settled Phase 4 decision — delta transport, not one-body replay.** For every matched snapshot, overwrite `predicted[A]` with authority, transport its pose and linear/angular-velocity delta through each retained state `A+1..current`, and apply that same delta once to the live local Jolt body. This keeps retained history coherent, so a later snapshot does not correct an already-corrected pre-delta trajectory a second time. - -Do **not** analytically replay stored actions. That approximation cannot reproduce Jolt integration or contact manifolds (friction, restitution, walls, ships, and ball), therefore it becomes least trustworthy exactly where reconciliation is most noticeable. This is still neither whole-world rollback nor a change to server physics: it is client-only state transport around a server-authoritative simulation. - -For reset generation changes, place exact authority, begin a new history epoch, and do not consume pre-reset actions. For missing or overflowed history, place authority once and suppress stale acknowledgements until a new matched sequence is recorded; never manufacture future history by filling it with one stale authority state. - -> Same-sequence **pre-correction** residual remains diagnostic telemetry. With a server input jitter buffer, it is not by itself a presentation-quality gate: the server may have integrated an action at a different physical instant from the client. Acceptance must report it separately by free-flight/contact/reset/resync cohort, while gating post-correction/presentation error and hard-snap behaviour. -> -> That the two sides integrate the **same action** for a given sequence is a separate claim, and a checkable one — it is what the action marker and task 4.11's `--exercise-input-transitions` gate exist for. Keep the two apart: "right action, different instant" is expected here; "wrong action" is a bug, and was one. - -### 4.5 Camera and visuals - -**The camera must follow `$Visual`, not the body.** `ship_camera.gd:115`, `:149`, `:150` read `target.global_transform` directly. Left as-is, every soft correct makes the *camera* jump the full error while the *mesh* smoothly lags — strictly worse than snapping, because the world lurches around a player whose ship slides inside the frame. - -**And it must read `$Visual.get_global_transform_interpolated()` from `_process`, not `global_transform` from `_physics_process`** (task 0.16, rationale in §5.4). `Node3D.get_global_transform_interpolated()` exists precisely for a camera tracking a physics-interpolated body; `global_transform` returns the last physics tick's pose, so a `_process` camera reading it would chase a 60 Hz staircase at 240 fps. - -> **Ordering hazard**, straight from the engine docs: `get_global_transform_interpolated()` "creates an interpolation pump on the `Node3D` the first time it is called, which can respond to physics interpolation resets… be sure to call it at least once before resetting the `Node3D` physics interpolation." Every hard snap calls `reset_physics_interpolation()` on `$Visual`. **Prime the pump when the camera's `target` is assigned**, not lazily on the first frame, or the first snap of the match streaks the camera. - -`project.godot` has `physics_interpolation=true`, and `$Visual`'s own local transform is interpolated too — so `reset_physics_interpolation()` must be called on `$Visual` as well as the body, or every snap smears the mesh for a frame. (This is the same artefact `game_mode.gd:263` already exists to prevent.) - -### 4.6 Remote bodies on the client - -- `freeze = true`, `freeze_mode = FREEZE_MODE_KINEMATIC` — **not `STATIC`**, or Jolt cannot derive contact velocity from the per-tick transform delta and your predicted ship hits a static wall instead of a moving ship. -- `net_interpolator.gd` samples the snapshot buffer (last 8 per body); collider at `server_time_est` (§4.1), `$Visual` at `server_time_est - INTERP_DELAY`. -- **The two samples run on different clocks *and* different callbacks.** The collider is a physics concern: `_physics_process`, 60 Hz. `$Visual` is a render concern: `_process`, sampled at true render time with `physics_interpolation_mode = OFF` so Godot does not interpolate an already-per-frame transform. On a 240 Hz client this is 240 distinct remote-ship positions per second instead of 60, and one fewer tick of lag, for no extra cost — the buffer lerp is happening either way (§5.4). -- `INTERP_DELAY = one_way_ms + snapshot_interval * 1.5 + 2.5 * jitter_ewma`, clamped `[25, 200] ms`. At 60 ms RTT / 60 Hz / 5 ms jitter that is 30 + 25 + 12.5 ≈ **68 ms**. - -> **The `one_way_ms` term is not optional, and omitting it is a silent architectural failure.** `server_time_est` (§4.7) estimates what the server clock reads *right now*. The newest snapshot in hand was stamped `one_way` ago — §4.1 says exactly this when it extrapolates the collider forward "by ~one-way + half a snapshot interval". So rendering `$Visual` at `server_time_est - INTERP_DELAY` only interpolates if `INTERP_DELAY ≥ one_way`. Set it to the buffer alone (~38 ms at 60 Hz) and the render cursor lands *on or past* the newest sample: the bullet below about extrapolating past the newest snapshot becomes the steady state rather than the exception, and every remote entity is permanently dead-reckoned. **The 25 ms clamp floor is reachable on LAN only.** -- Past the newest snapshot, extrapolate on last known velocity for at most 150 ms, then hold. **Never extrapolate indefinitely** — a stuck ship reads better than one flying through a wall. -- **Never write `linear_velocity` to a frozen body.** Godot/Jolt zeroes and holds velocity on frozen bodies, so `ball.gd:35`'s `linear_velocity.length()` trail driver will not work that way. Add `Ball.set_visual_speed(speed)` mirroring the `Ship.set_visual_action(thrust_z, turbo)` pattern. Don't route presentation data through a property the physics server owns. -- Call `reset_physics_interpolation()` on remote bodies at every kickoff. - -### 4.7 Clock - -`server_time_est = local_ms + clock_offset`, `clock_offset` from ping/pong on channel 0 every 1 s using the **minimum-RTT sample in a rolling 5 s window** (the min-RTT sample has the least queueing error). - -**Freeze `tick_offset` at match start.** Seed it exactly from the handshake (`server_tick + round(one_way / tick_ms)`) and absorb all subsequent drift into `input_lead` alone. The prediction ring is indexed in server-tick space, so slewing `tick_offset` during play silently reinterprets every historical entry and produces sporadic, unreproducible false snaps. Re-seed only across a kickoff boundary. - ---- - -## 5. Latency and frame-rate budget - -Three of the largest terms are invisible to a netcode document that only counts network hops. Record the budget so future changes are argued against a number. - -Client at 60 Hz physics, 60 ms RTT, 5 ms jitter, 60 Hz snapshots. **Display at 60 Hz with vsync on** — the Godot default, and the worst case. §5.4 redoes the display-dependent rows for 120/144/165/240/360 Hz. - -### 5.1 Own ship (predicted) — input to pixel - -| Stage | ms | | scales with fps? | -|---|---:|---|---| -| OS input → `Input.is_action_pressed` | 10 | 0.5 × frame interval + device polling | partly — see below | -| wait for next physics tick | 8 | avg of 0–16.7 | **no — 60 Hz physics** | -| physics step applies force | 0 | | | -| Godot physics interpolation | 8 | `physics_interpolation=true`; mean, worst case 16.7 | **no — 60 Hz physics** | -| render + vsync present | 25 | 1.5 refresh intervals, vsync defaults on | yes | -| **Total** | **≈52** | | | - -This is the **existing single-player floor**, unchanged by netcode — and ~43 of those 52 ms are things no netcode document discusses. A low-latency present would take it to ~35 ms (§5.4). - -Two notes on the model, both corrected from an earlier draft that read ≈45: - -- **Input freshness is 0.5 of a frame interval, not 0.25.** Godot pumps OS input once per main-loop iteration and `Ship._integrate_forces` (`ship.gd:347`) consumes it once per physics tick; for arrivals distributed uniformly between pumps the mean staleness at the pump is half the interval. On top sits **device polling**, which does not scale with fps at all: ~1 ms at a 1000 Hz mouse or gamepad, ~8 ms at a 125 Hz USB device. The table assumes ~2 ms. -- **Physics interpolation's 8 ms is a mean.** Rendering happens between the two most recent completed ticks, so displayed pose lags the newest state by `(1 − fraction)` of a tick — 0 to 16.7 ms, averaging 8.3. The worst case matters for §5.4's discussion of frame-time variance. - -Note the right-hand column: **16 of the 52 ms do not move no matter how many frames the client draws.** That is the price of a 60 Hz simulation. - -### 5.2 World response — the number that decides whether this ships - -| Stage | ms | | -|---|---:|---| -| input freshness | 10 | 0.5 × frame interval + ~2 ms device polling | -| wait for next physics tick | 8 | | -| manual multiplayer flush | ~0 | **~8 with default idle-frame poll** — see §7 task 1.3 | -| client → server transit | 30 | RTT/2 | -| jitter buffer, `target_depth = 1` | 17 | | -| server tick + flush | 8 | | -| **server → client transit** | **30** | **RTT/2 — the return leg** | -| interpolation buffer beyond arrival | 38 | `interval × 1.5 + 2.5 × jitter`; the `one_way` half of `INTERP_DELAY` is the row above | -| client physics interpolation | 8 | | -| render + present | 25 | vsync on, 60 Hz display | -| **World response, opponents** | **≈174** | | -| **Ball, with local prediction** | **≈52** | same as own ship | -| Both, at 144 Hz + low-latency present | **148 / 26** | §5.4 | - -> **Correction — this table previously read ≈138 ms and omitted the server→client transit row entirely.** `INTERP_DELAY` was quoted as 38 ms, which is the interpolation buffer measured *from snapshot arrival*, while §4.6 defines the render cursor relative to `server_time_est` — server-*now*. The 30 ms return leg fell between the two definitions and was never counted. §4.6's formula is corrected to include `one_way`; this table keeps the two terms on separate rows because that is clearer to budget against. - -For reference, Rocket League runs 120 Hz physics and predicts both car and ball locally; its equivalent at 60 ms RTT is roughly 90–110 ms. - -**≈174 ms as designed here is not competitive, and this document should not pretend otherwise.** It is also not the end state: **§5.6 gets to ≈127 ms with two changes that touch no graphics setting and require no bot retrain, and to ≈103 ms with 120 Hz simulation** — inside the reference band. Read §5.6 before treating this table as a verdict. - -What *is* settled is the shape of the design: a locally-predicted ball and own ship at ≈52 ms is the difference between this being playable and not, and a 30 Hz / default-poll / interpolated-ball design would land near ≈250. - -### 5.3 Why 60 Hz snapshots, not 30 - -- Interpolation buffer: the `interval × 1.5` term is **50 ms at 30 Hz vs 25 at 60**, on top of the one-way term both share (§4.6), plus a half-interval of cadence quantisation. -- Interpolation fidelity: at `MAX_SPEED = 32` the ball moves **1.07 m between samples at 30 Hz** — more than its own diameter, so any wall bounce landing between two samples gets lerped as a straight line *through the wall*. At 60 Hz it is 0.53 m. -- Cost: 300 kbit/s. Per §1.4, bandwidth is not the constraint. - -Keep `--snapshot-hz 30` as an explicit degraded mode. - -### 5.4 High-refresh-rate clients — 120 / 144 / 165 / 240 / 360 Hz - -Players on high-refresh displays are the ones most sensitive to everything in this document, and the current code has three places where **the client draws 240 frames but only 60 of them contain new information**. Those are bugs, not tuning. - -#### What frame rate actually buys - -Modelling present as ~1.5 refresh intervals with vsync on (§5.1), and input freshness as 0.5 of a frame interval plus ~2 ms of device polling: - -| Display | present | own ship / ball (§5.1) | world response (§5.2) | with low-latency present | -|---|---:|---:|---:|---:| -| 60 Hz | 25.0 | **52** | **174** | 35 / 157 | -| 120 Hz | 12.5 | **35** | **158** | 27 / 149 | -| 144 Hz | 10.4 | **33** | **155** | 26 / 148 | -| 165 Hz | 9.1 | **31** | **153** | 25 / 147 | -| 240 Hz | 6.3 | **27** | **149** | 23 / 145 | -| 360 Hz | 4.2 | **24** | **146** | 21 / 144 | - -> **This table assumes the client can actually produce those frames. It cannot — see §5.5.** As configured today the project runs SDFGI, SSIL, SSAO, a 5-level glow pyramid, five shadow-casting lights, MSAA 4× *and* FXAA, and an unconditional full-screen backbuffer pass, none of which any player can switch off. Read §5.5 before treating any row below 60 Hz's as reachable. - -Three conclusions to design around: - -1. **60 → 144 Hz is worth ~19 ms on own-ship feel. 144 → 360 Hz is worth ~9.** The curve flattens hard, because 16 ms of the remaining budget is the 60 Hz physics tick plus its interpolation and does not move. -2. **A low-latency present is worth more at 60 Hz (−17 ms) than the entire jump from 144 to 360 Hz.** It costs one settings dropdown. -3. **Frame rate barely moves world response** — 174 → 146 across the whole 60–360 range, because that budget is dominated by RTT and the interpolation buffer. Frame rate is an *own-ship feel* lever, not a netcode one. Say this to players plainly; someone who buys a 360 Hz monitor to see opponents sooner has been mis-sold. - -#### Three things that must run per rendered frame, not per physics tick - -**a. The camera rig.** `ship_camera.gd:86` runs the entire rig in `_physics_process`. Global `physics_interpolation=true` smooths the resulting camera *transform*, so this is not visible as judder — but it costs an extra tick of camera latency on top of the ship's, and two things it does are **not** transforms and therefore **not** interpolated: `camera.fov` (`:182`) and the `PostFX` shader parameters (`:186-187`). At 240 fps those step at 60 Hz, which reads as a faint pulse in the turbo FOV kick. - -The rig moves to `_process`, reading `target.get_global_transform_interpolated()` (and `$Visual`'s, post-task 0.2) instead of `target.global_transform`, with `physics_interpolation_mode = PHYSICS_INTERPOLATION_MODE_OFF` on the rig itself so Godot does not re-interpolate an already-per-frame transform. - -**The move is cheap but it is not tuning-neutral.** Cost first: one call is ~15 engine-bound operations (2 × `get_noise_1d`, 2 × `set_shader_parameter`, `Basis.looking_at`, `slerp`, `orthonormalized`, `signed_angle_to`, `rotated`, several `global_basis` accesses) plus ~60–100 bytecode ops — call it 5–15 µs. At 360 Hz that is **1.8–5.4 ms/s, under 0.5% of a core.** Negligible, but negligible *because the absolute work is tiny*; `1-exp(-k·delta)` is a correctness property, not a cost argument, and it does not license moving arbitrarily expensive code into `_process`. - -> **The impact shake must be re-tuned, and in the opposite direction to what you would guess.** `ship_camera.gd:204` advances the noise coordinate by `delta * 60.0`, and `:64` sets `frequency = 2.5`, so each sample steps `delta × 150` noise units. At 60 fps that is **2.5 units per sample** — simplex noise decorrelates over roughly 1 unit, so the shake is currently *white noise*, and physics interpolation is lerping between independent samples. At 360 fps in `_process` it becomes **0.42 units per sample**, which is strongly correlated: the shake turns into a slow, smooth wobble that gets softer the better your monitor is. Re-derive `frequency` (or the `* 60.0`) for constant noise-units-per-*second*, then re-check amplitude by eye at 60 and 240 fps. - -Everything else in the rig genuinely is rate-independent and needs no attention: `1.0 - exp(-k * delta)` at `:126, 137, 156, 172, 177` and `move_toward(…, shake_decay * delta)` at `:212`. - -Two pre-existing bugs sit in the code this task touches, so fix them here rather than discovering them in Phase 5: - -- **The rig has no snap path.** `camera.global_position` is smoothed at `camera_smoothing = 10.0` (`:14, 137, 156`) with no reset anywhere in the file. At a kickoff teleport (`game_mode.gd:256-263`, becoming an `_integrate_forces` write under task 0.15) the camera *lerps across the arena* over ~300 ms. Add `snap_to_target()` — set `global_position`/`global_basis` directly, zero `_last_shake_offset` — and call it from the kickoff path. -- **Shake decay stalls during a goal cut.** `:94-96` returns before `_apply_shake`, so `_shake_strength`'s `move_toward` decay never runs for the length of the cinematic. Task 0.12 proposes building goal feel on exactly this system. - -**b. Remote-entity visuals.** §4.6's interpolator samples a snapshot buffer between two known states. Driving that from `_physics_process` quantises every remote ship and the ball to 60 distinct positions per second and then leans on Godot to interpolate between them — an extra tick of lag for no benefit, since we are *already* interpolating. Sample the buffer at true render time in `_process` instead: 240 distinct positions per second and one fewer tick of lag. - -The split is clean because the two consumers want different times anyway (§4.1): the **collider** is a physics concern and stays in `_physics_process` at `server_time_est`; **`$Visual`** is a render concern and moves to `_process` at `server_time_est - INTERP_DELAY`, with `physics_interpolation_mode = OFF`. Setting it `OFF` is coherent precisely *because* the node's `global_transform` is overwritten every rendered frame — there is nothing left for the engine to interpolate. Note this is the opposite of §4.5's rule for the **local** ship's `$Visual`, which is written per physics tick and therefore must stay interpolated and must be reset on snap. Same node name, two different regimes; task 0.16 lands in Phase 0 against local-ship semantics, task 2.4 adds the remote case. - -It is not free, though it is cheap: per body per frame you bracket-search a ring of 8, run two `Vector3.lerp`s and a `Quaternion.slerp`, build a `Transform3D`, and assign `global_transform` (which dirties and propagates to children). Estimate 3–6 µs per body → **~21–42 µs/frame for 7 bodies, ~1.5% of a core at 360 Hz.** That is 4–6× the work of sampling at 60 Hz. Measure it in task 0.15b rather than asserting it. - -**c. Receive polling.** Task 1.3 already flushes sends from `_physics_process`. Receiving is the other half: with (b) in place, a snapshot that lands 2 ms after a physics tick can be rendered 2 ms later at 240 fps instead of waiting 14 ms for the next tick. **Poll for receive unconditionally at the top of both `_process` and `_physics_process` — no rate limiter.** A zero-timeout `enet_host_service` on an empty socket is one non-blocking `recvfrom` returning `EWOULDBLOCK`, on the order of 1 µs; 360 of those per second costs ~0.36 ms/s. An earlier draft proposed a 2 ms limiter, which is worse than useless: at 240 fps the frame interval is already 4.17 ms so it never fires, and it only engages above ~500 fps where polling was already cheaper than the limiter. - -> **Manual polling relocates the connection signals.** With `set_multiplayer_poll(false)`, `peer_connected` / `peer_disconnected` now fire from inside your `poll()` call — mid-`_process`, during a render frame — rather than on the idle-frame boundary. Any handler that mutates the scene tree must defer. - -#### Frame-time variance, not mean frame rate, is the real target - -At 240 fps the frame budget is **4.17 ms**, and physics runs at 60 Hz — so **one frame in four carries the entire physics tick** and must still fit in 4.17 ms. On that frame the client pays, in one go: the Jolt step over 7 dynamic bodies against a 172-shape compound; 7 × `Ship._integrate_forces` (`ship.gd:346-357`), each running `apply_thruster_forces`, a full `ArenaBoundary.get_surface_pull` with five `_falloff` calls (`arena_boundary.gd:183-198`), `apply_rotation_forces`, `apply_righting_torque` and `apply_drag_and_limits` with two `pow()` calls via `_tick_scaled` (`:450`); 6 × `_update_movement_vfx` (`:296-315`, writing two material params and two `OmniLight3D` energies per ship); and on decision ticks, bot inference — `policy_network.gd` is a pure-GDScript MLP at **31→64→64→7 ≈ 6.5k multiply-accumulates per bot**, so five bots landing together is ~33k GDScript float ops in one frame. - -Task 0.8's decision stagger is framed above as a cosmetic hitch. It is not — **the physics tick sets a floor on 1%-low frame time that no graphics setting can lower.** A game that averages 240 fps but drops one frame in four to 8 ms is not a 240 fps game. Profile p99, not mean (task 0.15b). - -The same term matters at the bottom of the range, where most players actually are: see gotcha 22 and task 0.22 for the client-side `Engine.max_physics_steps_per_frame` cap that stops a hitching client from spiralling. - -#### What frame rate does *not* buy, so nobody optimises the wrong thing - -**Input sampling does not improve.** `player_ship_controller.gd:15-38` reads seven `Input.is_action_pressed` calls — all digital, all held-state — and `Ship._integrate_forces` pulls them once per physics tick. The state read at the tick *is* the freshest state; sampling it 240 times a second returns the same value 4 times in a row. The only thing lost is a press-and-release entirely inside one 16.7 ms tick, which is below human tap duration. **Do not build a sub-tick input accumulator.** If analog stick support is added later this changes, and the right answer is then a time-weighted average over the tick, not a higher sample rate. - -**Physics interpolation stays on.** It costs ~8 ms (§5.1) and is the single largest fps-independent term after the tick wait, so it will look like a target. It is not: without it a 60 Hz simulation presents 60 distinct world states per second regardless of frame rate, which is precisely the stepping a 240 Hz display was bought to avoid. Leave it on; do not expose a toggle. - -#### Why physics stays at 60 Hz, and what a bump would cost - -The honest answer to "our players want 240 fps responsiveness" is that **simulation rate, not frame rate, is the binding constraint** — 16 ms of own-ship latency and ~33 ms of world response sit behind it, and §5.2 shows frame rate alone cannot get world response under ~146 ms. Doubling to 120 Hz (Rocket League's rate, with snapshots raised alongside) would take world response from ≈174 to **≈141 ms** and own-ship from 52 to **≈44**, at 60 Hz display — or **≈115 ms** combined with a 144 Hz display and a low-latency present: - -| Term | 60 Hz sim | 120 Hz sim | | -|---|---:|---:|---| -| wait for next tick | 8.3 | 4.2 | | -| physics interpolation | 8.3 | 4.2 | | -| jitter buffer, depth 1 | 16.7 | 8.3 | | -| server tick + flush | 8 | 4 | | -| interpolation buffer | 37.5 | 25.0 | only the `interval × 1.5` term halves; the jitter term does not | -| client ↔ server transit | 60 | 60 | **does not move** | - -That is a bigger win than every tuning parameter in §3 and §4 combined. It is nonetheless **out of scope for v1**, for reasons that are about the project rather than the netcode: - -- **Every policy in `Game/bots/` is invalidated.** `ship.gd:450`'s `_tick_scaled` is defined against a 60 Hz reference and `ai_ship_controller.gd`'s `reaction_ticks` counts ticks. A bump means a full retrain — and per `TODO.md` the generation-5 curriculum is still running. -- **Server density halves**, ~6–10 matches per core to ~3–5 (§1.4). -- **Bandwidth roughly doubles**: input 43 → 86 kbit/s up, snapshots 105 → 210 kbit/s per client, 631 kbit/s → 1.26 Mbit/s per 6-player match. Still not the constraint, but 100 concurrent matches becomes ~126 Mbit/s of server uplink, which is a hosting-plan question rather than a rounding error. - -**The consequence for this plan is a hard rule: 60 is a constant named `NetCodec.TICK_HZ`, never a literal.** Ring sizes, `INTERP_DELAY`, `input_lead` clamps, seq-window bounds, snapshot cadence and the timeout constants all derive from it. Task 1.4's handshake already gates on `physics_ticks_per_second`, so a mismatched client is rejected rather than silently desynced. Done this way, a later bump is a config change plus a retrain — not a protocol rewrite. Done the other way, the literal `60` ends up in twelve files and the bump never happens. - -#### Client display settings - -`project.godot` sets neither `display/window/vsync_mode` (defaults to enabled/FIFO) nor `application/run/max_fps` (uncapped). `video_settings.gd:14-16` persists only AA, glow and brightness, and `settings_menu.gd` exposes only those three. Task 0.17 adds: - -**VSync**: Enabled (FIFO) · **Adaptive (default)** · Mailbox · Disabled. - -- **Adaptive** (`FIFO_RELAXED`) is FIFO while the renderer keeps up and tears only on a *missed* vblank. That is the right default for a game that will sometimes drop below refresh, because it avoids FIFO's half-rate cliff — miss 144 Hz by one millisecond under strict FIFO and you are pinned to 72. -- **Mailbox** only lowers latency when the renderer sustains *above* the refresh rate; below it there is never a second frame to replace the queued one, so it degenerates to FIFO latency at Mailbox power draw. Per §5.5 this build will not sustain above 144 Hz on typical hardware today, which makes Mailbox an opt-in for players with headroom, not a default. Defaulting to it would be a thermal regression for most players in exchange for nothing. - -**FPS cap**: derived from the display, not a fixed list. Query `DisplayServer.screen_get_refresh_rate(DisplayServer.window_get_current_screen())` and offer **"Match display" (default), the integer divisors of that rate, then Unlimited** — 144 Hz → 144/72/48, 165 Hz → 165/82/55, 240 Hz → 240/120/80/60. - -> **Non-divisor caps beat against scanout.** A fixed 60/75/90/…/360 list is wrong on every panel that is not 60 or 120 Hz. Cap at 100 on a 144 Hz display and `gcd(100,144) = 4`: the pattern repeats every 25 frames across 36 refreshes, with frames held for one or two intervals in an irregular sequence — visible micro-stutter. 120 on a 165 Hz panel is 8 frames per 11 refreshes, same failure. Offer the free-form list only behind an Advanced toggle with a warning. - -Three implementation constraints, all of which an earlier draft got wrong: - -- **`Engine.max_fps` is a throttle, not a pacer.** It pads each frame with a post-frame sleep to hit `1/max_fps`; it has no knowledge of scanout and never phase-locks to a vblank. *(Sleep-granularity jitter of roughly ±0.5–1 ms is inferred, not measured — verify on target platforms. The absence of phase locking is structural.)* -- **Grey out the FPS cap whenever VSync is not Disabled.** With both active, FIFO clamps presents to vblanks while `max_fps` pushes some frames past the next one and not others — frame pacing worse than either setting alone. The menu must not permit the combination. -- **Godot cannot report the *negotiated* present mode.** `DisplayServer.window_get_vsync_mode()` echoes back the mode you stored, not the `VkPresentModeKHR` the driver granted, and there is no GDScript API that exposes the latter. An earlier draft's "report what was actually applied" is not implementable, and neither is an in-engine present-latency measurement (that needs LDAT or a high-speed camera). Instead put a live `Performance.get_monitor(Performance.TIME_FPS)` readout next to the dropdown: whether the player is above or below their refresh rate is the fact every one of these settings depends on. - -The renderer is Forward+ (`project.godot:21`, `config/features=PackedStringArray("4.7", "Forward Plus")`), so the usual "Mailbox is unavailable on Compatibility" caveat does not apply as written — but `rendering/renderer/rendering_method` is not pinned in `project.godot`, so a `--rendering-method gl_compatibility` launch or a driver fallback loses it silently. Mailbox is also commonly unavailable on macOS/MoltenVK. *(Needs empirical verification on target OS versions.)* - -### 5.5 Can this build produce frames at all? - -**§5.4's table describes a machine this project is not.** Nothing in the repo has ever been profiled, and the render configuration is a showcase build, not a competitive one. Every item below is on by default and **none is reachable from `video_settings.gd`**, which persists exactly three values (`:14-16`: `aa_mode`, `glow_scale`, `brightness`). - -From `scenes/arena_base.tscn`, the Environment every arena inherits: - -| `arena_base.tscn` | Setting | Note | -|---|---|---| -| `:47-50` | `sdfgi_enabled`, `sdfgi_use_occlusion`, `sdfgi_bounce_feedback = 0.5` | Godot 4's most expensive GI path; cascades re-voxelise as the camera moves, and this camera never stops (`ship_camera.gd:126,137,156`) | -| `:42-46` | `ssil_enabled`, `ssil_radius = 4.0` | A full-resolution screen-space pass **on top of** SSAO | -| `:34-41` | `ssao_enabled`, `ssao_radius = 2.5`, `ssao_detail = 0.75` | | -| `:18-29` | `glow_enabled`, 5 levels | Mip pyramid built and resolved every frame | -| `:61, 78, 87, 96, 105` | 1 directional + **4 shadow-casting `OmniLight3D`s** | Omni shadows are cubemaps: **24 shadow-map faces per frame** before the directional | - -Plus `project.godot [rendering]`: `msaa_3d=2` (4×) **and** `screen_space_aa=1` (FXAA) **and** `use_debanding=true` — mirrored by `video_settings.gd:14` defaulting to `MSAA_FXAA`. Stacking FXAA on resolved MSAA is redundant blur, and the menu (`settings_menu.gd`) offers no 2× rung between "off" and "4×". - -Plus `shaders/post_process.gdshader:4`, `uniform sampler2D screen_texture : hint_screen_texture` — a **full-screen backbuffer copy every frame**, unconditionally. The shader's comment notes that non-turbo frames skip two texture taps, but the copy and the full-screen pass happen regardless because `vignette_strength` never reaches zero (`ship_camera.gd:187` writes `0.22 + …`, `:243` restores `0.22`). - -**What is *not* the problem**, so nobody optimises the wrong thing: - -- **The 168 colliders (§1.4) cost zero frame time.** They are `CollisionShape3D`s on a `StaticBody3D` — no draw calls, no vertices. The count is confirmed correct (168 generated + 4 authored slabs = 172 in `objects/arena_boundary.tscn`). -- **The scene is not geometry- or draw-call-bound.** `arena_boundary.gd`'s visual shell is ~1450 triangles in two surfaces of one `MeshInstance3D`; the whole match is on the order of 100–150 draw calls and well under 50k vertices. That is nothing. - -**The project is bound entirely by full-screen passes the player cannot switch off.** That inverts §5.4's conclusion about where the leverage is: the largest win per line of code is not a vsync dropdown, it is a graphics preset that gates SDFGI/SSIL/SSAO/omni shadows. Task **0.15b blocks 0.16 and 0.17** for exactly this reason — every number in §5.4 is a priori, and the first measurement may invalidate the fps list entirely. - -One mitigating subtlety, which cuts both ways: `project.godot [display]` sets `window/stretch/mode="viewport"` with a 1920×1080 base and `aspect="expand"`, so the 3D renders at a fixed ~1080p and is blitted to the window. A 1440p or 4K player therefore does **not** pay more for any of the above — but also **cannot render at native resolution**, and a 1080p player cannot render lower. Task 0.17c owns that decision; it interacts directly with render scaling (0.17b) and cannot be left implicit. - -#### 5.5.1 Measured (task 0.15b, 2026-08-18) - -6-ship Match, 1080p, non-headless. **Hardware: Apple M4 (Metal), 10-core — a development laptop, not a dedicated gaming reference machine**; treat absolute fps as directional, not a promise to players on other hardware. - -| | p50 | p99 | fps (p50 / p99) | -|---|---:|---:|---:| -| All effects on (project defaults) | 17.93 ms | 20.39 ms | 55.8 / 49.0 | -| All effects off | ~17.2 ms | — | ~58 | - -**This invalidates the a priori §5.4/§5.5 fps list exactly as flagged.** Default settings cannot sustain even 60 fps on this hardware, let alone 144 — and the surprising part is *why*: turning every toggleable effect off (SDFGI, SSIL, SSAO, glow, all 5 shadow casters, MSAA, FXAA, PostFX) only recovers the difference between ~56 and ~58 fps. The ~17 ms floor is **not** made of the full-screen passes this section blamed — something else (base forward-clustered shading, the ~150 draw calls, per-ship VFX materials, or fixed engine/CPU overhead at 6 ships) dominates, and 5.4's framing ("the project is bound entirely by full-screen passes") is wrong as measured on this hardware. - -Per-effect isolated cost (each toggled off individually against a fixed baseline sample), for reference — treat these as low-confidence: they cluster tightly at 2.9–3.8 ms each with no clear outlier, which is consistent with most of that spread being sampling noise from a ~1 ms-jittery baseline rather than real per-effect attribution: - -| Setting | Cost (ms) | -|---|---:| -| SSAO | 3.77 | -| PostFX | 3.82 | -| Omni shadows (×4) | 3.69 | -| SSIL | 3.44 | -| FXAA | 3.37 | -| Directional shadow | 3.30 | -| SDFGI | 3.24 | -| MSAA 4× | 3.12 | -| Glow | 2.89 | - -**Consequence for 0.17/0.26/0.28**: a graphics preset alone will not reach a 144 fps target on hardware in this class — Low-preset gets to only ~58 fps by this measurement, not the 2×+ jump §5.4 assumed. **0.26 (bake GI) and 0.28 (separate physics thread) need to re-justify their expected win against this floor before implementation.** - -**Root-cause follow-up, attempted and inconclusive (2026-08-18).** Three further remote-automated profiling passes (via `godot-mcp` `game_eval` sampling `Performance.get_monitor()` against a live instance, no human at the editor) were run to find what the ~17 ms floor actually is. They did not converge: - -| Pass | Setup | Result | -|---|---|---| -| 1 (above) | 6-ship 3v3, sustained | 17.93 / 20.39 ms (p50/p99), all-off floor ~17.2 ms | -| 2 | Reportedly 6-ship, actually 1v1 (misconfigured) | CPU 17.64 ms + frame 10.75 ms — internally inconsistent (CPU time exceeding frame time from non-atomic sampling); agent also reported the game becoming unresponsive mid-run | -| 3 | 6-ship 3v3, atomic single-`eval` sampling, retried after pass 2's failures | 8.7–10.2 ms (98–115 fps), reported CPU time 0.013 ms — implausibly low for a frame running Jolt physics + GDScript bot inference across 6 ships, so not trusted either | - -Passes 1 and 3 supposedly measured the same scenario and differ by ~2×. **The likely explanation is the measurement method itself, not the game**: each `game_eval` round-trip through the MCP bridge has its own latency and can perturb the very frame timing it's sampling, and nothing here confirms the scene state (ship count, bot activity, camera framing) was identical across passes. Read the specific numbers in this subsection as *evidence a floor well under 144 fps exists*, not as an attributed cause — **the SSAO on/off screenshot check in pass 3 did confirm effect toggles are visually real** (ruling out "the toggles are no-ops" as an explanation), which is the one finding that survived across passes. - -**What this needs next, and why an agent can't finish it remotely:** a proper frame-time attribution needs either a human at the Godot editor reading the Debugger's built-in Monitors/Visual Profiler (which breaks GPU time down by pass — opaque, shadow, post-process, etc. — instead of one aggregate number), or an external GPU profiler (RenderDoc, Xcode GPU capture on this hardware). Both require eyes on a live UI, not remote `eval` polling. **This is now the concrete blocker for 0.26/0.28**, not further scripted measurement passes. **0.15b's original acceptance criterion (write a max-frame-rate number into §5.5) is still met by pass 1** — the floor is real and under both 60 and 144 fps — but the deeper "why" is open and parked here rather than guessed at. - -**Root cause of the pass-to-pass inconsistency, found (2026-08-18):** a Godot editor and an orphaned headless training process had both been running on the profiling machine, untouched, for 11 days (since 2026-08-08) — leftover from earlier local work, unrelated to this investigation. `godot-mcp`'s automated launches were plausibly contending with that stale editor instance rather than getting a clean process every pass, which is a much better explanation for a ~2× swing between "identical" scenarios than genuine frame-time variance. Both processes were killed and a clean re-check was run. - -**Is it just that we're on a Mac?** Partly, but not via the mechanism first suspected. HiDPI/Retina resolution inflation was checked directly and **ruled out**: the live viewport renders at 2036×1080 against a target of 1920×1080 — about 6% more pixels, non-uniformly (width only; the 2× multiplier a true Retina backbuffer would apply is not happening, `display/window/dpi/allow_hidpi=true` notwithstanding). A 6% pixel-count difference cannot produce the ~2× frame-time swings seen above, so resolution is not the explanation for this session's inconsistency — that was the stale-process contention above. It's still worth a one-line fix later (0.17c owns display/stretch decisions) since 2036×1080 is a mildly wasteful, non-native render target. - -What Mac hardware **does** plausibly bias is the *shape* of the result, not the run-to-run noise: Apple Silicon GPUs are tile-based deferred renderers (TBDR), architecturally unlike the immediate-mode AMD/Nvidia GPUs the target "reference hardware" (a Windows/Linux gaming PC) uses. TBDR keeps a frame in on-chip tile memory and is comparatively cheap at MSAA resolve, but any pass needing to read arbitrary neighbouring pixels across the whole frame — SSAO, SSIL, the glow downsample/upsample chain, the PostFX shader's `screen_texture` read — forces a break out of tile memory into a full system-memory resolve, an overhead that is largely constant per pass rather than proportional to what the pass computes. That lines up with pass 1's finding that SDFGI/SSIL/SSAO/MSAA/FXAA/shadows/PostFX all cost within a tight 2.9–3.8 ms band regardless of what each one actually does — consistent with a shared TBDR resolve tax dominating over each effect's real cost. **Numbers measured on this machine should be treated as informative about relative ordering at best, not as a stand-in for target-platform (desktop GPU) behaviour** — confirmed below. - -#### 5.5.2 Measured on real reference hardware — RTX 3090, Linux (2026-08-19) - -Same 6-ship 3v3 Match, 1080p, via a purpose-built harness (`Game/tools/gpu_profile_harness.gd`) run directly against a real GPU-bound X session (not Xvfb — an earlier attempt through Xvfb silently fell back to Mesa's `llvmpipe` **software** rasterizer, ~35x slower and completely unrepresentative; caught via the harness's own adapter-name check, not assumed). This is the number that matters — an actual discrete immediate-mode GPU, the architecture players will actually have: - -| | p50 | p99 | fps (p50) | -|---|---:|---:|---:| -| All effects on (project defaults) | 1.85 ms | 2.98 ms | 540 | -| All effects off | 0.53 ms | 1.53 ms | 1883 | - -**This overturns §5.5.1's conclusion, not just its numbers.** On real hardware, disabling every effect gives a **3.5×** speedup — the opposite of the Mac's ~1.03× — and the per-effect breakdown finally makes physical sense instead of clustering suspiciously: - -| Setting off | Frame time | Implied cost | -|---|---:|---:| -| (baseline, all on) | 1.85 ms | — | -| SDFGI | 1.49 ms | **0.36 ms** | -| SSIL | 1.60 ms | **0.25 ms** | -| Glow | 1.75 ms | 0.10 ms | -| Shadows (all 5 casters) | 1.76 ms | 0.09 ms | -| SSAO | 1.82 ms | 0.03 ms | -| MSAA 4×, FXAA, PostFX | 1.87–2.12 ms | noise-level (see below) | - -SDFGI and SSIL alone account for over half of the effects' total cost, matching §5.4's original expectation (voxel cone tracing and a full-res screen-space GI pass being the expensive ones) — the Mac's flat, undifferentiated cost profile was the anomaly, not this one. MSAA/FXAA/PostFX show *negative* "costs" (disabling FXAA measured as slightly slower than leaving it on) — at ~1-2 ms absolute frame times, OS scheduling jitter is larger than the real signal for cheap passes; those three need a longer sampling window or a proper GPU profiler to resolve, not this harness's coarse `get_process_delta_time()` sampling. Note also that all-off (0.53 ms) is faster than baseline-minus-sum-of-individual-savings (1.85 − 0.36 − 0.25 − 0.10 − 0.09 − 0.03 ≈ 1.02 ms) — the combined removal saves more than the parts, consistent with each full-screen pass carrying some fixed per-pass overhead (pipeline barriers, render-target switches) on top of its own work, which compounds when several stack. - -**Consequence for 0.17/0.26/0.28, revised**: at 540 fps p50 with every effect enabled, **this scene is nowhere near GPU-bound on reference-class hardware** — the entire "must hit 144 fps" framing in §5.4/§5.5 was solving a problem that doesn't exist on the hardware tier it was written for. That reframes the two gated tasks rather than clearing them outright: -- **0.26 (bake GI, retire SDFGI)** — the *relative* win is real and correctly targeted (SDFGI is the single largest line item, ~19% of the effects-on budget), and the preset design already bets on this being right (Low/Medium turn SDFGI+SSIL off first, matching exactly what this data says to cut). But "largest frame-time reduction of any task here" (its acceptance bar) oversells it on a 3090 — 0.36 ms off an already-tiny budget is not the headline win §5.7 implied. The task is worth doing for **lower-end/integrated GPUs**, where the same relative cost almost certainly scales to something that matters — but that's now the open question, unmeasured on this pass. -- **0.28 (physics/3d/run_on_separate_thread)** — its whole motivation is smoothing frame-time variance caused by the physics tick sharing the render thread; at a 1.85 ms p50 / 2.98 ms p99 baseline (both far under even a 240 Hz frame budget), there's no variance problem to fix on this hardware. Deprioritize below 0.26 unless a lower-end-hardware pass shows otherwise. -- The preset ladder itself (task 0.17, done) needs no changes — its bundle choices (drop SDFGI/SSIL first) are now empirically justified rather than just plausible-sounding. - -**Still open**: no low/mid-tier GPU has been profiled. The 3090 result rules out "the game is GPU-bound on reasonable hardware" as a near-term concern, but says nothing about a GTX 1660 or an integrated Iris/Vega part, which is where a real preset ladder earns its keep. Re-run `gpu_profile_harness.tscn` on weaker hardware before spending more effort on 0.26/0.28. - -### 5.6 Closing the gap to the reference — without lowering settings - -§5.2 lands at ≈174 ms against a ~90–110 ms reference band. The instinct is that reaching it means trading visual quality for frames. **It does not.** Decompose the 174: - -At 60 ms RTT, 60 ms is transit and irreducible in code. That leaves **114 ms of local overhead**, of which frame rate governs only two terms — input freshness (10) and present (25) — and *quality settings* govern neither directly. Present latency is a function of vsync mode and swapchain depth, not of how many effects are enabled; a 60 fps client with a shallow present queue beats a 240 fps client with a deep one. **The entire 60 → 240 fps range is worth ~12 ms once a low-latency present is in place** (§5.4). The other ~100 ms is netcode time model and simulation rate. - -Four levers, none of which touches a graphics setting: - -| | Lever | Saves | Risk | -|---|---|---:|---| -| **L1** | **Extrapolate remote *visuals* to present time** instead of interpolating the past | **−30** | Mis-prediction pops | -| **L2** | 120 Hz simulation | −21 | Bot retrain, ½ server density, 2× bandwidth | -| **L3** | Adaptive jitter-buffer depth, 0 on clean links | −8 | Starvation on jittery links | -| **L4** | Shallow present queue + Adaptive vsync | −17 | Throughput loss if GPU-bound | - -#### L1 is the big one, and it is nearly free - -§4.1 already computes remote entities' **present-time** state — that was the fatal correction that put the collider at `server_time_est`. `$Visual` is then deliberately rendered ~68 ms in the past for smoothness. **Render it at present time too and the whole 37.5 ms interpolation buffer disappears**, leaving only a residual for error smoothing. - -The reason this is safe here is that ships have bounded acceleration and the hull is large. Extrapolating with known velocity, error is `½·a·t²` over the full 68 ms horizon: - -| | max accel | error @ 38 ms | error @ 68 ms | -|---|---:|---:|---:| -| position, cruise | 30 m/s² (`thrust_power 150` / `mass 5`) | 0.022 m | **0.069 m** | -| position, turbo | 75 m/s² (`turbo_multiplier 2.5`) | 0.054 m | **0.173 m** | -| yaw | 20 rad/s² (`rotation_power 20` / `inertia.y 1`) | 0.8° | **2.6°** | -| pitch / roll | 2.9 rad/s² (`inertia.x/z 7`) | 0.1° | **0.4°** | - -**0.17 m and 2.6° worst case, against a 4 m hull.** That is well under the width of the ship and an order of magnitude smaller than the 3.5 m staleness §4.1 was written to eliminate. Feed the residual through the same soft-correct pipeline already specified for the local ship (§4.4) and remote ships are visually at present time with a sub-decimetre wobble. - -Two bonuses: it **collapses §4.1's dual clock back into one** — collider and visual both at `server_time_est`, so §5.4b's `_process`/`_physics_process` split and the two-regimes-for-one-node-name hazard both go away — and it applies to the ball, which is near-ballistic between contacts and therefore extrapolates better than ships do. - -The cost is real but narrow: a remote ship that *reverses input* at the moment you sample it mispredicts by the numbers above and then visibly corrects. Interpolation never mispredicts; it is just always late. This is the genuine trade, and it is the one the reference class makes. - -#### The reachable budget - -| Term | today | L1 + L4 (v1) | + L2 + L3 | at 144 fps | -|---|---:|---:|---:|---:| -| input freshness | 10 | 10 | 10 | 5.5 | -| wait for next tick | 8.3 | 8.3 | 4.2 | 4.2 | -| client → server | 30 | 30 | 30 | 30 | -| jitter buffer | 16.7 | 16.7 | 4.2 | 4.2 | -| server tick + flush | 8 | 8 | 4 | 4 | -| server → client | 30 | 30 | 30 | 30 | -| interp buffer → extrapolation residual | 37.5 | 8 | 8 | 8 | -| client physics interpolation | 8.3 | 8.3 | 4.2 | 4.2 | -| present | 25 | 8.3 | 8.3 | 3.5 | -| **World response** | **≈174** | **≈127** | **≈103** | **≈94** | - -**≈103 ms at 60 fps with every effect enabled**, and ≈94 at 144 fps. That is inside the reference band, reached without disabling SDFGI, SSIL, SSAO or shadows. Even a client struggling at 30 fps on maximum settings lands near ≈120 ms. - -Sequencing follows ms-per-unit-of-risk: **L4 then L1 for v1 (≈127 ms, no bot retrain, no protocol change)**; L2 and L3 after, when a retrain is affordable. §5.5's preset system remains worth building — but for *frame rate and thermals*, which is what it actually buys, not for latency. - -> **The largest lever is not on this list.** All of the above assumes 60 ms RTT. Regional server siting that puts most players on a 30 ms RTT takes ≈127 to ≈97 and ≈103 to ≈73 with no code at all. Phase 6 owns it, and it should be argued against these numbers. - -> **Perspective on where this matters.** Own ship and ball are already at ≈52 ms and are unaffected by every lever here — they are predicted locally. World response governs *opponent ships*. In a game whose subject is a ball, that ordering is favourable: the two objects a player tracks most closely are the two already at single-digit-tick latency. - -### 5.7 The next tier — and where it stops paying - -§5.5 and §5.6 are the first-order work. This section is what remains after them, and it is deliberately honest about the point where further effort stops being worth it. - -#### Frame rate: SDFGI is the wrong tool for this arena - -**The single largest available win, and it costs no visual quality.** `arena.gd` and `goal.gd` have **no `_process`, no `_physics_process`, no `AnimationPlayer` and no `Tween`** — the floor, walls, ceiling, goals and every light are static for the entire match. The only things that move are 6 ships and a ball, all small and all self-lit. - -SDFGI exists to light *dynamic* worlds, and it pays for that by re-voxelising cascades as the camera moves — and this camera never stops moving (`ship_camera.gd:126, 137, 156`). It is the most expensive thing in the frame, doing continuous work to solve a problem this project does not have. - -- **Replace `sdfgi_enabled` with baked GI** — `LightmapGI` for the static shell, or `VoxelGI` if bounce onto moving ships matters. Bake cost is offline; runtime cost is a texture fetch. The look is preserved or improved (baked bounce is higher quality than SDFGI's cascades), and it survives on the High preset rather than being the first thing a preset has to switch off. -- **`ssil_enabled` becomes largely redundant** once bounce is baked. It is a full-resolution screen-space pass duplicating information the lightmap already has. - -This is the answer to "lowest lag *and* highest fps without lowering settings": the expensive setting was solving the wrong problem. - -#### Frame rate: expensive defaults that `project.godot` never overrides - -`[rendering]` contains exactly three keys (`msaa_3d`, `screen_space_aa`, `use_debanding`). Everything else runs at engine defaults, including: - -| Setting | Default | Note | -|---|---|---| -| `lights_and_shadows/positional_shadow/atlas_size` | 4096 | Shared by **all** shadowed positional lights; 2048 is usually indistinguishable here | -| `lights_and_shadows/directional_shadow/size` | 4096 | | -| `lights_and_shadows/directional_shadow/soft_shadow_filter_quality` | high | | -| `occlusion_culling/use_occlusion_culling` | off | Low value in an enclosed arena — measure before adding bake time | -| `mesh_lod/lod_change/threshold` | — | Irrelevant: the scene is ~1450 triangles of arena plus low-poly ships (§5.5) | - -Also worth counting: `_build_movement_vfx` creates **two `OmniLight3D`s per ship** (`ship.gd:270-278`), so a 3v3 has 12 dynamic lights on top of the arena's 5. They are correctly `shadow_enabled = false` and `omni_range = 3.5`, so they are cheap — noted so nobody "discovers" them and disables engine glow for nothing. - -#### Frame rate: the CPU side, which §5.5 does not cover - -§5.5 establishes the project is GPU-bound on full-screen passes. Once those are fixed it becomes CPU-bound, and §5.4's frame-time variance becomes the ceiling. Three levers: - -- **`physics/3d/run_on_separate_thread`** (not set; defaults off). This decouples the physics step from the render thread and directly attacks "one frame in four carries the whole tick." It is the highest-leverage item here **and the riskiest** — it changes when `_integrate_forces` runs relative to script code, and this project puts real logic there (`ship.gd:346-357`) plus an RL training path. *Prototype and measure; do not enable on faith.* -- **`ArenaBoundary.get_surface_pull` has no early-out.** It runs a `to_local()` plus five `_falloff` calls for every dynamic body every tick, including for a ball sitting in the middle of the arena where every term is zero. A single bounds check against `wall_range`/`ceiling_range` skips almost all of it in open play — 7 bodies × 120 Hz once L2 lands. -- **Bot inference is ~6.5k GDScript multiply-accumulates per bot** (`policy_network.gd`). Task 0.8 staggers them; beyond that, the lever is network width, which is a training decision, not a rendering one. - -#### Latency: what is actually left - -After L1–L4 and 120 Hz simulation, at 144 fps, the budget is ≈94 ms — **and 60 of that is RTT.** The remaining 34 ms of local overhead breaks down as input freshness 5.5, tick wait 4.2, jitter 4.2, server 4, extrapolation residual 8, physics interpolation 4.2, present 3.5. Every one of those is at or near a floor set by physics rate or hardware. - -Two code ideas remain, both small and both with a cost: - -- **Forward-extrapolate the local `$Visual`** instead of interpolating between the last two ticks — render the predicted ship at present time rather than up to one tick behind. Worth ~4 ms. Risk: overshoot at the moment of a collision, which is the most visually sensitive moment in the game. -- **Tighten the extrapolation-error smoothing** (§5.6's 8 ms residual). Worth ~4 ms, paid for in more visible correction pops. - -**That is the whole remaining code budget: ~8 ms, both items trading visual stability for it.** Meanwhile: - -- **Regional server siting** takes a 60 ms RTT to 30 for most players: **−30 ms**, four times the remaining code budget, no code at all. -- **Ping-weighted matchmaking and a server browser sorted by measured ping** convert that into something players actually experience rather than something that is true on average. -- **Steam Datagram Relay (Phase 7)** is planned for NAT traversal and DDoS protection, but Valve's backbone frequently routes better than raw BGP paths — for some player pairs SDR is a *latency reduction*, not a tax. Measure it both ways rather than assuming it costs. - -#### Where this stops paying - -Two limits worth writing down before someone spends a month on the last 5 ms: - -1. **Past ~100 ms, you are optimising 3–4 ms at a time against a 60 ms constant.** The ratio of engineering effort to felt improvement collapses. Server siting and matchmaking dominate everything else from that point on. -2. **"Lowest lag" and "best feel" diverge at the end.** Both remaining code levers, and L1 itself, buy milliseconds by predicting further ahead and correcting harder. Past a point that makes the game feel *worse* — twitchier, less stable, more prone to visible snapping — while the latency number keeps improving. The number is a proxy, not the goal. **Task 4.7's tuning pass, with a human in the seat, is the authority; the budget table is not.** - ---- - -## 6. Match lifecycle - -### 6.1 State machine - -``` -LOBBY -> LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP -> ... - -> FULL_TIME -> OVERTIME_WARMUP -> OVERTIME -> ... - -> RESULTS -> LOBBY -``` - -Broadcast as the `match_state` byte in every snapshot, and on transition via `state_change(state, at_tick)`. - -### 6.2 Sequence - -1. **Connect.** Client sends `hello(protocol_version, physics_ticks_per_second, display_name, auth_ticket)`. Server rejects a mismatch on **either** version or tick rate, with a reason string, then `disconnect_peer`. (A client at 30 Hz advances its sequence numbers at half rate and confuses every control loop.) `auth_ticket` is an empty `PackedByteArray` until Phase 7 — reserve the field now. -2. **Welcome.** Server assigns `player_id`, balances teams, replies `welcome(player_id, server_info, roster, match_state, server_tick, score, end_tick)`, broadcasts `player_joined`. -3. **Lobby.** `ready_toggle()`; start when all ready, or `--auto-start` after `--min-players` plus a countdown. -4. **Config.** `match_config(match_id, arena_path, team_size, match_length_ticks, roster[], seed)`. `roster[i] = {slot, team, spawn_index, player_id, name, is_bot}` — **slot order here is the snapshot's body order for the whole match.** The client validates `arena_path` against `ArenaRegistry.ARENAS` before `load()`; a malicious or buggy server must not be able to make a client load an arbitrary `res://` path. -5. **Load.** Both sides load `networked_match.tscn`. Each peer loads the arena and spawns the roster in slot order. Client additionally spawns a camera rig on its own ship and adds `HUD.tscn` **in code** — `networked_match.tscn` must have no HUD child, because `GameMode._ready()` (`game_mode.gd:44-45`) would pick it up server-side. Client sends `scene_ready(match_id)`. -6. **Kickoff.** Server waits for all `scene_ready` (10 s timeout → proceed). Broadcasts `kickoff(reset_transforms[], countdown_start_tick, reset_gen)`. Both sides freeze bodies. HUD counts down from `server_tick`, not a local `Timer`. At `countdown_start_tick + 180` the server unfreezes and broadcasts `state_change(PLAYING)`. -7. **Play.** Inputs up, snapshots down. -8. **Goal.** Server's `Goal` sensor fires → `_handle_goal_scored` debounce → `goal_scored(scoring_team, score, goal_tick, resume_tick)`. Bodies freeze. Clients play the cinematic within `[goal_tick, resume_tick]`. At `resume_tick`: `kickoff(...)`. -9. **Clock.** Tick-derived: `remaining_ticks = end_tick - current_server_tick`. `end_tick` and a `running` flag ship in `match_config` and in `clock_state(running, end_tick, at_tick)`. -10. **Full time / overtime / results.** `RESULTS` holds, then `state_change(LOBBY)` and both sides load `lobby.tscn`. **Clients return to the lobby, not the main menu** — a community server that empties every 2.5 minutes is dead on arrival. - -**Every lifecycle message carries absolute ticks**, never durations. That is what makes reliable-channel latency harmless: on a lossy link ENet's RTO can stretch a `goal_scored` → `kickoff` → `state_change` burst to ~600 ms. **Specify the late-arrival case explicitly**: a `kickoff` that lands after its own `resume_tick` must apply the reset immediately and skip the countdown, not schedule it into the past. - -`NetworkedMatch` must declare all five signals `HUDController` duck-types on (`HUDController.gd:65, 88, 100, 103, 106`) — `timer_updated`, `score_changed`, `match_ended`, `kickoff_countdown`, `overtime_started` — and emit them from RPC handlers instead of from local logic. Otherwise the HUD silently omits rows. - -### 6.3 Late joiners and spectators - -`welcome` carries full state, so a late joiner reconstructs immediately. - -- Free slot and state is `LOBBY`/`WARMUP` → join as a player now. -- Free slot mid-match → **spectate now, take the slot at the next kickoff.** Swapping a controller at a kickoff boundary is free; mid-play it is not. -- No free slot → spectator. A spectator receives identical snapshots (the snapshot is already a broadcast — zero extra server work), spawns no ship, and points a camera rig at a chosen ship or the ball. Cap with `--max-spectators`. - -`HUDController._initialize_hud()` `push_error`s and bails when `ship` is null (`HUDController.gd:41-46`). Spectators need a path through that. - -### 6.4 Disconnects — no ship is ever despawned - -On `peer_disconnected` the server **keeps the ship and swaps its controller**: - -1. `--fill-bots`: replace with an `AIShipController` on the server's configured model. -2. `--no-fill-bots` (default for public servers, see §1.4): swap to the base `ShipController` — inert but simulated, exactly the placeholder `game_mode.gd:216` already uses. - -Set `flags.stalled` so clients can grey out the nameplate. Reserve the slot for 30 s keyed by identity so a reconnect gets its ship back. If the last human leaves, abort to `LOBBY`. - -**Justification is wire-format simplicity, not the bot cache.** Fixed slot order means the snapshot needs no add/remove machinery, no `MultiplayerSpawner`, and no re-indexing. That reason stands on its own. - -`ai_ship_controller.gd` currently caches teammate/opponent lists once with the comment "rosters never change mid-match (no despawn path exists anywhere in this codebase)". **Do not let that be the justification** — protecting a bot's implementation detail is tail-wagging-dog, and taken as an architectural constraint it permanently forecloses 3v3→2v2 shrink, mid-match rebalancing, and join-onto-a-new-slot. Fix the cache anyway (task 0.9): `filter(is_instance_valid)` plus a `roster_changed` signal, ~5 lines of cheap insurance. +*Sections 1–6 (architecture, wire format, input handling, prediction, +latency budget, match lifecycle) live in +[`MULTIPLAYER_SPEC.md`](MULTIPLAYER_SPEC.md). A bare `§N` below refers to +that document for `N` 1–6, and to this one for `N` 7+.* --- @@ -828,439 +107,142 @@ Set `flags.stalled` so clients can grey out the nameplate. Reserve the slot for `[P]` parallelisable within its phase · `[D:x.y]` hard dependency -### Phase 0 — Non-networked refactors - -Every task lands on `master` independently, is verifiable in single-player today, and cannot break anything. Near-total parallelism. - -| # | Task | Files | Acceptance | -|---|---|---|---| -| 0.1 `[P]` | **DONE.** Added `ShipAction.copy()`. Audit: the only `get_action()` call site (`ship.gd:347`) reassigns `_current_action` fresh each tick rather than buffering it, so no aliasing bug exists yet — `copy()` is a no-op today, ready for Phase 4's prediction ring | `ship_action.gd`, `player_ship_controller.gd` | Free Play unchanged; `copy()` returns a distinct object with equal fields | -| 0.2 `[P]` | **DONE.** Inserted `Visual` (`Node3D`) into `ship.tscn`, reparented `Nose`/`TailFin` under it, redirected all four code-driven `add_child` calls onto `$Visual` (now a public `@onready var visual`), resolved `_apply_team_color`'s lookup to `"Visual/" + mesh_name` | `objects/ship.tscn`, `scripts/ship.gd` | Child-type assertion holds; ship looks identical in Free Play; team colours still apply on both teams | -| 0.3 `[D:0.2]` | **DONE.** `ship_camera.gd`'s three `target.global_transform` reads (ball cam, ship cam ×2) now read `target.visual.global_transform` | `scripts/ship_camera.gd` | Camera behaviour unchanged in Free Play and Match — `visual` has identity transform relative to the body until Phase 4 writes an offset, so this is a no-op today | -| 0.4 `[P]` | **DONE.** `can_sleep = false` on Ship and Ball | `objects/ship.tscn`, `objects/ball.tscn` | No behaviour change | -| 0.5 `[P]` | **DONE.** `continuous_cd = true` on Ship (Ball already had it) | `objects/ship.tscn` | No tunnelling at max speed into the ball or walls | -| 0.6 `[P]` | **DONE.** Spawned ships renamed to `Ship_T%d_S%d` | `game_mode.gd` | Names are `(team, spawn_index)`-derived, not insertion-order | -| 0.7 `[P]` | **DONE.** `_jittered` now uses an owned `RandomNumberGenerator`, self-randomized in `_ready()` unless `kickoff_rng_seed` is set explicitly (a fresh `RandomNumberGenerator` defaults to a fixed internal state, unlike the global `randf_range` Godot auto-randomizes at startup — call this out for whoever reads the diff and expects `.new()` alone to be enough) | `game_mode.gd` | Kickoff jitter unchanged in feel; a fixed seed reproduces kickoffs exactly | -| 0.8 `[P]` | **DONE.** `_ticks_until_decision = randi_range(1, reaction_ticks)` at spawn, after `load_policy()` (which still resets to 0 on later calls, e.g. league opponent swaps — harmless, those land at reset boundaries) | `ai_ship_controller.gd` | Six-bot Spectate shows no periodic frame spike | -| 0.9 `[P]` | **DONE.** Roster validity checked (`Array.any()`) once per decision tick, not every physics tick; `filter(is_instance_valid)` + `roster_changed` signal only fire on an actual stale reference | `ai_ship_controller.gd` | Bots behave identically; freeing a ship mid-match no longer corrupts observations | -| 0.10 `[D:0.12]` `[P]` | ~~Add virtuals `_owns_goal_logic()`, `_allows_time_scale_effects()`, `_goal_pause_seconds()`, `_owns_world_simulation()`~~ **DONE, narrower than drafted.** `_allows_time_scale_effects()` dropped: 0.12 deletes `Engine.time_scale` from the file entirely, so there is nothing left for it to gate. Implemented `_owns_goal_logic()`, `_owns_world_simulation()`, `_goal_pause_seconds()`, all behaviour-preserving (default `true`/`GOAL_CELEBRATION_SECONDS`), gating the goal-signal connection and `_respawn_escaped_bodies()` | `game_mode.gd` | Free Play, Match, Spectate and Training all behave identically — verified no other virtual was load-bearing today; these exist for a future networked-client mode | -| 0.11 `[P]` | **DONE.** `_handle_goal_scored` checks `is_inside_tree()` after each `await` and bails before touching arena/hud state | `game_mode.gd` | A scene change mid-celebration cannot strand the flag | -| 0.12 `[P]` | ~~Replace `Engine.time_scale` hit-stop and goal slow-mo with camera-only effects~~ **DONE.** Added `ShipCameraRig`'s "Impact Punch" group (`punch_fov_kick`/`punch_vignette_kick`/`punch_chroma_kick`/`punch_decay`, applied additively after `_update_speed_feel` each tick, decaying via `move_toward` over real `delta`) triggered from the existing `_on_target_ball_contact`; goal moments now rely on the pre-existing `begin_goal_cut`/`end_goal_cut` cinematic cut alone, no separate slow-mo effect needed. All `Engine.time_scale` fields/methods deleted from `game_mode.gd` (`_hit_stop_*`, `_goal_slowmo_active`, `_restore_hit_stop`, `_run_hit_stop`, `GOAL_SLOWMO_SCALE`) | `game_mode.gd`, `ship_camera.gd` | Goal and impact feel is at least as good; `Engine.time_scale` is never written — confirmed via `grep -rn time_scale scripts/` | -| 0.13 `[P]` | **DONE.** `physics_jitter_fix = 0.0` set. `CLAUDE.md`'s architecture section had stale prose dimensions ("inner x ±12, z ±18, height 12, goal lines z ±17") — corrected to reference the actual named constants (`INNER_HALF_X` 18, `INNER_HALF_Z` 27, `INNER_HEIGHT` 18, `GOAL_LINE_Z` = `INNER_HALF_Z`) instead of restating numbers that can drift out of sync again | `project.godot`, `CLAUDE.md` | Flight feel unchanged; `CLAUDE.md` matches `arena_boundary.gd:8-14` | -| 0.14 `[D:0.2]` | **DONE.** Added `Ship.set_visual_action(thrust_z, turbo)`, `Ball.set_visual_speed(speed)` (with a `_visual_speed_override` field the trail prefers when ≥0), and `Ship.net_vel_correction`/`net_visual_offset` fields plus the guarded hook at the top of `_integrate_forces` (decays `net_visual_offset` via `_tick_scaled`, writes it to `visual.position`) | `ship.gd`, `ball.gd` | No-op until Phase 4; single-player unchanged — nothing calls any of these yet | -| 0.15 `[P]` | **DONE.** Ship/Ball gained `queue_teleport(to)`; `_integrate_forces` applies it via `state.transform` + zeroed velocities + `reset_physics_interpolation()`. `GameMode._reset_body` now calls `body.call("queue_teleport", to)` (dynamic dispatch — `RigidBody3D` itself has no such method) instead of `set_deferred` | `game_mode.gd`, `ship.gd`, `ball.gd` | Kickoff resets in Match are visually identical, with no interpolation smear | -| **0.15b** | **DONE, superseded by §5.5.2 — read that, not the Mac numbers below.** First pass measured a live 6-ship Match, 1080p, on an Apple M4 dev laptop (§5.5.1): all-on p50 17.93 ms, all-off floor ~17.2 ms, with per-effect costs clustered suspiciously flat (2.9–3.8 ms each). That data turned out to be a poor stand-in for the target platform — Apple's tile-based GPU architecture, not a real bottleneck — and was superseded by a same-scenario re-run on real reference hardware (RTX 3090, §5.5.2): all-on p50 1.85 ms / all-off 0.53 ms, SDFGI+SSIL clearly dominant as originally expected, everything else cheap. Keep §5.5.1 for the record of what was tried and why it was distrusted, not as a performance reference | `scenes/arena_base.tscn`, `shaders/post_process.gdshader`, `Game/tools/gpu_profile_harness.gd` | **Measured max frame rate written into §5.5.2 from real reference hardware.** At 540 fps p50 with everything on, this scene is nowhere near GPU-bound on a 3090-class GPU — the a priori §5.4 fps list was solving for a constraint that doesn't hold at that hardware tier. 0.17 (done) needed no changes: its preset bundle choices are now empirically validated. 0.26 stays open (real but smaller win than assumed); 0.28 closed (no variance problem exists to fix) | -| 0.16 `[D:0.3]` | **DONE.** Camera rig moved `_physics_process` → `_process`; reads `target.visual.get_global_transform_interpolated()` in both ball-cam and ship-cam; rig itself has `physics_interpolation_mode = OFF` (it writes its own transform every rendered frame now, so Godot's built-in interpolation would just fight the manual smoothing). `target` setter primes interpolation (`target.visual.reset_physics_interpolation()`) and calls the new `snap_to_target()` so a freshly-assigned target (or a Spectate switch) doesn't lerp in from wherever the rig was previously. **Shake re-derivation, implemented differently than drafted**: rather than rescale `frequency`, `_apply_shake` now quantizes the noise-domain input to whole 60Hz ticks (`floori(_shake_time * SHAKE_UPDATE_HZ)`) — every render frame within one 1/60s window reuses the identical noise sample, so consecutive *distinct* samples stay exactly `frequency` (2.5) domain-units apart at any render frame rate, reproducing 60fps's original jitter character everywhere instead of smoothing out at high fps. `snap_to_target()` is called from `game_mode.gd`'s `reset_ships()`, not directly from `ship_camera.gd`'s own kickoff-adjacent code — `reset_ships()` is now `async` and awaits one `get_tree().physics_frame` before snapping, because `_reset_body`'s `queue_teleport` (task 0.15) defers the actual transform write to the ship's next `_integrate_forces`; snapping immediately would read the pre-teleport position. Goal-cut shake decay extracted into `_decay_shake()`, called from the `_goal_cut_active` branch. Validated: scripts compile, Free Play renders correctly non-headless, reset produces no camera jump, all three headless scenes exit clean | `scripts/ship_camera.gd`, `scripts/game_mode.gd:reset_ships` | Turbo FOV kick and post-process are smooth at an uncapped frame rate; shake reads the same at 60 and 240 fps; a kickoff cuts the camera rather than lerping it across the arena | -| 0.17 `[D:0.15b]` | **DONE.** `VideoSettings` gains `Preset` (Low/Medium/High/Custom) driving a bundle (`sdfgi_enabled`, `ssil_enabled`, `ssao_enabled`, `shadows_enabled`, `glow_enabled`, `aa_mode`, `resolution_scale`) via `apply_preset()`; a `settings_changed` signal lets an already-loaded arena re-apply live (`arena.gd` connects in `_ready()`) rather than only affecting the next arena load — meets "settings persist and apply without a restart" without needing a scene reload. Shadow gating targets the actual `Light3D` nodes (found once at load via `find_children`, cached, re-applied on every settings change — deliberately *not* re-derived from current state each time, since a light this code just turned off would otherwise become indistinguishable from `FillLight`, which is authored `shadow_enabled = false` on purpose and must never be turned on by the preset ladder). `vsync_mode` (Disabled/Enabled/Adaptive, **Adaptive default**) and `fps_cap_divisor` (0 = uncapped, else divides the live refresh rate at apply time rather than storing a raw fps number, so the same preference re-derives correctly on a different display) added to the settings menu; FPS cap dropdown is `disabled` (greyed) unless VSync is Disabled; refresh-rate query ≤0 falls back to "Uncapped" only. Live fps readout via `_process` reading `Performance.TIME_FPS`. `main_menu.gd`'s `_leave_to_gameplay` now calls `VideoSettings.apply_fps_cap()` instead of hardcoding `Engine.max_fps = 0`, so the player's cap actually reaches gameplay scenes. **Acceptance numbers: not run as a literal Low-vs-High preset A/B, but strongly implied by §5.5.2** — real hardware (RTX 3090) runs the *High*-equivalent (all effects on) at 540 fps p50 already, so Low (which additionally turns off the two dominant costs, SDFGI+SSIL) clearing "≥2×" is close to guaranteed rather than measured directly; the flat-p99-histogram claim genuinely wasn't tested (`gpu_profile_harness.gd` measures per-toggle cost, not vsync/cap histograms) | `scripts/video_settings.gd`, `scripts/settings_menu.gd`, `scenes/settings.tscn`, `scripts/arena.gd`, `scripts/main_menu.gd` | Low preset ≥2× the frame rate of High on the same hardware; settings persist and apply without a restart; every offered cap gives a flat frame-time histogram (p99−p50 < 1 ms) with VSync disabled on a 144 Hz **and** a 165 Hz display; refresh-rate query returning `-1` falls back cleanly | -| 0.17b `[D:0.15b]` `[P]` | **DONE.** `VideoSettings.resolution_scale` (0.5–1.0, default 1.0) drives `Viewport.scaling_3d_mode`/`scaling_3d_scale`/`fsr_sharpness` via `apply_resolution_scale()` — `SCALING_3D_MODE_FSR2` below 1.0 (chosen over bilinear: this project already gave up native resolution at the fixed-1080p blit per 0.17c, so FSR2's sharpening recovers more of that loss than a plain bilinear upscale at the same internal scale), `SCALING_3D_MODE_BILINEAR` with scale pinned to 1.0 at the top of the range (a no-op scaling mode when the scale is 1:1). Low preset defaults to 0.8. Exposed as a slider in the settings menu; **not yet measured against the "0.7 scale gives a large, measurable frame-time drop" bar** — same real-hardware caveat as 0.17 | `scripts/video_settings.gd`, `settings_menu.gd` | 0.7 scale gives a large, measurable frame-time drop with acceptable image quality; setting persists | -| 0.17c `[D:0.17b]` | **DONE — decided, not changed.** Kept `stretch/mode="viewport"` fixed at 1080p rather than moving to `"disabled"`, documented inline in `project.godot [display]` with rationale: 0.17b's `scaling_3d_scale` already covers "render lower than the window" independently of stretch mode (it scales the 3D viewport's internal resolution before this blit, not the window itself), and separately, task 0.15b found an unexplained ~6% non-uniform width scaling on the one machine this was tested on (2036×1080 measured against a 1920×1080 target — see §5.5.1) that needs understanding before stretch mode is touched, not blindly carried into a resolution-dependent change | `project.godot` | The decision and its rationale are written into §5.5; render resolution follows the player's setting | -| 0.17d `[P]` | **INVESTIGATED — no such lever exists in Godot 4.7.** Searched the full `project.godot` schema (`read_project_settings`) for `rendering/rendering_device/vsync/frame_queue_size` and every variant (`frame_queue`, `swapchain`, `present`, `present_queue`) — none exist as a project-settable parameter in this engine version; the RenderingDevice backend may manage its own present queue internally but doesn't expose it. Adaptive vsync (task 0.17, done) is the only half of "L4" actually achievable through project settings. The §5.6 ~17 ms figure for a shallow present queue is therefore **not obtainable as specced** — closing this without a code change is correct here, not a shortfall; reaching it would need engine-level (C++/RenderingDevice) changes out of scope for a project-settings task | -| 0.18 `[P]` | **DONE, with one discovered GDScript constraint.** New `scripts/sim_constants.gd` (`class_name SimConstants`, plain `const TICK_HZ := 60`, not an autoload) is the source of truth for `ship.gd`'s `_tick_scaled` and `training_mode.gd`'s `TICKS_PER_SIM_SECOND` — both reference it via `const SimConstants = preload("res://scripts/sim_constants.gd")` rather than the bare global `class_name` symbol, because a cross-script `const X := f(OtherClass.CONST)` initializer needs the reference resolved before the global class table is guaranteed populated. **`@export_range()` upper bounds cannot take even a preloaded reference** — export hint arguments must be true literals — so `reaction_ticks`/`bot_*_reaction_ticks` (`ai_ship_controller.gd`, `match_mode.gd`, `spectate_mode.gd` ×2) stay at a literal `60`; these are editor-inspector slider bounds, not the timing math itself, so this doesn't reopen the bug the task exists to close, but it means the acceptance criterion below is met for tick-rate math and not for export-hint bounds | `ship.gd`, `training_mode.gd`, new `scripts/sim_constants.gd` | Tick-rate-derived timing math has no bare `60`; changing `TICK_HZ` changes `_tick_scaled` and `TICKS_PER_SIM_SECOND` coherently. `reaction_ticks` export bounds remain literal by GDScript necessity | -| 0.19 `[P]` | **DONE.** `AAMode` gained `MSAA_2X`, appended (not inserted) so existing `user://settings.cfg` ordinals keep their meaning; default `aa_mode` changed to `FXAA`; `settings_menu.gd`'s `AA_OPTIONS` now lists five entries | `video_settings.gd`, `settings_menu.gd` | Five AA options; default is FXAA; existing saved preferences migrate without resetting | -| 0.20 `[P]` | **DONE.** New autoload `scripts/perf_overlay.gd` (`PerfOverlay`), toggled by a new `toggle_perf_overlay` input action (F3 default). Headless-guarded; builds its own `Label` in code rather than touching `HUD.tscn` | new `scripts/perf_overlay.gd`, `project.godot [input]` | `TIME_PROCESS` vs total frame time tells the player whether they are CPU- or GPU-bound | -| 0.21 `[P]` | **DONE.** Shared `HudInstrument._throttled_redraw(delta)` paces `queue_redraw()` to ~60/s; value smoothing itself still runs every `_process` call, only the repaint is throttled | `scripts/hud_instrument.gd`, `scripts/hud_gauge.gd`, `scripts/hud_attitude_indicator.gd`, `scripts/hud_heading_tape.gd` | HUD is visually identical; instrument `_draw` call count is capped at ~60/s regardless of frame rate | -| 0.22 `[P]` | **DONE.** `Engine.max_physics_steps_per_frame = 4` set in `GameMode._ready()`, applies to every mode including headless Training | `scripts/game_mode.gd` | A client throttled to 20 fps degrades smoothly instead of compounding | -| 0.23 `[P]` | **DONE.** New autoload `scripts/background_fps.gd` (`BackgroundFPS`) drops to 30 fps on `NOTIFICATION_APPLICATION_FOCUS_OUT` / restores on focus-in, independent of scene. `main_menu.gd`/`settings_menu.gd` each cap to `DisplayServer.screen_get_refresh_rate()` in `_ready()` (falling back to uncapped on a `-1` query); leaving the main menu for a gameplay scene uncaps again via a new `_leave_to_gameplay()` helper, since gameplay has no cap of its own yet (0.17) | new `scripts/background_fps.gd`, `main_menu.gd`, `settings_menu.gd` | An unfocused window and an idle menu both stop rendering at 900 fps | -| 0.24 `[P]` | **DONE.** Both guarded with `if DisplayServer.get_name() == "headless": return` — `arena.gd:_ready()` skips the whole Environment block, `video_settings.gd:_ready()` skips `apply_aa()` | `scripts/arena.gd`, `scripts/video_settings.gd` | `--headless` allocates no Environment and no AA state | -| 0.25 `[P]` | **DONE.** `_process` still calls `to_local()` every frame (needed for the comparison itself) but skips `set_shader_parameter()` — the actual GPU-facing cost — below a 0.05 m movement threshold | `scripts/arena_boundary.gd` | Field shader behaves identically; the expensive call is skipped on most frames | -| **0.26** `[D:0.15b]` | **Bake the arena GI and retire SDFGI** (§5.7). `arena.gd`/`goal.gd` have no `_process`, no animation — the arena is fully static, and SDFGI is paying continuously to solve a dynamic-world problem this project does not have. Add UV2 to the arena shell, bake `LightmapGI` (or `VoxelGI` if bounce onto ships matters), disable `sdfgi_enabled` and re-evaluate `ssil_enabled` | `scenes/arena_base.tscn`, `scenes/arena_0*.tscn`, `scripts/arena_boundary.gd` | **Largest frame-time reduction of any task here, with equal or better image quality**; High preset keeps its look; bake is reproducible from a documented step | -| **0.27** `[P]` | **DONE.** `lights_and_shadows/positional_shadow/atlas_size` and `directional_shadow/size` set to 2048 (from the 4096 engine default), `soft_shadow_filter_quality=2` | `project.godot` | Measurable frame-time reduction; no visible shadow-quality regression at 1080p | -| **0.28** `[D:0.15b]` | **CLOSED, not implemented — the problem it targets doesn't exist.** Was: prototype `physics/3d/run_on_separate_thread` (§5.7) to attack frame-time variance from the physics tick sharing the render thread — **the riskiest item in this phase**, since it changes when `_integrate_forces` runs relative to script code, and both `ship.gd:346-357` and the RL training path depend on that. §5.5.2's real-hardware measurement (RTX 3090) found a 1.85 ms p50 / 2.98 ms p99 baseline with every graphics effect enabled — both comfortably under even a 240 Hz frame budget, with no meaningful p99-over-p50 variance to explain away. Taking on this task's real risk (reordering `_integrate_forces` relative to script code, with the RL training path depending on today's ordering) for a variance problem that isn't measurably present is a bad trade. Reopen only if a lower-end-hardware pass (§5.5.2's "still open" item) finds real physics-tick-driven variance that 0.26 and the preset ladder don't already cover | — | *(closed without a code change; see §5.5.2 for the evidence)* | -| **0.29** `[P]` | **DONE.** Bounds check against `wall_range`/`ceiling_range` at the top of `get_surface_pull`, returning `Vector3.ZERO` before `to_local()` and the five `_falloff` calls whenever every term would be zero mid-arena | `scripts/arena_boundary.gd` | Identical flight feel and identical RL observations; measurable tick-time reduction with 7 bodies | - -> **These tasks exist because of the high-refresh-rate mandate, and their order matters.** **0.15b blocked everything else, and did invalidate the a priori fps list** — but not in the direction first assumed (see §5.5.1 vs §5.5.2): on the Mac the game looked GPU-bound and undifferentiated; on real reference hardware (RTX 3090, §5.5.2) it runs at 540 fps p50 with everything on, nowhere near bound by anything. 0.17/0.17b/0.19 (done) are still the right frame-rate levers — SDFGI/SSIL genuinely dominate the optional-effects cost, exactly as originally assumed, just at a much smaller absolute scale than feared on this hardware tier. 0.16 and 0.20–0.25 are the per-frame hygiene that makes a high frame rate worth having. 0.18 buys nothing today — it is what keeps a future 120 Hz simulation a config change plus a retrain rather than a protocol rewrite. 0.28 closed without a code change (§5.5.2) — the frame-time variance it targeted isn't measurably present on reference hardware. -> -> **0.19–0.29 are all pure single-player wins with no netcode content.** If the multiplayer effort is ever paused, they should still land. Within them, **0.26 (bake the GI) is the largest single frame-time win in the document and costs no image quality** — the arena is fully static, so SDFGI is paying continuously for a problem this project does not have (§5.7). **0.28 is the riskiest**; it is the only Phase 0 task that can plausibly need reverting. - -> **Correction — task 0.2 is wider than an earlier draft claimed.** That draft argued the refactor was "narrow" because `_build_merged_hull` and `_build_movement_vfx` "only `add_child()`". That is exactly the problem: they `add_child()` onto **`self`, the `RigidBody3D`** — `ship.gd:208` (MergedHull: Hull, Canopy, EngineGlowL/R), `:241` (engine cores), `:268` (flames), `:278` (lights). Leave those and §4.4's soft correct offsets only `Nose` and `TailFin` while the hull, canopy, glows, flames and lights stay welded to the corrected collider — **every correction visibly tears the ship in half.** The old acceptance criterion ("looks identical in Free Play") passes either way, which is why the criterion is now a child-type assertion. `ship.gd:218`'s controller `add_child` correctly stays on the body; `CollisionShape3D` stays on the body. -> -> Still true from that draft, and re-verified: `ship.gd:44-47` documents why `Nose`/`TailFin` remain separate `MeshInstance3D`s, and **the RL path is untouched** — `ship_observations.gd` reads only `global_position`, basis, velocities and `PhysicsServer3D` contacts, and `training_mode.gd`'s only `get_node` is `arena.get_node("Boundary")`. - -**Phase gate:** the game plays identically to `master` in Free Play, Match, Spectate, and headless Training, with `Engine.time_scale` never written — **and additionally: §5.5 contains a real measured frame-time table (0.15b), the Low preset roughly doubles the frame rate of High (0.17), and the game looks correct uncapped on a high-refresh display** with no 60 Hz stepping in FOV, shake or post-process. - -### Phase 1 — Transport, connection, lobby - -| # | Task | Acceptance | -|---|---|---| -| 1.0 | **DONE, two real bugs found and fixed after adversarial review.** `tests/test_runner.tscn` + `test_runner.gd`: discovers every `*.gd` under `tests/cases/`, instances it, calls every `test_*()` method via `get_method_list()`, aggregates failures, `get_tree().quit(1 if failed else 0)`. `tests/test_case.gd` is the assertion base (`assert_true`/`assert_eq`/`assert_almost_eq`); case scripts use `extends "res://tests/test_case.gd"` (path-based) and the runner uses `preload()`, not a bare `class_name` reference — the global script-class cache isn't guaranteed populated on a fresh headless run (same class of issue as task 0.18's `SimConstants`). `tests/cases/test_smoke.gd` proves discovery/dispatch/aggregation and is the first real case file. **An Opus subagent's adversarial review found**: (1) GDScript has no exceptions, so a test that hit a runtime error before its first `assert_*` call left `failures` empty — exactly like every assertion passing — and was silently counted as a PASS. Fixed: `TestCase` now tracks `assertions_made`, incremented by every `assert_*`; the runner treats zero assertions as a failure in its own right ("made no assertions"). (2) A case file with a parse/compile error hung the whole runner forever — `load()` on a broken script does **not** return null here, it returns a non-null but uninstantiable `GDScript` resource, so a plain null check doesn't catch it; calling `.new()` on it threw an error severe enough to abort `_ready()` before ever reaching `quit()`. Fixed with `Script.can_instantiate()` as the real guard | `godot --headless --path Game res://tests/test_runner.tscn` runs and exits 0; verified exit 1 with a deliberately-failing assertion, then removed. Re-verified both fixes with scratch case files (not committed): a test that null-derefs before asserting now correctly fails with "made no assertions" (exit 1, not a false pass); an uncompilable case file now fails loudly and promptly (exit 1, not a 124-timeout hang) while the *other* valid case files in the same run still execute normally | -| 1.1 `[D:1.0]` `[D:0.18]` | **DONE.** `scripts/net_codec.gd`: protocol constants, `PacketType` enum, channel ids, i16/i8/thrust-z-bin quantisers, `pack_input`/`unpack_input`, `pack_snapshot_body_segment`/`pack_snapshot_client_header`/`pack_snapshot`/`unpack_snapshot`. New `scripts/net_body_state.gd` is the plain per-body data holder the snapshot functions read/write (not Ship/Ball themselves, so the codec stays callable with no scene tree). `NetCodec.TICK_HZ` derives from `SimConstants.TICK_HZ` via `preload()` (same cache-timing reason as 0.18); ring sizes / seq windows / `INTERP_DELAY` / timeouts don't exist as constants yet — they land with the tasks that consume them (3.1+), so "derives from `TICK_HZ`" is satisfied for what exists today | `scripts/net_codec.gd`, `scripts/net_body_state.gd`, `tests/cases/test_net_codec.gd` | 14 tests pass (`godot --headless --path Game res://tests/test_runner.tscn`, exit 0): input round-trip (1 and 4-entry, redundancy clamp), snapshot round-trip across 7 bodies incl. quaternion sign-fold and ship→ball angular-velocity rescale, thrust-z bin edges, type/version nibble round-trip. Byte counts asserted against §2.3/§2.4's numbers directly: 40 B input (max redundancy), 169 B snapshot (7 bodies) | -| 1.2 `[D:1.1]` | **DONE, strengthened after adversarial review.** `scripts/network_manager.gd` autoload (`NetworkManager` in `project.godot [autoload]`): `host(port, max_clients)`/`join(address, port)`/`shutdown()`, `client_connected`/`client_disconnected`/`connected_to_server`/`connection_failed`/`disconnected_from_server` signals forwarded from `multiplayer`'s own, `server_relay = false` set the moment a peer exists, `is_server`/`is_client` state. Gained a `shutting_down()` signal, emitted at the top of every `shutdown()` regardless of role or reason — see task 1.4's row for why | An Opus subagent's adversarial review (independently verified by the primary session before applying fixes) found the original `tests/net_smoke.gd` only proved each process exits cleanly on its own initiative, never that the OTHER peer actually observes the disconnect. Rewrote it: the host now waits for **both** `client_connected` and `client_disconnected` before passing; the client explicitly calls `shutdown()` mid-test (not just on process exit) and gives it a beat before quitting, same reasoning as §9 gotcha 26 for connects — a clean disconnect notice still needs a few `poll()` cycles to reach the wire, or the other side falls back to its ~5s peer timeout (gotcha 11) instead of a prompt one. Re-verified passing with both directions actually observed | -| 1.3 `[D:1.2]` | **DONE for what exists today.** `NetworkManager._ready()` calls `get_tree().set_multiplayer_poll_enabled(false)` (Godot 4.7's actual method name — the doc's `set_multiplayer_poll(false)` was shorthand) and exposes `NetworkManager.poll()` as the one entry point every caller uses instead. Verified against `tests/net_smoke.gd`, updated to poll from both `_process` and `_physics_process` every frame — connect/disconnect still works cleanly under manual-only polling (§9 gotcha 26 still applies: give a beat after a connect signal before shutdown). **The per-call-site placement this task specifies (client: end-of-physics-tick flush after input send, top-of-frame receive; server: tick-start drain, tick-end flush) has no real per-tick caller yet** — there is no input/snapshot traffic until tasks 1.4+/Phase 2 exist to send any, so there's nothing to place a flush *after*. That placement, and the RTT/staleness measurement below, land with the input pipeline, not as a separate task | `godot --headless` two-process test still connects/disconnects cleanly with automatic polling off (verified). RTT/staleness improvement **not yet measured** — deferred until Phase 2/3's real per-tick traffic exists to measure against, same honesty as task 1.1's "constants that don't fully exist yet" | -| 1.4 `[D:1.2]` | **DONE.** `scripts/match_net.gd` autoload (`MatchNet`): `_hello`/`_welcome`/`_player_joined`/`_player_left`/`_rejected` RPCs, `protocol_version` (`NetCodec.PROTOCOL_VERSION`) and `physics_ticks_per_second` (`SimConstants.TICK_HZ`) checked on the server before a peer is added to `roster`; on mismatch, server sends `_rejected` with a readable string then `disconnect_peer()`s after a 0.3s beat (§9 gotcha 26 applies here too — a bare RPC then immediate disconnect would drop the rejection message). `roster: Dictionary[int, PlayerInfo]` never contains peer 1 (§1.1 decision 2). A new peer is told about the existing roster via targeted RPCs before the broadcast that tells everyone (including itself) about the new peer, so no client ever observes an unexplained peer_id | Verified with a real two/three-process test (`tests/match_net_smoke.gd`/`.tscn`): matched client → both sides see `player_joined`/`welcomed`; deliberately wrong protocol version → client receives `rejected("protocol version mismatch: server=1 client=100")` and is disconnected. Caught and fixed one real bug in the process: the server's own `roster` update in `_hello()` didn't locally emit `player_joined` (the broadcast RPC is `call_remote`, never loops back to the sender) | -| — | **Two more real bugs found by an Opus subagent's adversarial review, both confirmed independently and fixed.** (1) `_hello`'s `player_name` was completely unvalidated and broadcast verbatim to every peer — a demonstrated DoS: a multi-MB name relayed to all peers head-of-line-blocked the reliable control channel hard enough that a concurrently-joining client's own `_welcome` never arrived. Fixed with a hard `MAX_INPUT_LENGTH = 256` reject (any legitimate client only ever sends `local_player_name`, which the UI already keeps short — anything past this is a bug or an attacker, not a name to politely truncate) followed by `_sanitize_player_name()`: strips control/formatting characters, clamps to `MAX_PLAYER_NAME_LENGTH = 24`, falls back to `"Player"` if empty. (2) `MatchNet.roster` was never cleared when a HOST stopped hosting — only the client-side disconnect path cleared it, so Host → Lobby → Leave → Host again left a phantom player in `roster` permanently, mis-balancing teams and getting broadcast to every future joiner. Fixed via `NetworkManager`'s new `shutting_down()` signal (task 1.2), which `MatchNet` now clears `roster` on unconditionally, regardless of role or reason | `_sanitize_player_name` is `static` (pure function of its argument) with 5 dedicated unit tests in `tests/cases/test_match_net.gd`, plus a live rejection test (`match_net_smoke.gd --role=client-longname`, a 500 KB name, confirmed rejected before ever reaching a broadcast). New regression test `match_net_smoke.gd --role=host_recycle`: host, client joins (`roster.size()==1`), host leaves and re-hosts, confirms `roster.is_empty()` before any new connection — reproduced the bug pre-fix, confirmed fixed post-fix | -| 1.5 `[D:1.4]` | **DONE, strengthened after adversarial review.** `scenes/lobby.tscn` + `scripts/lobby.gd`: roster split into two team columns (dynamically rebuilt `Label` rows on `MatchNet.player_joined`/`player_left`/`player_state_changed`/`welcomed`), Switch Team + Ready `CheckButton` (server process gets a read-only view — never a roster member, §1.1 decision 2), Leave. `MatchNet` grew `team`/`ready` fields on `PlayerInfo`, a balanced-team auto-assign on join (`_pick_balanced_team`), and `request_set_team`/`request_set_ready` + their server-authoritative RPCs, broadcasting `_state_changed` the same way `_player_joined` already did | Verified with a real two-process test (`tests/lobby_smoke.gd`/`.tscn`) that loads `lobby.tscn` via `change_scene_to_file` exactly as `main_menu.gd`'s Host/Join flow (task 1.7) does, then presses the real `%SwitchTeamButton`/`%ReadyButton` nodes via a persistent test-only helper (`tests/lobby_test_hooks.gd`, not a project autoload — parented under `get_tree().root` so it survives the scene swap, never referenced by production code). **An Opus subagent's adversarial review found the original test's host role never actually loaded `lobby.tscn` at all** — it only hosted and waited, so `lobby.gd`'s `is_server` branch (the read-only view a self-hosting player reaches via `main_menu.gd`'s own Host button — a real, production-reachable path, not a hypothetical) had never run under this task's own suite. Fixed: the host role now loads `lobby.tscn` too and a new `run_host_test()` in the shared test helper verifies `%ControlsRow` is hidden, the roster row renders, and the status text is correct, holding the connection open long enough (`MIN_HOST_LIFETIME_SECONDS`) for the client's own longer flow to finish against it. Confirmed: roster renders correctly server- **and** client-side (now genuinely, not just asserted), team switch moves the row to the other column, ready toggle updates the checkbox and the label's ✓ marker, row count matches roster size on both peers | -| 1.6 `[D:1.4]` `[P]` | **DONE.** `scenes/server_boot.tscn` + `scripts/server_boot.gd`: `--port=`/`--max-clients=`/`--log-level=` from `OS.get_cmdline_user_args()`, `Engine.max_fps = 60`, structured `[elapsed] LEVEL event key=value…` log lines for `server_started`/`peer_connected`/`player_joined`/`player_left`/`peer_disconnected`, and a physics-overrun watchdog comparing `Engine.get_physics_frames()` deltas frame-to-frame. Does not spawn a match yet — that's Phase 2's `networked_match.gd` — this is just the process shell: listen, log, idle cheaply. **Two real bugs caught and fixed while verifying, both in the watchdog**: (1) the very first `_process()` after boot compared against a pre-`_ready()` baseline and logged a spurious one-time `steps=5`; skip the first measurement. (2) the initial `steps > 1` threshold fired continuously (every 30–100ms) on a perfectly idle, healthy server — because §9 gotcha 6 means frames legitimately alternate between 0 and 2 physics ticks under `physics_jitter_fix = 0.0`, not a flat 1/frame; that's quantisation, not backlog. Raised the threshold to `steps > 2` (3+ ticks = the accumulator actually failing to drain), which produced zero false positives over a 4.8s idle run | Verified with real headless runs: idle CPU measured via `ps -o %cpu` at 0.0% (bar is <5%); a real client connect/disconnect via `tests/net_smoke.gd --port=` produces exactly the expected 4-line log sequence with no spurious warnings | -| 1.7 `[D:1.5]` `[P]` | **DONE.** `main_menu.tscn` gained a Multiplayer section (Host button; Join row with an IP `LineEdit`, default `127.0.0.1`; inline error label) and a full-screen `ConnectingOverlay` (status label + Cancel). `main_menu.gd`: `_on_host_pressed` calls `NetworkManager.host()` then goes straight to `lobby.tscn` (synchronous — no overlay needed); `_start_join` calls `NetworkManager.join()`, shows the overlay, and starts an app-level `CONNECT_TIMEOUT_SECONDS = 6.0` timer; `_on_connected_to_server`/`_on_connection_failed`/Cancel/timeout each resolve to the overlay hiding and either `lobby.tscn` or a visible error, gated by a token counter so a late/stray signal after the attempt was already resolved is a no-op | Verified with real multi-process runs of `scenes/main_menu.tscn` itself (not a wrapper — driven by a temporary-autoload test helper, `tests/main_menu_test_hooks.gd`, pressing the real `HostButton`/`JoinButton`/`ConnectingCancelButton`) across all four paths: Host → `lobby.tscn`; Join → connects → `lobby.tscn`; Join with nothing listening → times out → error shown, stays on menu; Join → Cancel → overlay hidden, stays on menu, `is_client` false. **Two real bugs found and fixed in the process, both pre-existing from earlier Phase 1 tasks, not new to 1.7**: (1) `NetworkManager`'s clock ping (task 1.8) gated only on `is_client`, which turns true the instant `join()` is called — a slow or refused connect attempt spammed "Trying to call an RPC via a multiplayer peer which is not connected" every frame; fixed by also requiring `_peer.get_connection_status() == CONNECTION_CONNECTED`. (2) ENet's own `connection_failed` proved **unbounded in practice** — verified empirically against a genuinely refused loopback connection, it hadn't fired even 14s in — which would have left a player staring at "Connecting…" indefinitely; task 1.7's own `CONNECT_TIMEOUT_SECONDS` is what actually satisfies "connection-refused reaches a sane UI state", not the built-in signal alone | -| 1.8 `[D:1.2]` `[P]` | **DONE, strengthened after adversarial review.** Folded into `network_manager.gd`: client pings the server once a second (`_ping`/`_pong` RPCs, reliable, channel 0); `clock_offset_ms` is the min-RTT sample in a rolling 5s window (`_clock_samples`, pruned by wall time); `get_server_time_estimate_ms()` is the public API later phases (`INTERP_DELAY`, `tick_offset` seeding) will actually call; `clock_updated(rtt_ms, offset_ms)` signal for observers. New `scripts/net_debug_overlay.gd` autoload (F4, `toggle_net_overlay` input action) mirrors `perf_overlay.gd`'s headless-guarded pattern, shows RTT + offset client-side or peer count server-side | Verified with a real two-process test (`tests/clock_smoke.gd`/`.tscn`) on localhost: first sample at t=0.95s, offset converged to 1534.50ms by t=2.0s (well inside the 2s bar), and stayed within 1.5ms of that value through t=3.96s — comfortably under the ±1 tick (16.67ms) bar. **An Opus subagent's adversarial review correctly pointed out this self-consistency check couldn't have caught a *systematically*-wrong-but-stable offset** (e.g. a missing `/2` on RTT, or a sign flip — it would converge just as cleanly). Fixed by adding an independent ground-truth cross-check: both host and client compute `Time.get_unix_time_from_system()*1000.0 - Time.get_ticks_msec()` (each process's own offset from the shared OS wall clock — the *same* real clock on both, since they're on the same machine), exchanged via a shared temp file written by the host, purely for test orchestration and touching no production code. The true required offset is just the difference of those two numbers; re-run measured the converged offset against it and found **0.99ms of error**, comfortably inside a deliberately loose 250ms tolerance (OS wall-clock read resolution and sampling-instant skew, not NetworkManager's own precision, is what sets the tolerance floor here). Note the converged offset *value* itself is large and arbitrary (~1.5s) because `Time.get_ticks_msec()` counts from each process's own start, not a shared epoch — expected, and exactly what `clock_offset_ms` exists to absorb | - -> `main_menu.gd` gains its **first async flow**. Every existing handler is `GameSettings.x = y; change_scene_to_file(...)` — there is no loading screen, no error state, and no back-navigation state machine to extend. Budget for that. - -**Phase gate:** two clients connect to a headless server, appear in a shared lobby, ready up, and disconnect cleanly. - -### Phase 2 — Server-authoritative simulation, dumb client - -No own-ship prediction yet: the client renders everything, including its own ship, from the interpolation buffer. Unplayable over the internet, fine on LAN, and it proves the whole state pipeline before prediction complicates the picture. - -**This phase is load-bearing, not throwaway** — the codec, slot mapping, snapshot pipeline, interpolator and HUD signal surface all survive into Phase 4. Roughly ten lines get discarded. - -| # | Task | Acceptance | -|---|---|---| -| 2.1 `[D:1.4]` | **DONE.** New `MatchSim` autoload (`scripts/match_sim.gd`) carries all Phase 2 hot-path RPCs (`match_config`, `input`, `snapshot`, `score_update`) per §1.1's "hot RPCs live on autoloads" decision — `NetworkedMatch` itself (`scripts/networked_match.gd` + `scenes/networked_match.tscn`, no HUD child) stays a plain scene node with no networking identity of its own. Server builds deterministic team/spawn-index slots by iterating `MatchNet.roster.keys()` sorted, loads a random arena via `ArenaRegistry.random_path()`, spawns ball/ships, then `send_match_config()`s. Client validates the received `arena_path` against `ArenaRegistry.ARENAS` before loading it | Both peers spawn an identical tree in real two-process runs (`tests/networked_match_smoke.gd`/`.tscn`); an invalid arena path is refused before load | -| 2.2 `[D:2.1]` | **DONE.** Server reuses **`RLShipController`** as the remote-input controller exactly as the architecture doc anticipated — each connected peer's real `Ship` is driven by one, fed by `MatchSim.input_received`. `_broadcast_snapshot()` runs every physics tick (60 Hz), packing `NetBodyState` for every ship + ball via `NetCodec.pack_snapshot_body_segment` and sending per-slot, filtered through `multiplayer.get_peers()` so a disconnected peer doesn't get an RPC send attempt | Server-side snapshot cadence confirmed stable at 60 Hz across multiple two-process runs; no "unknown peer ID" spam after the `get_peers()` filter fix (found via a real disconnect-mid-test case) | -| 2.3 `[D:2.2]` | **DONE.** New `scripts/net_interpolator.gd` (`class_name NetInterpolator`, `RefCounted`) buffers up to `MAX_SAMPLES=16` timestamped `NetBodyState`s per remote body and produces interpolated (or clamped-extrapolated, `MAX_EXTRAPOLATION_MS=150`) states at any fractional server tick via `sample_at()`. Client-side `_on_snapshot_received` feeds each body's decoded state into its interpolator; ships/ball spawn `FREEZE_MODE_KINEMATIC` so they never call `_integrate_forces`/`get_action()` | Client observed 31.43 m of real, physics-verified movement over a 2s held-thrust drive purely from interpolated snapshots, no local simulation | -| 2.4 `[D:2.3]` | **DONE — dual-time remote entities** (§4.1). Collider updates happen in `_physics_process` at `server_time_est` (present-time, correct contact resolution); `$Visual` updates happen separately in `_process` at `server_time_est - INTERP_DELAY` (`physics_interpolation_mode = OFF`, since the node's transform is overwritten every rendered frame). `_current_interp_delay_ms()` computes a simplified `INTERP_DELAY` (`one_way + interval*1.5`, clamped `[25,200]` ms) — no jitter term yet, that lands with Phase 3's jitter buffer | Verified via the smoke test's separate collider/visual checks; `Engine.get_physics_frames()`/`Time.get_ticks_msec()` epoch correlation (`NetInterpolator.to_tick()`) confirmed working with no extra sync handshake needed | -| 2.5 `[D:2.3]` `[P]` | **DONE.** `_send_local_input()` samples via a stateless, never-added-to-tree `PlayerShipController` instance (reading real `Input` state) and sends the resulting `ShipAction` every physics tick, no redundancy/buffering yet (Phase 3) | Input reaches the server and visibly moves the ship — confirmed via a real held `move_forward` keypress driving 31.43 m of server-authoritative movement | -| 2.6 `[D:2.3]` `[P]` | **DONE**, and empirically verified, not just inferred — turned out to already be satisfied as a natural consequence of 2.1–2.5's implementation (`_apply_ship_visual_state` already calls `set_visual_action` for remote ships; `_process`'s ball branch already calls `set_visual_speed`) | Smoke test explicitly reads `interpolator.latest().thrust_z` mid-drive and asserts `>0.5` while `move_forward` is held (not inferred from movement alone) — measured `thrust_z=1.00` | -| 2.7 `[D:2.3]` `[P]` | **DONE**, also a natural consequence of the above — `spawn_camera_rig(_my_slot.ship)` and `_spawn_hud()` are called once the client's own ship is identified in `_on_match_config_received` | Smoke test asserts `_camera_rig` and `hud` both `is_instance_valid()` on the client; confirmed true in every clean run | -| 2.8 `[D:1.1]` `[P]` | **DONE.** New `NetSim` autoload (`scripts/net_sim.gd`): seeded (`--net-sim-seed=`, fixed default so a bad run reproduces), CLI-driven (`--net-sim-latency=`/`--net-sim-jitter=`/`--net-sim-loss=`/`--net-sim-dup=`), a pure passthrough (`send()` calls the dispatch immediately) unless at least one flag is non-zero — confirmed byte-for-byte inert against every Phase 1/2 regression test with no flags set. Wraps `MatchSim.send_input`/`send_snapshot` per this row's original scope, **plus `NetworkManager`'s `_ping`/`_pong` dispatch** — a deliberate scope addition, since that's the only RTT measurement that already exists and is already tested (task 1.8), so it's what makes this task's own acceptance criterion checkable today without waiting on Phase 3's per-peer snapshot echo. "Asymmetric-capable" needs no special-case code: each process reads only its own CLI args and delays only its own outgoing sends, so hosting and joining with different flags is already asymmetric. **One correctness subtlety, caught before it shipped**: callers that embed a timestamp in a wrapped RPC (`_ping`/`_pong`) must capture `Time.get_ticks_msec()` *before* calling `NetSim.send()`, not inside the wrapped `Callable` — capturing it inside would silently absorb that process's own added delay out of the round-trip measurement instead of adding to it, since the timestamp would then reflect "after my delay" rather than "when I actually tried to send". **A second real bug, found by actually running Phase 2's own milestone gate** (a full `networked_match_smoke` run under `--net-sim-latency=80 --net-sim-jitter=20`, not just the isolated ping/pong test above): a delayed send can outlive the window its target was valid in — the host hit "Attempt to call RPC with unknown peer ID" (the client had disconnected during the ~80-100ms hold, after `_broadcast_snapshot`'s existing `get_peers()` filter had already passed at *schedule* time) and the client hit "'_recv_input' on yourself is not allowed by selected mode" (its own `shutdown()` had already reset `multiplayer_peer` to a fresh `OfflineMultiplayerPeer` before a still-pending delayed send fired, so peer id 1 now meant itself). Fixed by having `send()` accept an optional `target_peer_id` and re-validating it — plus that this process still has a real (non-Offline) peer at all — at *fire* time inside a new `_fire()`, not just at schedule time; the synchronous/inactive path is deliberately left unvalidated so NetSim stays a true no-op when idle | Verified with a real two-process test (`tests/net_sim_smoke.gd`/`.tscn`): baseline (no flags) observed `rtt_ms=7.00` on loopback; `--net-sim-latency=80` on the host alone raised the client's observed `rtt_ms` to `83.00` (want ≥70, confirmed measurably higher than baseline); `--net-sim-loss=1.0` on the host produced **zero** pong samples over 7s (`rtt_ms` stayed `-1`, confirmed the drop path actually drops rather than relabels). **Phase 2's own milestone gate re-run and passing**: `networked_match_smoke` under `--net-sim-latency=80 --net-sim-jitter=20` on both peers — client still observed 26.08m of clean server-authoritative movement via interpolation, `thrust_z=1.00` confirmed mid-drive, zero RPC errors (the fire-time-revalidation fix above). Re-ran the full Phase 1 + Phase 2 regression suite (`test_runner`, `net_smoke`, `match_net_smoke`, `clock_smoke`, `lobby_smoke`, `server_boot`, `networked_match_smoke`) with NetSim present but inactive — all still pass with unchanged behaviour (clock offset converged to the same value, `networked_match_smoke` still showed clean server-authoritative movement) | - -| — | **An Opus subagent's adversarial review of all of Phase 2 found real, verified bugs the smoke tests couldn't catch, since constant-velocity dead reckoning still moves a ship far enough to pass a `moved > 1.0` check.** Fixed, all independently re-verified with real two-process runs and temporary instrumentation (removed after confirming):

**(1) The interpolator never actually interpolated — every `sample_at()` call took the extrapolation branch, 100% of the time, LAN or under simulated latency alike.** `NetInterpolator.to_tick()` assumes `Time.get_ticks_msec() == physics_frame * TICK_MS` on the server; real engine/autoload startup work before the first physics step (plus any dropped tick, which only ever widens it) breaks that by a steady +45-55ms in practice. `networked_match.gd` now tracks one shared `_tick_bias_ms` estimate (`_update_tick_bias`, called from `_on_snapshot_received`) — the **minimum** `to_tick(server_time_est) - server_tick` over a rolling 5s window, same rationale as `NetworkManager`'s own min-RTT filtering: the least-delayed sample best isolates the constant bias from per-packet transit noise, and a rolling (not all-time) window still tracks a real future increase. `_estimated_tick()` subtracts it before every `to_tick()` call. Verified: bias converged to ~50-56ms (matching the bug's own measured magnitude exactly) and real interpolation rose from 0% to ~70% of calls (`interp=436 extrap=182` out of 618, up from `interp=0 extrap=617`). **A first attempt at this fix was itself broken and made the lead ~30x worse (90+ ticks, ~1.5s)**: early snapshots arrive before `NetworkManager`'s first clock pong lands (`rtt_ms < 0`, `clock_offset_ms` still `0.0`), so `server_time_est` briefly means "my own raw local uptime" — a wildly wrong bias sample that the 5s rolling-min then locked onto for a whole short test, since 5 real seconds never fully elapsed before the test ended. Fixed by skipping bias recording entirely while `rtt_ms < 0`.

**(2) Goals caused a ~27m visual slide.** `_on_goal_scored` bumped `_reset_gen` immediately, but `reset_ball()`/`reset_ships()` only *queue* teleports (task 0.15, applied on each body's next `_integrate_forces`) — so the broadcast that same tick carried the NEW gen with the OLD (still-in-goal) position, and the client's buffer-clear-on-reset kept exactly that stale sample and lerped a full-arena slide to the next, genuinely-reset one. **The first fix attempt (defer the bump to "the next `_physics_process`" via a plain boolean) didn't work either** — emperically, the goal Area's `body_entered` signal fires as part of physics tick N's own step, *before* tick N's `_physics_process` callback, so a flag set in the handler is already true by the time that same tick checks it: no delay was actually introduced. Fixed by recording the tick the goal was detected on (`_pending_reset_gen_bump_tick`) and only bumping once `Engine.get_physics_frames() > _pending_reset_gen_bump_tick` — i.e. strictly on a later tick, which guarantees the queued teleport's `_integrate_forces` has already run. Verified by forcibly teleporting the ball into a goal mid-test and logging the server's own broadcast stream tick-by-tick: gen change and the already-reset position now land in the identical broadcast, every time.

**(3) `_local_input_sampler` (a `PlayerShipController`, i.e. a plain `Node`) was created but never added to the tree and never freed** — this was the unexplained "3 resources still in use at exit" warning on every prior Phase 2 test run, confirmed by `--verbose` naming the exact leaked script chain and by the warning disappearing once a `_exit_tree()` cleanup was added. Also leaked on the **server** despite its "client only" comment, since the field initializer is unconditional.

**(4) Ball angular velocity decoded 8x too small** — `NetCodec.rescale_avel()` exists specifically to correct a ball's decoded `angular_velocity` from the ship-range assumption `unpack_snapshot()` decodes every body with, and was never called. Dormant today (nothing read decoded `angular_velocity` yet) but silently wrong the moment ball-spin VFX or Phase 4 prediction reads it; now called in `_on_snapshot_received`.

**(5) `get_server_time_estimate_ms()` was used unguarded before the clock had synced**, contradicting its own doc comment — against a long-running dedicated server this freezes every remote body at the oldest buffered pose for the whole first second of every match (`clock_offset_ms == 0.0` compares this process's own short uptime against the server's much larger tick count). Both `_physics_process` and `_process` now skip their collider/visual update entirely while `NetworkManager.rtt_ms < 0.0`.

**Smaller fixes, all confirmed via the regression suite**: `net_sim.gd`'s delayed-send timer now uses `process_always = true` (a simulated wire shouldn't stop just because the local game pauses) and `_fire()` also checks `get_connection_status() == CONNECTION_CONNECTED`, not just non-`Offline`, before dispatching (a known, accepted residual gap remains: a shutdown-then-reconnect inside one delayed send's hold window isn't fully closed, judged disproportionate to fix for debug-only tooling); `_broadcast_snapshot()` now appends one body per slot unconditionally (a zeroed placeholder for a momentarily-invalid ship) so the ball's fixed index assumption can't silently break if "no ship is ever despawned" (§6.4) ever stops holding; `networked_match.gd` now only declares `score_changed` (the one signal it actually emits) instead of also declaring `timer_updated`/`match_ended`/`kickoff_countdown`/`overtime_started`, which — despite never being emitted — made `HUDController` show a permanently frozen timer widget purely because `has_signal("timer_updated")` was true.

**Confirmed fine, not just assumed**, via a real hostile-client stress test and a real 3-process multi-client run: a malformed/garbage/oversized `_recv_input` payload cannot crash the server (Godot's `StreamPeerBuffer` silently zero-fills past EOF; `count` is a bounded `u8`); `NetworkedMatch` skipping `GameMode._ready()`'s `super()` call drops nothing load-bearing; deterministic team/spawn-index slot assignment is correct with 2 simultaneous clients (verified with a real 3-process host+2-client run); RPC authority enforcement on `_match_config`/`_score_update`/`_snapshot` genuinely rejects a forging client server-side | Full Phase 1 + Phase 2 regression suite (`test_runner`, `net_smoke`, `match_net_smoke`, `clock_smoke`, `lobby_smoke`, `networked_match_smoke` baseline and under the `--net-sim-latency 80 --net-sim-jitter 20` milestone gate, `net_sim_smoke`) re-run clean after every fix | - -> **`net_sim.gd` belongs in this phase, not Phase 3.** A LAN-only phase gate passes even with §4.1's flaw fully present, because LAN `INTERP_DELAY` sits at the clamp floor and closing-speed error is small. Phases 2 and 3 would both go green and Phase 4 would discover the architecture is wrong. - -**Phase gate — MILESTONE:** a real 1v1 **at `--net-sim-latency 80 --net-sim-jitter 20`**, not just on LAN. Ships fly, the ball moves, goals detect server-side. - -### Phase 3 — Input pipeline hardening - -| # | Task | Acceptance | -|---|---|---| -| 3.1 `[D:2.5]` | **DONE.** Client sends the last `NetCodec.MAX_REDUNDANCY` (4) ticks' actions per packet, newest-first (the wire format already supported this from Phase 1 — Phase 2 just wasn't using it). Server gains a real per-slot ring buffer, new standalone `scripts/input_jitter_buffer.gd` (`InputJitterBuffer`, `RefCounted`, no scene dependency — same reason `net_codec.gd`/`net_interpolator.gd` are pure classes), consuming exactly one sequence number per physics tick | Verified both by unit test (`test_redundancy_survives_3_packet_burst_loss`) and live: 25% random simulated input loss produced zero observed starvation ticks; 100% loss correctly produced zero seeding/consumption (no crash, ship simply never receives a command) | -| 3.2 `[D:3.1]` | **DONE.** `InputJitterBuffer.consume()`: repeat-last on starve, zero + `stalled=true` only after `STARVE_ZERO_TICKS` (30 = 500ms). `input_buffer_depth`/`last_input_seq`/`echo_client_send_ms` are now genuinely per-peer in every snapshot (`_broadcast_snapshot` builds them from each slot's own `InputJitterBuffer`), replacing Phase 2's hardcoded zeros | One real bug found wiring this into a live match: the server's ring buffer started counting its own "expected sequence" from a local 0 the instant a slot was created — well before that player's first real packet could possibly arrive (connection/spawn setup takes real time) — so the two numberings never converged and the ship silently never moved. Fixed by seeding `last_applied_seq` from the client's own numbering on first real `ingest()`, not assuming a shared from-zero baseline. Verified with real two-process runs before and after the fix | -| 3.3 `[D:3.2]` `[P]` | **DONE.** New standalone `scripts/input_lead_controller.gd` (`InputLeadController`, unit-tested like `InputJitterBuffer`): clamp `[1,12]`, fast attack (+3, debounced to once per 30 ticks) on any server-reported starve, slow release (−1 per 60 ticks) gated behind a one-time 2s clean-surplus bar. A lead change is realized as extra distance between the client's own outgoing seq and what the server has consumed — attack skips extra seq numbers, release duplicates (re-sends) the current one; the server's ring buffer needs no special handling for either, since a skip is an ordinary drop and a duplicate is a same-seq resend already discarded | Verified live: on a clean LAN, one early attack (a momentary connection-setup hiccup) recovered via two releases within ~4s, settling back near minimum; under sustained 30% simulated loss, lead climbed to 7 via repeated attacks and never released while genuine loss continued — confirming debounce, attack, and release gates all fire correctly on real conditions | -| 3.4 `[D:3.1]` `[P]` | **DONE.** `MatchSim._recv_input` validates before decoding: per-peer rolling-1s rate limit (packet count AND byte budget, §3.1's own numbers), disconnect after 3 consecutive over-budget seconds; framing (redundancy count + payload size checked against `NetCodec`'s own layout, since `StreamPeerBuffer` silently zero-fills past EOF instead of erroring — a Phase 2 adversarial-review finding), disconnect after 20 malformed packets. `networked_match.gd` additionally rejects `seq > server_tick + 20` and counts (rather than silently ignoring) input from a peer with no slot. Server-side `input_lead` enforcement from arrival times was scoped down to observability rather than active enforcement — see the note below the table | Two new **permanent** regression tests (`networked_match_smoke.gd --role=client-abuse-malformed` / `client-abuse-flood`) call `MatchSim._recv_input` directly with garbage bytes and a legitimate-but-too-frequent flood respectively — bypassing the honest client encoder entirely, i.e. exactly what a hostile custom client sending raw ENet packets looks like. Both confirm a real disconnect, not just tolerance. Found and fixed two smaller bugs getting these to pass cleanly: a GDScript lambda-captures-by-value mistake in the tests themselves (fixed by capturing a single-element `Array` instead of a plain `bool`), and a real race where `NetworkManager`'s own ping/pong reply could target a peer a concurrent abuse-disconnect had just removed from the same `poll()` batch (now guarded); the post-shutdown physics path now also uses the safe NetworkManager lifecycle flag and stays error-free | -| 3.5 `[D:3.2]` `[P]` | **DONE.** `tests/cases/test_input_jitter_buffer.gd` and `test_input_lead_controller.gd`: sequential consumption, redundancy surviving a 3-packet burst loss (3.1's own acceptance text, verbatim), starvation repeat-then-zero timing, stale/reordered-packet handling, buffered-depth reporting, ring-wraparound slot-tagging safety, and the full attack/debounce/release state machine including a starve mid-release-window forcing a fresh clean-surplus wait | 14 new tests, all passing (`test_runner.tscn`: 33 total, 0 failed) | -| 3.6 `[D:2.8]` | **DONE**, with one honest scope note. `networked_match.gd`'s client can swap its input sampler for a real `AIShipController` (`--test-bot`, optionally `--test-bot-model=`) instead of `PlayerShipController` — parented onto the client's own ship via `Ship.set_controller()` since (unlike the human sampler) it needs real scene context. **Known limitation, documented in code**: this client's ships are all `FREEZE_MODE_KINEMATIC`, driven purely by transform writes, so nothing ever writes `linear_velocity`/`angular_velocity` onto them — the bot's observations always see every ship as stationary. It still produces well-formed, bounded actions from that degraded input (the policy network's output layer is bounded regardless of input quality), sufficient for this task's actual job (CI traffic generation, not bot skill). New CI driver `tests/networked_match_ci.gd`/`.tscn`: headless server + two headless `--test-bot` clients. **This task's own original acceptance text names "p95/p99 prediction error" and "snap count" — both Phase 4 concepts that don't exist yet** (no client-side prediction or hard-snap threshold exists before Phase 4); asserting on data that doesn't exist would be fabricated, so those two are explicitly not checked, with the gap called out in the driver's own header comment rather than silently dropped | Real 3-process runs: both bots' independently-written final scores agreed after a deterministically forced goal (bot-vs-bot scoring isn't reliable enough within a short run to gate on), both saw 500+ snapshots over an 8s run (well above the 60Hz-scaled floor), all three processes exited 0. "Clean stderr" is the external invocation's job (grep the captured output), same as every other smoke test in this project — verified manually, not self-asserted by the script | -| 3.7 `[D:2.8]` `[P]` | **DONE**, with prediction error deliberately omitted (documented, not silently dropped — same Phase 4 gap as 3.6). Extends `net_debug_overlay.gd` with jitter (new RFC3550-style EWMA in `NetworkManager`, from raw per-sample RTT — Phase 1's `rtt_ms` is a min-filtered sample, deliberately jitter-insensitive by design, so it can't answer this on its own), snapshot loss (new EWMA in `networked_match.gd` over each received snapshot's own `server_tick` gap — snapshots go out at a steady one-tick cadence, so a gap is direct evidence of a drop or reorder), snapshot age (computed on demand from the same bias-corrected tick estimate the interpolator itself uses), input buffer depth and `input_lead` (both already tracked client-side for 3.3), and bandwidth (new rolling per-second byte counters in `MatchSim`, the two 60Hz hot-path channels only) | Verified values are live and plausible, not just present, by calling `get_net_debug_stats()` directly in a real two-process test: bandwidth matched the wire format's own byte math almost exactly (measured ≈2400 B/s sent against a computed 40B×60Hz, ≈3540 B/s received against 59B×60Hz for a 1v1), and buffer depth/lead/loss all moved in the correct direction between a clean LAN run and one under simulated 60ms latency + 10% loss | - -> `AIShipController` runs a policy in pure GDScript with no Python or ONNX dependency, so 3.6 gets a competent automated player for free. - -> **Server-side `input_lead` enforcement from arrival times (§3.3's closing paragraph) was scoped down to observability, not built as active enforcement.** The concrete, mechanically well-specified parts of task 3.4 (rate limiting, malformed-packet counting, seq-range rejection, disconnect policy) fully close the load-bearing security gaps; the advantage a client gains from claiming a dishonestly low `input_lead` is explicitly described in the doc itself as "small" (reduced apply latency, not an outright cheat — there's no prediction/reconciliation yet for a bad lead to actually corrupt), and building real arrival-jitter-derived enforcement well — without risking a third, subtly-interacting control loop on top of the two §3.3 already warns against — is a genuine design task in its own right, not a mechanical one. Revisit if Phase 4's prediction work turns "slightly lower latency" into a sharper edge. - -**Phase gate — MET.** Both `networked_match_smoke` and the CI driver (task 3.6) re-run under the gate's own exact condition, `--net-sim-latency 80 --net-sim-loss 0.05`, on every peer: the human-driven smoke test still shows clean server-authoritative movement (22.61m over the usual 2s drive), and the two-bot CI run still shows both clients independently agreeing on the final score after a forced goal, 495+/504 snapshots received each, all three processes exiting 0 with clean stderr. - -| — | **A second adversarial review of the fix commit above found that two of its nine fixes silently cancelled each other out, re-creating the original critical bug at a *lower* failure threshold — plus four smaller real issues, all re-verified with real two- and three-process runs.**

**CRITICAL — the seq-range guard fix (round 1's MEDIUM item, gotcha 42) made the ring-overflow resync fix (round 1's CRITICAL item, gotcha 39) unreachable in production.** The guard bounded every accepted `seq` at `last_applied_seq + RING_SIZE` — the *consumer's* position — which in turn caps `InputJitterBuffer`'s own `highest_ingested_seq` at that same ceiling, since nothing above the bound is ever allowed to reach `ingest()` at all. But `consume()`'s resync condition needs `highest_ingested_seq` to reach `expected + RING_SIZE`, one full ring past that same ceiling — arithmetically impossible on the only call path that exists. The two fixes read as independent (one in the jitter buffer, one in the caller) but shared a variable and quietly defeated each other; the round-1 commit's own new unit test for the resync never caught it because it called `ingest()` directly, bypassing the guard entirely — the exact composition the bug lived in. Verified failing on the committed code: a 0.6s `SIGSTOP` host freeze reproduced the original 0.00m death, at a *lower* threshold than the pre-round-1 bug (~0.6s vs ~0.7s), reachable via ordinary server tick loss with no external trigger at all (`Engine.max_physics_steps_per_frame = 4` means a server that falls behind wall-clock time during any stall never catches back up on its own). Fixed by rebinding the guard to `highest_ingested_seq` (now a public field, matching `last_applied_seq`'s own convention) instead of `last_applied_seq` — the client's actual send epoch, which `ingest()` updates once per accepted packet regardless of how far the consumer has fallen behind, rather than the consumer's own lagging position. Re-verified against a real 2-bot CI match with a 1.5s host `SIGSTOP` freeze injected mid-run (well past the 0.6s failure threshold): both peers kept moving (47.46m / 10.19m and, on a repeat run, 25.62m / 17.96m), `stalled=false`, sampled while genuinely still connected.

**HIGH — the `InputLeadController` release fix (round 1's HIGH item, gotcha 40) was itself incomplete.** Round 1 added a real depth check (`input_buffer_depth > TARGET_DEPTH`) but left the *original* gate, `and lead > LEAD_MIN`, still ANDed onto the same final condition — so a backlog the controller never caused (lead pinned at its own floor) still could never release, since that old clause always failed regardless of what the new depth check found. Confirmed by the round-1 commit's own new unit test, whose assertion text literally read "lead cannot release below its own floor even under large surplus" as if that were the intended behaviour. Fixed by splitting the one gate into two independent decisions: whether to duplicate this tick's seq (the only thing that actually narrows real buffered depth) now follows the depth signal alone; whether to keep decrementing `lead`'s own bookkeeping below its documented `[LEAD_MIN, LEAD_MAX]` floor is a separate, purely cosmetic choice made inside that branch.

**MEDIUM — task 3.6's CI gate (round 1's own fix for gotcha 43) still sampled after both bots had legitimately disconnected.** The fix used a `run_seconds - 0.5` margin, narrower than the original bug (sampling after the full run) but still not enough: `multiplayer.get_peers()` at sample time was already empty, and the check was only passing on `STARVE_ZERO_TICKS`'s own ~200ms of residual starvation grace, not because it was genuinely still connected as its own print claimed. Widened the margin to `run_seconds - 2.0` and added an explicit `slot.peer_id in multiplayer.get_peers()` assertion at sample time, so a future regression in either direction fails loudly here instead of silently passing on residual grace.

**LOW — the human smoke test's movement bar was beatable by gravity alone.** `moved > 1.0` measured full 3D distance; a 1.2s window of completely dead input still registered ~1.07m from pure vertical settling (spawn height dropping to the floor) — above the bar, with only the separate `thrust_z_ok` check actually catching the failure. Forward thrust is a horizontal force, so switched to XZ-only displacement, which gravity alone cannot satisfy regardless of spawn height or timing.

**LOW — "clean stderr" wasn't actually clean.** Every disconnect logged `ERROR: Unable to send packet on channel 0, max channels: 0` from `match_net.gd`'s `_remove_player`, which broadcasts `_player_left` to every peer in `multiplayer.get_peers()` — including, transiently, the peer that just disconnected (whose own ENet connection can still be momentarily present in that set with its channels already torn down), and — found only after the first fix still left an error in the 2-bot CI scenario specifically — including a *second* still-connecting peer when two clients disconnect within the same `poll()` batch, since `get_peers()` hadn't yet been updated for the one not currently being handled. Fixed by deferring the whole notification (`call_deferred`) to the next idle frame, by which point `poll()` has fully returned and every disconnect event in the batch has actually settled, then explicitly excluding the peer that left. Re-verified clean (grep for `ERROR`) across both the basic 2-process smoke test and a real 2-bot CI run with a mid-match host freeze injected.

**Noted, not fixed — a related but distinct stderr source in `_broadcast_snapshot`.** The deliberately-adversarial `client-abuse-malformed` smoke test still logs one `Unable to send packet` from `networked_match.gd`'s snapshot broadcast, racing a host-forced `disconnect_peer()` in `match_sim.gd`'s abuse-disconnect path against the same tick's `connected_peers.has(slot.peer_id)` snapshot — a different call site than the one just fixed, only reachable via the abuse-detection disconnect path rather than a normal client-initiated one, and out of scope for this pass. Left for a dedicated look rather than a rushed fix under this round's time pressure.

**Confirmed fully correct, not just re-asserted**: the `_input_history` fix (round 1's LOW-MEDIUM item) was re-verified via a synthetic-marker harness stamping a computable value into every outgoing action and checking it through a real 3-process match under 60ms+40ms jitter+18% loss — 1080 marker checks, 0 mismatches, including real attacks and releases; the leaky-bucket rate limiter (round 1's MEDIUM-HIGH item) cannot false-positive on honest traffic (~1.8x measured margin under real impairment); the resync boundary arithmetic itself is correct under packet reordering and duplication. **The lesson that mattered most this round wasn't any single fix — it was that two fixes landed in the same commit, each individually correct in isolation, that silently cancelled each other out** (see gotcha 45) | Full regression suite (35 unit tests, the basic 2-process smoke test, the malformed/rate-limit/duty-cycle abuse roles, and a real 2-bot CI run with a 1.5s host `SIGSTOP` freeze injected mid-match) re-run clean after every fix in this round - -| — | **An Opus subagent's adversarial review of all of Phase 3 found a critical, silent, permanent bug plus eight smaller real issues — all empirically verified with real two- and three-process runs, not just code reading.**

**CRITICAL — `InputJitterBuffer`'s 32-entry ring permanently bricked a player's input on any backlog bigger than the ring.** `consume()` advanced `last_applied_seq` by exactly 1 per tick with no resync; once the un-consumed backlog exceeded `RING_SIZE`, a fresh arrival would land in the exact slot `consume()` was still waiting on, and since both counters only ever advance, the gap never closed — the affected player's ship silently went to zero thrust for the rest of the match. The reviewer reproduced this with a real `SIGSTOP`/`SIGCONT` host freeze (a faithful stand-in for a GC/IO/scheduler hitch on a listen-server host): client movement dropped from ~26m to a flat 0.00m at ~0.7s of freeze, reproducible 4/4 times, and found the cliff got *worse* under real network conditions (a lossy link that had already pushed `input_lead` up lowered the fatal threshold to ~400ms) and could be reached with **no external trigger at all** via ordinary client/server clock drift (~1.7% faster client death-spiraled within ~60s). Fixed with a real resync mechanism: `ingest()` now tracks the highest seq ever seen regardless of ring capacity, and `consume()` detects when the gap to that value exceeds `RING_SIZE` and jumps directly to what the ring can still actually provide, instead of starving through an unrecoverable span. **Re-verified with the reviewer's own reproduction**: a 3-second `SIGSTOP` freeze mid-drive now fully recovers (27m+ movement), both via the human smoke test and a real 2-bot CI match. New unit test `test_ring_overflow_resyncs_to_fresh_data_instead_of_starving_forever` covers the exact under-tested direction the reviewer flagged (the original suite only exercised the *under*-full ring case).

**HIGH — `InputLeadController`'s release logic couldn't drain a backlog it didn't itself create.** Release was gated on `lead > LEAD_MIN` — this controller's own memory of past attacks — so a backlog from an external cause (a server hitch, persistent clock drift) left `input_buffer_depth` elevated indefinitely while `lead` (and the release gate) never moved, since the controller never itself attacked. Fixed by gating release on the actual server-reported `input_buffer_depth > TARGET_DEPTH` (§3.3's own `target_depth = 1`), not on self-tracked state. New unit test `test_release_drains_a_backlog_it_never_caused_itself` reproduces the scenario directly.

**MEDIUM-HIGH — the rate limiter was trivially evaded by a duty-cycled flood.** The original design tracked "N consecutive over-budget seconds" and hard-*reset* that streak to 0 on any single clean window, so a burst-then-idle attacker (flood hard, one clean window, repeat) evaded it indefinitely — the reviewer sustained ~33x the packet budget for 28.5s with zero disconnect warnings against the real `MatchSim._recv_input`. Replaced with a leaky-bucket excess accumulator (grows by each window's actual total, drains by exactly one window's worth of budget every window, regardless of how the excess is distributed in time) — immune to the same evasion by construction. New permanent regression test `client-abuse-flood-dutycycle` reproduces the reviewer's exact attack shape (0.35s burst / 3.0s cycle) and confirms it now disconnects.

**MEDIUM — the `seq > server_tick + 20` guard compared two unrelated epochs.** `Engine.get_physics_frames()` counts from the SERVER PROCESS's own start; a client's `_input_seq` starts at 0 when ITS match scene loads — `input_jitter_buffer.gd`'s own seeding logic exists specifically because these share no baseline. Bounding against server uptime meant the guard could never fire on a long-running dedicated server (no real protection, despite the comment's claim) and could silently drop an honest client's input forever once enough accumulated server tick loss closed whatever accidental head-start margin existed. Fixed by bounding against the slot's own `last_applied_seq + RING_SIZE` — the client's actual epoch, using the same capacity the ring-overflow fix itself treats as "unrecoverably far ahead."

**MEDIUM — `InputJitterBuffer.stalled` was computed but never reached the wire.** `_ship_to_net_body_state` never set `NetBodyState.stalled` even though `NetCodec` already packed/unpacked the bit — the one signal that would have made the ring-overflow bug visible to the client, the debug overlay, and the CI gate was silently dropped between the buffer and the snapshot builder. Now wired through.

**MEDIUM — task 3.6's own CI gate passed with a completely dead input pipeline.** Its assertions (snapshot count, a server-*forced* goal's score agreement) don't depend on client input reaching the server at all; the reviewer confirmed it kept reporting `SMOKE PASS` with the ring-overflow bug actively triggered mid-run. Fixed by recording each bot's ship position before the run and asserting real server-side movement plus a non-stalled jitter buffer — sampled *while clients are still actively connected*, not after (an early attempt sampled too late and caught each bot's own legitimate end-of-match disconnect instead of the bug, since a departed peer's buffer starves too — that's correct behaviour, not a regression, just the wrong moment to check it). Re-verified: the fixed CI gate still passes cleanly under a real mid-match host freeze now that the underlying bug is fixed, and (checked by inspection during the fix) would have caught the original bug had it still been present.

**LOW-MEDIUM — a lead change silently mislabelled the redundancy history.** `_input_history` was always `push_front`'d regardless of the seq delta, but the wire format has no per-entry seq field (`actions[i]` is implicitly `seq - i`) — a duplicated tick (release) shifted older entries under a label that no longer matched what was actually there, and a skip-ahead (attack) left the whole history discontiguous with the new seq, so the server could replay already-applied input or apply the wrong redundant copy. The original code comment's claim that this "only ever degrades a backup copy, never the real per-tick record" was itself wrong. Fixed by handling each delta case on its own terms: ordinary ticks still push; a release replaces the front entry in place instead of shifting everything back; an attack resets the window to just the current sample, which rebuilds naturally over the next few ticks (the same way it does at connection start).

**LOW — bandwidth and snapshot-loss overlay metrics froze at their last value instead of decaying during a total outage** — exactly when they matter most. `MatchSim.bytes_sent_per_sec`/`bytes_received_per_sec` are now read through `get_bytes_sent_per_sec()`/`get_bytes_received_per_sec()`, which report 0 once meaningfully more than one window has passed with nothing tracked; `get_net_debug_stats()`'s `snapshot_loss_pct` now reports 100% once more than `SNAPSHOT_STALE_MS` has passed since the last actual snapshot receipt. Verified live: all three read their honest post-outage values (0, 0, 100%) after a real ~2.5s gap in traffic, not the frozen pre-outage numbers.

**LOW — a guard comment on `NetworkManager._ping` misdescribed what the code actually does**, claiming `disconnect_peer(..., now=true)` when the real call uses the default `force=false` (an earlier attempt at `force=true`, tried and reverted elsewhere this session, made Godot's own peer bookkeeping *more* inconsistent, not less). Comment corrected to match reality.

**Confirmed fine, not just assumed**: the jitter metric's magnitude is honest (cross-checked against real 60Hz snapshot-stream jitter on the same impaired link, same order of magnitude), just slow to converge from a 1Hz sampling cadence — worth documenting as "steady-state link quality" rather than a live indicator, not worth rebuilding; `--test-bot`/`AIShipController` wiring on a frozen kinematic ship is fully safe, no NaN/Inf even under the bot's permanently-zero-velocity observations; both original abuse-detection regression tests are genuine, confirmed via a working control (a continuous flood still disconnects in ~4.5s); redundancy + adaptive lead + real loss alone (no hitch) is solid over long runs | Full regression suite — including the net-sim-latency milestone gate, all three abuse roles, the CI driver, and the reviewer's own `SIGSTOP`/`SIGCONT` reproduction at 3s (well past the original ~0.7s failure threshold) — re-run clean after every fix | - -### Phase 4 — Prediction and reconciliation, ship **and ball** - -| # | Task | Acceptance | -|---|---|---| -| 4.1 `[D:3.1]` | **DONE.** Immediate local input with immutable sequence/redundancy bookkeeping; server/training action semantics unchanged | 60 unit tests and 60s LAN/jitter/loss runs pass | -| 4.2 `[D:4.1]` | **DONE.** 128-entry sequence-tagged prediction history and snapshot matching | Same-sequence free-flight samples resolve in all 60s runs | -| 4.3 `[D:4.2, 0.14]` | **DONE.** Atomic staged reconciliation, delta rebase, epoch/reset and missing-history recovery | No free-flight hard snaps across LAN, 80±20ms, or 5% loss 60s runs | -| 4.4 `[D:4.3, 0.2]` | **DONE.** Client-only bounded position and rotation visual offsets/decay; interpolation reset | Free-flight p99 raw residual ≤0.207m; exposed visual p99 0m in final matrix | -| 4.5 `[D:4.3]` `[P]` | **REJECTED / SUPERSEDED.** Analytic one-body action replay was removed in favour of same-sequence delta transport | Jolt/contact nondeterminism makes replay unsuitable; see §4.4 | -| 4.6 `[D:4.3]` | **DONE.** Client-only dynamic proxy, authoritative shadow, RTT-limited reveal, 150 ms blend and 3 m handoff | Final contact run: same-frame reveal, 4 blends, max 152ms, no hard handoff | -| 4.7 `[D:4.4]` `[P]` | **DONE.** Client-only debug keys tune thresholds, decay, visual offset and remote-present A/B | Defaults remain 2m/60°/0.4m and render-only tuning never reaches server/training | -| 4.8 `[D:4.4]` `[P]` | **DONE.** p50/p95/p99 residual telemetry, elapsed-time snap rate, reason/cohort counters | Final free-flight p99: LAN .150m; 80±20ms .149m; 5% loss .207m; zero hard snaps | -| **4.9** `[D:4.4]` | **DONE.** Present-time remote visual extrapolation, angular integration, and render-only residual correction; delayed interpolation remains an A/B debug mode | Final two-bot present-time p99 ≤.208m / 3.146°, below .3m / 5° gate | -| **4.10** `[D:4.9]` `[P]` | **DONE.** Signed starvation sentinel and client hysteresis/cooldown; headless `--test-bot` remains target depth 1 | Jitter run observed starvation fallback; stable runs preserve safe target behavior | - -> **Ball prediction is not optional and not deferrable to a later phase.** With §4.1 in place the touch registers correctly on the server, but the ball still *renders* a third of a beat late — your ship visibly passes through it before it moves. In a game whose entire point is hitting a ball, that is the difference between "networked" and "broken", and it is the same machinery as own-ship prediction applied to one more body. Do it while the prediction code is warm. Buffering server ball state into a *shadow* copy (rather than discarding it) is what lets you measure disagreement continuously instead of discovering a 3 m error at window end. - -| 4.11 `[D:4.2]` | **DONE.** Prediction history is filed under the **issuing** sequence, and a forced-input-transition trace gates the label | Marker mismatch 0.00–1.3% (was 9.3% LAN / 24% at 80±20ms); control run at the old label fails the same gate at 50% | -| 4.12 `[D:4.11]` | **DONE.** Issued-but-unsimulated (attack-gap) sequences are recorded and skipped rather than diagnosed as history loss; the release path no longer re-files an already-issued sequence | Free-flight hard snaps 0 across all three 60 s conditions, down from 25/8/4 `missing_not_recorded` | -| **4.13** `[D:4.12]` | **DONE — two server-side input-death bugs found by adversarial review, both reproduced and fixed with controls.** A starve no longer advances past a sequence the client has not sent; the seq-range guard can no longer latch shut permanently | Marker 0.00% in all three conditions (was 1.7–2.5%); 2.0 s and 3.5 s host freezes now recover; control runs with each fix reverted fail the gate | -| **4.14** `[D:4.3,4.8]` | **DONE.** Prediction startup distinguishes the server's pre-history sequence-0 acknowledgement from genuine missing/evicted history, so warm-up cannot arm hard-snap recovery | 143 Godot tests pass; two-process ENet match passes 173 prediction samples with 0 hard snaps, 0% snapshot loss and authoritative movement; the 80±20 ms impaired-link run passes the near-surface gate with p95 0.682 m / p99 0.717 m and no free-flight hard snap; the 5% loss run passes with 222 samples, 7.1% observed snapshot loss, p99 0.716 m and 0 hard snaps | - -**Phase gate — correctness gates MET; the milestone's felt-quality half remains untested.** The action-sequence-correctness gap is closed and permanently gated (4.11), the two seq-delta paths it exposed are fixed (4.12), and an adversarial review's two server-side input-death bugs are fixed with controls (4.13). What has *not* happened is the original milestone's actual subject: nobody has played this with hands on a controller at ~100 ms RTT to judge whether ship and ball feel local and whether contact corrections read as bumps. Numbers cannot answer that, and the contact cohort is where the remaining known weakness lives (see the shadow-world note below). Sign off after a human playtest, not before — item **A** of §0. - -> **Read 4.13 before trusting any earlier Phase 4 evidence.** Until this session the server was silently discarding a connected player's input for ~30 ticks roughly every 6.5 seconds on a clean LAN, and permanently after any ~2 s host hitch. Every Phase 4 number recorded before 4.13 was measured through that, and the gates reported green throughout — for the same reason they missed the label bug in 4.11: a steady input cannot distinguish "the server repeated my last action" from "the server applied my real action". - -**The mislabelled prediction history, and why every earlier gate missed it.** `_send_local_input` filed each post-step predicted state under `_local_net_controller.last_applied_seq` — the timeline's *estimate of the sequence the server would consume this tick*, which trails issuance by `input_lead`. The body had actually integrated the current raw intent, issued under `_input_seq`. So `predicted[S]` held "state after integrating the intent from now" while the server's authority for `S` is "state after integrating `action(S)`", sampled `input_lead` ticks earlier. The two agree **only while the commanded action is constant** — and every Phase 4 acceptance trace held its input steady (`move_forward` held, or the free-flight hover alternating on a 0.7 s/0.3 s period). A steady input cannot falsify a sequence label: the marker reads 0/N under a correct and an incorrect label alike. The 60-second free-flight runs genuinely reported `marker=0/3784`; the instrument was fine, the trace was blind. - -Filing the state under `_input_seq` fixes it and costs nothing. The code comment that had rejected this ("avoids turning client prediction into an input-delay queue") conflated *which action the ship uses* — decided in `LocalNetShipController.get_action()`, still the raw current intent, still immediate, untouched by this change — with *which sequence its resulting state is filed under*. Measured with `--exercise-input-transitions` (below): - -| condition | `input_lead` | old label | filed under `_input_seq` | -|---|---|---|---| -| LAN | 1 | 35/376 (9.3%) | 0–6/456–582 (0–1.3%) | -| LAN, adversarial toggle phase | 1 | 289/576 (50.2%) | — | -| 80±20 ms | 3 | 97/404 (24%) | 0/424 (0%) | - -Mismatch scales with `input_lead`, exactly as the mechanism predicts. It also **cut pre-existing `missing_not_recorded` hard snaps 4×** on the 60 s LAN free-flight run (25 → 6, 24.8/min → 6.0/min): the old label's per-tick +1 cursor was an estimate that could drift off the sequence the server actually acknowledged, while an issued sequence is by construction the thing the server acknowledges. - -**Task 4.12 — the two seq-delta paths, and what is left.** Relabelling exposed two further places where the history disagreed with the wire, both now fixed: - -- **Attack gaps (`delta > 1`).** The lead controller skips sequence numbers to buy server buffer margin. Those sequences are filled with repeat-last actions and genuinely **sent**, and the server genuinely acknowledges them — but the client took exactly one physics step that tick, so no post-step state exists for them. They were simply absent from the ring, which `compare_authoritative` could only report as `missing_not_recorded`: indistinguishable from real ring loss, and therefore a hard snap, a full authority teleport, and armed resync suppression **several times a minute during ordinary play**. They are now recorded stateless via `record_unsimulated()` and report their own `unsimulated_gap` status, which `NetShipPredictor.decide()` answers with a new `"skip"` mode — no correction, no teleport, no suppression, no snap counted, its own metrics cohort. The next simulated sequence (a tick or two later) reconciles normally. **Result: free-flight hard snaps went from 25 / 8 / 4 to 0 / 0 / 0** across the LAN, 80±20 ms and 5%-loss 60-second runs; the doc's own long-standing target was <1/min and LAN was measuring 24.8/min. -- **Release (`delta == 0`).** `_send_local_input` re-recorded at the unchanged `_input_seq`, filing the *current* intent under a sequence that had already gone out carrying a different action. `LocalInputTimeline.issue()` deliberately refuses to mutate an already-issued sequence ("may be in flight or consumed"), so the ring was contradicting the wire outright. Recording is now skipped entirely on a release tick; the existing `predicted[S]` is already correct, and the extra unlabelled local step is precisely the tick of latency the release exists to recover. - -**The residual is solved — it was not a prediction bug at all.** An adversarial review intersected every sequence the server starved on against every sequence the marker flagged, across five two-process runs: **151 of 151 mismatches were the server repeating a stale action on a starve**, zero unexplained. When the server starves on seq `S` it repeats `action(S-k)` but still acks `S`, so the snapshot's `thrust_z` honestly describes a different action than `predicted[S]` — the marker was correctly reporting a real client/server disagreement that prediction did not cause and could not fix. The apparent correlation with `input_lead` was a confound: the conditions that raise the lead are the conditions that produce starves. Fixing the starvation cause (task 4.13 below) took the marker to **0.00% in all three conditions**, including 80±20 ms and 5% loss where it had been 1.7–2.5%. - -Two sub-findings from that investigation, recorded because both are counter-intuitive: `dequantize_thrust_z_bin(quantize_thrust_z_bin(0.0))` returns **0.142857**, not 0.0 (7 bins over [-1,1], `roundi(3.5) == 4`), so a server-reported `thrust_z` of 0.14 literally means "exactly zero" — the 0.26 threshold absorbs it, as designed. And `_pending_local_reconciliation` keeps only the newest snapshot, so acks are dropped whenever two snapshots land in one physics tick: **the marker under-samples, and the true action-disagreement rate is higher than it reports.** - -> **The client-only shadow Jolt world is still the open question (item F of §0), but it is now scoped to the contact cohort alone.** Even perfectly labelled, the client predicts contacts against remote ships and the ball sitting at interpolated-*delayed* positions, so a contact-cohort prediction cannot be sequence-correct in the live world — no amount of bookkeeping fixes that, and a shadow world is the only thing that does. It is a large subsystem and effectively the whole-world rollback §1's locked decisions set out to avoid, so **do not build it before a playtest says the contact cohort actually reads badly to a human.** Free flight no longer needs it. - -**New smoke role — `--exercise-input-transitions`.** Toggles forward thrust every 6 physics ticks (~100 ms) with alternating yaw, and asserts the action marker stays under 5% mismatch over ≥200 samples. This is the **only** gate here that can catch a sequence-label regression, for the reason above, so it must not be folded into the steady-input free-flight run: - -``` -godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=host --drive-seconds=8 -godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=client --drive-seconds=8 --exercise-input-transitions -``` - -Verified against a working control: reverting the one-line label makes this gate fail at 50.2% mismatch, so it is not vacuous. - -### Task 4.13 — two server-side input-death bugs the Phase 4 gates could not see - -Both are **Phase 3 code**, both predate this session, and both were found by an adversarial review of the Phase 4 changes rather than by any gate. Neither was caused by 4.11/4.12; both are squarely in the way of Phase 4's *feel* milestone, so they are fixed here. - -**(a) A starve stranded the input stream one sequence ahead of arrivals — permanently.** `InputJitterBuffer.consume()` set `last_applied_seq = expected` on **every** tick, including a starve. Because `ingest()` discards anything `seq <= last_applied_seq`, a single starve on a sequence the client had not sent yet left the server permanently one ahead: both sides then advance one per tick, the gap never closes, and **every honest packet is discarded on arrival**. The client's own `input_lead` RELEASE (`delta == 0`, which deliberately issues no new sequence for one tick) is sufficient to trigger it — so this fired roughly **every 6.5 seconds of ordinary play on a clean LAN**, blacking out input for 30 ticks until the lead controller's `MIN_CHANGE_INTERVAL_TICKS` debounce permitted a +3 attack to jump the client clear. The reviewer measured 2 blackouts in a 23 s run and 2 in a 30 s run, with the host applying the *same repeated action* for 30 consecutive ticks while the wire carried fresh input every one of them. Fixed by only giving up on `expected` when strictly newer data has arrived, which proves it lost rather than merely late. Both escape paths are untouched: a silent client still zeroes and stalls on `STARVE_ZERO_TICKS`, and a far-behind consumer still hits the ring-overflow resync. - -**(b) The seq-range guard was a one-way door.** `_on_input_received` bounded incoming `seq` against `jb.highest_ingested_seq + RING_SIZE` — but `highest_ingested_seq` only ever advances *inside* `ingest()`, which that same guard gates. Once a client's live sequence got more than 32 ahead (a host stall drops the intervening packets wholesale, since input is unreliable), every subsequent packet was rejected, the bound could never move again, and **that player's input was dead for the rest of the match with no diagnostic**. Reproduced with a 2 s `SIGSTOP` host freeze: 600+ consecutive rejections, the server applying zero thrust across 1300 sequences while the client's wire carried full thrust throughout. This is the **third** iteration of this guard, and the structural lesson is that each previous version bounded against a value only the accepted path could advance. Fixed by keeping the bound but adding an escape: after `SEQ_REJECT_RESYNC_LIMIT` (10) consecutive rejections, accept and let the existing resync machinery re-establish the baseline. This grants an attacker nothing — walking the epoch forward by sustained rejection costs the same packets as walking it forward by acceptance, and §3.4's rate limiter already bounds that rate. - -**(c) The gate printed PASS while input was permanently dead.** The `--exercise-input-transitions` gate reported `SMOKE PASS` at 3.76% mismatch on a run where input was completely dead, because *suppressed reconciliation stops calling `_record_metrics`* — so the worse the outage, the fewer marker samples and the **lower** the reported mismatch rate. Every other assertion in that path (`local_prediction_ok`, `moved > 1.0`) reads the client's own action and position, which a client flying purely on prediction satisfies perfectly. Fixed by scaling the required sample count with run length (`max(200, drive_seconds * 30)`, half of nominal 60 Hz) and asserting the wire's `server_stalled` bit. **Verified non-vacuous:** reverting both fixes and re-running the 3.5 s freeze fails at `samples 292/600` with `server_stalled=true` and `input_lead=12` (LEAD_MAX) — while reporting `marker=1/292 = 0.34%`, which the old gate would have passed. - -**QA matrix, re-run in full after 4.11 + 4.12 + 4.13** (all green): **72 unit tests**; 60 s free-flight at LAN / 80±20 ms / 5% loss — p99 raw **0.141 / 0.168 / 0.154 m**, exposed visual p99 0.000 m, **0 hard snaps in every condition**, marker 0/3484, 0/3049, 0/3397; forced-input-transition gate at LAN, 80±20 ms **and** 5% loss, all **0.00%**; 2.0 s and 3.5 s `SIGSTOP` host-freeze recovery; ball contact ×5; two-bot CI ×3; all three abuse roles; `net_smoke`, `match_net_smoke` (incl. `host_recycle`), `clock_smoke`, `lobby_smoke`. - -Note the free-flight p99 **improved** (0.170/0.176/0.184 → 0.141/0.168/0.154) and `input_lead` now sits at 1 on LAN instead of oscillating to 3–4. Both are downstream of 4.13(a): the periodic blackouts were degrading prediction accuracy and driving the lead controller. - -**Two test defects fixed alongside, both pre-existing and both surfaced by 4.13(a):** - -- **Ball-contact gate flaked 2 in 5.** `ball_proxy_moved_before_authority_count` requires the predicted proxy to have visibly moved *before the next authoritative ball state arrives* — but snapshots land every ~16.7 ms at 60 Hz, so on a loopback LAN the entire pre-authority window is about one physics tick and observing it is a coin flip. It is also the least interesting case: the counter measures RTT-masking, and LAN has no RTT to mask. Measured 5/5 passes (count 2–3) at `--net-sim-latency=80`. Now asserted only when `NetworkManager.rtt_ms >= 20`, with the same-frame reveal — the test's real claim, correct in every run either way — carrying the gate on LAN. Runs asserting the masking behaviour should pass `--net-sim-latency`. -- **Two-bot CI compared scores across a 3–5 s window.** The host checked each client's recorded score against its own score at *read* time, but clients write theirs several seconds earlier; any goal in between failed the run with both bots agreeing perfectly with each other. Latent until 4.13(a) made the bots effective enough to reliably score a second goal — then it failed 2 of 3 runs, every failure `server=2` vs `both clients=1`. The host now polls and records every score it actually holds, and asserts both clients agree **with each other** and that what they saw is a state the server genuinely passed through. 3/3 green, including a run ending 1–1 where the clients had recorded 0–1. (Polling, not `score_changed`: that signal is emitted only in `_on_score_update_received`, the *client* path — the server mutates `score` directly in `_record_goal` and never emits. Connecting to it recorded nothing but the initial 0–0.) - -> **Follow-up, not done:** `LocalNetShipController.last_applied_seq` is now write-only and `LocalInputTimeline.consume()` is vestigial to the reconciler (still unit-tested, still advancing `_last_applied_action`, but nothing reads the result). Left in place rather than removed as unreviewed scope — but it now looks load-bearing and is not. - -### Phase 5 — Match lifecycle - -| # | Task | Acceptance | -|---|---|---| -| 5.1 `[D:2.1]` | **DONE.** `scripts/match_state.gd` (enum + validated transition table, pure/unit-testable), server-driven machine in `NetworkedMatch`, `state_change` RPC on reliable channel 0 carrying an absolute `at_tick`, and the snapshot `match_state` byte populated for real | Client observed `LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP` with monotonic ticks in a real two-process run; every consecutive pair legal; wire byte asserted independently of the RPC | -| 5.2 `[D:5.1]` | **DONE.** `_end_tick`/`_clock_running`, `clock_state` RPC, `timer_updated` emitted from absolute ticks on both peers; goal pause shifts `end_tick` rather than pausing anything | No `Timer` and no `_process` polling remain in the networked path; both peers derive `remaining = end_tick - now` from the same server-tick estimate | -| 5.3 `[D:5.1]` | **DONE.** `kickoff` RPC carrying resulting transforms (never a seed, per §1), deferred freeze, `reset_gen` bump, countdown from `server_tick`, late-arrival skip | Real two-process run: `LOADING -> WARMUP -> PLAYING`, countdown ticks match `WARMUP_TICKS` exactly; a kickoff past its own resume tick unfreezes immediately and emits `0` | -| 5.4 `[D:5.1]` | **DONE.** `goal_scored(scoring_team, score, goal_tick, resume_tick)`, freeze on the goal tick, reset moved out of the sensor path into the kickoff at `resume_tick`; cinematic is presentation-only | `PLAYING -> GOAL_PAUSE -> WARMUP -> PLAYING` observed on the client; bodies stay where the goal left them for the whole window; `Engine.time_scale` untouched | -| 5.5 `[D:5.1]` `[P]` | **DONE.** Clock expiry -> `FULL_TIME` -> sudden death on a draw or `RESULTS`, golden goal in overtime, then `LOBBY` on both peers. `get_tree().paused` is never used in the networked path | Full run observed end to end: `LOADING -> WARMUP -> PLAYING -> FULL_TIME -> OVERTIME_WARMUP -> OVERTIME -> GOAL_PAUSE -> RESULTS -> LOBBY`, both peers returning to the lobby scene | -| 5.6 `[D:5.1]` `[P]` | **DONE.** Controller swap on disconnect (ship never despawned), 30 s identity-keyed reservation, reclaim on reconnect, `--fill-bots`/`--no-fill-bots`, `stalled` set immediately for the nameplate | Real 3-process run: ship survived, controller valid, slot reserved, reclaimed by name, same ship instance | -| 5.7 `[D:5.6]` | **DONE.** `_swap_slot_controller()` rebinds in the same transaction; `slot.controller` retyped to the base `ShipController`; every use `is_instance_valid`-guarded | The disconnect test caught the real bug: the narrower `RLShipController` type made the swap assignment fail, leaving a freed reference | -| 5.8 `[D:5.1]` `[P]` | **DONE.** A slotless peer spectates (no ship spawned, same snapshot stream), `HUDController.spectator_mode` keeps clock/score/celebration and hides only the ship cluster, camera cycles ships then ball, `--max-spectators` cap. **§6.3's "take the slot at the next kickoff" is now implemented, not just printed** — the line claiming it was there from the start while `_is_spectator` was assigned once and never revisited (see the Phase 5 note below) | Spectator path exercised by the mid-match joiner; HUD no longer `push_error`s and bails with a dead HUD; promotion verified 4/4 from both sides, with a control proving §6.4's reservation outranks the queue | -| 5.9 `[D:5.3]` `[P]` | **DONE.** New `GameMode._on_bodies_respawned()` virtual; `NetworkedMatch` bumps `reset_gen` through Phase 2's deferred path so the bump and the respawned pose land in the same broadcast | Single-player modes unaffected (base is a no-op) | -| 5.10 `[D:5.1]` `[P]` | **DONE.** `scripts/replay_log.gd`, `--replay-log=`, storing wire bytes verbatim in both directions — plus, after a review found three recording gaps, REJECTED packets with their reason in the kind byte (capped per window so the log cannot become a remote disk-fill amplifier), a failed write that ends the log instead of desyncing its framing, an explicit `close()` with a summary, and `tools/replay_dump.gd` to read one back. The reject recording immediately found a real bug: the server was rate-limiting a stall backlog it had caused itself, losing 8.88% of a player's input | Live 6 s match recorded 1115 records (557 inputs / 558 snapshots); a stored snapshot decodes back to `server_tick=100 match_state=WARMUP bodies=2`; 6 unit tests incl. truncation and foreign-file rejection | - -> `Ship.set_controller` (`ship.gd:213-218`) calls `queue_free()` on the outgoing controller. Task 5.7 exists because the takeover path in 5.6 otherwise leaves `MatchNet` holding a freed reference — the exact class of bug that surfaces as a random server crash weeks later. - -> Task 5.10 is the highest-value debuggability investment here. The packets are already flat bytes, so it is ~50 lines. Without it, "my ship snapped" is permanently unreproducible from a field report — the CI gate catches regressions, but it cannot debug a player's bad night. - -#### Task 5.1 notes - -`scripts/match_state.gd` holds the enum and the §6.1 transition table as pure data with no scene/RPC dependency — the same reason `net_codec.gd` and `input_jitter_buffer.gd` are standalone — so the table is checked exhaustively (every state reachable, every state has an exit, no self-transitions, abort-to-LOBBY from anywhere, illegal shortcuts rejected) rather than by example. **The enum's integer values are the wire format**, pinned by a test: `match_state` has been a `u8` in the snapshot header since §2.4, so renumbering an existing state silently reinterprets packets from an older peer. Only append. - -The server validates every transition and `push_error`s an illegal one rather than following it, because the symptom otherwise — clients faithfully following into a state the server's own code never meant to reach — is near-impossible to diagnose from a field report. - -**Two channels carry the state, deliberately.** `state_change` (reliable, channel 0) is prompt and carries the absolute `at_tick`; the snapshot's `match_state` byte is the catch-up path for a client that has not been sent a transition yet — a late joiner (§6.3), or the window between scene load and the first RPC. **The byte needs a tick guard**: snapshots are `unreliable_ordered` on channel 2 and ordering holds only *within* a channel, so a `state_change` for tick N routinely arrives before an in-flight snapshot from tick N-2. Without the guard the client applies the new state and is immediately dragged back by the older byte, oscillating on every transition — observed directly (`LOADING -> WARMUP -> LOBBY -> PLAYING -> LOBBY -> ...`) while running a deliberately-broken-byte control. Only a byte at least as new as `match_state_since_tick` is accepted. - -The client deliberately does **not** enforce the transition table — authoritative state must be accepted, and a late joiner legitimately jumps straight to `PLAYING`. The table is a server-side invariant. The smoke test asserts legality of what the client *observes*, seeding its first sample from whatever state the client converged to rather than counting that as a transition, so late-loading clients (seen seeding at `WARMUP` rather than `LOADING`) still pass. - -**5.1 does not gate physics, freezing or input on state.** Tasks 5.3 and 5.4 own freeze/unfreeze at kickoff and goal; doing it here would both duplicate that work and change the conditions every Phase 4 prediction gate was measured under. `MatchState.is_live()` exists for them to use. `WARMUP_TICKS`/`GOAL_PAUSE_TICKS` are honest placeholders so 5.1 drives *real* transitions to verify against — 5.3 replaces the first with the broadcast kickoff (reset transforms + countdown from `server_tick`), 5.4 the second with `_goal_pause_seconds()` and the client-cinematic split. The server also leaves `LOADING` immediately rather than waiting for `scene_ready`, which does not exist yet (5.3). - -New smoke flag `--exercise-match-state` (pass to **both** roles — the host forces a goal to drive a `GOAL_PAUSE` cycle, the client records and validates the sequence): - -``` -godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=host --drive-seconds=6 --exercise-match-state -godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=client --drive-seconds=6 --exercise-match-state -``` - -Verified against a control: hardcoding the snapshot byte back to `0` fails both the byte assertion and the transition-legality assertion. That control is why the gate asserts the wire byte separately from the RPC at all — everything else in the check is RPC-driven and would pass identically with a dead byte, which is exactly how Phase 4's mislabelled history survived every gate (gotcha 47). - -#### Phase 5 notes - -**Task ordering caught three ordering bugs of the same shape**, all found by a failing run rather than by review, and all worth remembering as a class: *a value consumed by one per-tick updater and cleared by another is order-dependent.* `_update_kickoff_countdown()` clears the `_kickoff_resume_tick` that `_update_match_state()` reads to leave `WARMUP` (match froze forever); `_apply_match_state()` resets `_state_deadline_tick` on every transition, so a `GOAL_PAUSE` deadline assigned *before* `_set_match_state` was wiped (match never resumed); and a `set_deferred("freeze", true)` landed before the queued kickoff teleport could apply, stranding every body where the goal left it. - -**Freezing is asymmetric between server and client, and this is not optional.** On the server every body is a real dynamic simulation and all of them freeze. On a client, `freeze` is *already* load-bearing for something else: remote ships and the ball are permanently `FREEZE_MODE_KINEMATIC` and driven by transform writes, with only the local ship unfrozen for prediction. Freezing "all bodies" on a client therefore **unfreezes the remote ones on the way back out** — they fall under gravity while the interpolator fights them for the transform. Measured: 210 hard snaps and an infinite p99. A client freezes only the one body it actually simulates. - -**Prediction is suspended while the match is not live.** During a countdown or goal pause the local ship is frozen on both peers, so there is nothing to predict — but the reconciler still ran delta transport and visual-offset maths over those frozen states and produced a p95 position error of **2.4e10 m** while the instantaneous error stayed small. Input keeps flowing so the server's jitter buffer does not starve into `stalled`. - -**§6.4's two rules conflict and the reservation has to win.** "Reserve a departed player's slot for 30 s" and "abort to the lobby once the last human leaves" applied naively means the abort fires instantly in a 1v1 — the moment the only player drops, the match is torn down and the reservation can never be redeemed, making the reconnect path unreachable exactly when it matters (one player whose connection blipped). Abort now waits until nobody is connected **and** no reservation is outstanding. - -**Task 5.7's bug was real and the test found it.** `SlotInfo.controller` was declared `RLShipController`, but §6.4's takeover swaps in an `AIShipController` or the base controller — a narrower declared type makes that assignment fail its type check, leaving the field pointing at the controller `set_controller()` just `queue_free()`d. It surfaced as `controller_valid=false` on the first disconnect run. The per-tick `slot.controller.action` write is now also gated on `is RLShipController`: a disconnected slot's bot drives itself, and overwriting its action from a permanently-starving buffer would pin it to the departed player's last input. - -**`--check-only --script` is the only thing that catches a parse error in `networked_match.gd`.** The unit runner never loads it, so `bot_model_path` being undefined (and later `ReplayLog` being unregistered) both passed 81/87 unit tests while breaking every two-process run. Validate touched scripts directly. A newly added `class_name` also needs `godot --headless --path Game --import` before anything can resolve it — and the same `--import` is the fix when a *previously working* `class_name` stops resolving, which happens on its own: `.godot/global_script_class_cache.cfg` silently lost `MatchState` between sessions, and every two-process run then died with `Cannot infer the type of "live" variable` at the `MatchState.is_live()` call, with nothing in `git status` to explain it. Read that error as "the class cache is stale", not "the code is wrong". - -**The reviewer's p95 0.688 was real, and the three-process framing was a red herring — mine as much as the reviewer's.** The report was "a 3-process run failed the free-flight gate at p95 0.688 (bar 0.5) with roughly a third of snapshots missing", so the first investigation compared process counts: two-process p95/p99 0.084/0.098, idle third process 0.084/0.094, spectator 0.084–0.098 / 0.094–0.146 over four runs, and 0.0% snapshot loss even under deliberate 2x CPU oversubscription (20 spinners on 10 cores, where only `snapshot_age` moved, 14ms → 32.3ms). Every one of those runs passed, so the conclusion recorded here was "not reproducible". **That conclusion was wrong, and it was wrong because every probe used `--exercise-free-flight` — the one mode the 0.5 bound was calibrated on.** - -It reproduces on *two* processes, on an idle machine, with 0.0% snapshot loss: **the plain `--role=client` drive fails the free-flight gate roughly a third of the time.** Eight plain-role runs measured a free-flight cohort of 12–257 samples with p95 0.275–0.726, failing the 0.5 bound in 3 of 8. The harness's own `_run_free_flight_trace` comment had already said why — "a straight forward trace reaches the goal/wall in seconds and turns the supposed free-flight QA run into a contact test" — but the plain role went on asserting the open-volume bound against whatever free-flight samples that contact-heavy drive happened to leave behind, sometimes as few as 12. - -The underlying difference is not noise. Prediction error near the arena's surface-pull field is genuinely several times higher than in open air: the same build measures 0.084–0.111 under `--exercise-free-flight` and 0.275–0.726 on the plain drive. Both are honest numbers about different flight profiles, and one bound cannot serve both. `--exercise-free-flight` keeps the calibrated 0.5/2.0 gate (~5x margin). The plain role now asserts the **all-cohort** percentiles instead — always well-sampled (545–696, versus a free-flight cohort that can collapse to 12) and much tighter in spread (raw_p95 0.354–0.609, raw_p99 0.362–0.742) — at 1.2/2.0, ~2x above the worst observed, and prints the free-flight numbers explicitly marked *reported, not asserted*. `free_flight_hard_snaps == 0` is still asserted in both modes, and anything past 2.0m is a hard snap by definition, so a genuine free-flight regression cannot hide behind the looser bound. Verified: 6/6 plain-role runs pass where 3/7 previously failed, all four other modes (free-flight, 80±20ms latency, input transitions, ball contact, match state) still pass, and tightening the new bound to 0.3 makes it fail — the gate is evaluated, not skipped. - -The other durable improvement from the first investigation still stands: a percentile alone cannot distinguish "the predictor regressed" from "the client never received the data", so the client gate prints `snapshot_loss` / `snapshot_age` / `rtt` on every run and, on a quality failure with >20% loss, says explicitly that the run was transport-starved — **without converting the failure into a pass**. Both directions verified non-vacuously. It is also what proved the 0.688 was not transport: every reproduction reported 0.0% loss. - -**Lesson worth more than the fix: probing only with the purpose-built mode is how a flaky gate stays invisible.** The first pass ran eight variations of process count and CPU load and never once ran the plain role that the reviewer had actually run. - -**Task 5.10's three recording gaps, and the real bug closing them found.** The review flagged that the replay log ignored `store_*` failures, never recorded the packets the server *rejected*, and had no caller for `close()`. All three are fixed: a failed write now ends the log permanently rather than desyncing every later record's framing (`write_failed`, checked via `FileAccess.get_error()` once per record); `close()` is called from `_exit_tree` with a summary line, because letting the RefCounted's destructor do it implicitly never tells anyone whether the log is complete; and rejected packets are recorded with their reason in the kind byte (`REJECTED_MALFORMED` / `REJECTED_RATE_LIMIT` / `REJECTED_SEQ_GUARD`, framing unchanged, `FORMAT_VERSION` 2 so "no rejects" can be told from "this build never recorded them"). Recording is capped at 8 per peer per rate-limit window — without that cap the diagnostic is a remote disk-fill amplifier, since the attacker chooses the packet rate. Verified end to end: an honest client logs 0 rejects; `client-abuse-malformed` sends 25 and logs exactly 8; `client-abuse-flood` sustains ~2400 packets/s and logs exactly 8. Uncapped totals are kept separately (`MatchSim.get_reject_totals()`) and survive the peer's disconnect — the first version stored them on `_PeerInputState`, which is erased on disconnect, so every summary printed an empty dictionary. - -**And the bug the recording immediately found: the server rate-limited a backlog it caused itself.** A 2s host stall (`SIGSTOP`, standing in for a GC/IO/scheduler hitch) has the client sending at 60Hz throughout, and ENet delivers the whole backlog in the first window after resume — **70 of an honest client's input packets rejected as "rate limit exceeded"**, against a limit that client never came close to violating. Redundancy does not cover it, and that was the assumption worth checking rather than asserting: the dropped packets are *contiguous*, so each one's redundancy window falls inside the same dropped run. Measured with the new log: **0 of 70 rescued, and 82 of 923 sequences (8.88%, ~1.4s of that player's input) never reached the server at all**, versus 0.00% on an otherwise identical run with no stall. Every prediction gate still passed — this is the same class as the Phase 3/4 input-death bugs, invisible to every gate that reads only the client's own state. - -Fixed by not policing a backlog the server caused: `MatchSim._physics_process` watches for a wall-clock gap over `STALL_DETECT_MS` (a stalled process doesn't run that callback either, so the first frame after the stall sees the whole gap, which is exactly the size of the backlog about to arrive) and grants each *already-tracked* peer a capped, two-window packet grace. The leaky bucket drains against the same graced budget, or a stall would still accumulate excess toward a disconnect for traffic the server just explicitly allowed. Results: 2s stall, rate-limit rejects 70 → **0**, sequences missing 8.88% → **0.00%**, and `REJECTED_SEQ_GUARD` 9 → 0 as a second-order confirmation (the guard was firing partly *because* the dropped backlog let the client's epoch run away). Across eight stall runs on the fixed build, 7 measured 0.00% missing; the eighth measured 23.54% with zero rate-limit rejects and the seq-guard resync visibly doing its job — a separate, occasional transport-level loss during the stall that this change does not address and does not make worse (**item D of §0**). The three control runs on the unfixed build lost 4.34%, 7.52% and 7.86%, every time. - -Abuse detection is unweakened and this was checked rather than argued: all three abuse roles still disconnect, and **no flood induced a server stall in any run**, so the grace cannot be farmed by flooding. An attacker who *can* induce server stalls to earn budget already has a strictly worse capability than sending extra input packets. - -**§6.3's "spectate now, take the slot at the next kickoff" was a print statement, not a feature.** The server logged *"joined mid-match; spectating until the next kickoff"* and then never did anything about it; on the client, `_is_spectator` was assigned once during `_on_match_config_received` and never revisited — and that handler returns early whenever `_slots` is non-empty, so no rebroadcast could ever promote an in-match spectator. The reconnect path only worked because a returning player is a *fresh process* that runs `_on_match_config_received` from scratch. - -Implemented on both sides. The server queues late joiners in arrival order and drains the queue from `_begin_kickoff()` — before the reset transforms are read, so a promoted player's ship is placed by that same kickoff instead of being left wherever its previous owner abandoned it, and the controller swap lands on an already-frozen body, which is the entire reason §6.3 puts this at a kickoff boundary. A slot only becomes available once its player has gone **and** their 30s reservation has lapsed: §6.4 outranks §6.3, because taking a still-reserved slot would quietly break the reconnect promise. `_abort_if_abandoned` now counts a waiting spectator as somebody still present, for the same reason it already counts an outstanding reservation — otherwise the one person queued for the slot that just opened gets dumped to the lobby at the exact moment they were about to receive it. - -The client gets a new broadcast `slot_assigned` (reliable, channel 0). Broadcast rather than addressed to the new owner, because every client holds its own slot list and one that names the wrong peer keeps flying somebody else's ship as a remote body; reliable, because unlike `match_state` there is no per-snapshot field that would re-converge a client that missed it. The promoted client undoes everything that made that body remote — fresh interpolator (its buffered samples describe the *previous owner's* flight), Godot's own physics interpolation switched back on, visual offsets cleared — and then deliberately does **not** unfreeze: it clears `_local_prediction_ready` so the next snapshot teleports it to a genuine authoritative pose and starts prediction there, exactly as a fresh client does. The controller-attach block was factored out of `_on_match_config_received` into `_take_local_ownership()` rather than copied, since a copy is a copy that drifts. - -New `--role=host-latejoin` / `--role=client-latejoin` and `--slot-reservation-seconds=` (a server-side override in the same shape as `--match-length`, because the interesting moment is otherwise 30 real seconds away). Verified 4/4 from both sides: the joiner is queued, is **not** promoted merely because the reservation lapsed, takes the slot at the forced goal's kickoff, keeps the same ship instance, and both peers independently measure ~45.7m of movement under its input — the client's own number and the server's agree, so the promoted seat is real rather than relabelled. Control with a 90s reservation: the kickoff fires and nothing is promoted, the slot still reads the departed player's name, and the joiner stays a spectator. The existing spectator test is a second control — a spectator with no free slot is never promoted. - -Two test-side races were fixed while getting there, both worth remembering because they produced confident false failures: sampling `predicting` at an arbitrary frame reported `false` for a client that then flew 45m, because unfreezing is *queued* and applied on the body's next `_integrate_forces` (task 0.15), so there is a real window where the state is PLAYING and `_local_prediction_ready` is set but `ship.freeze` has not flipped yet. Poll the whole condition with a deadline, never a proxy signal, and never one instant. - -**§6.4's reconnect was only ever graded from the server's side, and the client's side was failing the whole time.** `run_disconnect_host_check` ticked 60 physics frames (1.0s) past the reclaim and then shut the server down — so the reconnecting client, whose wiring check waits a 2.0s settle before it looks at anything, had its peer torn out from under it every single run and reported `current_scene is not NetworkedMatch after 2.0s`. The host printed PASS throughout, and the host was the side anyone read. The hold is now a real window (default 8s), and the host additionally asserts that the reconnected player's input reaches the server and moves the ship the server owns — every other assertion there is slot bookkeeping that would hold identically for a client whose input pipeline came back dead, which is the exact failure the reservation exists to prevent. Both the position and the connection state are sampled *while the peer is still connected*, not once at the end of the hold: the client leaves on its own schedule, and an end-of-hold sample reported `still_connected=false` for a perfectly good run — the same mis-timed sampling a Phase 3 review caught in the CI gate. - -New `--role=client-reconnect` grades the returning player: not a spectator, owns a slot whose `peer_id` is its own, has a real ship, rejoined a live match with the clock already known (`_end_tick >= 0` — §6.2 step 2's bootstrap, since a player who must wait for the next goal to learn the score has not really rejoined), and its input still moves its ship. That set is chosen because a stale `_last_match_config` once made a reconnecting player a spectator, and *that bug was visible in this scenario's own logs while it reported PASS*. Verified 3/3 both sides, with a control that rejoins while the slot is still occupied and correctly fails on `is_player=false`. The first version of that control failed with the generic "lost its ship mid-drive", so the spectator case is now reported before the drive rather than after. - -`tools/replay_dump.gd` reads a log back — record counts by kind, plus how much of the input sequence stream actually reached the server once redundancy is counted. It is committed rather than left in a scratch directory because it is what turned "the server dropped some input" into the numbers above, and a log nobody can read is half a feature. - -**New/changed test surface:** `--exercise-match-state` (both roles; host forces a goal, client validates the whole observed sequence and the wire byte), `--role=host-disconnect` for the 5.6/5.7 three-process scenario (paired with `--role=client-reconnect`, which grades the returning player), `--role=host-latejoin`/`--role=client-latejoin` plus `--slot-reservation-seconds=` for §6.3's kickoff promotion, `--match-length=` to reach `FULL_TIME` in a short run, `--replay-log=`, `--fill-bots`/`--no-fill-bots`, `--max-spectators=`. The ball-contact scenario now **steers at the ball with closed-loop real input** instead of a hand-tuned fixed-heading burst, which 5.3 broke by adding `KICKOFF_YAW_JITTER` (0 contacts in 3/3 runs); it thrusts while turning rather than hovering to aim, which took it from 2/3 to 5/5. - -**Phase gate:** a full 3v3 start-to-finish including a mid-match disconnect and a late joiner. **Not yet run** — every scenario above was verified at 1v1 (plus a two-bot CI match). The 3v3 gate needs a real multi-client session; it is item **B** of §0, alongside Phase 4's un-run human playtest (item **A**). - -### Phase 6 — Dedicated server productionisation - -| # | Task | Acceptance | -|---|---|---| -| 6.1 `[P]` | **DONE.** Export preset (`dedicated_server=true`, `custom_features="dedicated_server"`) and `run/main_scene.dedicated_server`, mirroring the existing `run/main_scene.training` mechanism | `Linux Dedicated Server` builds | -| 6.2 `[D:6.1]` | **DONE.** Verify the stripped export boots and scores a goal | Docker smoke runs two exported-server matches and observes server-owned goals from two headless clients | -| 6.3 `[P]` | **DONE.** Full CLI surface plus a config-file fallback | Unit tests cover precedence, validation, and `--help` | -| 6.4 `[P]` | **DONE.** Structured logging (join, leave, goal, kick, rate-limit, tick overrun) with `--log-level` | Greppable stdout/stderr events exercised in the smoke | -| 6.5 `[P]` | **DONE.** Arena rotation between matches; `--max-matches N` drain-and-exit | Smoke asserts two different arenas and `server_draining` | -| 6.6 `[P]` | **DONE.** systemd unit, Dockerfile, `SERVER.md` (ports, firewall, sizing per §1.4, and the SIGTERM caveat) | A third party can host from the docs alone | -| 6.7 `[D:3.6]` `[P]` | **DONE.** CI builds the server export and runs the smoke test against the **exported binary**, not source | `.github/workflows/dedicated-server-smoke.yml` runs `make verify-phase6` on clean checkout | - -> `dedicated_server=true` enables Godot's strip-visuals export mode, which replaces meshes and textures with placeholders per resource. Every relevant site is already headless-guarded — `ship.gd:167`, `ball.gd:25`, `goal.gd`, `arena_boundary.gd` — so the code should be safe. **Verify it against a real stripped build anyway**; this is the kind of thing that fails silently. - -> **Docker/VPS is the primary v1 deployment path.** Raw ENet self-hosting needs port forwarding, and SDR is Phase 7 — so Phases 1–6 ship something that works on LAN or a VPS and nowhere else. That is fine, but say it out loud rather than letting a player discover it. - -> Godot 4 gives GDScript no SIGTERM hook. `SIGTERM`/`Ctrl-C` kills the process immediately and clients see an ENet timeout (~5 s). Acceptable — but document it rather than letting it be discovered. `--max-matches N` under a process supervisor covers planned drains. - -> **Rcon is deferred past v1.** An authenticated remote command channel is a real security surface, and `--max-matches` plus a supervisor covers most of the need with none of it. - -**Phase gate:** `docker run` a server, connect from another machine over the internet, play a full match. **Precondition, not a footnote:** §0 item **C** — slot reservations keyed on display name alone — is fixed by task 7.4, so exposing this build to strangers is gated on that, not on this phase. +### Phases 0–6 — complete + +Every task in Phases 0–6 is implemented and verified locally: non-networked +refactors, transport/connection/lobby, server-authoritative simulation with +a dumb client, input pipeline hardening, prediction and reconciliation for +ship and ball, match lifecycle, and dedicated-server productionisation +(Docker export, rotation/drain, CI). The two remaining gates on this work +are human verification, not code — see §0 gates A and B. Task-by-task +acceptance evidence for Phases 0–6 has been trimmed from this document; +`git log -- multiplayer-next.md` has the full history if a past task's +reasoning is needed. ### Phase 7 — Steam transport, browser, identity -| # | Task | Acceptance | +**In progress.** GodotSteam requires custom engine builds and export +templates — **including for the headless server**; budget for it. The +`NetTransport` boundary (ENet + feature-gated `steam_transport.gd`) is +already extracted so this phase adds a second implementation rather than +retrofitting one. + +| # | Task | Remaining | |---|---|---| -| 7.1 `[D:1.2]` | **IN PROGRESS.** GodotSteam integration and custom export templates — **client *and* headless server** | Pinned build inputs and the reproducible validation command are documented; awaiting the custom binaries/SDK access | -| 7.2 `[D:7.1]` | **IN PROGRESS.** `NetTransport` boundary extracted with ENet and feature-gated `steam_transport.gd` (`SteamMultiplayerPeer`, SDR); advertising waits for `ISteamGameServer` work | `NetworkManager.host/join(..., transport)` selects explicitly; stock builds reject Steam without ENet fallback | -| 7.3 `[D:7.2]` `[P]` | **IN PROGRESS.** Server-browser UI and `ISteamMatchmakingServers` adapter remain intentionally unimplemented until the pinned GodotSteam client API is available; ENet direct-IP remains the supported browser-free path | No `server_browser.tscn` or fake Steam API has been added; implementation must wait for real Steam SDK/API access so Internet/LAN/favourites/history behavior can be verified against the actual service | -| 7.4 `[D:7.2]` `[P]` | **IN PROGRESS.** `TicketVerifier` now supports a synchronized backend ban decision before single-use ticket consumption; auth tickets in `hello` → `BeginAuthSession`, Steam identity in the roster and persistent ban list remain | `server/domain/auth.go` and adversarial tests reject banned identities without consuming their ticket and allow a later verification after unban; GodotSteam auth integration, server-side VAC state and durable ban storage remain | -| 7.5 `[D:7.2]` `[P]` | **IN PROGRESS.** `SteamBootstrap` gates initialization on the `steam` feature, `SteamMultiplayerPeer` class and Steam singleton; explicit Steam selection fails closed, while ENet remains the default and never becomes an implicit fallback | `test_net_transport.gd` proves stock builds keep ENet available and reject unavailable Steam requests without returning an ENet peer; the full local ENet multi-process gate passes with Godot 4.7.1, while the custom Steam client/server export smoke remains blocked on pinned GodotSteam binaries | -| 7.6 `[D:7.4]` | **IN PROGRESS.** Pure Go `AuthCoordinator` models pending/accepted/rejected/cancelled backend auth sessions around the single-use verifier, including maintenance expiry sweeping; `store.PostgresSessions` persists only token digests; the authenticated API issues sessions only from an injected verified-identity provider; Godot `ControlPlaneClient.login_steam()` now submits only the Web API ticket, validates the opaque response and stores the session in memory | `server/domain/auth.go`, `server/store/session_sql.go`, `server/api/service.go`, `control_plane_client.gd` and adversarial tests keep identity unavailable while pending, reject cancellation/expiry/wrong-attempt/replay, expire abandoned attempts at the boundary, release identity only after verifier success, reject invalid session inputs/extra identity fields, prove API delegation, validate ticket/session header boundaries and issue opaque sessions; real Steam BeginAuthSession/EndAuthSession adapter, login UI and live PostgreSQL/session integration remain | -| 7.7 `[D:7.1]` `[P]` | Obtain the production App ID, publisher key, SDR coordinator SDK/signing approval, certificates and hosted-data-centre support from Valve | Production prerequisites and rotation owners are recorded; no Spacewar credential or development certificate can reach a release build | -| 7.8 `[D:7.6,7.7]` | Ticketed Hosted Dedicated Server SDR: routing registration, coordinator-issued player→server relay tickets, client ticket installation, reconnect and expiry | Two real accounts complete and reconnect to an assigned dedicated match through SDR; server/player IPs are not exposed; ENet gates remain green | - -> **The transport interface is written here, not in Phase 1.** Eight virtual methods (`begin_auth`, `advertise`, `get_identity`, `supports_server_browser`…) designed against an API nobody on the project has used will be wrong. Write `NetworkManager._make_peer()` concretely in Phase 1 and extract the boundary once there are two real implementations. Locked decision 3 guarantees the ENet path is never deleted, so there is no migration risk in waiting. - -> GodotSteam requires custom engine builds and export templates — **including for the headless server**. That is the part people discover three weeks in. Budget for it. +| 7.1 `[D:1.2]` | GodotSteam integration and custom export templates, client *and* headless server | Awaiting the custom binaries/SDK access | +| 7.2 `[D:7.1]` | `NetTransport` Steam implementation (`SteamMultiplayerPeer`, SDR) | Server advertising waits for `ISteamGameServer` work | +| 7.3 `[D:7.2]` `[P]` | Server-browser UI and `ISteamMatchmakingServers` adapter | Unimplemented until real Steam SDK/API access is available; ENet direct-IP remains the supported browser-free path meanwhile | +| 7.4 `[D:7.2]` `[P]` | Auth tickets in `hello` → `BeginAuthSession`, Steam identity in the roster, persistent ban list | Real GodotSteam auth integration, server-side VAC state, durable ban storage remain. **Fixes known defect C** for direct/community servers once landed | +| 7.5 `[D:7.2]` `[P]` | `SteamBootstrap` gating (stock builds keep ENet, explicit Steam selection fails closed) | Custom Steam client/server export smoke remains blocked on pinned GodotSteam binaries | +| 7.6 `[D:7.4]` | Backend `AuthCoordinator`, session persistence, `ControlPlaneClient.login_steam()` | Real Steam `BeginAuthSession`/`EndAuthSession` adapter, login UI, live PostgreSQL/session integration remain | +| 7.7 `[D:7.1]` `[P]` | Obtain the production App ID, publisher key, SDR coordinator SDK/signing approval, certificates and hosted-data-centre support from Valve | Not started | +| 7.8 `[D:7.6,7.7]` | Ticketed Hosted Dedicated Server SDR: routing registration, coordinator-issued player→server relay tickets, client ticket installation, reconnect and expiry | Not started; depends on 7.6 and 7.7 | ### Phase 8 — Matchmaking, ranked ladder, per-match server autoscaling -**1.0 launch blocker.** Full design and reasoning: [`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). -The local control-plane, durable-store, allocated-server, and verification -paths are substantially implemented; the per-row status below distinguishes -that evidence from the remaining live Steam, PostgreSQL/Redis, Agones, release, -and human-playtest gates. Unlike Phases 0–7 this phase adds a component outside -the Godot project — a backend service — and that is the largest architectural -departure in the project's history, so read the design doc before picking up -any task below. +**1.0 launch blocker.** Full design and reasoning: +[`docs/MATCHMAKING.md`](docs/MATCHMAKING.md). This is the first phase to add +a component outside the Godot project — a Go backend service — and that is +the largest architectural departure in the project's history; read the +design doc before picking up any task below. The local control-plane, +durable-store, allocated-server, and verification paths are substantially +implemented; every row below lists only what's still open, not what's +built. **The critical path is task 8.31 — see §0's root blocker.** -This inverts the server model. Phases 1–7 build a **community server**: it -runs forever, waits for `--min-players`, plays a match, rotates arena, repeats, -and players find it by IP or (7.3) the server browser. Matchmaking makes the -*player* durable instead — queue, get grouped by rating, and a server is -**allocated for that one match** and destroyed after. Both models ship; they -are different playlists, not a replacement. +**Hard dependency on 7.6 and 7.8.** The local allocated path binds slot +reclaim to a control-plane-signed player identity and locks its team/slot +pair, but production Steam ticket verification is still required before a +rating can be trusted. Production allocation also depends on the ticketed +Hosted Dedicated Server SDR route; ENet remains the local/CI/community +transport, not a silent production fallback. -**Hard dependency on 7.6 and 7.8.** The local allocated path now binds slot -reclaim to a control-plane-signed player identity and locks its team/slot pair, -but production Steam ticket verification is still required before a rating can -be trusted. Production allocation also depends on the ticketed Hosted Dedicated -Server SDR route; ENet remains the local/CI/community transport, not a silent -production fallback. +This inverts the server model from Phases 1–7's **community server** (runs +forever, waits for `--min-players`, plays a match, rotates arena, repeats). +Matchmaking makes the *player* durable instead — queue, get grouped by +rating, and a server is **allocated for that one match** and destroyed +after. Both models ship; they are different playlists, not a replacement. + +Tasks 8.1–8.4 (versioned contracts, state transitions, an ADR locking the +Go/PostgreSQL/Redis/Agones stack, and launch SLOs) and 8.11 (threat model) +are done; everything below is what's left on the tasks still open. #### 8A — Architecture, contracts and data -| # | Task | Acceptance | +| # | Task | Remaining | |---|---|---| -| 8.1 | **DONE.** Add an ADR locking **Go + PostgreSQL + Redis**, provider-portable Kubernetes, Agones, ticketed Hosted Dedicated Server SDR, EU/NA fleets and independently runnable API, matcher, allocator and maintenance roles; keep `README.md`/`docs/TECH_STACK.md` consistent | [`docs/ADR-001-matchmaking-platform.md`](docs/ADR-001-matchmaking-platform.md) names boundaries/rejected alternatives; current docs name the locked stack and replaceable provider; no application code calls a provider allocation API | -| 8.2 `[D:8.1]` | **DONE.** Encode the launch SLOs from `docs/MATCHMAKING.md`: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health | [`docs/MATCHMAKING-SLOs.md`](docs/MATCHMAKING-SLOs.md) defines each metric, denominator, percentile/window, owner, alert threshold and release evidence | -| 8.3 `[D:8.1]` | **DONE.** Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | [`server/contracts/v1/`](server/contracts/v1/) contains machine-readable REST/events contracts and dependency-free structural tests; REST resync is specified by the contract; `server/api/service.go` also exposes the documented `/api/v1` route names (including server-assigned idempotent queue ticket IDs and DELETE cancellation) alongside the existing client `/v1` routes, covered by `TestDocumentedContractRoutesAdaptToServiceAPI` | -| 8.4 `[D:8.3]` | **DONE.** Define opaque IDs, legal queue/match state transitions, revisions and idempotency keys | [`server/contracts/v1/state-transitions.json`](server/contracts/v1/state-transitions.json) locks terminal states, legal edges, stale-revision handling and same-key replay/conflict behavior; contract tests cover the invariants | -| 8.5 `[D:8.4]` | **IN PROGRESS.** Initial PostgreSQL migration now defines durable idempotency keys, queue ownership/active-participation fencing, identities, sessions/revocations, ranked seasons, ratings/events, matches/participants, penalties, results, audits and outbox; follow-up migrations persist server-derived queue probe RTT metadata, the allocator GameServer/allocation registry, matcher-selected proposal region/protocol/team/slot plans, leased allocating-match claims, optional shared regional allocation quotas, initial-connect timing, and participant disconnect lease timestamps | `server/migrations/0001_initial.sql` through `0013_validate_initial_connect_ready.sql`, `migrations/runner.go`, `cmd/migrate` and static checks cover the durable tables, uniqueness/check constraints, Redis-as-cache boundary and serialized forward migration recording. Migration 0011 backfills legacy connected participants to generation one; 0012 validates the lease check, and 0013 validates 0010's initial-connect timestamp backfill, so inconsistent legacy lifecycle rows halt rollout instead of surviving behind `NOT VALID` constraints. The arena constraints remain deliberately `NOT VALID` for historical ranked records created before arena identity existed; they still fence every new write. `migrations.Rollback` reverses N most-applied migrations via matching down files; prior live rollback/reapply verification remains valid, while the new validations await a live database rerun because local Docker storage is exhausted | -| 8.6 `[D:8.3,8.4]` | **IN PROGRESS.** Add allocated-mode `ServerConfig` compatibility fields as opt-in defaults | `ServerConfig` now validates allocation mode, match/server IDs, playlist version, client build, future assignment expiry, image digest, transport and EU/NA region; `server_boot.gd` fails closed for the not-yet-wired Steam SDR transport, constrains allocated processes to one match, and emits allocation identity/transport in `server_started`; signed-authorisation admission, dynamic endpoint wiring and full manifest/runtime tests remain | +| 8.5 `[D:8.4]` | PostgreSQL migrations 0001–0013 (idempotency, queue fencing, identities, ratings, matches, results, audits, outbox, allocator registry, proposal plans, leases, quotas) | New validations await a live database rerun — Docker storage exhausted locally | +| 8.6 `[D:8.3,8.4]` | Allocated-mode `ServerConfig` fields | Signed-authorisation admission, dynamic endpoint wiring, full manifest/runtime tests remain | #### 8B — Authentication and secure control plane -| # | Task | Acceptance | +| # | Task | Remaining | |---|---|---| -| 8.7 `[D:7.6,8.3]` | **IN PROGRESS.** Pure Go ticket policy binds the expected App ID and verified identity, rejects expiry/replay/wrong app/malformed tickets, and consumes each ticket once. `cmd/control-plane` now wires `SessionIssuer: store.PostgresSessions{DB: db}` (same discovery/fix pattern as §8.10's `ResultSubmitter`: the adapter already correctly implemented `Issue`, just wasn't wired, so `/v1/session/steam` 503'd even before considering whether `SteamLogin` — the real, still-correctly-unwired Steam blocker — was available) | `server/domain/auth.go` covers single-use and binding invariants; real `AuthenticateUserTicket` backend adapter, bans, publisher secret store and Steam verification remain. A separate test-only `server/cmd/testkit-api` binary (never referenced by any Dockerfile/K8s manifest) substitutes a fake Steam login accepting any non-empty ticket, enabling §8.40's real Go+Postgres+Godot integration test | -| 8.8 `[D:8.7]` | **IN PROGRESS.** Pure Go session policy issues opaque short-lived tokens, stores only digests, authenticates by verified player identity and supports revocation; the production control-plane uses bounded atomic account+IP request limits | `server/domain/auth.go` covers wrong-token/expiry/revocation behavior; `server/store/session_sql.go` provides durable digest/revocation persistence and `server/api/rate_limit.go` plus `cmd/control-plane` provide per-replica request limiting; distributed revocation coordination and live Steam/session integration remain | -| 8.9 `[D:8.4,8.7]` | **IN PROGRESS.** Pure Go join policy binds SteamID/player/match/server/team/slot/protocol/expiry, rejects duplicate roster slots, permits same-identity reclaim through backend loss and fences prior server-owned generations. A workload-authenticated durable lease API atomically claims generations and records exact-generation disconnects against the allocation/match/server/participant roster. Allocated Godot startup now fails closed without valid control-plane lease configuration, and admission awaits one bounded durable claim before publishing the roster entry; definitive conflicts fail closed, while a known nonzero same-process generation may reconnect during an outage and queues its connect/disconnect sequence for ordered reconciliation. A fresh process never guesses generation one offline and can adopt a later backend generation only from a durably disconnected lease | `server/domain/reconnect.go`, `server/store/server_connection_sql.go`, migration 0011, `/servers/{serverId}/{connect|disconnect}`, `connection_lease_client.gd`, and adversarial tests cover missing workload configuration, active duplicate claims, stale disconnect fencing, exact 60-second reclaim, process recovery, wrong binding, initial assignment expiry, malformed/skipped responses, ordered outage rules, retry-safe active receipts, and migration backfill. Admission rechecks drain, token expiry, and peer presence after the awaited claim and releases a claim that became unusable. Live PostgreSQL/Godot process-restart and outage recovery verification remains | -| 8.10 `[D:8.5,8.31]` | **IN PROGRESS.** Dependency-free workload credential policy validates an adapter-verified signature, issuer/audience/time bounds, namespace/service account, pod UID, GameServer UID, allocation ID, match ID and server ID before result submission; `server/workload` now parses projected compact JWTs, rejects unsigned/malformed/ambiguous tokens, verifies signing input through an injected trust callback, and delegates exact binding to the domain policy; the API exposes the workload-authenticated server result route. **`WorkloadVerify` is now wired for real** -- the blocker this row previously named (`cmd/control-plane/main.go` never wiring it, so `/register` and `/result` 503'd on every real request) is closed by taking a different, equally-valid approach instead of the Kubernetes-JWT one: `serverMutation` only ever compares `WorkloadBinding.ServerID`/`.MatchID` (and `AdvanceServerRegistration` only additionally needs `.AllocationID`) -- nothing downstream requires Namespace/ServiceAcct/PodUID/GameServerUID, so a Kubernetes-issued token was never actually required, just the assumption that it was. `server/workload/signed_token.go` mints and verifies a short-lived HMAC-signed token with a secret only the control plane holds (the same trust model `domain.SessionStore` already uses for player sessions), needing no cluster JWKS/TokenReview to validate -- signature and expiry are fully self-contained. The token binds only `allocation_id` (deliberately, not `match_id`/`server_id` too: it must be requestable in the same Agones request that asks for a server, before Agones has picked one) -- `store.AllocationBindingByAllocationID` resolves `match_id`/`server_id` durably from the `allocations` table at verify time, so a token can never claim a pairing that wasn't actually, durably allocated. `api.WorkloadVerifierFromSignedToken` combines both and is wired into `cmd/control-plane` (`--workload-secret`/`COSMIC_CLASH_WORKLOAD_SECRET`; a startup warning fires and the route stays 503 if it's left unset) and `cmd/testkit-api` (fixed test secret). **The delivery channel is now wired too**: `agones.Client` gains `WorkloadSecret`/`WorkloadTokenTTL` -- when set, `Allocate` mints a token for the allocation and requests it as a third `cosmic-clash.io/workload-token` annotation alongside match-id/allocation-id, wired from `cmd/allocator`'s own `--workload-secret`/`COSMIC_CLASH_WORKLOAD_SECRET` (must match the control plane's); `supervisor.Supervisor.workloadToken()` resolves the bearer credential with an explicit `--workload-token-path` (kept for a possible future Kubernetes-JWT path) always winning, otherwise falling back to that same annotation -- the same fallback pattern `matchID()` already used for `cosmic-clash.io/match-id`. `WorkloadTokenPath` is accordingly no longer required at construction time | `server/domain/workload.go`, `server/workload/jwt.go`, `server/api/service.go` and adversarial tests reject every binding mutation, missing/unverified signature, `none`/malformed JWT, ambiguous audience, server/match mismatch and time boundary; `server/testkit/pipeline_test.go` carries allocation identity through the offline result path; `cmd/control-plane` now wires `ResultSubmitter: store.PostgresResults{DB: db}` (a ready-made adapter that had been referenced from nowhere at all, not even a test); `server/workload/signed_token_test.go` covers round-trip, tampered payload, wrong secret, malformed input and the exact expiry boundary; `server/api/workload_verifier_integration_test.go` (opt-in, real PostgreSQL) covers acceptance against a real allocation row, rejection of an unrecorded allocation, that two distinct real allocations each resolve to their own and only their own match/server pairing, and the previously-503 `Service.WorkloadVerify` field itself now succeeding -- all verified clean with `-race` across multiple runs; `cmd/control-plane/main_test.go`'s `TestServerRoutesRequireWorkloadVerifyToBeWired` now documents and pins the misconfigured-secret case specifically, not the "always unwired" case; `server/agones/allocation_test.go`'s `TestAllocateRequestsAWorkloadTokenAnnotationWhenConfigured` and `server/supervisor/supervisor_test.go`'s new annotation-fallback/fails-closed pair cover the delivery channel end to end (mint -> annotation -> supervisor read -> Authorization header) short of a live cluster. **Duplicate/conflict alerting is now wired**: `observability.Metrics.ObserveServerConflict(kind)` adds a dedicated `cosmic_clash_api_server_conflicts_total{kind}` counter (bounded to `register`/`connect`/`disconnect`/`shutdown`/`result`, matching `serverMutation`'s own routes), incremented at every `domain.ErrConflict`/`ErrResultConflict` branch in `serverMutation` -- deliberately separate from `ObserveAPI`'s generic 4xx-class bucket, which also catches ordinary client noise (malformed bodies, expired tokens) that isn't a duplicate/conflict signal at all. `deploy/observability/prometheus-rules.yaml` adds `CosmicClashControlPlaneServerConflicts`, alongside the existing p95/5xx rules, firing on >3 conflicts of one kind in 15 minutes. **What's still missing**: this has never run against a real Agones cluster (only HTTP-level fakes), so the exact `object_meta` JSON casing Agones actually returns remains unverified from this sandbox (see `GameServer.ObjectMeta`'s existing caveat); the alert itself has only been validated statically (`scripts/verify_observability_manifests.py`), never against a live Prometheus/Alertmanager firing on real traffic | -| 8.11 `[D:8.1]` | **DONE.** Write the threat model: forged clients/replay/queues/results, floods/bots, pod/insider compromise, gameplay and API DDoS, SDR signing-key theft, dependencies and denial-of-wallet | [`docs/THREAT-MODEL.md`](docs/THREAT-MODEL.md) records prevention, detection/response, owner and residual risk for every threat; it separates offline CA/online signer, client/game-server/PostgreSQL/Redis trust boundaries and recovery behavior | -| 8.12 `[D:8.11]` | **IN PROGRESS.** Provider-neutral Kubernetes baseline enforces restricted namespace admission, non-root/read-only/no-capability workloads, separate service accounts, default-deny networking and explicit edge/data/DNS/Agones/metrics flows; application manifests consume externally populated Secret objects; the Go API and allocator now have bounded per-replica account+IP rate/quota boundaries and metrics endpoints, plus an atomic operator-controlled degraded-mode gate that sheds new login/queue/proposal work while preserving live-match paths | `deploy/k8s/base/` includes hardened control-plane, allocator, and game-server resources; the control-plane now has health probes, a disruption budget, explicit zero-unavailable/one-surge rolling updates with graceful termination, failure-domain spreading/anti-affinity, and Secret-backed `COSMIC_CLASH_POSTGRES_DSN`/`COSMIC_CLASH_WORKLOAD_SECRET` runtime wiring; the authenticated WebSocket now requires RFC 6455 version 13, enforces a bounded 64 KiB frame size, two-minute idle deadline, 120-message/minute inbound budget, and bounded per-player fan-out; `server/security/test_kubernetes_policies.py`, `server/api/rate_limit.go`, `server/api/admission.go`, allocator metrics tests and adversarial checks cover static hardening, secret-reference invariants, fixed-window limits, bounded labels/key memory, atomic account+IP charging, route classification, concurrent toggling and degraded responses; production `cmd/control-plane` now wires the bounded limiter by default with `--rate-limit`, `--rate-limit-window`, and `--rate-limit-max-keys`, and the base Deployment declares those defaults explicitly; start with control-plane `--degraded`, enable with SIGUSR1, disable with SIGUSR2; private-store provisioning, distributed/global quotas, edge DDoS/WAF/origin shielding, encrypted backups and live policy/load tests remain | -| 8.13 `[D:8.12]` | **IN PROGRESS.** Docker/Kubernetes references are digest-pinned, a dependency-free checker rejects mutable tags/plaintext credentials and concrete release overlays can reject template digests; release documentation defines SBOM, dependency/image scanning, signing, admission verification and a 24-hour critical-fix SLA | `scripts/verify_supply_chain.py`, `server/security/test_supply_chain.py`, `docs/SUPPLY-CHAIN.md` and `.github/workflows/supply-chain.yml` cover repository policy and provenance requirements; registry SBOM/scan/sign/admission execution and a concrete production overlay remain | +| 8.7 `[D:7.6,8.3]` | Ticket policy binding expected App ID/identity | Real `AuthenticateUserTicket` backend adapter, bans, publisher secret store, real Steam verification remain | +| 8.8 `[D:8.7]` | Session policy (opaque tokens, digests, revocation) | Distributed revocation coordination, live Steam/session integration remain | +| 8.9 `[D:8.4,8.7]` | Join policy, durable reconnect leases | Live PostgreSQL/Godot process-restart and outage recovery verification remains | +| 8.10 `[D:8.5,8.31]` | Workload credential policy (signed tokens, not Kubernetes JWTs), delivery channel, conflict alerting | Never run against a real Agones cluster; alert validated only statically, not against live Prometheus/Alertmanager traffic | +| 8.12 `[D:8.11]` | Kubernetes hardening baseline, rate/quota limiting, degraded-mode gate | Private-store provisioning, distributed/global quotas, edge DDoS/WAF/origin shielding, encrypted backups, live policy/load tests remain | +| 8.13 `[D:8.12]` | Digest-pinned images, supply-chain policy checker | Registry SBOM/scan/sign/admission execution and a concrete production overlay remain | #### 8C — Queueing, matchmaking, playlists and rating -| # | Task | Acceptance | +| # | Task | Remaining | |---|---|---| -| 8.14 `[D:8.4,8.5,8.8]` | **IN PROGRESS.** Pure Go queue policy and PostgreSQL enforce one active ticket per verified player, 10 s heartbeat/30 s expiry, retry-safe owner/revision-scoped create/heartbeat/cancel, and deterministic candidate projection. Client cancellation is limited to `QUEUED`/`PROPOSED`; it cannot overwrite match-owned `ACCEPTED` through `LIVE` lifecycle states. A locked rejection classifier maps missing ticket, wrong owner, expiry, stale revision, and invalid state to distinct domain/API outcomes without weakening the atomic mutation predicate. Queue admission also honors both pre-live and live ranked abandonment penalties, so an expired reconnect cannot immediately requeue after result completion. Redis is an optional rebuildable projection over authoritative PostgreSQL | Domain/store/API tests cover ownership, expiry, idempotency, candidate binding, exact mutation-state fences, live-ticket cancellation rejection, stale revision classification, abandonment cooldown selection, concurrent create/heartbeat races, durable-source cache repair, Redis TTL/lost-keyspace behavior, and playlist/build/protocol compatibility. PostgreSQL-tagged lifecycle regressions compile and prior live runs cover the queue races; live database reruns remain blocked by Docker storage. Live Redis failover-under-load and worker integration remain | -| 8.15 `[D:7.8,8.3]` | **IN PROGRESS.** Pure Go probe validation treats Steam location as opaque, requires nonce/freshness/region and server-computed RTT, and implements discrepancy quarantine/release; authenticated HTTP now accepts only opaque location/nonce input through a server-owned probe provider and records validated RTT into the active player's durable queue ticket; durable queue projections have a server-derived RTT JSON field for matcher reads | `server/domain/probes.go`, `server/migrations/0003_queue_probe_metadata.sql`, adversarial fixtures and `server/api/service.go`/`store/queue_sql.go` cover stale/wrong/forged evidence, the 25 ms/30% threshold, three-sample quarantine, five-clean release, authenticated provider arguments, rejection of client RTT fields, player/ticket/expiry binding, persistence failure, playlist-scoped candidate reads and bounded metadata decoding; Steam coordinator, regional probe adapters and multi-region probe population remain | -| 8.16 `[D:8.14,8.15]` | **IN PROGRESS.** Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction; the runnable matcher now supports explicitly enabled ranked six-player polling with durable verified-Steam identity lookup. **Fixed a real crash-loop**: `Worker.Run` treated every `RunOnce` error as fatal to the whole loop, including "no compatible candidates" (`FormFromQueue`'s completely routine answer when currently-queued players share no verified region) — found building a live two-player integration attempt (see below): two real players with no common region crashed the entire matcher process, taking matchmaking down for every other player in the playlist, and would crash-loop again on restart since the same incompatible candidates stay queued. Now only genuine static misconfiguration (`ErrWorkerNotConfigured`/`ErrUnsupportedPlaylist`/`ErrInvalidMatcherSize`) stops the loop; everything else retries next interval. **Fixed a second, quieter wedge in the same area**: `FormFromQueue`'s anchor is always the single oldest candidate, deterministically, so when `domain.PrepareProposal` rejected that exact formation for a reason specific to those particular players (mismatched protocol, incomplete ranked identity metadata, a duplicate-SteamID pair) rather than "no compatible batch exists", `RunOnce` returned immediately and the next interval reproduced the identical formation and failed again — forever, permanently head-of-line-blocking every other waiting player behind that anchor too, not just the players actually at fault (this is the "innocent-ticket restoration" gap task 8.20 named: the innocents were never stuck in the database, since no claim had happened yet, but they were durably starved of ever being tried). `RunOnce` now excludes a failed formation's players and retries with the remaining pool, bounded to 8 attempts per pass; a batch that has no viable formation at all (the pre-existing no-common-region case) still returns immediately rather than looping pointlessly. **This fix was inert without a companion one**: `RunOnce` was asking `Source` for exactly `w.Size` candidates -- `SelectCandidates` was always designed to search a larger pool (it takes an anchor plus an arbitrary remainder and widens through it), but the call site never gave it one, so there was never a "remainder" for the exclusion retry to fall back to in production. `RunOnce` now requests up to 10x `w.Size` (capped at 200) instead | `server/domain/matcher.go`, `teams.go`, `server/matcher/worker.go`, `server/store/queue_sql.go` and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure, queue-backed oldest-anchor formation and incomplete ranked identity metadata; two new tests cover `Run` (not just `RunOnce`) surviving a per-pass error via a real concurrent goroutine, and still stopping immediately on a real configuration error, both clean across repeated `-race` runs; three further tests cover the formation-exclusion retry (an 8-candidate batch whose oldest 4 are permanently doomed still forms and claims the remaining 4, excluding the doomed players from the claimed ticket set), that exhausting every attempt still surfaces the last real error rather than a silent `false,nil`, and that `Source` is actually asked for more than `w.Size` candidates. **The two-player Godot proposal integration now passes**: `Game/tests/control_plane_proposal_smoke.gd`/`.tscn` and `scripts/verify_control_plane_proposal_integration.sh` found the original crash-loop bug above; headless Godot testing was then paused for several sessions after a run of native engine crashes (macOS crash reporter, `EXC_BAD_ACCESS`/`SIGBUS`) that day. Reading the actual `~/Library/Logs/DiagnosticReports/Godot-*.ips` crash reports (rather than relying on temporal correlation) found every one of the 25 reports on the machine named `ChatGPT`/`codex` (17), an already-exited process under that same tree (6), or a manual `iTerm2` session (1) as the responsible/parent process — none named Claude Code. Godot testing was resumed on that evidence (with the user's explicit go-ahead) and re-verified clean: `test_runner.tscn` (212/212), the full `make verify-enet-integration` suite (all five cases including the 3-process match), `verify_control_plane_proposal_integration.sh` (passed twice, real matcher forms the proposal and both clients accept), and the complete `make verify-multiplayer-local` gate -- zero new crash reports across all of it. **"Arena selection" was also stale**: `domain.RankedArenaForProposal` selects the arena deterministically at proposal time for ranked, `agones.Client.Allocate` already requests it (and playlist/region/build/protocol/transport) as Agones annotations, and `supervisor.withAllocatedCompatibility` already overlays every one of those onto the allocated Godot process's launch flags -- overriding the Fleet's static defaults, since a shared pod template cannot vary per-match on its own -- fully tested (`supervisor_test.go`'s `TestWithAllocatedCompatibilityOverridesStaleFlagsAndRejectsUnsafeValues` proves stale static flags are overridden by live annotation values) and wired into `Supervisor.Start()`. Casual deliberately never sets an arena path at all (`proposal.ArenaPath` stays empty for `domain.Casual` in `formation.go`); the supervisor's flag-override is then a no-op and the allocated server falls back to its own `ArenaRegistry.path_for_match` rotation, the same mechanism the community server already used -- this was always the intended design for casual, not a gap. §8.41's "dynamic per-match launch flags... remain" note describing this same mechanism was equally stale and is corrected there too. Long-running worker integration remains | -| 8.17 `[D:8.14,8.16]` | **IN PROGRESS.** Pure Go proposal policy sends a 10-second response window to every selected human, requires unanimous acceptance, applies exact casual/ranked decline and timeout cooldowns with ranked escalation, and exposes revisioned idempotent responses through the authenticated API. Proposal closure now atomically separates offenders from innocents: a decliner's ticket is `CANCELLED`; a timed-out player's ticket is `EXPIRED`; accepted or otherwise innocent participants return to `QUEUED` with their original `enqueued_at` and refreshed expiry. Direct queue cancellation closes the open proposal and requeues remaining participants immediately. Late API responses commit expiry, timeout penalties, and ticket release before returning `ErrProposalClosed`; recovery of an old declined proposal cannot misclassify its pending innocents as timeouts. Cooldown history rejects future, foreign-playlist, and invalid-kind events, and database rows are closed before penalty writes | Domain/store/API fixtures cover partial/unanimous response, expiry, replay/conflict, stale revision, exact cooldown windows/escalation, corrupt history filtering, offender ticket termination, innocent precedence preservation, direct-cancel cascade, and the former late-response rollback. PostgreSQL-tagged regressions compile and assert the durable split and penalty rows; the full local Go suite passes. Live PostgreSQL execution and allocation integration remain | -| 8.18 `[D:8.5,8.14,8.17]` | **IN PROGRESS.** Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using `FOR UPDATE SKIP LOCKED` plus durable uniqueness/revision fences; proposal creation persists matcher-selected region/protocol/team/slot topology before acceptance, and the same serializable final-acceptance transaction now promotes the exact roster into one `ALLOCATING` match, closing the process-crash gap that could otherwise strand an accepted proposal before the former second promotion transaction. The API promoter remains a replay check. Promotion replay validates immutable playlist/region/protocol/arena, participant, ticket, team, and slot identity but deliberately ignores mutable match state/server ownership, so a retry after a lost response still succeeds after allocation has advanced. Result sets are closed before crossing into promotion writes, avoiding one-connection pool stalls. Redis remains a rebuildable candidate projection over PostgreSQL authority | Store/API tests cover retries, claims, owner/revision fencing, expiry, exact promotion replay/conflict, progressed-match replay, rollback of partial claims, concurrent contested-ticket formation, and lost-cache repair. PostgreSQL-tagged regressions compile and assert acceptance, ticket transitions, match creation, and roster insertion are one durable outcome; prior live runs covered queue/proposal promotion and races, while this atomic-promotion change awaits a live database rerun. Allocation runtime integration remains | -| 8.19 `[D:8.18]` | **IN PROGRESS.** Pure Go casual lineup requires 2–6 humans with at least one per team, fills missing slots with explicit bots, permits kickoff-only bot-slot backfill and assigns no backfill penalty/rating update; proposal preparation now derives the lineup from formed teams | `server/domain/casual.go`, `formation.go` cover both-team minimum, bot shape, live-play rejection, zero-penalty backfill and casual proposal composition; queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties and live integration remain | -| 8.20 `[D:8.18]` | **IN PROGRESS.** Pure Go ranked admission requires six unique verified solo humans, rejects bots/backfill/parties, and allows only random-enabled non-elevated arenas; proposal preparation requires matching metadata for every formed player | `server/domain/ranked.go`, `formation.go` cover count, identity, party, bot/backfill, arena eligibility and formed-player metadata rejection; **innocent-ticket restoration is fixed, see §8.16**: a formation rejected by ranked admission (or any other formation-specific `PrepareProposal` failure) no longer permanently wedges the matcher on the same doomed anchor group, starving every other waiting player behind it. `ArenaRegistry` integration and allocation wiring remain | -| 8.21 `[D:8.5,8.20]` | **IN PROGRESS.** Pure Go rating core implements canonical Glicko-2, daily inactivity, ranked 1/3 and casual 1/N human-opponent weights, deterministic opponent ordering, and authoritative draw/overtime/abandon scoring; certified result completion now applies per-player updates inside the durable transaction with lexical row locks and revision increments | `server/domain/rating.go`, `server/store/result_sql.go` and tests cover canonical/inactivity/weight/invalid-input, draw/OT/abandon, ordered participant snapshots, lock/value re-read and rating update SQL; live PostgreSQL rating and seasons execution now covered (§8.23), and concurrent result transaction cases are covered: `TestPostgreSQLConcurrentIdenticalResultSubmissionAppliesRatingsExactlyOnce` races 5 identical submissions and confirms one rating application, while `TestPostgreSQLConcurrentConflictingResultSubmissionsKeepOneReceipt` races different payloads and confirms exactly one winner, one conflict, one receipt and one completion event; live maintenance/DB execution remains | -| 8.22 `[D:8.21]` | **IN PROGRESS.** Pure Go ranked profile exposes the first ten games as provisional, derives tiers only through validated backend-owned rating bands, and keeps casual ratings outside the API; authenticated HTTP now returns the authoritative ranked view. **"Authoritative ranked view" was durable-adapter-shaped but had no durable adapter**: `rankedProfile`/`profile` only ever read an in-memory map, so every real `GET /v1/profile/ranked` 404'd regardless of a player's actual rating. `RankedProfileProvider` (interface) + `store.PostgresRankedProfiles` close it, preferred over the map when set so existing tests/literals are unaffected; `LastSeasonID`/`SeasonHistory` deliberately left unset (no season pointer on `ratings`, needs its own query/semantics) | `server/domain/rating.go`, `tier_test.go` and `server/api/service.go` cover provisional override, exact band boundaries, malformed policy rejection, session authentication and ranked-only response fields; `server/store/ranked_profile_sql.go`, verified against real PostgreSQL via curl (a fresh identity correctly 404s through the real adapter) and via §8.40's integration test (`control_plane_smoke.gd` now asserts this exact 404 round-trips before queueing); persisted tier policy, client UI and reconnect transport remain | -| 8.23 `[D:8.21]` | **IN PROGRESS.** Pure Go ranked-only season policy compresses 25% toward 1500, clamps RD to 200–350, preserves volatility/history, is idempotent by season ID, and defines exact 12-week windows/due detection; migration and Go store now persist a per-player/per-season marker and rating update atomically; `cmd/maintenance` runs bounded due-season batches with signal-bound shutdown | `server/domain/rating.go`, `season_test.go`, `server/migrations/0001_initial.sql` and `server/store/maintenance_sql.go` cover compression, floor/cap, duplicate replay, window boundary, completed-season idempotence, bounded enumeration, row locking and conflict-safe rollover markers; opt-in PostgreSQL execution now covers the durable rating update, marker creation and duplicate replay without a second compression — re-run live for the first time as part of the wider integration-suite verification below, after fixing a test setup gap (a missing `seasons` row tripped the `ranked_season_rollovers` foreign key before the rollover logic itself ran); live maintenance/DB execution remains | -| 8.24 `[D:8.9,8.20,8.21]` | **IN PROGRESS.** Ranked connection policy binds match/server/player/Steam identity/team/global slot/protocol/expiry, permits a 60-second same-token reclaim with monotonically increasing server-owned generations, and applies the rolling 7-day 5 m/15 m/1 h/24 h abandon ladder. Admission rejects active duplicates, zero/reversed clocks, disconnect-before-admit, and duplicate disconnects. PostgreSQL persists generation/disconnect leases with serializable CAS: stale disconnects cannot evict newer generations, active leases cannot be reclaimed, initial admission requires an unexpired assignment, and later reconnects use the durable grace boundary. Godot consumes that lease before admission, reconciles known-generation outage events in order, closes future admissions on reconciliation divergence, and rejects unsafe generation-zero outage fallback. The hardened two-replica Kubernetes maintenance deployment and allocated Compose topology run the reconciler, which turns a ranked `LIVE` lease expired beyond 60 seconds into `abandoned_at`, a durable `MATCH_ABANDONED` cooldown, and a revisioned, targeted `state_changed` outbox event without releasing the participant or `LIVE` ticket needed by the result transaction | Go/store/API/Godot adversarial fixtures cover signature tampering, every binding, replay/conflict semantics, active duplicate admission, repeated valid reclaim, stale-generation fencing, grace boundaries, process recovery, expiry, zero/reversed clocks, malformed JSON generations, deterministic cooldown ordering, legacy-row migration, rolling-upgrade 204 compatibility, result-roster retention, outbox dispatch compatibility, maintenance deployment/PDB hardening, and cursor-pool safety. The current local gate passed all 212 Godot tests; focused Go suites and the PostgreSQL-tagged abandonment regression compile. Allocated Compose now seeds and observes the durable live-abandonment path; live PostgreSQL/process-restart/outage execution remains blocked by Docker storage | -| 8.25 `[D:8.10,8.24]` | **IN PROGRESS.** Workload-bound result policy validates match/server identity, nonce, scores, integrity state, canonical digest, and idempotent receipt conflicts. Result ingestion now owns the complete serializable production transition: a `LIVE` match and its active tickets advance through `RESULT_PENDING`, certified ratings are computed from locked authoritative participant rows, then match/tickets become `COMPLETED`, the receipt is acknowledged, and one revisioned outbox event is inserted. The allocated Godot server now emits the authoritative score to that route at `RESULTS`; its deterministic score-bound nonce makes every retry identical, and the match cannot leave `RESULTS` or exit until the API returns its committed `202` acknowledgement. Inactive pre-match no-shows are excluded; an active participant's durable `abandoned_at` forces loss scoring. Missing rating rows or ticket-count divergence fail the whole transaction. Zero-time/incomplete receipts fail before database use and conflicts wrap `ErrResultConflict`. Signed workload credentials now correctly carry the durable allocation/match/server binding without pretending to be Kubernetes JWTs; partial Kubernetes identity claims remain rejected | Domain/store/API/outbox tests cover workload and digest binding, identical/conflicting concurrency, direct `LIVE` completion, active-ticket completion, abandonment rating input, incomplete roster failure, integrity suppression, ordered rating locks, receipt/outbox atomicity, fan-out retry/ack ordering, delivery health, and signed-binding versus partial-identity validation. The allocated Compose gate now exercises an authenticated certified result and identical retry against the real verifier/store. Production credentials, Agones annotation persistence/reconciliation, and integrity-evidence adapters remain | - -The allocated-runtime result reporter now keeps a completed match in `RESULTS` until its exact score-bound, workload-authenticated result has received the API's committed `202`. Its 15-minute, server-side sudden-death cap turns an unresolved draw into `REVIEW` (no rating update), while allocator-issued workload tokens now default to two hours and expose a positive `--workload-token-ttl` setting. These bounds cover ordinary allocation, play, and result retry without treating a permanently unavailable control plane as a completed match. +| 8.14 `[D:8.4,8.5,8.8]` | Queue policy (ownership, heartbeat/expiry, candidate projection) | Live Redis failover-under-load and worker integration remain | +| 8.15 `[D:7.8,8.3]` | Probe validation (RTT, nonce/freshness/region, quarantine) | Steam coordinator, regional probe adapters, multi-region probe population remain | +| 8.16 `[D:8.14,8.15]` | Candidate/team formation, matcher worker | Long-running worker integration remains | +| 8.17 `[D:8.14,8.16]` | Proposal policy (response window, cooldowns, offender/innocent split) | Live PostgreSQL execution and allocation integration remain | +| 8.18 `[D:8.5,8.14,8.17]` | Store layer (serializable retries, claim SQL, atomic promotion) | Allocation runtime integration remains | +| 8.19 `[D:8.18]` | Casual lineup (2–6 humans, bot backfill) | Queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties, live integration remain | +| 8.20 `[D:8.18]` | Ranked admission (six unique verified humans) | `ArenaRegistry` integration and allocation wiring remain | +| 8.21 `[D:8.5,8.20]` | Rating core (Glicko-2, weights, transactional updates) | Live maintenance/DB execution remains | +| 8.22 `[D:8.21]` | Ranked profile (provisional games, tiers) | Persisted tier policy, client UI, reconnect transport remain | +| 8.23 `[D:8.21]` | Ranked season policy (compression, rollover) | Live maintenance/DB execution remains | +| 8.24 `[D:8.9,8.20,8.21]` | Ranked connection policy, reconnect lease, abandon ladder | Live PostgreSQL/process-restart/outage execution remains, blocked by Docker storage | +| 8.25 `[D:8.10,8.24]` | Result policy (workload-bound, idempotent, transactional) | Production credentials, Agones annotation persistence/reconciliation, integrity-evidence adapters remain | #### 8D — Agones, allocation and regional scaling -| # | Task | Acceptance | +| # | Task | Remaining | |---|---|---| -| 8.26 `[D:8.1,8.6,8.12]` | **IN PROGRESS.** Provider-neutral Kustomize base now defines a restricted Agones Fleet with region/build/protocol/transport labels and UDP game port, plus distinct EU/NA overlays; the base grants the allocator only namespaced Agones CRD access through the Kubernetes API and wires the digest-pinned supervisor image, control-plane Service, dynamic roster volume, signing/drain secret references and required network flow | `deploy/k8s/base/fleet.yaml`, `control-plane-service.yaml`, `network-policies.yaml`, `rbac.yaml`, `overlays/eu`, `overlays/na` and the manifest policy tests cover labels, replica floor, UDP declaration, pod hardening, supervisor/runtime arguments, Service selection, egress policy, overlay distinction, Kustomize rendering and allocator-only RBAC; operator secret/image replacement, second-provider fixtures, edge/DNS and SDR POP/cert/public-UDP overlays remain | -| 8.27 `[D:8.26]` | **IN PROGRESS.** Go supervisor package provides local-safe Agones REST discovery, validates assigned address/port data, injects dynamic `SDR_LISTEN_PORT`/`SDR_IP`, performs explicit process-ready probing and Ready transition; direct mode bypasses Agones | `server/supervisor/` covers allocated/direct startup, invalid endpoint rejection, dynamic endpoint/Ready ordering and authenticated drain; allocated Godot now supplies a loopback readiness/drain control surface and `agones_sdk.gd` supplies sidecar Health/Ready/Shutdown/annotation REST operations; metadata watch, real Agones annotation/shutdown confirmation and emulator integration remain | -| 8.28 `[D:8.6,8.27]` | **IN PROGRESS.** Supervisor separates explicit process-ready from Agones Ready and never scrapes stdout; allocated mode refuses to mark Ready without a configured readiness probe. The durable control-plane counterpart now exists: a workload-authenticated `POST /v1/servers/{id}/register` (and its `/api/v1` contract alias) advances a match's `ALLOCATING -> PROCESS_READY -> ASSIGNMENT_READY` state, and every participant's queue ticket with it, as one idempotent SERIALIZABLE transaction, gated on every participant already holding a live, unexpired assignment. The supervisor now calls it: once Agones Ready succeeds, it POSTs process-ready (`assignment_ready=false`) using a workload token read fresh from disk each call (matching kubelet's in-place rotation of a projected service-account token), and a registration failure kills the child rather than leaving an Agones-Ready-but-control-plane-unregistered process running; `ControlPlaneURL` unset (the default) is a total no-op. It then reports assignment-ready too: `server_boot.gd` already verifies its mounted roster synchronously before `/ready` is ever exposed (so process-ready implies the roster was valid), and the API's `ASSIGNMENT_READY` gate checks only durable `assignments` rows server-side — so no new Godot-side state was needed, correcting an earlier overcautious note here. The supervisor retries assignment-ready (default 5 attempts, 2s apart, configurable) since those durable rows may lag process-ready slightly; a persistent failure there does not kill the child, unlike process-ready. Per-allocation data (currently `match-id`) now has a real channel to an already-Ready pod: `server/agones.Client.Allocate` requests `cosmic-clash.io/match-id`/`cosmic-clash.io/allocation-id` as `GameServerAllocation.spec.metadata.annotations` (Agones applies these to the allocated GameServer's own `object_meta` — the only channel that exists post-allocation, since env vars are fixed at pod creation), and the supervisor reads them back from its existing `/gameserver` SDK call, falling back to them only when `MatchID` isn't explicitly configured. The image now exists: a new `Dockerfile` `game-server` target packages the supervisor as ENTRYPOINT alongside the same dedicated-server export `server` produces; the base Fleet now invokes that target with the control-plane URL, server/image Downward API identity, roster/signing/drain material, and exported Godot executable. | `server/supervisor/` tests prove Ready follows the probe and direct mode remains functional; `server_control.gd`, `agones_sdk.gd` and process-level smokes prove loopback `/ready`, `/health`, bearer-protected `/drain`, sidecar-shaped Health/Ready calls and drain admission fencing; `server/api/service.go`, `server/store/allocation_match_sql.go` and adversarial tests cover the registration route, digest/protocol validation, idempotent replay/conflict and the assignment-count gate; `server/supervisor/supervisor_test.go` covers opt-in registration, the workload-token/body/idempotency-key shape, the kill-on-failure path, the match-ID annotation fallback (both that it's used and that its absence fails closed before any HTTP call), the full process-ready-then-assignment-ready sequence and body shapes, and a retry test that fails assignment-ready twice with 409 before succeeding, asserting `Start()` still succeeds and the child is never killed; `server/agones/allocation_test.go` covers the requested annotations. `docker build --target game-server` verified for real: both binaries present, correct permissions, supervisor prints its usage; `server/store/stalled_allocation_sql.go`/`_test.go` and a live `TestPostgreSQLStalledAllocationsAreReclaimedWithoutPenalisingPlayers` cover the deadline boundary (a recent match must survive untouched), the no-penalty requeue and refreshed expiry, participant release, and idempotence against a second pass. **Fixed in passing**: `Dockerfile`'s `server` stage's `ubuntu` base digest had gone dead on Docker Hub (`docker pull` returned "not found", verified independently) — `make verify-phase6` was silently broken for a clean build before the re-pin; confirmed fixed with a full `make verify-phase6` run (arenas rotated, both goals observed, clean teardown). `deploy/cosmic-clash-server` now wraps its exec in `stdbuf -oL -eL` (falling back to unwrapped if unavailable), fixing a real, live-confirmed bug: a genuinely detached (`docker run -d`) container showed zero `docker logs` output — not even the startup line — for 20+ seconds while the process ran normally, and `docker stop`'s SIGTERM lost that buffered output permanently rather than delaying it; re-verified fixed against the real launcher script, then a full `make verify-phase6` re-run confirmed no regression. Health-reclaim now exists: `store.ExpireStalledAllocations` reclaims a match stuck in `ALLOCATING`/`PROCESS_READY`/`ASSIGNMENT_READY` past a deadline (server crashed, or was reclaimed by Agones as unhealthy, before ever registering) by failing the match and requeuing every participant to `QUEUED` with a fresh expiry rather than penalising them — task 8.50's own "infrastructure-caused cases cannot penalise affected players" criterion directly settles the requeue-vs-fail design question this had been blocked on. Wired into `cmd/maintenance` alongside the season-rollover sweep (`--stalled-allocation-deadline` default 2m, `--stalled-allocation-batch`). **Superseding the `fleet.yaml` framing above**: §8.10's `WorkloadVerify` blocker, and its delivery channel, are both now closed — a control-plane-self-issued signed token (not a Kubernetes JWT), minted by `cmd/allocator` into a `cosmic-clash.io/workload-token` annotation and read back by the supervisor, exactly the way `match-id` already worked — see §8.10. `/register` and `/result` no longer 503 unconditionally once every `--workload-secret` (control plane, allocator) is set consistently. The remaining gates are live Agones annotation/shutdown behavior and production cluster readiness; those are covered by §8.49 and remain explicitly open. | -| 8.29 `[D:8.26,8.27]` | **IN PROGRESS.** Supervisor discovers and validates the Agones endpoint, propagates the actual dynamic `--port`, and exports `SDR_LISTEN_PORT`/`SDR_IP` only for Hosted-SDR while preserving an isolated ENet path | `server/supervisor/` tests cover invalid address/port rejection, dynamic port argument/env propagation and SDR-vs-ENet separation; real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT and multi-match fixture remain | -| 8.30 `[D:8.18,8.26,8.28,8.29]` | **IN PROGRESS.** PostgreSQL leases each `ALLOCATING` match under a deterministic allocation ID, derives immutable compatibility from its accepted roster, and atomically binds only a durably recorded provider allocation while advancing every ticket. Fresh and recovered provider results now share the same fail-closed validation of allocation/match/server identity, region, build, protocol, arena, transport, allocated state, and non-empty endpoint before persistence or binding. Workers bind the canonical allocation returned by durable reconciliation rather than the provider's pre-persistence object, preserving server-owned timestamps and normalization. Ambiguous provider outcomes retain the lease and recover by allocation ID before another external request. Agones request/response parsing and Fleet labels remain provider-portable | Unit/adversarial tests cover every fresh/recovered compatibility mismatch, empty endpoint, canonical durable result propagation, lease recovery, bind/release fencing, quota behavior, accepted-proposal gating, provider ambiguity, malformed responses, and immutable labels. PostgreSQL-tagged allocator/race/integration suites and the Agones-shaped HTTP runner remain committed; this provider-validation change awaits live database/cluster reruns while Docker storage, kind, and Helm are unavailable. Full unknown-outcome cluster recovery and signed roster metadata remain | -| 8.31 `[D:8.9,8.30]` | **IN PROGRESS.** Assignment exposure requires Allocated state, exact allocation/match/server/region/build/protocol/transport compatibility, a hosted endpoint, and verified manifest/signature. Signed roster persistence now runs as one serializable transaction and proves the submitted set exactly equals the active durable match roster before writing any player row: allocation/server compatibility, player and Steam identity, canonical global slot, and team must all match. Partial rosters, unknown/substituted players, duplicate slots, mixed match/server/manifest batches, and zero revisions fail closed. Player recovery remains owner-, match-state-, server-, and expiry-scoped | Domain/store/allocator/API tests cover early exposure, tampered manifests/signatures, wrong compatibility, partial/mixed/duplicate rosters, durable Steam/team/slot mismatch, atomic no-row-on-failure behavior, expiry, and identical replay. PostgreSQL-tagged exact-roster regressions compile; live database and Agones reruns remain environment-dependent. **"Production signer... remain" understates this badly -- this is the actual root blocker of the whole allocation-to-connect pipeline, found 2026-09-04, flagged rather than fixed at the user's explicit direction (see §0)**: `store.SaveAssignment`/`SaveAssignments`/`SaveVerifiedAssignmentRoster` -- the only functions that ever write the `assignments` table this whole row describes -- are called only from tests, never from `allocator/worker.go`, `cmd/allocator`, or anywhere else in the real service; `allocator.Service.PublishRoster` (wired to `store.PostgresRosterStore`) is likewise never called from production code. `allocation_match_sql.go`'s `AdvanceServerRegistration` SQL requires an `assignments` row for every match participant before allowing the `ASSIGNMENT_READY` transition -- with nothing ever creating those rows, a real match cannot advance past `PROCESS_READY`, which also means §8.41's connect-wiring fix (`ControlPlaneClient._connect_when_assigned`) has nothing real to fetch in production even though it is itself correct. `TestRealSupervisorRegistersAllocatedServerThroughControlPlane` (the test that was supposed to prove this end to end) manually seeds `store.SaveAssignment` in its own setup rather than exercising the real write path, which is why this has never been caught. Closing it needs new security-relevant design, not just wiring: a join-signing key shared between the allocator (to sign) and the game server (`fleet.yaml` already mounts one for verification via `--join-authorisations-key-file`, but no control-plane binary has a matching signing flag), roster-digest computation, and per-player `domain.JoinAuthorisation` construction (slot/team from `match_participants`, `steam_id` from `identities`, reconnect generation) via the already-built `domain.SignJoinAuthorisationHMAC` | -| 8.32 `[D:8.2,8.26,8.30]` | **IN PROGRESS.** Provider-neutral FleetAutoscaler baseline preserves a two-process Ready buffer, caps warm capacity, and leaves Allocated scale-down independent of the Ready floor; Fleet image references remain digest-pinned for current/rollback pre-pull | `deploy/k8s/base/fleet-autoscaler.yaml` and manifest tests cover Fleet ownership, Buffer policy and floor/cap invariants; regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99 and N+1 certification remain | -| 8.33 `[D:8.26,8.32]` | **IN PROGRESS.** Fleet scheduling now requires on-demand capacity and spreads Ready processes across zones with skew 1; the autoscaler preserves the two-process Ready floor | `deploy/k8s/base/fleet.yaml` and manifest tests reject interruptible placement and single-zone concentration structurally; regional node pools, forced node-loss testing and measured N+1 headroom remain | -| 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog | -| 8.35 `[D:8.17,8.19,8.20,8.30,8.31]` | **IN PROGRESS.** Allocated Godot servers now claim each signed-roster admission through a match-bound workload-authenticated/idempotent lease API before publishing it locally, record exact-generation disconnects, and preserve ordered same-process reconciliation through a control-plane outage. PostgreSQL persists `connected_at`/generation/disconnect state, starts the fair deadline at durable `ASSIGNMENT_READY`, and atomically starts complete rosters, applies ranked 30 s no-show cancellation/abandon ladders, or applies casual bot/cancel outcomes after 60 s. The maintenance role evaluates pre-live outcomes and expired live reconnect leases every second. Godot's local clock is armed only after the same durable readiness transition and applies the same complete/partial roster policy | Domain/store/API/supervisor/Godot tests cover forged workload/allocation/player bindings, stale and concurrent lease fencing, replay after response loss, malformed rosters, complete ranked/casual starts, relaxed 2–5-human bot starts, canonical team/global-slot preservation, empty-team cancellation, stale-snapshot races, retryable datastore outages, unsafe fresh-process outage fencing, and readiness-clock ordering. Migration `0010_initial_connect_ready_at.sql` gives deployed in-flight matches a fresh window; 0011 preserves/backfills durable lease state. Live PostgreSQL execution, allocated process termination evidence, and real Agones multi-client verification remain | -| 8.36 `[D:8.10,8.25,8.28,8.30]` | **IN PROGRESS.** Supervisor exposes an authenticated loopback-only drain request boundary that never places the token in command arguments/logs and rejects remote/partial/query-bearing configurations; the base now includes a two-Ready PodDisruptionBudget; `Supervisor.Run` and `cmd/game-server-supervisor` now orchestrate signal-bound drain-before-kill with a bounded grace deadline | `server/supervisor/`, `server/cmd/game-server-supervisor/`, `server_control.gd` and `deploy/k8s/base/game-server-pdb.yaml` cover bearer-token enforcement, loopback URL validation, secret-safe configuration, missing drain credentials, readiness transitions, Ready-floor disruption protection, graceful child exit after drain and force-kill of an unresponsive child; live 300 s/285 s lifecycle, PDB/Fleet drain and infrastructure-abort classification remain | -| 8.37 `[D:8.5,8.10,8.25,8.26,8.31]` | Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO <=5 m/RTO <=30 m; signed result annotation/reconciliation preserves delivery outages | Restore/failover meet targets; live simulation continues; valid delayed result commits exactly once after recovery | -| 8.38 `[D:7.7,8.26,8.36,8.37]` | Provider migration after Valve approves both providers' EU/NA POPs/certs and public UDP: restore, validate coordinator trust, switch allocations, drain old | Both geographies complete Hosted-SDR matches on new provider and no old-provider live match is terminated | +| 8.26 `[D:8.1,8.6,8.12]` | Provider-neutral Fleet, EU/NA overlays, RBAC | Operator secret/image replacement, second-provider fixtures, edge/DNS, SDR POP/cert/public-UDP overlays remain | +| 8.27 `[D:8.26]` | Supervisor package (Agones discovery, Ready transition) | Metadata watch, real Agones annotation/shutdown confirmation, emulator integration remain | +| 8.28 `[D:8.6,8.27]` | Process-ready/Agones-Ready separation, control-plane registration | Remaining gates are live Agones annotation/shutdown behavior and production cluster readiness — see task 8.49 | +| 8.29 `[D:8.26,8.27]` | Dynamic port/SDR env propagation | Real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT, multi-match fixture remain | +| 8.30 `[D:8.18,8.26,8.28,8.29]` | Allocation leasing, compatibility validation | Full unknown-outcome cluster recovery and signed roster metadata remain | +| **8.31** `[D:8.9,8.30]` | Signed assignment/roster persistence, player recovery | **This is the actual root blocker of the whole allocation-to-connect pipeline (see §0).** `store.SaveAssignment`/`SaveAssignments`/`SaveVerifiedAssignmentRoster` and `allocator.Service.PublishRoster` are built and tested but never called from `allocator/worker.go`, `cmd/allocator`, or anywhere else in production — only tests seed the table directly. A real match cannot advance past `PROCESS_READY`. Closing it needs new security-relevant design: a join-signing key shared between the allocator (to sign) and the game server (`fleet.yaml` already mounts one for verification via `--join-authorisations-key-file`, but no control-plane binary has a matching signing flag), roster-digest computation, and per-player `domain.JoinAuthorisation` construction via the already-built `domain.SignJoinAuthorisationHMAC`. Flagged rather than fixed at the user's explicit direction, pending a decision on that design | +| 8.32 `[D:8.2,8.26,8.30]` | FleetAutoscaler baseline, Ready buffer | Regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99, N+1 certification remain | +| 8.33 `[D:8.26,8.32]` | Fleet scheduling, zone spread | Regional node pools, forced node-loss testing, measured N+1 headroom remain | +| 8.34 `[D:8.28,8.29]` | Native x86_64 benchmark of boot-to-process-ready/assignment-ready, p99 CPU/RSS/network, node cap with 30% headroom | Not started | +| 8.35 `[D:8.17,8.19,8.20,8.30,8.31]` | Admission lease, disconnect/reconnect generations, no-show/bot policy | Live PostgreSQL execution, allocated process termination evidence, real Agones multi-client verification remain | +| 8.36 `[D:8.10,8.25,8.28,8.30]` | Authenticated drain, PodDisruptionBudget | Live 300 s/285 s lifecycle, PDB/Fleet drain, infrastructure-abort classification remain | +| 8.37 `[D:8.5,8.10,8.25,8.26,8.31]` | Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO ≤5 m/RTO ≤30 m | Not started | +| 8.38 `[D:7.7,8.26,8.36,8.37]` | Provider migration (restore, coordinator trust, switch allocations, drain old) | Not started; needs Valve approval for both providers' EU/NA POPs/certs and public UDP | #### 8E — Client experience and recovery -| # | Task | Acceptance | +| # | Task | Remaining | |---|---|---| -| 8.39 `[D:8.3,8.14,8.17]` | **IN PROGRESS.** `MatchmakingState` now projects queue → proposal → allocation/process-ready/assignment-ready/connect/live plus terminal failure states; autoload `ControlPlaneClient` provides authenticated queue create/recovery/heartbeat/cancel and proposal response requests with idempotency/revision headers; `matchmaking.tscn`/`matchmaking.gd` expose the state and authoritative actions from the main menu; authenticated proposal recovery now reconciles missed proposal events and expires them at read time; the Go API publishes targeted authenticated revisioned queue/proposal/assignment events and the Godot client consumes state/proposal/assignment events, fetching the authoritative assignment after assignment readiness; the API can now use the durable participant-scoped proposal provider for both recovery and accept/decline mutations; durable allocation/no-show transitions write targeted state outbox rows and production/testkit dispatchers deliver them after commit; the client now explains queue wait progress and connection latency quality | `test_matchmaking_state.gd`, `test_control_plane_client.gd`, `test_matchmaking_ui.gd`, `TestProposalRecoveryIsParticipantScopedAndExpiresAtReadBoundary`, `TestAuthenticatedWebSocketDeliversOnlyTargetedRevisionedEvents`, `TestStateChangingAPIActionsPublishTargetedEvents`, `TestProposalRecoveryUsesDurableBackendAndRemainsParticipantScoped`, and state outbox tests reject stale/gapped/conflicting updates, validate endpoint/token/payload normalization, preserve idempotent duplicates, guarantee visible phase/terminal copy, and target every participant; live PostgreSQL-backed dispatcher/fan-out verification remains | -| 8.40 `[D:8.3,8.14]` | **IN PROGRESS.** Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated `/v1/events` WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot `ControlPlaneClient` can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return, including a deferred recovery when an event arrives during an in-flight HTTP mutation; `OutboxDispatcher` now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match; proposal creation writes one durable targeted `proposal_changed` outbox event per proposal aggregate revision, while result completion writes `match_completed` and production `cmd/control-plane` plus the test-only API harness dispatch both event types through separate filtered consumers | `server/domain/sync.go`, `server/api/events.go`, `server/api/outbox.go`, `server/api/service.go`, `server/store/outbox.go`, `server/store/proposal_sql.go`, `server/cmd/control-plane`, `server/cmd/testkit-api`, `service_test.go`, `outbox_test.go`, `matchmaking_state.gd`, `control_plane_client.gd`, `local_prediction_history.gd`, `net_ship_predictor.gd` and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation, prediction warm-up/hard-resync separation, deferred proposal recovery and proposal/result outbox filtering and delivery failures; Godot 4.7.1 headless project parse and 150-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; `cmd/control-plane` provides a signal-bound API role, `cmd/matcher` now supports casual and explicitly enabled ranked roles with durable identity lookup, and `cmd/maintenance` provides bounded season maintenance. **Live multi-process control-plane/game verification now exists**: `scripts/verify_control_plane_integration.sh` runs a real `postgres:17-alpine`, the real `api.Service` (via the new test-only `server/cmd/testkit-api`, wired identically to `cmd/control-plane` except for a fake Steam login — see §8.7), and a real headless Godot client (`control_plane_smoke.gd`) round-tripping login → `fetch_ranked_profile` (expect 404, §8.22) → `queue_create` → heartbeat → `cancel_queue` over an actual network connection — the first time this boundary was exercised end to end rather than against a mock on either side. **The two-player proposal extension now also passes**: `scripts/verify_control_plane_proposal_integration.sh` starts isolated PostgreSQL, the real matcher and testkit API, launches two headless Godot clients, seeds deterministic RTT projections, observes the real WebSocket `OPEN` proposal, accepts from both clients, and verifies both durable tickets plus the proposal reach `ACCEPTED` (including authoritative recovery when the concurrent revision advances). `scripts/run_result_fanout_integration.sh` additionally verifies a real PostgreSQL-backed authenticated WebSocket receives a completed-match event; allocator and Redis fan-out live verification remain | -| 8.41 `[D:7.8,8.9,8.31,8.40]` | **IN PROGRESS.** Authenticated `GET /v1/assignments/{matchId}` now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot `AssignmentState`/`ControlPlaneClient.fetch_assignment()` bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and `connect_to_assignment()` starts only the validated ENet/Steam transport after assignment readiness. **Found and fixed a severe gap this row had previously described as already closed**: `connect_to_assignment()` existed, fully validated, with its own `assignment_connection_started`/`assignment_connection_failed` signals -- but nothing anywhere in the client ever called it. A player who completed the entire queue -> proposal -> allocate -> assign pipeline would reach `ASSIGNED` and see "Your match server is ready" and then simply sit there forever; the transport was never actually started. `ControlPlaneClient._connect_when_assigned()` now calls it automatically the moment `state.phase` reaches `ASSIGNED` (wired into the one call site every queue-shaped HTTP response -- heartbeat, recover, and resync-triggered recover -- already shares, so both the REST poll and the WebSocket-triggered-resync path are covered without a second call site), deferring via `_pending_connect_match_id` if the assignment fetch triggered earlier by `ASSIGNMENT_READY` hasn't completed yet, and guarding against a duplicate/replayed `ASSIGNED` event reattempting the connection. The opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; `AssignmentProviderFromStore` wires that durable projection into the API injection point; `SaveAssignments` publishes a complete signed roster atomically instead of allowing partial player visibility; `SaveVerifiedAssignmentRoster` rechecks signed claims before deriving player rows; a workload-authenticated `GET /v1/servers/{serverId}/roster` now returns the complete signed envelope set only for the bound allocation, and the allocated supervisor atomically materializes it before launching Godot; live authenticated Godot/PostgreSQL assignment verification now exists via `scripts/verify_assignment_integration.sh`, which seeds a durable player-scoped `ASSIGNMENT_READY` row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary); `scripts/run_supervisor_integration.sh` additionally verifies the allocated-server process-ready → roster fetch/materialization → assignment-ready registration path against real PostgreSQL and the real API, with the workload token recovered from an Agones-shaped GameServer annotation | `server/api/service.go`, `store_adapters.go`, `service_test.go`, `assignment_state.gd`, `control_plane_client.gd`, `match_net.gd`, `server_boot.gd`, `server_config.gd`, `test_assignment_state.gd`, `test_control_plane_client.gd`, `test_match_net.gd`, `server/contracts/v1/openapi.json`, `server/migrations/0002_assignments.sql`, `server/allocator/allocator_integration_test.go`, `server/supervisor/supervisor_integration_test.go`, `server/supervisor/supervisor.go` and `server/store/assignment_sql.go` cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior, workload-authenticated roster delivery, atomic file installation, real allocator reconciliation, real supervisor registration and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; two new `test_control_plane_client.gd` tests cover the connect-wiring fix directly: `test_client_starts_the_transport_once_the_ticket_reaches_assigned` proves a ready, fresh assignment plus an `ASSIGNED` ticket update actually starts the transport (`state.phase` advances to `CONNECTING`, `connect_to_assignment`'s own signal fires) and that a duplicate attempt is refused, `test_client_defers_the_connect_until_the_assignment_fetch_completes` proves the opposite ordering (an `ASSIGNED` update before the assignment fetch completes) defers rather than either connecting with stale data or erroring; verified against the real Godot 4.7.1 binary (216/216, no crash), the full local gate and the ENet integration suite. **"Dynamic per-match launch flags" was stale, corrected in §8.16**: `agones.Client.Allocate` already requests arena-path/playlist/region/build/protocol/transport as Agones annotations and `supervisor.withAllocatedCompatibility` already overlays them onto the launch command, fully tested and wired into `Supervisor.Start()`. SDR relay-ticket installation and live Agones cluster integration remain | -| 8.42 `[D:8.22,8.23,8.24,8.40]` | **IN PROGRESS.** `RankedProfileState` and `ControlPlaneClient.fetch_ranked_profile()` expose the backend-authoritative rating/RD/volatility/games/tier/provisional/season view; matchmaking UI displays provisional/tier status without client-side rating math; runnable API binaries now wire the backend-owned default tier policy; live authenticated profile verification now exists via `scripts/verify_ranked_profile_integration.sh`, which seeds a durable rating and verifies the populated profile over real PostgreSQL, Go HTTP, and Godot JSON | `test_control_plane_client.gd` validates profile shape, numeric safety and provisional display; the real populated profile path now covers durable rating/games/tier/provisional decoding and caught the missing runtime tier-policy wiring; committed revision after reconnect, abandon status and season countdown remain dependent on live auth/backend events and Godot runtime verification | -| 8.43 `[D:8.39,8.40,8.41]` | **IN PROGRESS.** Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits `session_expired` and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling | `MatchmakingState` and `ControlPlaneClient` tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path. **This row's own "remain" list was stale**: `matchmaking.gd`'s decline button/handler already existed (`%DeclineButton`, `_on_decline_pressed`, visibility toggled by `MatchmakingState.PROPOSED` alongside accept) with no test gap; `ControlPlaneClient.can_retry_last_mutation()`/`retry_last_mutation()` -- the generic "duplicate-action recovery beyond proposals" and "regional outage retry" mechanism -- also already existed (any mutation, not just a proposal response, becomes retryable on a transport failure or 503/408/429, and the matchmaking queue button already fell back to it), it simply had zero test coverage proving the transition actually happens for a non-proposal mutation; two new tests close that (`test_generic_mutation_retry_recovers_after_a_transient_failure`, `test_generic_mutation_retry_is_not_offered_for_unsafe_failures`). `MatchmakingClient`'s real dispatch (`HTTPRequest.request()`) needs a live SceneTree that `test_runner.tscn`'s synchronous single-`_ready()` execution model cannot provide mid-suite, so the two new tests exercise the `can_retry_last_mutation()` decision boundary and the `ERR_INVALID_DATA` fail-closed path rather than the literal network call; verified against the real Godot 4.7.1 binary (214/214 tests, no crash, no engine-level error), plus a full `make verify-multiplayer-local` re-run. **Version-mismatch messaging is now built**: before this, there was no server-side protocol rejection at all -- `queue_create` accepted any `protocol_version >= 1` unconditionally, so an outdated client could only ever discover the mismatch by waiting forever unmatched (the matcher's own compatibility check requires every formed player to share an identical `protocol_version`), with no error and no explanation. `Service.MinProtocolVersion` (opt-in, zero by default) now rejects a below-floor `queue_create` with `426 Upgrade Required`/`client_outdated` before ever reaching the candidate provider, wired via `cmd/control-plane`'s `--min-protocol-version` flag; `ControlPlaneClient` recognises 426 on `queue_create` specifically and sets a distinct "Your client is out of date -- please update to continue searching" message, clearing `_last_queue_create` so the generally-available "Retry Search" affordance is never offered for a failure retrying can't fix. `TestQueueCreateEnforcesMinProtocolVersion`/`TestQueueCreateMinProtocolVersionZeroIsDisabled` (Go) and `test_outdated_client_receives_a_distinct_message_and_no_retry_offer` (Godot) cover the floor end to end: below-floor rejection before the candidate provider is ever reached, exactly-at-floor acceptance, the opt-in zero-disables-it default, the client message and the suppressed retry. **Failed-reconnect UX is now built too**: `connect_to_assignment()`'s synchronous failures (assignment missing/expired, invalid endpoint, `NetworkManager.join()` erroring immediately) only ever emitted `assignment_connection_failed` -- a signal nothing in the client listened to, leaving `state.phase` stuck at `ASSIGNED` and the UI showing "Your match server is ready" forever with no way back to a fresh search. Worse, the likelier real-world failure -- `NetworkManager.join()` returning `OK` immediately while the actual ENet handshake fails asynchronously later (unreachable server, refused connection, ENet's own ~5s connect timeout) -- had no handler at all for a matchmaking-driven connect, even though `main_menu.gd`'s own `_on_connection_failed` exists specifically to cover this exact async gap for the direct-join flow. `ControlPlaneClient` now connects both `assignment_connection_failed` and (guarded to `state.phase == CONNECTING`, so it never misattributes an unrelated direct-join failure) `NetworkManager.connection_failed` to `state.fail(...)`, so either failure mode now surfaces as a failed search the player can retry from, instead of a silent hang. `test_synchronous_assignment_connection_failure_surfaces_as_a_failed_search`, `test_async_network_connection_failure_after_assignment_ready_surfaces_as_a_failed_search` and `test_network_connection_failure_is_ignored_outside_a_matchmaking_driven_connect` cover both failure modes and the CONNECTING guard; verified against the real Godot 4.7.1 binary (220/220, no crash, stable across repeated runs), the full local gate and the ENet integration suite, zero new crash reports. Long-running worker integration (§8.16) remains | +| 8.39 `[D:8.3,8.14,8.17]` | `MatchmakingState`/`ControlPlaneClient`, queue/proposal UI, targeted revisioned events | Live PostgreSQL-backed dispatcher/fan-out verification remains | +| 8.40 `[D:8.3,8.14]` | Revisioned event stream, REST resync, outbox dispatcher | Allocator and Redis fan-out live verification remain | +| 8.41 `[D:7.8,8.9,8.31,8.40]` | Player-scoped assignment API, `connect_to_assignment()` wiring, join-authorisation verification in `MatchNet` | SDR relay-ticket installation and live Agones cluster integration remain | +| 8.42 `[D:8.22,8.23,8.24,8.40]` | `RankedProfileState`, backend-authoritative rating/tier display | Committed revision after reconnect, abandon status, season countdown remain dependent on live auth/backend events and Godot runtime verification | +| 8.43 `[D:8.39,8.40,8.41]` | Error/expiry UX, generic mutation retry, version-mismatch and failed-reconnect messaging | Long-running worker integration (§8.16) remains | #### 8F — Observability, verification, cost and rollout -| # | Task | Acceptance | +| # | Task | Remaining | |---|---|---| -| 8.44 `[D:8.3,8.4,8.28,8.31]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Go observability package encodes queue/proposal/match/server IDs and lifecycle stage while recursively redacting auth/relay tokens and credentials. `Service.Log` is wired to mutation and read routes at every outcome, and `cmd/control-plane` writes those events as JSON lines to stderr | `server/observability/` covers correlation fields, nested secret redaction, content-aware credential canaries and unnamed-event rejection; API tests cover lifecycle event wiring without logging error text. A production metrics/traces backend and dashboard/alert routing remain open; the local logger is intentionally stderr-only | -| 8.45 `[D:8.2,8.44]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Go observability package turns the documented RTT, allocation/connect latency, result-success, API-latency and tick/headroom thresholds into executable window checks; the API exporter emits a bounded cumulative latency histogram suitable for querying the documented p95 API SLO | `server/observability/slo.go`, `metrics.go`, `deploy/observability/prometheus-rules.yaml` and adversarial tests cover healthy/violating/empty windows, fixed operation/status labels, cumulative bucket boundaries, arbitrary-path cardinality safety, and optional API p95/5xx alerts. Production scrape configuration, alert routing, wait/MMR/proposal/flood/cost series and runbooks remain | -| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | **IN PROGRESS.** Go unit/race coverage spans the current domain/store/supervisor policies, and fuzz targets now exercise queue input, result payload hashing and revision events. The local gate uses the pinned headless Godot container when the native executable is unavailable or crashes by signal, while preserving ordinary nonzero test failures, so its full cross-language suite remains runnable without an image export | `scripts/verify_multiplayer_local.sh` passed end to end on the current tree: Go normal/race/vet, all three bounded fuzz targets, 212 Godot tests, contracts, migrations, and manifests. `server/domain/*_test.go`, `server/store/*_test.go`, `server/supervisor/*_test.go`, `server/migrations/*_test.go` and `server/domain/fuzz_test.go` provide the underlying coverage; PostgreSQL live migration execution now runs clean (§8.5), and five real-concurrency cases are covered against a live database with `-race`: §8.14's queue-heartbeat revision race, §8.18's two-matcher contested-ticket race, §8.30's cross-allocator-replica capacity race, §8.21/§8.25's concurrent identical-result-submission race, and now `TestPostgreSQLConcurrentProposalExpiryRecoveryAppliesCooldownsExactlyOnce`, which races 8 concurrent `GetProposal`/`RespondToProposal` calls (mixed read-recovery and a late accept) against one already-expired proposal and proves the design's own defense holds: `ProposalParticipantExpireSQL` only ever flips a still-PENDING row once, so a losing racer's `now` never matches `recordProposalTimeoutCooldowns`' `responded_at = $2` filter and cannot double-apply a `PROPOSAL_TIMEOUT` penalty -- verified against a real PostgreSQL container, `-race`, 3 repeated runs plus a full store-package integration run, all clean; the "lost Redis" fixture is covered live against a real server (§8.14: real TTL expiry, repair-after-`FLUSHALL`; fake Steam/allocator fixtures are §8.47's testkit, already done). **Fixed a real Redis-failover bug found while chasing this gap**: `CandidateProjection.Snapshot` funnelled "the index errored" (Redis unreachable) and "the index came back empty" (ambiguous: truly empty, or a lost keyspace) into the same `Repair` path -- but `Repair` itself calls `Index.Rebuild`, a second Redis round-trip that fails for exactly the same reason the first one did. A genuine Redis outage or mid-failover window therefore made `Snapshot` fail outright even though PostgreSQL, the documented authoritative source, was completely healthy -- contradicting Redis's own documented status everywhere (`RedisCandidateIndex`'s comment, `cmd/matcher`, `cmd/control-plane`'s `--redis-addr` help text) as an optional, rebuildable acceleration layer. `Snapshot` now falls back to serving `Source` directly whenever the index errors or comes back empty, attempting to repopulate Redis only best-effort (its outcome is deliberately ignored) — verified with both a killed miniredis instance and a real `redis:7-alpine` container (existing `TestRealRedisCandidateIndexUpsertSnapshotRemove`/`TestRealRedisCandidateProjectionRepairsAfterFlush` still pass unmodified). Live matcher-worker-under-load-during-failover integration remains | -| 8.47 `[D:8.7,8.30]` | **IN PROGRESS.** Offline testkit provides deterministic fake Steam verification and fake allocation with forced failure injection; the independent Compose runner drives fake-Steam session issuance, real HTTP queue create/heartbeat/cancel, matcher-backed six-player proposal formation/acceptance, and idempotency-conflict checks | `server/testkit/` covers verified identity/replay, unknown identity, wrong App ID, expiry, no capacity, compatibility-key conflict, idempotent allocation replay and cloud-free forced allocation failure; the Compose API/matcher slice is wired into CI, while live exhaustive matrix and production Steam remain | -| 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** Offline testkit exercises verified queue projection → ranked six-player proposal → ENet allocation → assignment-ready manifest → certified durable result receipt; `compose.allocated-smoke.yml` independently runs the real testkit API, matcher, allocator, maintenance, HTTPS Agones-shaped provider, PostgreSQL, and game-server supervisor with generated TLS, roster, and signed workload credentials. It verifies an expired ranked reconnect becomes one durable abandonment/cooldown, queue/proposal/allocation binding, authenticated result, idempotent retry, shutdown acknowledgment, durable receipt/audit rows, and supervised game-process stop without repurposing the Phase 6 fixture | `scripts/verify_allocated_compose.sh` passed on 2026-09-04 in this workspace; `.github/workflows/allocated-compose.yml` runs `make verify-allocated-compose`. Real Agones/kind and production evidence remain open | -| 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | **IN PROGRESS.** `scripts/verify_kind_agones.sh` creates a disposable kind cluster, installs pinned Agones, loads the real `game-server` image, applies the Fleet in an explicitly separate Agones-only supervisor/UDP readiness mode, and verifies readiness plus allocation of a dynamic UDP endpoint; `.github/workflows/agones-integration.yml` runs it for infrastructure changes and on demand | The cloud-free runner is committed and fails clearly when Docker/kind/Helm are unavailable. CI/live evidence for production control-plane registration, roster/no-show, both readiness stages, races, multi-match node, result-pending reconciliation, drain, and rollback remains open | -| 8.50 `[D:8.25,8.37,8.43,8.49]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** `make verify-chaos-recovery` provides a disposable PostgreSQL + real testkit API + real maintenance flow: it restarts the API, injects a stale allocation, and verifies no-penalty requeue plus a durable participant-targeted lifecycle event | Passed on 2026-09-04 in this workspace. 100 ms RTT/jitter/loss, matcher/client restart, game-pod death, node drain, Redis failover, control-plane loss, and live chaos evidence remain | -| 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | **LOCAL COMPLETE; PRODUCTION GATE OPEN.** The opt-in `make verify-multiplayer-load` gate drives 10,000 real HTTP queue-create requests through the service with 256 in flight and records p95/p99, plus 100 concurrent proposal formations through the real matcher/domain path; the handler and in-process ownership boundary are exercised without weakening normal tests | Passed on 2026-09-04 in this workspace. PostgreSQL saturation, durable matcher fencing under load, forecast launch concurrency x2, and replica scaling remain live infrastructure gates | -| 8.52 `[D:8.32,8.34,8.45,8.51]` | **IN PROGRESS.** Allocator supports both an opt-in per-replica fixed-window quota (`--allocation-quota` / `--allocation-quota-window`) and an optional PostgreSQL-backed EU/NA quota table consumed inside the serializable allocation transaction before any provider call; idempotent replays do not double-count. The allocator now exposes bounded EU/NA Prometheus counters at an explicit `/metrics` listener (`--metrics-addr`), including quota denials; the checked-in rule warns on regional denial activity, and hardened Kubernetes Deployment/Service/ServiceMonitor/PDB/placement resources plus an observability Kustomization provide the provisioning, health, rollout, disruption, discovery, and failure-domain-spreading contract | Normal/race/vet tests cover local quota exhaustion, window reset, region isolation, invalid input, atomic concurrent consumption, bounded metrics labels, read-only endpoint behavior, and hardened deployment/network policy/lifecycle/PDB/placement invariants; migration/SQL and manifest coverage define the shared quota/metrics boundaries; real image digest/secrets, measured regional cost model, threshold tuning, and denial-of-wallet rehearsal remain | -| 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | **IN PROGRESS.** `scripts/verify_release_gate.py` provides a fail-closed promotion check for the ordered development → internal → casual canary → casual → provisional ranked → ranked stages, requiring an evidence report for SLO, security, cost, rollback, EU+NA playtests, and both legacy gates | Validator and adversarial tests cover skipped stages, unknown stages, missing gates, non-boolean gate values, and blank release IDs; the actual reports, production rollback rehearsal, regional playtests, and live promotion remain open | +| 8.44 `[D:8.3,8.4,8.28,8.31]` | Structured logging, redaction | **Local complete; production gate open** — production metrics/traces backend and dashboard/alert routing remain | +| 8.45 `[D:8.2,8.44]` | SLO window checks, API latency histogram | **Local complete; production gate open** — production scrape configuration, alert routing, wait/MMR/proposal/flood/cost series, runbooks remain | +| 8.46 `[D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25]` | Go unit/race/fuzz coverage, local verification gate | Live matcher-worker-under-load-during-failover integration remains | +| 8.47 `[D:8.7,8.30]` | Offline testkit (fake Steam, fake allocation) | Live exhaustive matrix and production Steam remain | +| 8.48 `[D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47]` | Allocated Compose end-to-end (queue → proposal → allocation → assignment → result) | **Local complete; production gate open** — real Agones/kind and production evidence remain open | +| 8.49 `[D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36]` | Disposable kind+Agones cluster runner | CI/live evidence for production control-plane registration, roster/no-show, both readiness stages, races, multi-match node, result-pending reconciliation, drain, rollback remains open. Blocked locally on Docker storage/kind/Helm availability | +| 8.50 `[D:8.25,8.37,8.43,8.49]` | Chaos recovery (stale allocation, no-penalty requeue) | **Local complete; production gate open** — 100 ms RTT/jitter/loss, matcher/client restart, game-pod death, node drain, Redis failover, control-plane loss, live chaos evidence remain | +| 8.51 `[D:8.17,8.18,8.30,8.31,8.45]` | 10,000-client API load gate | **Local complete; production gate open** — PostgreSQL saturation, durable matcher fencing under load, forecast launch concurrency ×2, replica scaling remain live infrastructure gates | +| 8.52 `[D:8.32,8.34,8.45,8.51]` | Per-replica + shared regional allocator quota | Real image digest/secrets, measured regional cost model, threshold tuning, denial-of-wallet rehearsal remain | +| 8.53 `[D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52]` | Fail-closed release-gate promotion validator | Actual reports, production rollback rehearsal, regional playtests, live promotion remain open | Implementation invariants for every task above: @@ -1272,91 +254,79 @@ Implementation invariants for every task above: deterministic local/CI and direct-IP path. - One process serves one match. Warm processes/nodes absorb startup variance; capacity and cost are determined from 8.34 measurements, not old estimates. -- Implementation evidence is appended under the completed task as in earlier - phases; design changes first update `docs/MATCHMAKING.md` and dependencies. +- Design changes first update `docs/MATCHMAKING.md` and dependencies. --- ## 8. What needs refactoring, not extending -| # | Location | Why extension is insufficient | -|---|---|---| -| 1 | `objects/ship.tscn`, `ship.gd:175-180, 189-208, 241-278` | No node exists to carry a render-only offset — meshes hang directly off the `RigidBody3D`. Needs `$Visual`. | -| 2 | `ship_camera.gd:115, 149, 150` | Camera reads the body's transform, so it would jump the full correction error while the mesh smoothly lags. | -| 3 | `match_mode.gd:36, 59-64, 76-82, 93-96, 107-109` | The `Timer` + `_process` clock is frame-rate **and** `time_scale` coupled. Must become tick-derived. Five call sites. | -| 4 | `match_mode.gd:162-171` | `get_tree().paused = true` stops the client's own send loop and snapshot processing, and the return-to-lobby RPC lands in a tree that cannot act on it. | -| 5 | `game_mode.gd:95-121, 171-194` | `Engine.time_scale` is fundamentally incompatible with a shared tick clock — sequence numbers ride on `Engine.get_physics_frames()`, so a hit-stop at 0.06 starves the jitter buffer within a few frames. The *effects* must be reimplemented, not merely disabled. | -| 6 | `game_mode.gd:85-92` | `_handle_goal_scored` interleaves timing with presentation. On a headless server `_play_goal_celebration` returns **synchronously**, so the reset fires on the same frame as the goal — while clients are 1.6 s into a cinematic. | -| 7 | `game_mode.gd:248-263` | `_jittered` uses global RNG; `_reset_body` uses `set_deferred`. Both must become authoritative-broadcast plus a Jolt-correct teleport. | -| 8 | `game_mode.gd:54-55, 284-285` | Unconditional goal-signal connection (an interpolated ball entering a client's local `Goal` would score locally) and unconditional escape-respawn both write authoritative state on clients. | -| 9 | `main_menu.gd` (all handlers) | Every mode launch is a synchronous `change_scene_to_file`. Connecting is async and can fail — a genuinely new UI state, not another button. | -| 10 | `HUDController.gd:41-46` | Hard-requires a ship; spectators have none. | -| 11 | `player_ship_controller.gd` | Single reused `ShipAction` instance; buffering aliases every history entry. | -| 12 | `ship_camera.gd:86` (whole rig) | Runs in `_physics_process`, so on a 240 Hz display the FOV kick (`:182`) and `PostFX` parameters (`:186-187`) step at 60 Hz — neither is a transform, so global physics interpolation does not cover them — and the shake noise (`:200-212`) loses its high-frequency character. Must become `_process` + `get_global_transform_interpolated()` (§5.4a, task 0.16). | -| 13 | `video_settings.gd:14-16`, `settings_menu.gd` | Persists AA, glow and brightness only — three values. The three genuinely expensive settings (SDFGI, SSIL, SSAO) and the five shadow-casting lights are unreachable, and neither `vsync_mode` nor `max_fps` is set anywhere. A player chasing 240 fps has exactly one lever: turn glow off. Needs a preset system, not another checkbox (§5.5, tasks 0.17/0.17b). | -| 14 | `scenes/arena_base.tscn:18-50, 61-105` | The Environment every arena inherits enables SDFGI + SSIL + SSAO + a 5-level glow pyramid simultaneously, with four shadow-casting `OmniLight3D`s (24 cubemap faces/frame). Not tunable per-arena around a preset; the preset must gate the shared base (§5.5). | -| 15 | `shaders/post_process.gdshader:4` | `hint_screen_texture` forces a full-screen backbuffer copy **every frame**, not only during turbo — `vignette_strength` never reaches 0 (`ship_camera.gd:187, 243`). Either bake the static vignette into `Environment.adjustment_*` and hide `PostProcess` when `chromatic_aberration` is at rest, or drop the screen read for a plain gradient overlay and keep it only for the turbo chroma. | -| 16 | `project.godot [display]` | `stretch/mode="viewport"` + 1920×1080 base + `aspect="expand"` fixes the 3D render at ~1080p and blits. A 4K player cannot render native; a 1080p player cannot render lower. Blocks any render-scaling setting until decided (task 0.17c). | +Historical note: this table described Phase 0's non-networked refactors, +all of which are now implemented (see §7's Phase 0 summary). Kept for the +underlying reasoning where it's still relevant to Phase 7/8 work touching +the same files. -**On `Engine.time_scale`:** replace hit-stop and goal slow-mo with the camera-based effects **in single-player as well** (task 0.12), so there is one code path and one game feel to maintain rather than a networked variant that drifts away from the single-player one. `ShipCameraRig` already has `_shake_strength`, `shake_decay`, `max_shake_offset` and a `PostFX` `ShaderMaterial` to build on. +**On `Engine.time_scale`:** replaced with camera-based effects in +single-player as well, so there is one code path and one game feel to +maintain rather than a networked variant that drifts away from the +single-player one. -**What does not need surgery:** the `ShipController` seam, the Arena/GameMode split, code-driven spawning, group-based discovery, and the dumb `Goal` sensor all extend cleanly. `CLAUDE.md`'s claim about the three load-bearing seams is accurate — they hold. `rl_ship_controller.gd` is *already* the remote-input controller (a public `action` field that something else writes, pulled each tick), so no new class is needed for it. +**What does not need surgery:** the `ShipController` seam, the Arena/GameMode split, code-driven spawning, group-based discovery, and the dumb `Goal` sensor all extend cleanly. `CLAUDE.md`'s claim about the three load-bearing seams is accurate — they hold. `rl_ship_controller.gd` is *already* the remote-input controller (a public `action` field that something else writes, pulled each tick), so no new class was needed for it. --- ## 9. Godot 4.7 + Jolt gotchas 1. **`ENetMultiplayerPeer.server_relay` defaults to `true`** — clients can RPC each other through your server. Set it `false`. -2. **`MultiplayerAPI.poll()` runs on the idle frame**, so an `rpc()` from `_physics_process` waits up to a full frame — and `Engine.max_fps = 60` on the server is what creates that delay on the return leg. Take manual control (task 1.3). **~16–33 ms of round-trip, for ~10 lines.** +2. **`MultiplayerAPI.poll()` runs on the idle frame**, so an `rpc()` from `_physics_process` waits up to a full frame — and `Engine.max_fps = 60` on the server is what creates that delay on the return leg. Take manual control. **~16–33 ms of round-trip, for ~10 lines.** 3. **Jolt sleeps bodies.** A ship corrected to near-zero velocity can sleep and then ignore `state.linear_velocity` writes. `can_sleep = false` on Ship and Ball. 4. **Teleporting a rigid body**: `state.transform` inside `_integrate_forces` is the only path with no frame of lag. `set_deferred("global_transform", …)` lands between frames and interacts badly with Jolt's sleep/wake ordering. 5. **`reset_physics_interpolation()` is not automatic for `state.transform` writes** (it is when you set `global_transform` directly). Call it explicitly, on the body **and** on `$Visual`. 6. **`physics_jitter_fix = 0.0` does not give you "a flat 60 Hz."** You still get occasional 0-tick and 2-tick frames, because frame time is never exactly 16.667 ms. The real reason to set it to 0 is that you never want a tick's input *delayed* by the accumulator smoother. **The send path must therefore transmit both ticks' actions on a 2-tick frame** — redundancy-4 covers this, but only if you actually send both. 7. **`_integrate_forces` is not called on frozen bodies**, so remote ships never pull `get_action()` — hence `set_visual_action`. Use `FREEZE_MODE_KINEMATIC`, **not `STATIC`**, or contact velocity transfer breaks. 8. **Never write `linear_velocity` to a frozen body** — Godot/Jolt zeroes and holds it. -9. **`Engine.max_physics_steps_per_frame` defaults to 8.** If a server tick overruns 16.7 ms the accumulator backs up and the next frame runs multiple ticks, spiking CPU further. Log overruns (task 1.6). +9. **`Engine.max_physics_steps_per_frame` defaults to 8.** If a server tick overruns 16.7 ms the accumulator backs up and the next frame runs multiple ticks, spiking CPU further. Log overruns. 10. **ENet channel indices** are offset by Godot's reserved system channels — verify the mapping empirically. 11. **ENet peer timeout** defaults to ~5 s. Tune via `ENetPacketPeer.set_timeout()` for faster drop detection. 12. **Jolt is not bit-deterministic** across platforms or across differing contact orderings. Never rely on it anywhere, including in "obviously safe" places like a client-side goal check. -13. **`dedicated_server=true` exports strip visual resources.** Verify against a real stripped build (task 6.2). +13. **`dedicated_server=true` exports strip visual resources.** Verify against a real stripped build. 14. **MTU**: ENet fragments above ~1400 B. At 219 B/snapshot there is ~6× headroom; recheck if per-body cosmetic state is ever added. 15. **RPC NodePath caching**: the first `rpc()` to a node sends the full path, later calls send a cached int. Routing hot paths through autoloads warms the cache once at connect and never invalidates it on scene change. -16. **Physics tick rate is 60 for v1 — and must never be a literal.** Every policy in `Game/bots/` is tick-coupled through `ship.gd:450`'s `_tick_scaled` (defined at a 60 Hz reference) and `ai_ship_controller.gd`'s `reaction_ticks`, so raising it toward Rocket League's 120 invalidates every trained model and halves server density. But it is the largest single latency term left (§5.4), so it *will* be revisited: derive everything from `TICK_HZ` (tasks 0.18, 1.1) so that day is a config change plus a retrain. +16. **Physics tick rate is 60 for v1 — and must never be a literal.** Every policy in `Game/bots/` is tick-coupled through `ship.gd:450`'s `_tick_scaled` (defined at a 60 Hz reference) and `ai_ship_controller.gd`'s `reaction_ticks`, so raising it toward Rocket League's 120 invalidates every trained model and halves server density. But it is the largest single latency term left (§5.4), so it *will* be revisited: everything derives from `TICK_HZ`, so that day is a config change plus a retrain. 17. **`Node3D.get_global_transform_interpolated()` is the only correct way to track a physics-interpolated body from `_process`.** `global_transform` returns the last physics tick's pose, so a per-frame camera reading it chases a 60 Hz staircase. Per the engine docs the method "creates an interpolation pump… the first time it is called" — **call it once before any `reset_physics_interpolation()` on that node**, or the first hard snap streaks (§4.5). 18. **Physics interpolation covers transforms only.** `camera.fov`, shader parameters, light energy and anything else written from `_physics_process` steps at 60 Hz on a 240 Hz display. Either write them from `_process` or accept the stepping deliberately. 19. **`display/window/vsync_mode` defaults to enabled (FIFO) and `max_fps` to uncapped.** Neither is set in `project.godot`. FIFO present latency is **1.5–3 refresh intervals** depending on swapchain image count (2 vs 3) and whether the present queue is full — §5's tables use the optimistic 1.5, which assumes the renderer is *not* GPU-bound. **The model does not hold below refresh**, where a missed vblank under strict FIFO halves the effective rate and roughly doubles present latency. Prefer **Adaptive** as the default, not Mailbox (§5.4). *(Swapchain image count per platform needs empirical verification.)* 20. **`Engine.max_fps` is a throttle, not a frame pacer.** It pads each frame with a post-frame sleep; it has no vblank phase lock. Caps that are not integer divisors of the refresh rate beat against scanout, and combining a cap with an active vsync paces *worse* than either alone (§5.4). Derive the offered caps from `DisplayServer.screen_get_refresh_rate()`. 21. **`DisplayServer.window_get_vsync_mode()` echoes your request, not the driver's grant.** There is no GDScript API for the negotiated `VkPresentModeKHR`, so a UI cannot honestly report what was applied. Show a live fps readout instead and let the player infer it. -22. **`Engine.max_physics_steps_per_frame = 8` is a client problem too**, not just a server one (gotcha 9). A client hitching to 20 fps runs 3 ticks per frame, and each of those frames also runs the per-frame camera rig and remote-visual sampling. Set it to 4 client-side (task 0.22). On a multi-tick frame the send path must transmit **every** tick's action (gotcha 6) — §4.3's `_physics_process` sampling does this naturally, but nothing else guarantees it. +22. **`Engine.max_physics_steps_per_frame = 8` is a client problem too**, not just a server one (gotcha 9). A client hitching to 20 fps runs 3 ticks per frame, and each of those frames also runs the per-frame camera rig and remote-visual sampling. Set it to 4 client-side. On a multi-tick frame the send path must transmit **every** tick's action (gotcha 6) — §4.3's `_physics_process` sampling does this naturally, but nothing else guarantees it. 23. **`hint_screen_texture` forces a full-screen backbuffer copy on every frame the node is drawn**, regardless of what the shader then does with it. Branching inside the shader saves taps, not the copy. Hide the node when the effect is at rest. 24. **`physics_jitter_fix` matters less the higher the frame rate.** Its purpose is smoothing when frame rate ≈ tick rate; at 240 fps against 60 Hz physics most frames run zero ticks and the accumulator is never near an edge. Gotcha 6's reasoning for setting it to `0.0` still holds, but do not expect a visible difference on a high-refresh machine — test that change at 60 fps. -25. **`MultiplayerAPI.multiplayer_peer`'s default value is an `OfflineMultiplayerPeer` sentinel, not `null`.** Resetting it with `multiplayer_peer = null` (rather than a fresh `OfflineMultiplayerPeer.new()`) leaves the API in a state distinct from its own default and is a known source of "the server never sees `peer_connected`, `get_peers()` stays empty" bugs (godotengine/godot#81540) — confirmed the hard way while building task 1.2's `NetworkManager.shutdown()`. Always reset to a real `OfflineMultiplayerPeer`. -26. **Don't tear down a peer the instant its own connect signal fires.** `connected_to_server` (client-side) fires once the client's *local* view of the handshake completes, but the final ACK the server needs to consider *its* side complete may not have hit the wire yet — closing the peer or quitting the process in the same callback can drop it, and the other side then never sees `peer_connected`/`connected_to_server` at all, even though your own side looked successful. This isn't a corner case: it reproduced on **every** attempt until fixed, is easy to misdiagnose as a server-side bug (the server-side symptom — `get_peers()` staying empty — is identical to gotcha 25's), and cost significant debugging time before the actual cause (client-side premature teardown) was found. Give at least one frame — in practice `tests/net_smoke.gd` uses 0.3 s — between a fresh connect signal and calling `shutdown()`/`quit()`. Directly relevant to task 5.6's disconnect/reconnect controller swap and any CLI test client that connects, asserts, and exits quickly. -27. **`change_scene_to_file()` must be called on (or from a descendant of) the actual `get_tree().current_scene`, and never synchronously from `_ready()`.** Both failure modes were hit building task 1.5's `lobby.tscn`/`tests/lobby_smoke.gd`: (a) a test harness that instantiated `lobby.tscn` as a plain child of a driver node — rather than loading it as the real current scene, the way `main_menu.gd`'s Host/Join flow will — caused `lobby.gd`'s own (entirely correct, standard-pattern) `change_scene_to_file(ScenePaths.MAIN_MENU)` disconnect handler to hang the process completely on a real disconnect, with near-zero CPU (blocked, not spinning) and no error output; the fix was to load the scene the way production actually will, not to change the production code. (b) calling `change_scene_to_file()` (or `add_child()` on `get_tree().root`) synchronously from inside `_ready()` throws "Parent node is busy … Consider using `.call_deferred()`", because the tree is still mid-traversal adding the very node whose `_ready()` is running; `main_menu.gd`'s real button-press handlers won't hit this (they run outside any `_ready()`), but anything that needs to trigger a scene change during its own initialization must `.call_deferred()` it. -28. **`ENetMultiplayerPeer`'s `connection_failed` signal is not bounded to anything a UI should make a player wait for.** Verified empirically (task 1.7): against a genuinely refused loopback connection (nothing listening on the target port), `connection_failed` had still not fired 14 seconds in. Don't rely on it alone to end a "Connecting…" state — run your own app-level timeout (`main_menu.gd`'s `CONNECT_TIMEOUT_SECONDS = 6.0`) that shuts the peer down and shows an error regardless of whether ENet ever gets around to reporting failure itself. -29. **A `MultiplayerPeer`'s "am I a client" flag (however you track it — `NetworkManager.is_client` here) turns true the instant `join()`/`create_client()` is called, not once the connection actually completes.** Anything gated on that flag alone (task 1.8's clock ping, in `network_manager.gd`'s `_process`) will try to `rpc_id()` on a peer that's still `CONNECTING` — or has already failed — during a slow or refused connect attempt, and Godot logs "Trying to call an RPC via a multiplayer peer which is not connected" every single frame until it resolves. Gate on the peer's actual `get_connection_status() == MultiplayerPeer.CONNECTION_CONNECTED`, not just the higher-level intent flag. -30. **`load()` on a `.gd` file with a parse/compile error does not return `null`.** Found via adversarial review of `tests/test_runner.gd`: it returns a non-null but uninstantiable `GDScript` resource, so `if script == null` silently fails to catch the failure — and the natural next line, `script.new()`, throws "Invalid call: Nonexistent function 'new'", severe enough to abort the *entire calling function* (not just that statement) without ever reaching whatever cleanup/exit code follows. In a loop over multiple files with no per-iteration error boundary, this reads as a hang: the loop that would have moved to the next file, and the code that would have called `quit()`, both never run. The real guard is `Script.can_instantiate()`. -31. **An `@rpc` method named `_input` collides with `Node`'s built-in `_input(event: InputEvent)` virtual.** Found building task 2.1's `MatchSim` autoload: naming the client→server input RPC `_input(bytes: PackedByteArray)` produced a parse error ("function signature doesn't match the parent") — and because this was on an autoload, the error broke the **entire autoload from loading**, cascading into unrelated failures across every scene that touched `MatchSim` at all, none of which mentioned RPCs or `_input` in their own error output. Renamed to `_recv_input`. General lesson: on an autoload especially, treat any bare virtual-sounding method name (`_input`, `_process`, `_ready`, `_unhandled_input`, …) as reserved regardless of what you intend it to do — a signature mismatch there doesn't fail locally, it fails the whole autoload. -32. **Disabling automatic multiplayer polling (task 1.3) is global, not autoload-scoped — every scene that touches an RPC, not just `NetworkManager`-adjacent code, must call `NetworkManager.poll()` itself every frame it wants traffic to move.** Building task 2.1–2.3, `networked_match.gd`'s `_physics_process`/`_process` sent and listened for RPCs (`MatchSim.request_match_config`, `send_input`, snapshot RPCs) but never called `poll()` — nothing sent via `rpc()` in this scene ever reached the wire in either direction, silently, with no error in either process's log. Confirmed via debug prints: the client's request fired, but the host's handler print never appeared. The first (wrong) hypothesis was a startup race between the server's broadcast and the client's listener connecting — that fix (a request/response retry pattern, still worth keeping for the genuine late-join case) didn't resolve it alone. The real fix was adding `NetworkManager.poll()` at the top of both `_physics_process` and `_process` in the new scene. If a scene sends or receives RPCs and nothing arrives with no errors at all, check for a missing `poll()` before anything else. -33. **A request/response fallback for a one-shot broadcast can double-deliver, and the receiving handler must be idempotent.** Once gotcha 32's fix made polling actually work, `_on_match_config_received` ran **twice** per client — once from the server's original one-shot `_match_config.rpc()` broadcast (queued the whole time, since it had been sent before polling was fixed) and again from the request/response retry — producing two arenas, two ship sets, two HUDs (`_slots.size() == 2` instead of 1). Any handler reachable via both an original broadcast and a "resend on request" path needs its own guard (here: `if not _slots.is_empty(): return` at the top) rather than assuming "only sent once" from the RPC design alone. -34. **An `Area3D`'s `body_entered` signal fires as part of physics tick N's own step, strictly *before* tick N's `_physics_process` callback — not "on the next frame."** Found while fixing the goal-reset-ordering bug above: a boolean "handle this on the next `_physics_process`" flag set from inside a `body_entered` handler is a no-op, because that same tick's `_physics_process` hasn't run yet and sees the flag already true — it "defers" to the same tick it was set on, not the next one. If you actually need next-tick-or-later semantics, compare `Engine.get_physics_frames()` against the tick the flag was set on and require strictly-greater, not just "check a boolean at the top of `_physics_process`." -35. **A queued `queue_teleport()` (task 0.15) can take one tick longer to land than "the very next `_integrate_forces`" suggests, when the call originates from a signal handler mid-physics-step rather than from a `_physics_process` callback.** Empirically confirmed by teleporting a body into a goal and logging the server's own per-tick broadcast: the goal was detected on tick N (per gotcha 34, during tick N's own step), but the reset position didn't appear in a broadcast until tick N+1's, one tick later than "queued during N, applied on N+1's `_integrate_forces`" alone would predict. Don't assume queued-teleport timing without checking a real tick-by-tick log for your specific call site — the exact tick it lands on depends on where in the physics step the queuing call happens, not just "next frame" intuition. -36. **`NetworkManager.get_server_time_estimate_ms()` (and anything derived from it) is not just imprecise before the first clock pong lands — it's actively wrong in a way that can persist far longer than the sync window implies.** `clock_offset_ms` is `0.0` until the first pong, so a value derived from `get_server_time_estimate_ms()` during that window means "my own raw process uptime," not a server-synced estimate — and if that value feeds a rolling-window filter (e.g. a min-tracked bias, per the interpolator epoch-bias fix in Phase 2's adversarial review), the bad early sample can dominate the window for the filter's *entire* configured duration if a short test or a short match doesn't run long enough for real time to age it out. Always gate recording, not just consuming, anything derived from this estimate on `rtt_ms >= 0.0`. -37. **Anything that deliberately delays an RPC dispatch (task 2.8's `net_sim.gd`) must re-validate its target at *fire* time, not just at the moment it was scheduled.** Found by actually running Phase 2's own gate (`networked_match_smoke` under `--net-sim-latency=80 --net-sim-jitter=20`), not the isolated ping/pong test alone: `_broadcast_snapshot`'s existing `get_peers()` filter (gotcha from task 2.2's own fix) only proves the target was valid *when the send was queued* — a target that legitimately disconnects during the ~80–100ms hold produces "Attempt to call RPC with unknown peer ID" anyway, and if the delay outlives this *process's own* `shutdown()`, `multiplayer_peer` has already been reset to a fresh `OfflineMultiplayerPeer` (§9 gotcha re: never resetting to raw `null`), so a stale `rpc_id(1, …)` now means "call yourself" and Godot rejects it. Any deliberate-delay layer needs to re-check both "do I still have a real peer at all" and "is this specific target still in `get_peers()`" inside the delayed callback itself, immediately before dispatching — not reuse whatever validation the caller did before the delay was added. -38. **A GDScript lambda captures an enclosing local variable BY VALUE at the moment the lambda is created, not by reference.** Bit two separate Phase 3 test scripts the same way: `var disconnected := false; some_signal.connect(func(): disconnected = true)` compiles and runs with no error or warning, but the assignment inside the lambda mutates only *that lambda's own captured copy* — the enclosing function's `disconnected` stays `false` forever, even after the signal genuinely fires (confirmed firing via an extra debug print before the real cause was found). The underlying disconnect-detection code was correct the whole time; only the test's own assertion logic was broken. The fix is to capture a container instead of a value — `var disconnected := [false]` and `disconnected[0] = true` inside the lambda — since capturing an `Array`/`Dictionary`/`Object` captures a reference to the same instance, and mutating its *contents* from inside the lambda is visible outside it. Relevant anywhere a lambda is used to flip a flag or accumulate a result for a caller to read later (a `connect(func(): ...)` one-liner is the single most common place this bites). -39. **A fixed-size ring buffer fed by an unbounded-rate producer needs an explicit resync path, not just "wait for the next expected slot."** `InputJitterBuffer`'s 32-entry ring assumed the consumer (`consume()`, one call per server physics tick) would never fall more than `RING_SIZE` ticks behind the producer (`ingest()`, driven by real wall-clock packet arrival, unrelated to the consumer's own tick rate) — but a server-side stall, or even ordinary client/server clock drift with zero external trigger, breaks that assumption, and once broken, a design that only ever advances its "expected" pointer by exactly one per call can never catch up: newer arrivals silently overwrite the exact slot still being waited on, and the wait never ends. If a ring's producer and consumer rates aren't provably bounded relative to each other, the consumer needs a way to detect "the data I'm waiting for no longer exists in the ring at all" (track the newest value ever seen, independent of ring capacity) and jump directly to what's still available, rather than assuming "keep waiting" is always eventually correct. -40. **A client-owned adaptive control loop must react to the actual ground-truth signal it's regulating, not to its own memory of past decisions.** `InputLeadController`'s release logic was gated on `lead > LEAD_MIN` — a count of the controller's own past attacks — rather than on the real server-reported `input_buffer_depth` it exists to keep near target. Any elevated depth the controller didn't itself cause (an external stall, drift, a burst redelivery) was invisible to that gate and so never got drained, even while the "real" signal sat well above target the whole time. When a control loop's condition for acting can be satisfied or blocked by state the loop itself controls, rather than by the environment it's meant to respond to, it can silently stop responding to the environment. -41. **A "consecutive N over-budget windows" streak counter that hard-resets to 0 on any single clean window is trivially evaded by a duty-cycled attacker** (burst hard, one clean window, repeat) — confirmed sustaining ~33x a stated packet budget indefinitely with zero disconnect warnings. A leaky-bucket accumulator (grows by each window's actual total, drains by exactly one window's worth of budget every window, disconnect once the accumulated excess crosses a threshold) is immune to the same evasion by construction, since it doesn't matter how the excess is distributed in time — only the sustained average matters. -42. **Two counters that don't share an epoch must never be compared directly, even when both are monotonically increasing integers that "look like" the same kind of thing.** `seq > Engine.get_physics_frames() + 20` compiled, ran, and looked like a sane bound — but `Engine.get_physics_frames()` counts from the SERVER PROCESS's own start while a client's `_input_seq` starts at 0 when ITS match scene loads, so the check either never fires (on a long-running server, no real protection despite its own comment's claim) or fires wrongly and silently drops an honest client's input forever, depending entirely on how much unrelated head-start or drift has accumulated between the two clocks. Bound a value against another value that shares its own actual epoch (here: the receiving buffer's own `last_applied_seq`), not against a same-typed number from a conceptually different clock. -43. **A regression test that doesn't independently exercise the specific mechanism it claims to gate will pass even when that mechanism is completely broken.** Task 3.6's CI driver asserted snapshot throughput and a server-*forced* goal's score agreement — neither of which depends on client input ever reaching the server — and kept reporting PASS with a real, reproduced bug (§7's ring-overflow) actively zeroing both bots' input for the whole run. A CI gate's assertions should trace back to the specific claim in the task's own acceptance text, not just "the match ran and didn't crash." -44. **When adding a "is this connection still healthy" check to a test, sample it while the peer is still actively connected — not after its own end-of-run disconnect, which produces symptoms indistinguishable from the bug being checked for.** Fixing gotcha 43 first sampled `InputJitterBuffer.stalled` and ship movement *after* the full test run (plus a buffer for score-file writes), which meant both readings came from ~4s after the bot had already legitimately shut down — a departed peer's input naturally starves and goes `stalled=true` too, and that's correct, expected behaviour, not the bug. Move the check to a moment still comfortably inside the peer's own active connection window. -45. **Two fixes landed in the same commit, each individually correct in isolation, can share a variable and silently cancel each other out — and a fix's own unit test can miss it by testing the mechanism in isolation from the thing that defeats it.** Gotcha 39's resync fix and gotcha 42's guard rebound were reviewed, tested, and verified independently, each against its own scenario, both passing. Combined, the guard caps the exact variable (`highest_ingested_seq`) the resync's own trigger condition depends on, making it permanently unreachable — recreating the original critical bug at a *lower* failure threshold than before either fix existed. The resync's own new unit test called `InputJitterBuffer.ingest()` directly, which is correct in isolation but bypasses the guard entirely, so it could never have caught this regardless of how thorough it was on its own terms. **When two fixes in the same round touch the same subsystem, explicitly re-test the combination end-to-end** (here: a real `SIGSTOP` freeze against the actual production RPC call path, not a direct unit-level call into the class the fix lives in) — passing tests for each fix individually is not evidence the pair composes correctly. -46. **A guard that bounds an incoming value against the *consumer's* position, rather than against the *producer's* own epoch, re-introduces exactly the "consumer can never catch up past a stall" failure it's often added specifically to prevent.** The seq-range guard bounded `seq` against `last_applied_seq` (advanced only by `consume()`, i.e. gated on however fast the physics tick loop is actually running) rather than `highest_ingested_seq` (advanced by `ingest()`, i.e. gated on however fast packets are actually arriving and being processed by `poll()`) — during a stall where ticks fall behind but polling keeps pace (the common case: a single-frame hitch, or `Engine.max_physics_steps_per_frame` capping tick catch-up while `poll()` itself isn't similarly capped), bounding against the lagging consumer rejects the very packets that would let the buffer refill and the resync condition ever trigger. Bound against whichever side of a producer/consumer pair is not the one already known to be falling behind. -47. **A trace that holds its inputs steady cannot falsify anything about *which sequence* a prediction is filed under — and "we hold thrust for 60 seconds" describes almost every prediction test people write.** Phase 4's history was filed under the wrong sequence (the estimated server-consumption seq, `input_lead` ticks behind issuance, instead of the issuing seq), and the dedicated action-marker instrument built to catch exactly that reported a flawless `marker=0/3784` across 60-second LAN, 80±20 ms and 5%-loss runs. It was not broken: while the commanded action is constant, "the intent from this tick" and "the action the server consumes for seq S" hold the same *value*, so a right and a wrong label are indistinguishable. Only an input **edge** separates them, and only for about `input_lead` ticks per edge. The bug then scales with `input_lead` — 9.3% mismatch at lead 1, 24% at lead 3 — meaning it was worst precisely on the impaired links the test matrix existed to cover, and invisible in all of them. **When a test is meant to validate a label, an index, or a phase relationship rather than a magnitude, the trace has to change that quantity frequently**; a steady-state trace validates the magnitude and silently asserts nothing about the label. -48. **A guard whose bound is derived from a value only the ACCEPTED path can advance is a latch, not a guard.** The seq-range check has now been written three times — bounded against server uptime, then `last_applied_seq`, then `highest_ingested_seq` — and all three could permanently reject an honest client's input, because in every version the quantity being compared against could only move forward via a packet that got through. Once enough drift or loss accumulated, nothing could ever move it again. The property to check when writing a guard like this is not "is the bound correct?" but "**if this guard rejects everything from now on, what advances the bound?**" If the answer is "an accepted packet", it needs an independent escape path (here: resync after N consecutive rejections) regardless of how well-chosen the bound is. -49. **Advancing a consumer cursor past data that has not arrived is not a lossy shortcut — it is permanent, because the producer-side filter then rejects the very data being waited for.** `InputJitterBuffer.consume()` advanced `last_applied_seq` on a starve, and `ingest()` discards `seq <= last_applied_seq`. One starve on a sequence the client had not sent yet therefore stranded the stream one ahead of arrivals *forever* — both sides advancing in lockstep, the gap never closing, every packet discarded on arrival. The client's own routine `input_lead` release was enough to trigger it, roughly every 6.5 s on a clean LAN. **Only give up on an expected item once strictly newer data proves it lost**; "it hasn't arrived yet" and "it will never arrive" are different states and must not share a code path. -50. **A metric that stops sampling during a failure will report that failure as healthy.** The action-marker gate printed `SMOKE PASS` at 3.76 % on a run where the player's input was permanently dead — because reconciliation suppression stops `_record_metrics` being called, so the worse the outage, the fewer samples and the *lower* the computed mismatch **rate**. Every rate-shaped assertion needs a companion assertion on the **denominator** (here: a sample count scaled to run length), or an outage silently becomes an absence of evidence and then evidence of absence. -51. **An architectural blocker inherited from a previous session is a claim to verify, not a premise to build on.** Phase 4 was handed over blocked on approval for a client-only shadow Jolt world — a large subsystem, and effectively the whole-world rollback §1's locked decisions rule out. The actual same-sequence defect turned out to be a one-line mislabel, falsifiable in about an hour with instrumentation that already existed; the shadow world remains genuinely necessary for the *contact* cohort but nothing else, which is a far smaller commitment than "Phase 4 is blocked on it." Reconstruct the failing invariant from the code and reproduce it against a control before accepting a scope estimate attached to it — especially when the recommendation arrives without the cheaper alternative recorded as tested. +25. **`MultiplayerAPI.multiplayer_peer`'s default value is an `OfflineMultiplayerPeer` sentinel, not `null`.** Resetting it with `multiplayer_peer = null` (rather than a fresh `OfflineMultiplayerPeer.new()`) leaves the API in a state distinct from its own default and is a known source of "the server never sees `peer_connected`, `get_peers()` stays empty" bugs (godotengine/godot#81540). Always reset to a real `OfflineMultiplayerPeer`. +26. **Don't tear down a peer the instant its own connect signal fires.** `connected_to_server` (client-side) fires once the client's *local* view of the handshake completes, but the final ACK the server needs to consider *its* side complete may not have hit the wire yet — closing the peer or quitting the process in the same callback can drop it, and the other side then never sees `peer_connected`/`connected_to_server` at all, even though your own side looked successful. This reproduced on **every** attempt until fixed and is easy to misdiagnose as a server-side bug (the server-side symptom — `get_peers()` staying empty — is identical to gotcha 25's). Give at least one frame (in practice `tests/net_smoke.gd` uses 0.3 s) between a fresh connect signal and calling `shutdown()`/`quit()`. +27. **`change_scene_to_file()` must be called on (or from a descendant of) the actual `get_tree().current_scene`, and never synchronously from `_ready()`.** (a) instantiating a scene as a plain child of a driver node, rather than loading it as the real current scene, breaks its own disconnect-handling `change_scene_to_file()` calls with a silent hang. (b) calling `change_scene_to_file()` (or `add_child()` on `get_tree().root`) synchronously from inside `_ready()` throws "Parent node is busy … Consider using `.call_deferred()`", because the tree is still mid-traversal adding the very node whose `_ready()` is running. +28. **`ENetMultiplayerPeer`'s `connection_failed` signal is not bounded to anything a UI should make a player wait for.** Verified empirically: against a genuinely refused loopback connection, `connection_failed` had still not fired 14 seconds in. Don't rely on it alone to end a "Connecting…" state — run your own app-level timeout. +29. **A `MultiplayerPeer`'s "am I a client" flag turns true the instant `join()`/`create_client()` is called, not once the connection actually completes.** Anything gated on that flag alone will try to `rpc_id()` on a peer that's still `CONNECTING` — or has already failed. Gate on the peer's actual `get_connection_status() == MultiplayerPeer.CONNECTION_CONNECTED`, not just the higher-level intent flag. +30. **`load()` on a `.gd` file with a parse/compile error does not return `null`.** It returns a non-null but uninstantiable `GDScript` resource, so `if script == null` silently fails to catch the failure. The real guard is `Script.can_instantiate()`. +31. **An `@rpc` method named `_input` collides with `Node`'s built-in `_input(event: InputEvent)` virtual.** On an autoload especially, treat any bare virtual-sounding method name (`_input`, `_process`, `_ready`, `_unhandled_input`, …) as reserved regardless of what you intend it to do — a signature mismatch there doesn't fail locally, it fails the whole autoload. +32. **Disabling automatic multiplayer polling is global, not autoload-scoped — every scene that touches an RPC, not just `NetworkManager`-adjacent code, must call `NetworkManager.poll()` itself every frame it wants traffic to move.** If a scene sends or receives RPCs and nothing arrives with no errors at all, check for a missing `poll()` before anything else. +33. **A request/response fallback for a one-shot broadcast can double-deliver, and the receiving handler must be idempotent.** Any handler reachable via both an original broadcast and a "resend on request" path needs its own guard rather than assuming "only sent once" from the RPC design alone. +34. **An `Area3D`'s `body_entered` signal fires as part of physics tick N's own step, strictly *before* tick N's `_physics_process` callback — not "on the next frame."** If you actually need next-tick-or-later semantics, compare `Engine.get_physics_frames()` against the tick the flag was set on and require strictly-greater, not just "check a boolean at the top of `_physics_process`." +35. **A queued `queue_teleport()` can take one tick longer to land than "the very next `_integrate_forces`" suggests, when the call originates from a signal handler mid-physics-step rather than from a `_physics_process` callback.** Don't assume queued-teleport timing without checking a real tick-by-tick log for your specific call site. +36. **`NetworkManager.get_server_time_estimate_ms()` (and anything derived from it) is not just imprecise before the first clock pong lands — it's actively wrong in a way that can persist far longer than the sync window implies.** Always gate recording, not just consuming, anything derived from this estimate on `rtt_ms >= 0.0`. +37. **Anything that deliberately delays an RPC dispatch must re-validate its target at *fire* time, not just at the moment it was scheduled.** Any deliberate-delay layer needs to re-check both "do I still have a real peer at all" and "is this specific target still in `get_peers()`" inside the delayed callback itself, immediately before dispatching. +38. **A GDScript lambda captures an enclosing local variable BY VALUE at the moment the lambda is created, not by reference.** Capture a container instead of a value — `var disconnected := [false]` and `disconnected[0] = true` inside the lambda — since capturing an `Array`/`Dictionary`/`Object` captures a reference to the same instance. +39. **A fixed-size ring buffer fed by an unbounded-rate producer needs an explicit resync path, not just "wait for the next expected slot."** If a ring's producer and consumer rates aren't provably bounded relative to each other, the consumer needs a way to detect "the data I'm waiting for no longer exists in the ring at all" and jump directly to what's still available, rather than assuming "keep waiting" is always eventually correct. +40. **A client-owned adaptive control loop must react to the actual ground-truth signal it's regulating, not to its own memory of past decisions.** When a control loop's condition for acting can be satisfied or blocked by state the loop itself controls, rather than by the environment it's meant to respond to, it can silently stop responding to the environment. +41. **A "consecutive N over-budget windows" streak counter that hard-resets to 0 on any single clean window is trivially evaded by a duty-cycled attacker.** A leaky-bucket accumulator is immune to the same evasion by construction, since it doesn't matter how the excess is distributed in time — only the sustained average matters. +42. **Two counters that don't share an epoch must never be compared directly, even when both are monotonically increasing integers that "look like" the same kind of thing.** Bound a value against another value that shares its own actual epoch, not against a same-typed number from a conceptually different clock. +43. **A regression test that doesn't independently exercise the specific mechanism it claims to gate will pass even when that mechanism is completely broken.** A CI gate's assertions should trace back to the specific claim in the task's own acceptance text, not just "the match ran and didn't crash." +44. **When adding a "is this connection still healthy" check to a test, sample it while the peer is still actively connected — not after its own end-of-run disconnect, which produces symptoms indistinguishable from the bug being checked for.** +45. **Two fixes landed in the same commit, each individually correct in isolation, can share a variable and silently cancel each other out — and a fix's own unit test can miss it by testing the mechanism in isolation from the thing that defeats it.** When two fixes in the same round touch the same subsystem, explicitly re-test the combination end-to-end — passing tests for each fix individually is not evidence the pair composes correctly. +46. **A guard that bounds an incoming value against the *consumer's* position, rather than against the *producer's* own epoch, re-introduces exactly the "consumer can never catch up past a stall" failure it's often added specifically to prevent.** Bound against whichever side of a producer/consumer pair is not the one already known to be falling behind. +47. **A trace that holds its inputs steady cannot falsify anything about *which sequence* a prediction is filed under — and "we hold thrust for 60 seconds" describes almost every prediction test people write.** When a test is meant to validate a label, an index, or a phase relationship rather than a magnitude, the trace has to change that quantity frequently; a steady-state trace validates the magnitude and silently asserts nothing about the label. +48. **A guard whose bound is derived from a value only the ACCEPTED path can advance is a latch, not a guard.** The property to check when writing a guard like this is not "is the bound correct?" but "**if this guard rejects everything from now on, what advances the bound?**" If the answer is "an accepted packet", it needs an independent escape path regardless of how well-chosen the bound is. +49. **Advancing a consumer cursor past data that has not arrived is not a lossy shortcut — it is permanent, because the producer-side filter then rejects the very data being waited for.** **Only give up on an expected item once strictly newer data proves it lost**; "it hasn't arrived yet" and "it will never arrive" are different states and must not share a code path. +50. **A metric that stops sampling during a failure will report that failure as healthy.** Every rate-shaped assertion needs a companion assertion on the **denominator**, or an outage silently becomes an absence of evidence and then evidence of absence. +51. **An architectural blocker inherited from a previous session is a claim to verify, not a premise to build on.** Reconstruct the failing invariant from the code and reproduce it against a control before accepting a scope estimate attached to it — especially when the recommendation arrives without the cheaper alternative recorded as tested. --- @@ -1370,293 +340,28 @@ godot --headless --path Game res://scenes/server_boot.tscn -- --port 27015 --tea godot --path Game -- --connect 127.0.0.1:27015 --name Alice ``` -**CI smoke test (task 3.6).** Headless server plus two headless `--test-bot` clients, driven by the existing `AIShipController`. Asserts: +**CI smoke test.** Headless server plus two headless `--test-bot` clients, driven by the existing `AIShipController`. Asserts: - snapshots received ≥ `N * snapshot_hz * 0.9` - own-ship prediction error p95 < 0.5 m, p99 < 2.0 m, hard-snap count < 3 - final score identical on the server and both clients - no `push_error` emitted (scrape stderr) -**Network conditions.** `net_sim.gd` (task 2.8) is first-class: seeded so failures reproduce, works in CI, needs no display, and can be applied *asymmetrically* — which OS tools make painful. `tc netem` / Network Link Conditioner / `clumsy` for a pre-release realism pass only. A real remote host once per phase from Phase 4 onward is the only true test of the jitter buffer's adaptivity. +**Network conditions.** `net_sim.gd` is first-class: seeded so failures reproduce, works in CI, needs no display, and can be applied *asymmetrically* — which OS tools make painful. `tc netem` / Network Link Conditioner / `clumsy` for a pre-release realism pass only. A real remote host once per phase from Phase 4 onward is the only true test of the jitter buffer's adaptivity. -**Unit tests (task 1.0).** No test framework exists today, so keep it minimal — a scene that runs pure-function assertions and exits with a code. High-value targets, all zero-engine-state: codec quantise/dequantise round-trip and bounds; quaternion max error; snapshot pack→unpack identity; input packet framing; jitter-buffer policy against scripted arrival traces; `ShipAction.copy()` non-aliasing. These are exactly where a bug is invisible in play and catastrophic in aggregate. +**Unit tests.** `godot --headless --path Game res://tests/test_runner.tscn`. High-value targets, all zero-engine-state: codec quantise/dequantise round-trip and bounds; quaternion max error; snapshot pack→unpack identity; input packet framing; jitter-buffer policy against scripted arrival traces; `ShipAction.copy()` non-aliasing. These are exactly where a bug is invisible in play and catastrophic in aggregate. --- ## 11. Flagged, not solved -**Former item C — display-name slot takeover — RESOLVED locally.** `_try_reclaim_slot` now compares the verified signed `PlayerID` retained in the server roster and slot; late-join promotion carries the same identity. A changed display name can reconnect, but a same-name peer with a different identity cannot. Direct unauthenticated servers retain a documented display-name fallback for backwards-compatible community hosting. Live public-internet verification still remains a separate Phase 6 gate. - -**Low-latency present and graphics presets** — *now specified*, see §5.4, §5.5 and tasks 0.17/0.17b. Left here as a pointer because they are the largest wins in the document per line of code changed, and they are video settings rather than netcode. - **120 Hz simulation** — deliberately deferred, not dismissed. §5.4 and §5.6 record what it would buy (≈21 ms of world response once L1 has taken the interpolation buffer out, plus ≈8 ms of own-ship feel — the difference between ≈127 ms and ≈107 ms), what it costs (a full bot retrain, half the server density, double the bandwidth), and the one rule that keeps the door open: `TICK_HZ`, never `60`. -**The latency gap to the reference has a plan but not yet a measurement.** §5.2 lands at ≈174 ms as designed; §5.6 routes that to ≈127 ms (tasks 0.17d, 4.9) and ≈103 ms (tasks 4.10 plus 120 Hz simulation), against ~90–110 ms for the reference class at the same RTT. Every figure in §5.6 is arithmetic on the budget, not a measurement — task 4.9's acceptance criterion exists to make it one. Beyond that the residual is RTT, which is a server-siting problem (§6) rather than a code one and is worth more than every remaining code lever combined. +**The latency gap to the reference has a plan but not yet an implementation.** §5.2 lands at ≈174 ms as designed; §5.6 routes that to ≈127 ms and ≈103 ms (tasks L1–L4 plus 120 Hz simulation), against ~90–110 ms for the reference class at the same RTT. Every figure in §5.6 is arithmetic on the budget, not a measurement or a shipped change — L1–L4 remain unimplemented. Beyond that the residual is RTT, which is a server-siting problem (§6, Phase 8) rather than a code one and is worth more than every remaining code lever combined. -**Audio.** The runtime now has dependency-free procedural placeholder hooks for UI, countdown, engine/thrust/turbo, impacts, wall contacts, goals, and camera/gameplay events. `TODO.md` still tracks authored engine/turbo/impact/wall/goal/crowd/music assets and production mixing/QA; remote-ship engine audio can build on `set_visual_action` / `set_visual_speed` (task 0.14), and “ball feel” (task 4.6) remains partly auditory. +**Audio.** The runtime has dependency-free procedural placeholder hooks for UI, countdown, engine/thrust/turbo, impacts, wall contacts, goals, and camera/gameplay events. `TODO.md` tracks authored engine/turbo/impact/wall/goal/crowd/music assets and production mixing/QA as still open. **Split-screen.** Tracked separately in `TODO.md`; unrelated to this effort, though the camera-outside-the-ship structure that enables it is the same structure this plan relies on. -**Item E of §0 — stale snapshot sends after forced disconnect — is now resolved locally.** `MatchSim.send_snapshot()` validates the live peer and `NetSim._fire()` revalidates delayed targets immediately before dispatch, covering the deliberately adversarial `client-abuse-malformed` path as well as normal disconnects. A full multi-process abuse smoke remains a useful runtime check, but the stale-target call sites no longer enter Godot's RPC path after peer teardown. -#### Deployment wiring update (2026-09-01) +**Graphics: baked GI (task 0.26) and low/mid-tier hardware profiling.** See §5.5 and §5.7 — real but smaller wins than originally assumed on reference-class desktop hardware; unmeasured on low-end/integrated GPUs. -The current working implementation now wires `deploy/k8s/base/fleet.yaml` to the digest-pinned `game-server` supervisor target, the in-cluster control-plane Service, workload roster materialization, signing/drain secret references, downward-API server/image identity, and the required game-server egress policy. `kubectl kustomize deploy/k8s/base` and `server/security/test_fleet_manifests.py` pass. The older 8.28 narrative above still records the pre-wiring state; live Agones, operator secret/image replacement, and real cluster readiness remain explicit gates. - -An adversarial Fleet-entrypoint review found that the supervisor invocation had -no executable after `--`, and that its required supervisor-level protocol flag -was missing. The Fleet now passes the exported Godot server explicitly and -sets `--protocol-version=1`; the NA overlay's positional patch and manifest -regression test were updated together. This is a local launch-contract fix, -not evidence of live Agones readiness. - -The NA overlay now also patches the allocated child’s `--region=NA` argument, keeping it aligned with the NA Fleet label; rendered EU and NA overlays and the adversarial manifest test verify that regional assignment validation cannot silently remain EU in the NA deployment. - -Allocated Godot startup now derives its `min-players` floor from the verified signed roster size, preventing the direct-server default of one player from starting a partially admitted allocated match. A focused regression test covers six-player, casual two-player, and direct-server behavior; the current full local gate passes all 212 Godot tests, using the pinned Linux fallback if the native macOS engine crashes. - -The former display-name reclaim weakness (flagged item C) is now closed for allocated matches: the signed `PlayerID` is retained in the server roster and slot, and both reconnect reclaim and late-join promotion carry that stable identity across peer-id changes. Display-name matching remains only as a legacy fallback for unauthenticated direct servers. A focused adversarial unit test covers changed names, same-name impostors, missing identities, and the direct-server fallback. - -Allocated supervisor launch arguments now have a direct regression guard: authoritative match/server/image/assignment-expiry values replace stale child placeholders without mutating the caller’s command slice or disturbing unrelated arguments; dynamic Agones port propagation remains covered by the existing startup test. This closes the local implementation portion of task 8.29; live Agones passthrough/NAT and multi-match validation remain infrastructure gates. - -Read-only authenticated queue, proposal, assignment, legacy profile, and ranked-profile routes now emit lifecycle-safe observability events for successful, rejected, and not-found reads. An API regression exercises all five real HTTP routes and verifies the event set; event fields remain free of credentials. This closes the local read-route portion of task 8.44; metrics/traces export, dashboards, and alert routing remain operational work. - -Allocated join admission now retains and applies the signed assignment’s authoritative team and global slot: peer order can no longer rebalance a valid allocation, and inconsistent team/slot claims are rejected before roster admission. The server exposes the verified assignment list for allocation-aware startup and uses the per-team spawn index derived from the assigned slot. Go verification is clean; Godot execution remains blocked by the documented macOS pre-test crash. - -Allocated boot now also validates the complete signed roster shape before opening the gameplay endpoint: malformed claims, duplicate player identities, duplicate slots, and team/global-slot mismatches fail closed rather than leaving a partially usable server. The Godot `--check-only` attempt still reaches the known macOS renderer/ZSTD crash before script parsing, so this startup guard remains statically reviewed and covered by the existing signed-claim tests pending a working Godot runtime. - -The control plane now mirrors that topology fence at roster publication: signed entries with duplicate players, duplicate slots, or a team inconsistent with the canonical global slot are rejected before durable assignment rows are written. Focused store tests cover forged topology and duplicate entries; normal/race Go suites and vet pass. - -The backend roster persistence boundary now enforces the same duplicate-player, duplicate-slot, and team/global-slot invariants as Godot startup. This closes the remaining local consistency gap in task 8.31; production signer/client-ticket publication and live Agones verification remain external gates. - -The no-show policy now has an explicit domain translation layer (`PlanInitialConnect`): `WAIT` remains non-mutating, ranked no-shows produce a `CANCELLED` match plan with innocent-player IDs, and eligible casual play produces a `LIVE` plan plus the complete bot-filled six-slot lineup. Normal/race domain tests cover both branches; applying the plan transactionally to durable tickets/matches and wiring it into the allocated server lifecycle remain task 8.35 work. - -The durable no-show boundary is now implemented by `ApplyInitialConnectPlan`: it locks the match and roster, validates that the plan covers every active participant, records deterministic no-show cooldown penalties, fails no-show tickets, requeues innocent tickets on cancellation or advances connected tickets to `LIVE` for eligible casual bot start, and emits a replayable state-change outbox event under the same serializable transaction. Idempotency keys reject conflicting retries. Focused store tests, race tests, and vet pass; the real PostgreSQL integration remains an environment-dependent gate. - -The maintenance command now invokes a bounded `ReconcileInitialConnect` sweep for `ASSIGNMENT_READY`/`ASSIGNED`/`CONNECTING` matches, carrying ranked no-show history into the domain ladder and skipping non-actionable WAIT plans. This closes the local control-plane trigger for task 8.35; actual allocated-server bot spawning, shutdown signaling, and live Agones integration remain separate gates. - -Allocation registration now writes a participant-targeted, revisioned `state_changed` outbox event for both `PROCESS_READY` and `ASSIGNMENT_READY` transitions. The production and test API binaries run a type-scoped dispatcher with delivery-before-ack semantics, so allocation lifecycle events survive WebSocket outages without competing with proposal or result consumers. Store/API adversarial tests cover event-type isolation, target validation, and revision mismatches; live allocator/Agones delivery remains an integration gate. - -The state-event implementation is now complete through the registration boundary: the durable registration SQL returns the authoritative match revision, includes every participant target in the payload, and the dispatcher validates aggregate/revision/state consistency before fan-out. Full Go tests, race checks, and vet pass after an adversarial database-cursor review. - -Allocated Godot runtime now applies the same initial-connect policy: ranked allocations cancel and exit after 30 seconds if the signed roster is incomplete; casual allocations wait 60 seconds, cancel when fewer than two humans or one team is absent, and otherwise start with a deterministic six-slot assignment-derived lineup containing explicit bots. The bot branch is opt-in and consumed once, so direct servers and ranked matches cannot inherit it. Godot parse plus the 155-test harness and manifest checks pass; durable no-show penalties/state reconciliation remain owned by the control-plane sweep. - -An adversarial transaction review found that cancellation released only no-show participant rows, which would leave innocent players marked active in the cancelled match and trip the active-match uniqueness fence on their next match. `ApplyInitialConnectPlan` now releases the complete participant roster on cancellation, while retaining cooldown penalties only for no-shows; the full Go suite, race checks, and vet pass. - -The documented `server_shutdown` reliable control message is now implemented in `MatchNet`, with bounded reason sanitisation and an authority-only receiver signal. Controlled drain broadcasts `server_draining`; allocated initial-connect cancellation broadcasts its policy reason and waits a transport-flush beat before closing. The 156-test Godot harness covers emission and bounds; full multi-process drain delivery remains a live integration gate. - -Clients now consume planned shutdowns: the reason is retained for presentation, an in-match client returns to the lobby after the notice, and the generic disconnect callback is fenced so it cannot overwrite that planned transition. Lobby clients surface the reason directly. The complete Godot harness remains green; real two-process drain delivery is still an external runtime gate. - -An adversarial UI review found the lobby’s generic disconnect handler still replaced that message with the main menu immediately afterward. Planned disconnects are now fenced in the lobby, and a lobby reached from an active match restores the retained reason on startup; unplanned disconnects keep the existing main-menu behavior. - -The workload-authenticated `POST /servers/{serverId}/shutdown` contract is now exposed for allocated servers. It validates the bound credential and reason, records an idempotent `SERVER_SHUTDOWN` audit event under a serializable transaction, and returns a stable acknowledgment on retry; match-state transitions remain owned by the no-show/result transactions. API/store tests cover authorization, validation, idempotency SQL, and audit wiring; live PostgreSQL delivery remains an integration gate. - -The allocated supervisor now calls that shutdown acknowledgment during signal-bound controlled drain, using the same workload credential and a deterministic idempotency key after the local drain request succeeds. The lifecycle test verifies the drain-before-ack ordering, credential separation, and bounded graceful child exit; live pod termination and control-plane outage behavior remain deployment gates. - -Allocator-selected region, build, protocol, and transport now travel with the allocation as Agones annotations and override stale child launch flags immediately before an allocated process starts. The overlay rejects control characters and preserves direct-server command behavior; focused supervisor/allocator tests cover precedence and annotation payloads, while live Agones passthrough remains an infrastructure gate. - -The same allocation path now carries the matcher-selected playlist, preventing a ranked match from inheriting the Fleet’s casual default. Durable allocation claims return the playlist, the worker includes it in Fleet selection metadata, Agones copies it to the allocated GameServer, and the supervisor overrides `--playlist` before launch; the existing compatibility tests remain green. - -The allocator’s durable bind now increments the match revision and writes a participant-targeted `state_changed(ALLOCATING)` outbox event in the same serializable transaction as the server binding and ticket transitions, so clients can recover allocation progress after a delivery outage. - -Ranked maintenance now marks expired seasons with no ranked profiles as rolled over, preventing an empty season from being selected and reconsidered on every maintenance pass; the boundary is covered by the integration-tag regression suite. - -Season rollover now computes from the row locked inside its serializable transaction rather than a stale caller snapshot; the PostgreSQL integration regression deliberately passes a 1900 profile against a durable 2000 rating and verifies the 1875 result is preserved. - -The production ranked-profile adapter now projects the active ranked season ID from the durable `seasons` table while keeping rollover history separate; the API prefers that current-season value and retains the legacy in-memory fallback for existing callers. - -Allocator quota accounting now charges only fresh provider attempts; recovery of a provider result after an ambiguous durable write does not consume the same regional quota a second time. - -Accepted-proposal allocation now binds the request back to the proposal’s playlist, arena, region, and protocol before any provider call; adversarial mismatches fail closed. - -Ranked proposal admission no longer trusts the matcher’s `--ranked-random-arena` boolean. The Go domain now owns a named allowlist for the three floor-goal `ArenaRegistry` entries, and rejects unknown and elevated IDs before any proposal is created. - -The arena hand-off is now durable: the matcher deterministically selects an eligible floor-goal arena from the proposal ID, migration 0008 stores that path on proposals and matches and enforces it for new direct SQL writes, migration 0009 retains it on provider allocations, domain/store/provider boundaries and recovery lookups recheck the same allowlist, allocation claims and idempotency digests retain it, Agones applies it as a match-scoped annotation, and the supervisor overlays the allocated child’s `--arena-path`. Godot accepts only the same floor-goal `ArenaRegistry` paths and requires one for allocated ranked matches, so a stale Fleet default, an elevated variant, or an altered retry cannot substitute a ranked arena. The recovery worker also rejects a provider-recovered allocation whose arena differs from the durable request before recording or binding it. - -The Godot control-plane client now retains the exact last idempotent mutation and exposes `retry_last_mutation()` for transport, timeout, rate-limit, and 5xx failures. Retries reuse the original idempotency key and expected revision, while 401 and 409 responses remain non-retryable; the harness covers the policy boundary. This closes the local duplicate-action recovery mechanism for heartbeat/cancel/proposal calls, with broader live UI retry verification still remaining. - -The matchmaking UI now exposes that retained replay through its existing action button as `Retry Request` while a heartbeat, cancellation, or proposal action has a retryable failure. Terminal, authentication, and revision-conflict paths remain ineligible, so the button cannot issue a stale blind command. - -The control plane now exports bounded Prometheus-compatible API request counters and latency summaries at `GET /metrics`, with fixed operation/status labels and no event-stream wrapping. Production and testkit services wire the collector; adversarial tests verify unknown paths cannot inject label cardinality or leak URL secrets, and full Go/race/vet checks pass. Durable SLO dashboards and alert routing remain operational work. - -The authenticated event stream now rejects client data/reserved opcodes and oversized control frames at the parser boundary; only RFC 6455 close, ping, and pong frames are accepted from clients, preserving the bounded v1 stream contract. - -Event delivery also applies a bounded write deadline, so a client that stops reading cannot strand the event handler after the bounded subscriber queue evicts it. - -`make verify-multiplayer-local` now provides one cloud-free regression gate for the current implementation: the complete Go suite, the Godot harness, OpenAPI parsing, and the migration/Fleet/Kubernetes/supply-chain checks. It falls back to the pinned Linux harness when the configured Godot executable is unavailable or crashes by signal, while ordinary test failures still fail the gate; PostgreSQL, Redis, Steam, Agones, and multi-process Internet gates remain separate. - -That local gate now also runs `go test -race ./...`, `go vet ./...`, and each declared domain fuzz target for a bounded 2-second interval, aligning the one-command gate with the separately recorded 8.46 verification requirements. - -Observability redaction now adds content-aware protection on top of denylisted field names: bearer values, compact JWT-like strings, PEM material, and long opaque mixed alphanumeric values are redacted recursively through arbitrary nested maps and string slices. Unknown-key credential canaries pass without leaking; false-positive risk is limited to custom long opaque fields, while canonical correlation IDs remain outside the free-form field map. - -The authenticated control-plane WebSocket now caps each player at two -simultaneous connections, releasing capacity on disconnect; this complements -the bounded per-player event queue and prevents connection fan-out from -becoming an unbounded account-level resource cost. Over-limit attempts fail -before upgrade with `429 websocket_connection_limited`, rather than becoming -ambiguous post-upgrade disconnects. - -Proposal decline and timeout cooldowns are now durable and matchable-state -safe: an offender's existing ticket becomes terminal (`CANCELLED` for decline, -`EXPIRED` for timeout), while innocent or already-accepted participants retain -their original queue precedence. Late response recovery commits before the API -returns `ErrProposalClosed`; deterministic penalty IDs preserve replay safety, -future/corrupt cooldown events are ignored, and reopening an old declined -proposal cannot create false timeout penalties for its innocent participants. - -### Current local completion index (2026-09-04) - -The following Phase 8 slices have local implementation and verification evidence in this document: 8.29 dynamic allocated launch flags and endpoint handling; 8.30 allocator claim/reconciliation including provider-outcome recovery fencing and durable arena identity (migrations 0008–0009); 8.31 signed assignment/roster validation; 8.35 initial-connect no-show and casual bot policy; 8.36 controlled drain and shutdown acknowledgment; 8.39–8.43 client state, assignment, profile, recovery, and idempotent action retry; 8.44 structured observability and content-aware redaction; 8.45 bounded API metrics export plus optional Prometheus scrape/alert rules; 8.46 normal/race/vet/fuzz coverage; 8.47–8.48 offline/testkit/Compose coverage; 8.50 atomic stalled-allocation recovery notifications; 8.51 the 10,000-client API load boundary; 8.52 the opt-in per-replica plus shared PostgreSQL regional allocator quota; and 8.53 the fail-closed promotion validator. Their remaining acceptance text is infrastructure or production dependent where explicitly noted below the corresponding row. - -The following are not locally certifiable from this workspace and remain open prerequisites rather than silently “done”: Valve/GodotSteam credentials and hosted SDR (7.1–7.8), live Agones/kind lifecycle (8.30–8.38, 8.49), public-network chaos/load/cost/release gates (8.50–8.53), and real-hardware graphics profiling (0.15b onward). `make verify-kind-agones` is the committed runner for 8.49; its response validator is unit-tested against malformed/ambiguous allocation payloads. On 2026-09-04, kind and Helm were installed and the runner reached its real Agones chart. That run found and repaired the chart's default 10,100 MiB `agones-extensions` ephemeral-storage request, which cannot schedule on a one-node kind cluster. The corrected extensions pod became Ready, but the Agones controller image then could not unpack because this Docker Desktop instance retains 2.71 GB of non-reclaimable BuildKit state and its internal disk filled despite pruning unused volumes, images, and cache. The live gate remains open pending Docker engine capacity; no kind/Agones success is claimed. `TODO.md`’s AI-training and presentation tasks remain separate from multiplayer and are not marked by this index. - -The live control-plane integration was retried on 2026-09-01 after Docker Desktop became available, but the disposable `postgres:17-alpine` container failed during `initdb` with `No space left on device`; Docker reported 10.2 GB of images and 3.3 GB of volumes. The user approved pruning the disposable volumes on 2026-09-04 (3.3 GB reclaimed), and `scripts/run_postgres_integration.sh` then passed against real PostgreSQL. That run caught and repaired a stalled-allocation outbox CTE without `RETURNING`, an untyped JSON timestamp parameter, a season-rollover scan arity mismatch, lifecycle-incompatible fixtures, and a rollback-test step count that did not actually reach migration 0006. - -On 2026-09-04, the client-facing control-plane, assignment, ranked-profile, and two-player proposal runners were made portable by falling back to the pinned Docker Godot harness when no native `godot` binary is available. The real PostgreSQL supervisor and result-fan-out runs then passed too. That adversarial pass caught a second registration-query defect: its `matched` CTE selected `revision` without returning it, and the ticket transition left `revision` ambiguous after the CTE was corrected. The query now returns the match revision and explicitly increments `q.revision`; the real supervisor test covers process-ready → roster materialization → assignment-ready, and the full local multiplayer gate (Go, race, vet, fuzz, Godot, contracts, manifests) passes. Production Steam/SDR, live Agones, and public-network gates remain open as listed above. - -The deferred teamplay TODO prerequisite is now implemented locally but not -enabled: team-touch credit is opt-in and the evaluator can run paired 2v2 -matches with `--team-size=2`. No Stage 7 training run or promotion is claimed; -teamplay still needs recorded behaviour thresholds and a working Godot runtime -for its end-to-end evaluation. - -The generation-5 environment now also has opt-in wall-play and pre-rebound -episode starts. Stage 6's next command enables each at 10% after the Stage 5 -aerial baseline; Stages 4–5 retain their prior distributions. The state -generator and configuration are covered statically, but no training pass or -promotion is claimed until the Godot runtime and telemetry gates are available. - -The three declared domain fuzz targets have now each completed a bounded 4-second run (`FuzzQueueCreateDoesNotPanic`, `FuzzResultDigestIsDeterministic`, and `FuzzSyncEventApplicationDoesNotPanic`) with no failures; this closes the locally runnable fuzz portion of 8.46. Live Redis failover, further transaction races, and cloud/runtime gates remain explicitly unverified. - -The real ENet integration gate now passes with `GODOT_BIN=/Applications/Godot.app/Contents/MacOS/Godot bash scripts/verify_enet_integration.sh`, covering the `net`, `match-net`, `clock`, `lobby`, and `networked match` process scenarios. The default `GODOT_BIN` remains the portable `godot` PATH lookup for CI; this machine requires the explicit app-bundle path. - -Stalled-allocation recovery now emits a participant-targeted `state_changed` -outbox event in the same serializable transaction that fails the abandoned -match, releases its participants, and requeues their tickets. The maintenance -adapter verifies that every reclaimed match produced its durable event, so an -API/WebSocket restart cannot turn a successful infrastructure recovery into a -silent client-side stale state. Normal, race, vet, and SQL-shape checks pass; -the live PostgreSQL chaos/restart gate remains part of 8.50. - -The ENet gate now auto-detects `/Applications/Godot.app/Contents/MacOS/Godot` when no PATH executable or `GODOT_BIN` override exists, while retaining explicit override precedence. The same gate passes without an environment override on this macOS host. - -The client matchmaking projection now preserves the server's `enqueued_at` timestamp through normalization, snapshots, and recovery, and uses it for the displayed queue wait when available. This prevents a client restart or delayed response from resetting the user's perceived wait to local process uptime; a local timer remains the fallback when older responses omit the timestamp. Godot state and normalization tests cover the projection and restore path. - -The ranked profile projection now also carries the active season's authoritative end timestamp from PostgreSQL through the API and Godot client. Ranked matchmaking displays a bounded days-remaining countdown, while providers without an active season remain compatible and omit the countdown. - -The versioned OpenAPI contract now declares the implemented `/profile/ranked` surface and its server-authoritative ranked profile schema, including optional active-season metadata. Contract tests reject omission of this operation, extra response fields, and credential leakage. - -The Godot client now defers reconnect-triggered authoritative recovery when an HTTP mutation is still in flight, closing the `ERR_BUSY` recovery-drop race. An adversarial client test verifies that the active ticket remains queued for recovery rather than silently staying stale. - -The Godot queue projection now includes the contract's `ACCEPTED` ticket phase. Accepted events are no longer rejected as an unknown state; the UI keeps the accepted status visible and proceeds through allocation recovery. State and WebSocket vocabulary tests cover the transition. - -The queue projection now also accepts the contract's post-allocation/result states (`ASSIGNED`, `RESULT_PENDING`, and `COMPLETED`). These states remain visible, cannot issue queue cancellation, and completed matches return the search action to a valid new-search state; adversarial lifecycle and WebSocket vocabulary tests cover them. - -Client ticket updates now enforce the versioned legal transition graph as well as revision ordering. Same-state heartbeat revisions remain valid, while higher-revision jumps and rewinds request authoritative recovery without mutating the visible phase; adversarial tests cover both boundaries. - -Reconnect recovery now treats `COMPLETED` as terminal, avoiding a needless queue read after a finished match. UI policy tests cover the complete expanded lifecycle, including the completed-to-new-search boundary. - -Ticket, proposal, and WebSocket revisions now fail closed unless they are finite, non-negative integers; fractional values are no longer silently truncated into valid revisions. Adversarial client tests cover fractional and negative inputs. - -Proposal updates now enforce the documented `OPEN → ACCEPTED/DECLINED/EXPIRED/CANCELLED` graph, including rejecting higher-revision reopen/accept attempts after terminal decisions while preserving same-state duplicates. Adversarial proposal-transition tests cover accepted and declined terminal paths. - -Proposal decline/expiry/cancellation now leaves a still-proposed ticket in `QUEUED`, matching the durable server requeue transaction; the proposal’s terminal message remains visible without making the ticket terminal. A cancelled ticket is never resurrected by a later proposal event, covered by adversarial cross-aggregate tests. - -Recovery targeting now follows the same boundary: only an `OPEN` proposal is polled as a proposal; terminal proposal outcomes fall back to the ticket recovery endpoint. This prevents repeated reads of a finished proposal from starving recovery of the requeued ticket. - -Client queue/proposal expiry and enqueue epoch metadata now fail closed on malformed, negative, or fractional values instead of being silently coerced to zero. Adversarial metadata tests cover string, negative, and fractional timestamps. - -All client resync entry points now apply the open-proposal boundary: a terminal proposal always recovers the durable ticket instead of polling the finished proposal. A direct-resync regression test covers this path. - -Ticket projections now validate playlist metadata on every update, rejecting unknown values before either phase or playlist state can mutate. An adversarial higher-revision update test covers this boundary. - -Client sessions now fail closed at the expiry boundary and proactively clear credentials before reconnects or authenticated requests. Boundary and malformed-expiry tests cover the lifecycle guard. - -Queue heartbeat and cancellation revision conflicts now schedule the same authoritative ticket recovery as proposal conflicts, preventing stale client actions from leaving the visible queue state unresolved. Adversarial operation/status/identity coverage is included. - -WebSocket event envelopes now require RFC3339 timestamps rather than merely non-empty text, matching the versioned contract; session-expiry format checks use the same boundary validator. Malformed-format adversarial coverage is included. - -WebSocket event resource identifiers now enforce the contract’s opaque 16–128 character allowlist, preventing path/separator text or undersized identifiers from entering the client projection. - -The Go event hub now enforces the same resource-ID allowlist before publication, so malformed identifiers are rejected at the server boundary rather than only discarded by clients. - -The matchmaking UI now displays the authoritative proposal countdown from the server expiry epoch, clamped at zero and retaining compatible copy when older responses omit expiry metadata. Adversarial countdown tests cover delayed and missing-expiry responses. - -The UI now provides explicit detail copy for every non-terminal allocation and connection phase (`ACCEPTED` through `LIVE`), so server progress remains understandable throughout assignment and transport startup. - -Reconfiguring the client with new credentials now clears the prior session expiry, preventing an expired session’s timestamp from invalidating a fresh authentication. A re-authentication regression test covers the boundary. - -Assignment expiry validation now fails closed on malformed non-empty timestamps before invoking the date parser, and fresh-assignment checks share the same format boundary. This prevents malformed assignment manifests from reaching transport startup. - -MatchNet join-authorisation admission now applies the same expiry format guard before parsing signed roster claims, closing the malformed-expiry gap at the transport handshake boundary. - -Ranked profile season metadata now validates optional expiry type and RFC3339 format before deriving the UI countdown, rejecting malformed server projections instead of silently displaying a profile without season context. - -Ranked profile `season_id` now enforces the OpenAPI opaque-ID shape and exact string type, preventing undersized or coerced identifiers from entering the client projection. - -Persisted matchmaking snapshots now validate field types, non-negative integral revisions/epochs, and proposal identity/state consistency before restoration; malformed restart data cannot be coerced into an active projection. - -Ticket timestamp normalization now preserves an invalid sentinel for malformed or non-string raw timestamps, allowing the projection to reject bad server metadata instead of silently converting it to epoch zero. - -Ranked profile projection now rejects fractional `ranked_games` values instead of silently truncating them, matching the OpenAPI integer contract. - -Ranked profile projection now enforces the OpenAPI tier enum, rejecting unknown tier labels before they reach the HUD. - -Assignment projections and assignment-changed events now enforce the published opaque-ID shape for match, server, and player identifiers; short or unsafe IDs fail closed. - -The WebSocket contract and Go event hub now enforce opaque match and server IDs on assignment notifications, keeping server publication aligned with the Godot client validator. - -Assignment projection now rejects fractional `slot` and `protocol_version` values instead of truncating them, matching the OpenAPI integer contract. - -Allocated `ServerConfig` startup now enforces the opaque match/server ID contract, rejecting short or unsafe allocation flags before process launch. - -Authenticated client REST methods now enforce opaque ticket, proposal, and match IDs before constructing request paths, preventing malformed identifiers from crossing the URL boundary. - -Persisted matchmaking snapshots now apply the same opaque-ID validation to ticket and proposal identities, preventing malformed restart state from entering recovery. - -Control-plane REST responses now fail closed on malformed ticket, proposal, or session player IDs before projection, covering the server-to-client JSON boundary as well as request paths. - -Session establishment now also requires a present, syntactically valid, future `expires_at`, preventing malformed authentication responses from creating an unbounded client session. - -MatchNet admission configuration now requires exact string opaque match/server IDs and a finite integral protocol version, preventing malformed server context from being coerced into a valid roster binding. - -The proposal wire contract now matches the real API participant-object shape (`player_id`, response, team, slot), with JSON tags on the Go model and client validation for count, uniqueness, identities, enums, and integer team/slot assignments. - -Proposal responses now require and normalize the contract's RFC3339 `expires_at`; malformed or missing expiry metadata fails closed while already-expired terminal proposals remain representable. - -Queue responses now validate the complete published shape before projection: opaque ticket/player IDs, playlist and lifecycle enums, integral revision, and RFC3339 enqueue/expiry timestamps. - -The public `/api/v1` route adapters now reject non-opaque queue, proposal, assignment, and server path identifiers before delegating to the legacy handlers; adversarial route tests cover short and separator-bearing IDs. - -The public queue adapter also rejects an explicitly supplied short or unsafe `ticket_id`; omitted IDs continue to be deterministically server-assigned for idempotent retries. - -RFC3339 validation now checks both wire syntax and calendar parseability, rejecting impossible dates before they can become epoch metadata. - -Matchmaking now explains queue progress (including bounded skill widening while preserving latency limits) and exposes live connection latency quality during connect/live phases; adversarial UI tests cover missing, infinite, negative, and threshold RTT values. - -Signed MatchNet claims now also require exact JSON string/integer types for every identity, protocol, expiry, slot, team, and generation field; string-number coercion is rejected before canonical signature verification. - -Presentation progress: a shared `Game/themes/cosmic_clash_theme.tres` now gives the menu, lobby, matchmaking, and settings surfaces consistent button, input, option, and label styling. The custom-font portion of `TODO.md` remains open until a distributable font asset is selected. - -The audio TODO now has a runtime foundation: `AudioManager` generates bounded placeholder tones for kickoff countdowns, camera-reported ball impacts, static-wall contacts, goals, UI clicks, a local thrust/turbo-pitched engine loop, and a rising-edge turbo cue without adding binary assets; authored sound design and production audio QA remain open. - -The video-settings TODO is likewise locally implemented: presets, vsync, refresh-derived FPS caps, and resolution scaling are wired through `VideoSettings` and the settings menu. The remaining acceptance work is low/mid-tier hardware frame-time and image-quality profiling, which cannot be certified from this workspace. - -Assignment handoff now has a non-circular recovery path. Match-scoped lifecycle events are no longer misapplied as queue-ticket resources: they trigger owner-scoped ticket recovery, and recovered active tickets include their durable `match_id`. An `ASSIGNMENT_READY` event or recovered ticket can therefore drive `GET /assignments/{matchId}` without already having fetched that assignment. Owner-scoped REST ticket snapshots may cross missed revisions only along a reachable forward lifecycle path, while incremental WebSocket updates remain strictly contiguous and neither path can rewind state. The OpenAPI queue projection includes the optional active match identity, Go tests cover the store/API projection, and the 199-test Godot harness covers match-resource separation, malformed identities, missed-revision recovery, illegal rewinds, and assignment-fetch scheduling. The real PostgreSQL assertion is committed with the store integration suite; rerunning it in this workspace is temporarily blocked by Docker storage exhaustion (`initdb` cannot create `pg_wal`), so live SQL evidence remains open rather than being claimed from the static/unit gates. - -Replica-independent client convergence now supersedes the earlier "at-least-once WebSocket delivery" wording in tasks 8.25/8.40 and the allocation-outbox progress notes. The database outbox guarantees ordered, replayable invocation of a replica's transient publication adapter, not receipt by a socket that may be absent or attached to another replica. Active clients now perform bounded five-second owner-scoped REST recovery; ticket recovery exposes the active `proposal_id` or `match_id`, so a missed proposal, allocation, assignment, or result notification cannot strand the client without the next resource key. A terminal proposal can be replaced by a later recovered proposal identity, while an open proposal cannot be overwritten. Network and malformed-JSON failures during recovery remain visible and retryable instead of falsely terminating matchmaking. WebSocket events remain the low-latency path; REST snapshots are the correctness path. Store/API and the 200-test Godot harness cover projection, transient failure, replacement, and hostile identity/state combinations, with live PostgreSQL execution still subject to the Docker storage gate recorded above. - -The production allocator now uses the API it actually implements: Kubernetes custom-resource paths at `https://kubernetes.default.svc`, rather than sending those paths to the distinct mTLS Agones Allocator Service. Its HTTPS client trusts the mounted cluster CA, rereads the projected service-account token for every request so rotation is honored, applies a ten-second request timeout, and refuses to forward the credential to another origin. The allocator pod explicitly mounts its token; namespaced RBAC permits only GameServer `list` and GameServerAllocation `create`; and its default-deny policy permits portable API-server egress only on TCP 443. Focused Go/auth, static policy, and `kubectl kustomize` checks pass. The real kind/Agones runtime gate remains open because kind and Helm are unavailable here and Docker storage is exhausted; no live-cluster success is claimed. - -Drain admission now fails at the handshake boundary: a new `_hello` is rejected with the actual RPC peer ID after `admissions_open` closes. Disconnects no longer perform the admission check (or try to reject an already-gone sender); they always invalidate transport state, release the signed join token, record the reconnect boundary, and remove the roster entry. Godot regressions cover the admission decision and cleanup while draining. Task 8.36's live lifecycle/PDB gates remain open. - -Allocated team and slot assignments are now immutable after signed admission. MatchNet rejects client `_set_team` requests whenever join authorisation is required, preserving the signed global-slot/team pairing and its derived spawn index; direct/community lobbies retain team switching and its existing unready behavior. The Godot regression asserts both sides of that compatibility boundary. - -Per-IP API limiting now resolves the client behind the edge gateway instead of charging every player to the gateway's socket address. `X-Forwarded-For` is ignored unless the immediate peer belongs to an explicitly configured `--trusted-proxy-cidrs` range; trusted chains are walked from right to left past known proxies, while malformed/oversized chains fail closed to the immediate peer. The base deployment supplies private/CGNAT/ULA pod ranges under its edge-only ingress NetworkPolicy and calls out that production overlays should narrow them to the actual gateway CIDR. Tests cover spoofing from an untrusted peer, chained proxies, malformed input, invalid configuration, and independent clients behind one gateway. - -Allocator probes now distinguish process liveness from useful progress. `/healthz` remains live during dependency outages, while `/readyz` starts unavailable and requires a fully successful provider-list, Ready-registration, and worker cycle within `--readiness-max-stale` (30 seconds in the base deployment). The Kubernetes/Agones HTTP path is bounded by `--provider-timeout=10s`, so an unavailable provider cannot leave readiness green indefinitely; startup rejects a freshness window shorter than the poll interval plus provider timeout, and the probe listener has its own header-read deadline. Boundary and HTTP tests cover startup, exact staleness, clock reversal, recovery, method rejection, and metrics coexistence. - -The timeout boundary is enforced inside both network adapters as well as in the production allocator wiring: an `agones.Client` or game-server `Supervisor` constructed without an injected HTTP client now receives a ten-second client rather than Go's unbounded `http.DefaultClient`. This prevents alternate binaries, tests, and future callers from restoring an infinite GameServer, roster, registration, or SDK wait by omission. - -Control-plane probes now separate liveness from datastore readiness too. `/healthz` proves the process can serve without restarting it during a PostgreSQL outage; `/readyz` runs a one-second-bounded `PingContext` and the Deployment routes traffic only to replicas whose core durable store responds. Probe and metrics routes bypass the player request limiter, so operator-selected low limits cannot make Kubernetes evict a healthy replica. Missing checks, datastore errors, non-GET methods, and successful recovery are covered by API tests. - -The task 8.35 adversarial pass closed the previously disconnected initial-connect implementations. An accepted signed player now produces a workload-authenticated `POST /servers/{serverId}/connect` receipt bound to the exact allocation, match, server, participant, and unexpired assignment; durable replay survives a lost response and keys include the match so a later match cannot conflict. Unknown datastore failures return retryable 503 responses. Player assignment reads are hidden until the match has durably reached `ASSIGNMENT_READY`, and the supervisor now fails closed if that transition never commits. - -Initial-connect timing and topology now agree across every layer. Migration 0010 records `initial_connect_ready_at` at the assignment-ready transition instead of using match creation time; maintenance polls that path independently every second; and an authenticated loopback signal arms Godot's local timeout only after the durable transition. Complete rosters enter `LIVE` immediately, relaxed two-to-five-human casual rosters immediately fill their disclosed vacant slots with bots, and six-human casual no-shows use the 60-second policy. Casual lineup, reconnect, signed-roster, matcher, store, and Godot validation all use canonical global slots 0–2 for team 0 and 3–5 for team 1; the earlier alternating-slot bot layout has been removed. Focused Go tests, contract/migration/manifest checks, and the 204-test Godot harness pass; the committed PostgreSQL integration assertion remains unexecuted locally while Docker storage is exhausted. +**Not locally certifiable from this workspace, and open prerequisites rather than done:** Valve/GodotSteam credentials and hosted SDR (Phase 7 tasks 7.1–7.8), live Agones/kind lifecycle (tasks 8.30–8.38, 8.49), public-network chaos/load/cost/release gates (tasks 8.50–8.53), and real-hardware graphics profiling on low/mid-tier GPUs. `make verify-kind-agones` is the committed runner for 8.49; it has not yet completed a full run against a real cluster from this workspace (blocked on local Docker/kind/Helm resource availability, not a code gap). `TODO.md`'s AI-training and presentation tasks remain separate from multiplayer.