351 KiB
Online multiplayer — architecture and task breakdown
The single tracking document for the online multiplayer effort: architecture
decisions, the wire format, current progress, and a numbered task breakdown
with checkboxes, all in one place. TODO.md points here for anything
multiplayer-related.
How to use this doc: start at §0 for what's outstanding right now. Pick up
a single numbered task, do it, verify it against its stated acceptance
criterion, mark it [x] DONE, and stop. Sections 1–6 are the decisions
those tasks assume; read them before picking up work in Phase 2 or later. §9
is a running gotchas list — check it before debugging something that looks
like a Godot/Jolt engine quirk, and add to it when you find a new one.
Status: every task in Phases 0–6 is implemented and verified locally. Both multiplayer milestones still need human verification — a playtest at ~100 ms RTT (Phase 4) and a real 3v3 session (Phase 5). Phase 6's public-internet gate is no longer blocked by the former display-name reclaim defect: allocated reconnects now use the signed identity, while the export, Docker, rotation/drain, and CI work remain complete. Phase 7's Steam foundation is in progress. Phase 8 — matchmaking, ranked, and per-match server autoscaling — remains a 1.0 launch blocker and is partially implemented: the Go domain policy, store boundaries, migration, supervisor, hardened Fleet baseline, testkit and offline end-to-end path are in place, while production API/DB/Redis/Steam/Agones wiring and runtime gates remain. It is the first phase to add a component outside the Godot project, and its design lives in docs/MATCHMAKING.md. The client has local-ship delta-rebase reconciliation, client-only ball touch prediction, adaptive input-depth signalling, and experimental remote present-time visuals; all server simulation, bot action/observation behaviour, collision resources, and tick rate remain unchanged. See the outstanding list immediately below for what is left and why, §7 for the implemented work and its evidence, and §11 for what is deliberately deferred.
0. Outstanding work — the short list
The one place to look before planning. Everything here is also written up where it belongs; this is the index, not the detail. Phases 0–5 contain no unfinished tasks.
Phase 8 (matchmaking, ranked, per-match server autoscaling) is a 1.0 launch blocker and is in progress. It is larger than anything below and adds a backend service outside the Godot project. Tasks 8.1–8.53 are in §7; the design is in docs/MATCHMAKING.md. Three findings would break a naive implementation:
| # | Finding | Why it bites |
|---|---|---|
| Task 8.28 | server_started never appearsdeploy/cosmic-clash-server now wraps the exec in stdbuf -oL -eL. Verified live — a real docker run -d container showed zero log output for 20+ seconds, including the startup line, and docker stop's SIGTERM lost it permanently rather than delaying it (Godot has no SIGTERM hook); the wrapped launcher shows the startup line within 3s of the same scenario. This affected the already-shipped community server (Docker and native systemd both route through this script), not only the not-yet-built Agones path |
Process-ready must be an explicit Agones call after static validation/listen, independent of this fix — the API/registration boundary never depended on log output either way, so this was a real operational bug (silent docker logs/journalctl), not a correctness gap in the process-ready design |
| Task 8.29 | --port defaults to 7777 and the Dockerfile hardcodes EXPOSE 7777/udp |
Live Agones passthrough/NAT and multi-match validation remain infrastructure gates |
| Task 8.48 | compose.phase6-smoke.yml hardcodes the port, first-come slots and --max-matches=2 |
The allocated flow needs its own fixture so Phase 6 behavior and invocations stay unchanged |
Blocking sign-off — the work exists, the verification does not
| # | What | Why it is not done | Detail |
|---|---|---|---|
| A | Phase 4 human playtest at ~100 ms RTT. Does the ship feel local? Does the ball? Do contact corrections read as bumps or as glitches? | Needs hands on a controller. Every numeric gate is green; feel is the milestone's actual subject and no percentile can answer it. | Phase 4 gate |
| B | Phase 5 3v3 gate: a full start-to-finish match with 6 players, a mid-match disconnect, and a late joiner. | Needs a real multi-client session. Every scenario is verified at 1v1 plus a two-bot CI match; nothing has run at 3v3. | Phase 5 gate |
These two are independent and can be done in either order, but B is the cheaper of the two to arrange and would also exercise A's conditions incidentally.
Known defects
| # | What | Severity | Detail |
|---|---|---|---|
| C | Slot reservation and takeover are keyed on display name alone. Any peer connecting with a departed player's name inside the 30 s window claims their slot, ship and team. | Real, demonstrated. Bounded by needing a genuine disconnect to race. | §11 |
| D | Input is still lost at the transport layer during a long server stall, variably — 7 of 8 runs measured 0.00 % of the sequence stream missing, the eighth 23.54 %. | Low. Distinct from the rate-limiter cause, which is fixed. The seq-guard resync visibly recovers it. | Phase 5 notes |
| E | A second Unable to send packet on channel N stderr race, in _broadcast_snapshot rather than the fixed site in _remove_player. |
Fixed. Server-side abuse disconnects invalidate the peer before closing it, and snapshot sends re-check that invalidation at the transport boundary. | §11 |
C is the one to plan around: it is fixed for free by task 7.4 (Steam auth tickets in hello), which is why it has not been given a bespoke solution. Anything that ships to strangers before Phase 7 needs it addressed first.
Open architectural question
| # | What | Detail |
|---|---|---|
| F | A contact-cohort-only shadow world. The remaining known prediction weakness is the contact cohort. Whether it is worth a client-side shadow Jolt world scoped to contacts alone is undecided — and deliberately so until A supplies the felt evidence. | Phase 4 notes |
Unstarted phases
- Phase 6 external gate: run the exported Docker server and clients from separate machines over the internet, then play a full match. This is a controlled test only until C is fixed.
- Phase 7 — Steam transport, browser, identity and production SDR (8 tasks): the optional bootstrap and
NetTransportfoundation now exist, but custom Steam client/server templates have not yet been supplied. Browser, verified tickets, bans, production credentials and ticketed Hosted Dedicated Server SDR await a project-owned Steamworks App ID and Valve coordination. Carries the fix for C and is the hard prerequisite for Phase 8.
Phase 6 has no dependency on Phase 7 and now turns a two-terminal game into something another person can host. Phase 7 is the next block because Steam identity is required before public exposure.
Deferred by choice, not forgotten
120 Hz simulation, the latency-gap measurement (task 4.9's acceptance criterion), audio hooks, split-screen — all in §11 with what each would buy and cost.
1. Architecture decisions
1.1 Locked decisions
| # | Decision | Why |
|---|---|---|
| 1 | Server-authoritative simulation, with client-side prediction of the local ship and ball. No world rollback / resimulation. | Jolt is not bit-deterministic across platforms or across differing contact orderings, and Godot exposes no world snapshot/restore API. Rollback netcode would be a research project. |
| 2 | Dedicated servers only. Headless Godot export; the server is never a player. | Fair for every player, no host advantage. Self-hostable community servers first, so nothing is blocked on paid infrastructure. |
| 3 | ENet first, GodotSteam later, behind a boundary. | ENet works in-editor, headless, on LAN, and in CI with no Steam client. Direct-IP connect stays permanently supported and must never become the degraded path. |
| 4 | Community discovery uses no custom backend; superseded for queued play by Phase 8. | Steam's server APIs remain enough for the community browser. Casual/ranked queues, durable ratings, allocation and authoritative results require the project-owned Go control plane specified in docs/MATCHMAKING.md; it does not replace the browser or direct-IP path. |
1.2 Rejected alternatives
-
Peer-authoritative ships (each client owns its own transform). Easiest to build, feels perfect locally, and is trivially cheatable — it directly contradicts
README.md's stated anti-cheat position. Ship-vs-ship collisions also become ambiguous with no arbiter. -
Deterministic lockstep / rollback. See decision 1.
-
MultiplayerSynchronizer/MultiplayerSpawner. The decisive objection is not bandwidth. It is thatlast_processed_input_seqmust arrive in the same packet as the state it describes, or reconciliation is off by a snapshot — and a synchroniser gives you nowhere to put it. It also writes replicated properties directly onto the node, which is exactly wrong for aRigidBody3Dunder prediction: incoming state has to enter a compare-against-history pipeline, not be stamped ontoglobal_transform. You would end up building the correction pipeline anyway, with the synchroniser as pure overhead. Secondary objections: one packet per body (7 bodies × ~50 B of UDP/IP/ENet framing versus one coalesced snapshot), no per-field quantisation, and no client-side interpolation.MultiplayerSpawneris unnecessary for a separate reason: the roster is fixed at match start and fully described by thematch_configmessage, and no ship is ever despawned (§6.4). -
Seeded RNG for kickoff jitter. Shared-seed determinism requires both sides to consume the RNG stream in exactly the same order forever. The first
randf()anyone later adds anywhere in the reset path — a spawn VFX variation, a cosmetic, a commentary line — silently desyncs kickoff positions with no error message. The server broadcasts the resulting transforms instead: 336 bytes, once per kickoff, cannot rot.
1.3 Derived decisions
All hot-path RPCs live on autoloads. /root/NetworkManager and /root/MatchNet exist at identical paths on every peer regardless of which scene is loaded, which side is headless, or whether a client is mid-scene-transition. This deletes the entire "NodePaths must match across peers" class of bugs, kills a family of late-join races where an RPC arrives before its target node exists, and warms Godot's RPC path cache once at connect so it never re-sends a full path on scene change.
Entities are addressed by integer slot, never by path. The snapshot is [slot 0..N-1] in a fixed order established by match_config. MatchNet holds an Array[Node] _slots populated at spawn.
One server process hosts exactly one match. This is forced, not chosen: ship.gd:162 resolves the arena boundary via get_tree().get_first_node_in_group("arena_boundary") and ai_ship_controller.gd discovers its roster via get_tree().get_nodes_in_group("ship"). Both are tree-global, so two matches in one scene tree would cross-wire instantly. It is recorded here because it determines the RAM figure in §1.4.
1.4 Server sizing — bandwidth and CPU are not the constraint
Worth establishing up front, because §2 and §3 repeatedly trade bandwidth for latency and somebody will eventually want to trade back.
ArenaBoundary.bake_colliders() generates roughly 168 box colliders (corner fillets, base wrap, ceiling, end walls) plus the scene's own slabs, 2 goal backstops, 2 Area3D sensors, and 7 dynamic bodies (the ball with continuous_cd). Estimated per-tick cost:
| Component | ms/tick |
|---|---|
| Jolt step | 0.15 – 0.4 |
| Godot headless main loop | 0.1 – 0.3 |
| Bot inference, amortised (see task 0.8) | ~0.3 |
| Total, of a 16.7 ms budget | 0.6 – 1.1 |
→ ~6–10 concurrent matches per modern core, ~150–250 MB RSS per process. 100 concurrent matches ≈ 12–16 cores and ~20 GB — a single mid-tier VPS. Upstream bandwidth for a full 6-player match is ~630 kbit/s (§2.4).
Neither CPU nor bandwidth is scarce. Latency is. Optimise accordingly.
2. Wire format
Two peers must agree byte-for-byte, so this is specified rather than sketched.
2.1 Channels
| Channel | Transfer mode | Contents |
|---|---|---|
| 0 | reliable | handshake, match_config, kickoff, goal, clock, state changes, chat, admin |
| 1 | unreliable-ordered | client → server input |
| 2 | unreliable-ordered | server → client snapshots |
Unreliable-ordered (ENet sequenced-unreliable, drops stale) rather than plain unreliable for both hot paths: we carry explicit sequence numbers, and a reordered late packet is worthless work. Separating them stops a large reliable match_config from head-of-line-blocking state on a lossy link.
Verify at implementation time. Godot's
ENetMultiplayerPeerreserves low ENet channels for its own system messages and offsetstransfer_channelon top. The intent above is "three logically distinct channels"; the concrete indices may need an offset. Confirm empirically, don't assume.
Set ENetMultiplayerPeer.server_relay = false. It defaults to true, which lets any client rpc() any other client through your server. With it off, clients can only talk to peer 1. Single highest-value one-line security change in this document.
2.2 Packet header
Every hot-path packet opens with a 1-byte type + version. A capture then decodes standalone, and a mismatched build fails loudly instead of decoding garbage straight into state.transform.
Hot paths carry a single PackedByteArray RPC argument (≈14 B of Godot RPC framing once the path cache is warm). Control messages on channel 0 use normal typed arguments — they are rare and readability beats bytes.
2.3 Input packet — client → server, channel 1, 60 Hz
u8 type_version
u32 seq server-tick-space sequence of the NEWEST action
u8 count 1..4 (MAX_REDUNDANCY)
u32 ack_snapshot_tick newest snapshot tick this client has processed
u16 client_send_ms wrapping ms clock, echoed back for RTT
--- repeated `count` times, newest first ---
i8 thrust_x, thrust_y, thrust_z value = clamp(round(v*127), -127, 127)
i8 rot_x, rot_y, rot_z
u8 flags bit0 = turbo
12 + 7×4 = 40 B payload, ~90 B on the wire with UDP/IP/ENet framing → ~43 kbit/s up per client.
- Redundancy 4 is what makes an unreliable input channel safe: starvation requires four consecutive losses (~66 ms).
i8per axis, not 3-bit bins. Bins matchingShipActionCodec.HEADSwould cut an action to 3 bytes, but they permanently foreclose analog gamepad sticks, which this game will want.round(v*127)/127round-trips-1/0/+1exactly, so today's digital input (player_ship_controller.gdisis_action_pressed-only) is lossless.- The encoding is itself a validator.
i8/127cannot express NaN, Inf, or a value outside[-1.008, 1.008]. Half of "sanitise untrusted client input" is solved by not using Variant encoding.
2.4 Snapshot — server → client, channel 2, 60 Hz default
Per-client header built per peer; body buffer built once per tick and reused across peers.
--- per-client header (7 B) ---
u32 last_input_seq newest input from THIS client the server has applied
i8 input_buffer_depth jitter-buffer occupancy; negative = starved
u16 echo_client_send_ms from that input packet, for RTT
--- shared body header (8 B) ---
u8 type_version
u32 server_tick Engine.get_physics_frames() on the server
u8 match_state see §6.1
u8 reset_gen increments on every authoritative teleport
u8 body_count
--- repeated body_count times, slot order fixed by match_config (22 B each) ---
i16 pos_x, pos_y, pos_z range ±64 m -> 1.95 mm
i16 quat_x, quat_y, quat_z w = ±sqrt(1-x²-y²-z²), sign in flags
i16 vel_x, vel_y, vel_z range ±64 m/s -> 1.95 mm/s
i8 avel_x, avel_y, avel_z ships ±4 rad/s; ball ±32 rad/s
u8 flags bit0 frozen, bit1 turbo, bits2-4 thrust_z bin,
bit5 stalled, bit6 quat_w sign
7 bodies → 8 + 7 + 7×22 = 169 B payload, ~219 B on the wire.
| per client down | server up, 6 clients | + 10 spectators | |
|---|---|---|---|
| 60 Hz | 105 kbit/s | 631 kbit/s | 1.68 Mbit/s |
MTU headroom is ~6× (ENet fragments above ~1400 B); a hypothetical 10v10 at 21 bodies is 477 B and still fits. This format does not need delta compression.
Plain i16 quaternion components, not smallest-three. Smallest-three saves 4 B/body and is the textbook answer. It is also exactly where a hand-rolled codec goes subtly wrong — off-by-one in the 2-bit index, sign of the dropped component, renormalisation drift — in a project that has no test framework yet. Three i16s plus a sign bit give ~3e-5 rad with no bit-shifting, for 2 B/body (≈3 kbit/s). Take the bytes.
Quantisation ranges derive from constants, not from prose. ArenaBoundary.INNER_HALF_X = 18.0, INNER_HALF_Z = 27.0, INNER_HEIGHT = 18.0 (arena_boundary.gd:8-10) plus GameMode.ESCAPE_MARGIN = 15.0; Ship.max_speed = 35.0 (ship.gd:16); Ball.MAX_SPEED = 32.0 (ball.gd:17).
CLAUDE.md's Architecture section states the play volume as "inner x ±12, z ±18, height 12, goal lines z ±17". That is stale — see the real constants above. Task 0.13 fixes the doc.
The flags byte must carry turbo and a 3-bit thrust_z bin. _integrate_forces is not called on frozen bodies, so remote ships on a client never pull get_action(), and Ship._update_movement_vfx() (ship.gd:293) reads _current_action.thrust.z and turbo. Without those bits, every remote ship flies with dead engines.
2.5 Reliable control messages, channel 0
hello · welcome · player_joined · player_left · ready_state · match_config · scene_ready · kickoff · state_change · goal_scored · clock_state · match_ended · chat · server_shutdown.
3. Server-side input handling
Per-player server state:
class PlayerSlot:
var peer_id: int
var slot: int # snapshot index
var ring: Array[ShipAction] # FIXED 32 entries, indexed seq % 32
var ring_seq: PackedInt32Array # 32 entries, seq stored at each index (-1 = empty)
var last_applied_seq: int
var last_action: ShipAction
var starved_ticks: int
var packets_this_second: int
var remote_controller: RLShipController # see §7 task 5.7 — null on takeover
3.1 Ingestion
@rpc("any_peer", "unreliable_ordered", channel = 1), in order:
multiplayer.get_remote_sender_id()→ look up slot. Unknown sender → drop and count.- Rate limit.
packets_this_second > 110(60 Hz × 1.5 + 20) → drop. Three consecutive seconds over budget → disconnect withRATE_LIMIT. Same for a byte budget. - Framing.
count > 4orpayload_size != 12 + count*7→ drop, count malformed. 20 malformed → disconnect. - Sequence range.
seq > server_tick + 20→ drop. (Not 120:input_leadis clamped to 12, so anything above ~20 is broken or hostile.) This is why the ring is fixed-size and indexedseq % 32— a client can never make the server allocate. - For each action, newest first at descending seq:
seq <= last_applied_seq→ discard (already consumed); else writering[seq % 32]. - Decode with per-axis clamp only:
action.thrust = Vector3(b[0]/127.0, b[1]/127.0, b[2]/127.0).clampf(-1.0, 1.0)
Never normalise the thrust vector. A player holding W+A+E legitimately produces
thrust = (1,1,1), length 1.73, and each axis uses a different power constant —thrust_power 150,maneuvering_thrust 75,vertical_thrust 120(ship.gd:12-14). Normalising would silently change the flight model for honest players. Per-axis clamp combined with thei8encoding is complete validation: the reachable value space is exactly what a legitimate client can produce.
3.2 Consumption — once per server physics tick, before the step
expected = last_applied_seq + 1
if ring holds expected:
action = ring[expected % 32]; starved_ticks = 0
else:
action = last_action # REPEAT — do not zero
starved_ticks += 1
if starved_ticks > 30: # 500 ms
action = ZERO_ACTION; flags.stalled = true
last_applied_seq = expected
last_action = action
remote_controller.action = action
Repeat-last, not zero. Player inputs are heavily autocorrelated at 60 Hz — the odds that a held thrust was released on exactly the dropped tick are low, and the client predicted with the real input either way, so repeating minimises expected divergence. It is also consistent with AIShipController, which already holds its action between decisions. Zeroing after 500 ms stops a disconnecting player's ship flying into a wall at full throttle forever.
3.3 Jitter buffer — one control loop, not three
An earlier draft had the server adapting target_depth, the server fast-forward-dropping queued actions, and the client slewing input_lead. Three integrators acting on one plant (buffer occupancy) with different time constants is a textbook oscillation; on a jittery link it hunts, and it presents to the player as intermittent sticky controls that are nearly impossible to attribute.
The server reports input_buffer_depth in every snapshot and does nothing else adaptive. The client owns input_lead exclusively.
target_depth = 1(16.7 ms), not 2. With redundancy-4 you have already bought the insurance depth 2 provides; depth 2 is 16.7 ms of pure input latency for nothing.- Client
input_leadclamp[1, 12], fast attack / slow release: on any starve, increase by up to 3 immediately; decrease by 1 per 60 ticks only after 2 s of clean surplus. A symmetric ±1-per-500 ms slew takes two seconds to absorb a wifi spike, during which the player steers and the ship does not turn — the most rage-inducing failure mode in any netcode. - Changing
input_leadmeans skipping or duplicating one tick's sequence number. Never change it more than once per 30 ticks.
Enforce input_lead server-side from observed arrival times. A client that fakes starvation to drive input_lead to 1 gets its inputs applied with less server-side buffering than honest players — a small but real responsiveness edge. The i8 encoding does nothing about this; only observing actual arrival timing does.
4. Prediction and reconciliation
4.1 Two clocks for remote entities — the load-bearing correction
The obvious design runs remote ships and the ball as frozen kinematic proxies at server_time_est - INTERP_DELAY while predicting the local ship to now. That is wrong, and it is wrong in a way that only shows up over real latency:
- Two ships closing at 50 m/s put the opponent's collider 3.5 m from truth. The hull is a
BoxShape3Dof(1.6, 0.6, 4)(ship.tscn:12) — that is most of a ship length of positional lie. - A fast ball is 2.2 m off against a 0.5 m radius — four ball diameters.
ship.tscn:16hascollision_mask = 7: ships collide with ships, the ball, and the arena. Ship-vs-ship contact is constant in vehicle soccer, not incidental.
So prediction would not diverge occasionally due to timing noise. It would diverge deterministically and in the same direction on essentially every contact, and the hard-snap threshold would become the steady state rather than a backstop.
Fix: separate the collider clock from the render clock.
| runs at | why | |
|---|---|---|
| remote body collider | server_time_est, extrapolated forward from the newest snapshot by ~one-way + half a snapshot interval |
Extrapolation error over ~45 ms at real accelerations (thrust_power 150 / mass 5 = 30 m/s², 75 m/s² on turbo — ship.gd:12,15, ship.tscn:17) is ~0.03–0.08 m. Two orders of magnitude better than 3.5 m. |
remote $Visual |
server_time_est - INTERP_DELAY |
Smooth, jitter-free rendering. |
This is the same trick applied to the local ship, pointed the other way. It costs one extra transform write per remote body per tick.
4.2 Where each piece lives
| Concern | Location |
|---|---|
| sample + send input | LocalNetShipController._physics_process — runs before the physics step, guarantees exactly one sample/tick |
| record predicted state | same, at top of tick N (state = result of N−1) |
| apply velocity / teleport correction | Ship._integrate_forces, ~15 guarded lines — the only Jolt-safe place to write state.transform / state.linear_velocity |
| visual smoothing | Ship/$Visual.global_transform, set in _physics_process |
| snap-vs-blend decision | net_ship_predictor.gd (child node) |
| remote bodies | net_interpolator.gd |
4.3 Per-tick, own ship
predicted[current_tick - 1] = {transform, linear_velocity, angular_velocity}— ring of 128.var a := _player.get_action().copy()— must copy.player_ship_controller.gdreuses a singleShipActionacross ticks (its own header warns about this); buffering it aliases every history entry to the same object. See task 0.1._action = a, returned byget_action()this tick soShip._integrate_forcessamples input exactly once.input_history[seq] = a,seq = predicted_server_tick + input_lead.- Build and send the packet with the last 4 entries.
Ship._integrate_forces then runs completely unchanged.
4.4 On snapshot arrival
A = last_input_seq
if reset_gen changed OR predicted[A] missing OR flags.frozen != local frozen:
HARD SNAP
else if pos_err > 2.0 m OR rot_err > 60°:
HARD SNAP
else:
SOFT CORRECT
Comparing server state at tick A against predicted[A] — the client's own state at that same tick — makes the delta latency-free by construction. That is the entire reason for keeping the prediction ring, and it is why this works acceptably without resimulation: never blend current state toward stale state.
SOFT CORRECT
- Velocity: applied in full, immediately.
net_vel_correction += (srv.linvel - predicted[A].linvel), consumed once in_integrate_forces. Velocity error is invisible to the player but is the cause of future position error; blending it just prolongs divergence. - Position/rotation: physics moves in full, rendering does not. Queue the body teleport, and simultaneously offset
$Visualby the negation. Net visual movement at the instant of correction: zero. The body is where the server says; the rendered ship catches up. - Decay each physics tick, reusing the existing convention at
ship.gd:450:var k := _tick_scaled(0.88, delta) # 63% gone in ~130 ms, 95% in ~280 ms MAX_VISUAL_OFFSET = 0.4 m, not 2.0. The hull is 4 m long; a 2 m offset means being rendered half a ship-length from your own collider for ~280 ms, so you clip walls you visibly cleared — a felt bug in a game built around wall-riding. Beyond 0.4 m, show the correction. A visible correction is honest; an invisible 2 m lie is not.
HARD CORRECT
- Apply the same sequence-matched authoritative pose and velocity delta to the current local body, reset body and
$Visualinterpolation, and clear the visual offset. It is physically the same correction as soft correction; only its presentation differs.
Settled Phase 4 decision — delta transport, not one-body replay. For every matched snapshot, overwrite predicted[A] with authority, transport its pose and linear/angular-velocity delta through each retained state A+1..current, and apply that same delta once to the live local Jolt body. This keeps retained history coherent, so a later snapshot does not correct an already-corrected pre-delta trajectory a second time.
Do not analytically replay stored actions. That approximation cannot reproduce Jolt integration or contact manifolds (friction, restitution, walls, ships, and ball), therefore it becomes least trustworthy exactly where reconciliation is most noticeable. This is still neither whole-world rollback nor a change to server physics: it is client-only state transport around a server-authoritative simulation.
For reset generation changes, place exact authority, begin a new history epoch, and do not consume pre-reset actions. For missing or overflowed history, place authority once and suppress stale acknowledgements until a new matched sequence is recorded; never manufacture future history by filling it with one stale authority state.
Same-sequence pre-correction residual remains diagnostic telemetry. With a server input jitter buffer, it is not by itself a presentation-quality gate: the server may have integrated an action at a different physical instant from the client. Acceptance must report it separately by free-flight/contact/reset/resync cohort, while gating post-correction/presentation error and hard-snap behaviour.
That the two sides integrate the same action for a given sequence is a separate claim, and a checkable one — it is what the action marker and task 4.11's
--exercise-input-transitionsgate exist for. Keep the two apart: "right action, different instant" is expected here; "wrong action" is a bug, and was one.
4.5 Camera and visuals
The camera must follow $Visual, not the body. ship_camera.gd:115, :149, :150 read target.global_transform directly. Left as-is, every soft correct makes the camera jump the full error while the mesh smoothly lags — strictly worse than snapping, because the world lurches around a player whose ship slides inside the frame.
And it must read $Visual.get_global_transform_interpolated() from _process, not global_transform from _physics_process (task 0.16, rationale in §5.4). Node3D.get_global_transform_interpolated() exists precisely for a camera tracking a physics-interpolated body; global_transform returns the last physics tick's pose, so a _process camera reading it would chase a 60 Hz staircase at 240 fps.
Ordering hazard, straight from the engine docs:
get_global_transform_interpolated()"creates an interpolation pump on theNode3Dthe first time it is called, which can respond to physics interpolation resets… be sure to call it at least once before resetting theNode3Dphysics interpolation." Every hard snap callsreset_physics_interpolation()on$Visual. Prime the pump when the camera'stargetis assigned, not lazily on the first frame, or the first snap of the match streaks the camera.
project.godot has physics_interpolation=true, and $Visual's own local transform is interpolated too — so reset_physics_interpolation() must be called on $Visual as well as the body, or every snap smears the mesh for a frame. (This is the same artefact game_mode.gd:263 already exists to prevent.)
4.6 Remote bodies on the client
freeze = true,freeze_mode = FREEZE_MODE_KINEMATIC— notSTATIC, or Jolt cannot derive contact velocity from the per-tick transform delta and your predicted ship hits a static wall instead of a moving ship.net_interpolator.gdsamples the snapshot buffer (last 8 per body); collider atserver_time_est(§4.1),$Visualatserver_time_est - INTERP_DELAY.- The two samples run on different clocks and different callbacks. The collider is a physics concern:
_physics_process, 60 Hz.$Visualis a render concern:_process, sampled at true render time withphysics_interpolation_mode = OFFso Godot does not interpolate an already-per-frame transform. On a 240 Hz client this is 240 distinct remote-ship positions per second instead of 60, and one fewer tick of lag, for no extra cost — the buffer lerp is happening either way (§5.4). INTERP_DELAY = one_way_ms + snapshot_interval * 1.5 + 2.5 * jitter_ewma, clamped[25, 200] ms. At 60 ms RTT / 60 Hz / 5 ms jitter that is 30 + 25 + 12.5 ≈ 68 ms.
The
one_way_msterm is not optional, and omitting it is a silent architectural failure.server_time_est(§4.7) estimates what the server clock reads right now. The newest snapshot in hand was stampedone_wayago — §4.1 says exactly this when it extrapolates the collider forward "by ~one-way + half a snapshot interval". So rendering$Visualatserver_time_est - INTERP_DELAYonly interpolates ifINTERP_DELAY ≥ one_way. Set it to the buffer alone (~38 ms at 60 Hz) and the render cursor lands on or past the newest sample: the bullet below about extrapolating past the newest snapshot becomes the steady state rather than the exception, and every remote entity is permanently dead-reckoned. The 25 ms clamp floor is reachable on LAN only.
- Past the newest snapshot, extrapolate on last known velocity for at most 150 ms, then hold. Never extrapolate indefinitely — a stuck ship reads better than one flying through a wall.
- Never write
linear_velocityto a frozen body. Godot/Jolt zeroes and holds velocity on frozen bodies, soball.gd:35'slinear_velocity.length()trail driver will not work that way. AddBall.set_visual_speed(speed)mirroring theShip.set_visual_action(thrust_z, turbo)pattern. Don't route presentation data through a property the physics server owns. - Call
reset_physics_interpolation()on remote bodies at every kickoff.
4.7 Clock
server_time_est = local_ms + clock_offset, clock_offset from ping/pong on channel 0 every 1 s using the minimum-RTT sample in a rolling 5 s window (the min-RTT sample has the least queueing error).
Freeze tick_offset at match start. Seed it exactly from the handshake (server_tick + round(one_way / tick_ms)) and absorb all subsequent drift into input_lead alone. The prediction ring is indexed in server-tick space, so slewing tick_offset during play silently reinterprets every historical entry and produces sporadic, unreproducible false snaps. Re-seed only across a kickoff boundary.
5. Latency and frame-rate budget
Three of the largest terms are invisible to a netcode document that only counts network hops. Record the budget so future changes are argued against a number.
Client at 60 Hz physics, 60 ms RTT, 5 ms jitter, 60 Hz snapshots. Display at 60 Hz with vsync on — the Godot default, and the worst case. §5.4 redoes the display-dependent rows for 120/144/165/240/360 Hz.
5.1 Own ship (predicted) — input to pixel
| Stage | ms | scales with fps? | |
|---|---|---|---|
OS input → Input.is_action_pressed |
10 | 0.5 × frame interval + device polling | partly — see below |
| wait for next physics tick | 8 | avg of 0–16.7 | no — 60 Hz physics |
| physics step applies force | 0 | ||
| Godot physics interpolation | 8 | physics_interpolation=true; mean, worst case 16.7 |
no — 60 Hz physics |
| render + vsync present | 25 | 1.5 refresh intervals, vsync defaults on | yes |
| Total | ≈52 |
This is the existing single-player floor, unchanged by netcode — and ~43 of those 52 ms are things no netcode document discusses. A low-latency present would take it to ~35 ms (§5.4).
Two notes on the model, both corrected from an earlier draft that read ≈45:
- Input freshness is 0.5 of a frame interval, not 0.25. Godot pumps OS input once per main-loop iteration and
Ship._integrate_forces(ship.gd:347) consumes it once per physics tick; for arrivals distributed uniformly between pumps the mean staleness at the pump is half the interval. On top sits device polling, which does not scale with fps at all: ~1 ms at a 1000 Hz mouse or gamepad, ~8 ms at a 125 Hz USB device. The table assumes ~2 ms. - Physics interpolation's 8 ms is a mean. Rendering happens between the two most recent completed ticks, so displayed pose lags the newest state by
(1 − fraction)of a tick — 0 to 16.7 ms, averaging 8.3. The worst case matters for §5.4's discussion of frame-time variance.
Note the right-hand column: 16 of the 52 ms do not move no matter how many frames the client draws. That is the price of a 60 Hz simulation.
5.2 World response — the number that decides whether this ships
| Stage | ms | |
|---|---|---|
| input freshness | 10 | 0.5 × frame interval + ~2 ms device polling |
| wait for next physics tick | 8 | |
| manual multiplayer flush | ~0 | ~8 with default idle-frame poll — see §7 task 1.3 |
| client → server transit | 30 | RTT/2 |
jitter buffer, target_depth = 1 |
17 | |
| server tick + flush | 8 | |
| server → client transit | 30 | RTT/2 — the return leg |
| interpolation buffer beyond arrival | 38 | interval × 1.5 + 2.5 × jitter; the one_way half of INTERP_DELAY is the row above |
| client physics interpolation | 8 | |
| render + present | 25 | vsync on, 60 Hz display |
| World response, opponents | ≈174 | |
| Ball, with local prediction | ≈52 | same as own ship |
| Both, at 144 Hz + low-latency present | 148 / 26 | §5.4 |
Correction — this table previously read ≈138 ms and omitted the server→client transit row entirely.
INTERP_DELAYwas quoted as 38 ms, which is the interpolation buffer measured from snapshot arrival, while §4.6 defines the render cursor relative toserver_time_est— server-now. The 30 ms return leg fell between the two definitions and was never counted. §4.6's formula is corrected to includeone_way; this table keeps the two terms on separate rows because that is clearer to budget against.
For reference, Rocket League runs 120 Hz physics and predicts both car and ball locally; its equivalent at 60 ms RTT is roughly 90–110 ms.
≈174 ms as designed here is not competitive, and this document should not pretend otherwise. It is also not the end state: §5.6 gets to ≈127 ms with two changes that touch no graphics setting and require no bot retrain, and to ≈103 ms with 120 Hz simulation — inside the reference band. Read §5.6 before treating this table as a verdict.
What is settled is the shape of the design: a locally-predicted ball and own ship at ≈52 ms is the difference between this being playable and not, and a 30 Hz / default-poll / interpolated-ball design would land near ≈250.
5.3 Why 60 Hz snapshots, not 30
- Interpolation buffer: the
interval × 1.5term is 50 ms at 30 Hz vs 25 at 60, on top of the one-way term both share (§4.6), plus a half-interval of cadence quantisation. - Interpolation fidelity: at
MAX_SPEED = 32the ball moves 1.07 m between samples at 30 Hz — more than its own diameter, so any wall bounce landing between two samples gets lerped as a straight line through the wall. At 60 Hz it is 0.53 m. - Cost: 300 kbit/s. Per §1.4, bandwidth is not the constraint.
Keep --snapshot-hz 30 as an explicit degraded mode.
5.4 High-refresh-rate clients — 120 / 144 / 165 / 240 / 360 Hz
Players on high-refresh displays are the ones most sensitive to everything in this document, and the current code has three places where the client draws 240 frames but only 60 of them contain new information. Those are bugs, not tuning.
What frame rate actually buys
Modelling present as ~1.5 refresh intervals with vsync on (§5.1), and input freshness as 0.5 of a frame interval plus ~2 ms of device polling:
| Display | present | own ship / ball (§5.1) | world response (§5.2) | with low-latency present |
|---|---|---|---|---|
| 60 Hz | 25.0 | 52 | 174 | 35 / 157 |
| 120 Hz | 12.5 | 35 | 158 | 27 / 149 |
| 144 Hz | 10.4 | 33 | 155 | 26 / 148 |
| 165 Hz | 9.1 | 31 | 153 | 25 / 147 |
| 240 Hz | 6.3 | 27 | 149 | 23 / 145 |
| 360 Hz | 4.2 | 24 | 146 | 21 / 144 |
This table assumes the client can actually produce those frames. It cannot — see §5.5. As configured today the project runs SDFGI, SSIL, SSAO, a 5-level glow pyramid, five shadow-casting lights, MSAA 4× and FXAA, and an unconditional full-screen backbuffer pass, none of which any player can switch off. Read §5.5 before treating any row below 60 Hz's as reachable.
Three conclusions to design around:
- 60 → 144 Hz is worth ~19 ms on own-ship feel. 144 → 360 Hz is worth ~9. The curve flattens hard, because 16 ms of the remaining budget is the 60 Hz physics tick plus its interpolation and does not move.
- A low-latency present is worth more at 60 Hz (−17 ms) than the entire jump from 144 to 360 Hz. It costs one settings dropdown.
- Frame rate barely moves world response — 174 → 146 across the whole 60–360 range, because that budget is dominated by RTT and the interpolation buffer. Frame rate is an own-ship feel lever, not a netcode one. Say this to players plainly; someone who buys a 360 Hz monitor to see opponents sooner has been mis-sold.
Three things that must run per rendered frame, not per physics tick
a. The camera rig. ship_camera.gd:86 runs the entire rig in _physics_process. Global physics_interpolation=true smooths the resulting camera transform, so this is not visible as judder — but it costs an extra tick of camera latency on top of the ship's, and two things it does are not transforms and therefore not interpolated: camera.fov (:182) and the PostFX shader parameters (:186-187). At 240 fps those step at 60 Hz, which reads as a faint pulse in the turbo FOV kick.
The rig moves to _process, reading target.get_global_transform_interpolated() (and $Visual's, post-task 0.2) instead of target.global_transform, with physics_interpolation_mode = PHYSICS_INTERPOLATION_MODE_OFF on the rig itself so Godot does not re-interpolate an already-per-frame transform.
The move is cheap but it is not tuning-neutral. Cost first: one call is ~15 engine-bound operations (2 × get_noise_1d, 2 × set_shader_parameter, Basis.looking_at, slerp, orthonormalized, signed_angle_to, rotated, several global_basis accesses) plus ~60–100 bytecode ops — call it 5–15 µs. At 360 Hz that is 1.8–5.4 ms/s, under 0.5% of a core. Negligible, but negligible because the absolute work is tiny; 1-exp(-k·delta) is a correctness property, not a cost argument, and it does not license moving arbitrarily expensive code into _process.
The impact shake must be re-tuned, and in the opposite direction to what you would guess.
ship_camera.gd:204advances the noise coordinate bydelta * 60.0, and:64setsfrequency = 2.5, so each sample stepsdelta × 150noise units. At 60 fps that is 2.5 units per sample — simplex noise decorrelates over roughly 1 unit, so the shake is currently white noise, and physics interpolation is lerping between independent samples. At 360 fps in_processit becomes 0.42 units per sample, which is strongly correlated: the shake turns into a slow, smooth wobble that gets softer the better your monitor is. Re-derivefrequency(or the* 60.0) for constant noise-units-per-second, then re-check amplitude by eye at 60 and 240 fps.
Everything else in the rig genuinely is rate-independent and needs no attention: 1.0 - exp(-k * delta) at :126, 137, 156, 172, 177 and move_toward(…, shake_decay * delta) at :212.
Two pre-existing bugs sit in the code this task touches, so fix them here rather than discovering them in Phase 5:
- The rig has no snap path.
camera.global_positionis smoothed atcamera_smoothing = 10.0(:14, 137, 156) with no reset anywhere in the file. At a kickoff teleport (game_mode.gd:256-263, becoming an_integrate_forceswrite under task 0.15) the camera lerps across the arena over ~300 ms. Addsnap_to_target()— setglobal_position/global_basisdirectly, zero_last_shake_offset— and call it from the kickoff path. - Shake decay stalls during a goal cut.
:94-96returns before_apply_shake, so_shake_strength'smove_towarddecay never runs for the length of the cinematic. Task 0.12 proposes building goal feel on exactly this system.
b. Remote-entity visuals. §4.6's interpolator samples a snapshot buffer between two known states. Driving that from _physics_process quantises every remote ship and the ball to 60 distinct positions per second and then leans on Godot to interpolate between them — an extra tick of lag for no benefit, since we are already interpolating. Sample the buffer at true render time in _process instead: 240 distinct positions per second and one fewer tick of lag.
The split is clean because the two consumers want different times anyway (§4.1): the collider is a physics concern and stays in _physics_process at server_time_est; $Visual is a render concern and moves to _process at server_time_est - INTERP_DELAY, with physics_interpolation_mode = OFF. Setting it OFF is coherent precisely because the node's global_transform is overwritten every rendered frame — there is nothing left for the engine to interpolate. Note this is the opposite of §4.5's rule for the local ship's $Visual, which is written per physics tick and therefore must stay interpolated and must be reset on snap. Same node name, two different regimes; task 0.16 lands in Phase 0 against local-ship semantics, task 2.4 adds the remote case.
It is not free, though it is cheap: per body per frame you bracket-search a ring of 8, run two Vector3.lerps and a Quaternion.slerp, build a Transform3D, and assign global_transform (which dirties and propagates to children). Estimate 3–6 µs per body → ~21–42 µs/frame for 7 bodies, ~1.5% of a core at 360 Hz. That is 4–6× the work of sampling at 60 Hz. Measure it in task 0.15b rather than asserting it.
c. Receive polling. Task 1.3 already flushes sends from _physics_process. Receiving is the other half: with (b) in place, a snapshot that lands 2 ms after a physics tick can be rendered 2 ms later at 240 fps instead of waiting 14 ms for the next tick. Poll for receive unconditionally at the top of both _process and _physics_process — no rate limiter. A zero-timeout enet_host_service on an empty socket is one non-blocking recvfrom returning EWOULDBLOCK, on the order of 1 µs; 360 of those per second costs ~0.36 ms/s. An earlier draft proposed a 2 ms limiter, which is worse than useless: at 240 fps the frame interval is already 4.17 ms so it never fires, and it only engages above ~500 fps where polling was already cheaper than the limiter.
Manual polling relocates the connection signals. With
set_multiplayer_poll(false),peer_connected/peer_disconnectednow fire from inside yourpoll()call — mid-_process, during a render frame — rather than on the idle-frame boundary. Any handler that mutates the scene tree must defer.
Frame-time variance, not mean frame rate, is the real target
At 240 fps the frame budget is 4.17 ms, and physics runs at 60 Hz — so one frame in four carries the entire physics tick and must still fit in 4.17 ms. On that frame the client pays, in one go: the Jolt step over 7 dynamic bodies against a 172-shape compound; 7 × Ship._integrate_forces (ship.gd:346-357), each running apply_thruster_forces, a full ArenaBoundary.get_surface_pull with five _falloff calls (arena_boundary.gd:183-198), apply_rotation_forces, apply_righting_torque and apply_drag_and_limits with two pow() calls via _tick_scaled (:450); 6 × _update_movement_vfx (:296-315, writing two material params and two OmniLight3D energies per ship); and on decision ticks, bot inference — policy_network.gd is a pure-GDScript MLP at 31→64→64→7 ≈ 6.5k multiply-accumulates per bot, so five bots landing together is ~33k GDScript float ops in one frame.
Task 0.8's decision stagger is framed above as a cosmetic hitch. It is not — the physics tick sets a floor on 1%-low frame time that no graphics setting can lower. A game that averages 240 fps but drops one frame in four to 8 ms is not a 240 fps game. Profile p99, not mean (task 0.15b).
The same term matters at the bottom of the range, where most players actually are: see gotcha 22 and task 0.22 for the client-side Engine.max_physics_steps_per_frame cap that stops a hitching client from spiralling.
What frame rate does not buy, so nobody optimises the wrong thing
Input sampling does not improve. player_ship_controller.gd:15-38 reads seven Input.is_action_pressed calls — all digital, all held-state — and Ship._integrate_forces pulls them once per physics tick. The state read at the tick is the freshest state; sampling it 240 times a second returns the same value 4 times in a row. The only thing lost is a press-and-release entirely inside one 16.7 ms tick, which is below human tap duration. Do not build a sub-tick input accumulator. If analog stick support is added later this changes, and the right answer is then a time-weighted average over the tick, not a higher sample rate.
Physics interpolation stays on. It costs ~8 ms (§5.1) and is the single largest fps-independent term after the tick wait, so it will look like a target. It is not: without it a 60 Hz simulation presents 60 distinct world states per second regardless of frame rate, which is precisely the stepping a 240 Hz display was bought to avoid. Leave it on; do not expose a toggle.
Why physics stays at 60 Hz, and what a bump would cost
The honest answer to "our players want 240 fps responsiveness" is that simulation rate, not frame rate, is the binding constraint — 16 ms of own-ship latency and ~33 ms of world response sit behind it, and §5.2 shows frame rate alone cannot get world response under ~146 ms. Doubling to 120 Hz (Rocket League's rate, with snapshots raised alongside) would take world response from ≈174 to ≈141 ms and own-ship from 52 to ≈44, at 60 Hz display — or ≈115 ms combined with a 144 Hz display and a low-latency present:
| Term | 60 Hz sim | 120 Hz sim | |
|---|---|---|---|
| wait for next tick | 8.3 | 4.2 | |
| physics interpolation | 8.3 | 4.2 | |
| jitter buffer, depth 1 | 16.7 | 8.3 | |
| server tick + flush | 8 | 4 | |
| interpolation buffer | 37.5 | 25.0 | only the interval × 1.5 term halves; the jitter term does not |
| client ↔ server transit | 60 | 60 | does not move |
That is a bigger win than every tuning parameter in §3 and §4 combined. It is nonetheless out of scope for v1, for reasons that are about the project rather than the netcode:
- Every policy in
Game/bots/is invalidated.ship.gd:450's_tick_scaledis defined against a 60 Hz reference andai_ship_controller.gd'sreaction_tickscounts ticks. A bump means a full retrain — and perTODO.mdthe generation-5 curriculum is still running. - Server density halves, ~6–10 matches per core to ~3–5 (§1.4).
- Bandwidth roughly doubles: input 43 → 86 kbit/s up, snapshots 105 → 210 kbit/s per client, 631 kbit/s → 1.26 Mbit/s per 6-player match. Still not the constraint, but 100 concurrent matches becomes ~126 Mbit/s of server uplink, which is a hosting-plan question rather than a rounding error.
The consequence for this plan is a hard rule: 60 is a constant named NetCodec.TICK_HZ, never a literal. Ring sizes, INTERP_DELAY, input_lead clamps, seq-window bounds, snapshot cadence and the timeout constants all derive from it. Task 1.4's handshake already gates on physics_ticks_per_second, so a mismatched client is rejected rather than silently desynced. Done this way, a later bump is a config change plus a retrain — not a protocol rewrite. Done the other way, the literal 60 ends up in twelve files and the bump never happens.
Client display settings
project.godot sets neither display/window/vsync_mode (defaults to enabled/FIFO) nor application/run/max_fps (uncapped). video_settings.gd:14-16 persists only AA, glow and brightness, and settings_menu.gd exposes only those three. Task 0.17 adds:
VSync: Enabled (FIFO) · Adaptive (default) · Mailbox · Disabled.
- Adaptive (
FIFO_RELAXED) is FIFO while the renderer keeps up and tears only on a missed vblank. That is the right default for a game that will sometimes drop below refresh, because it avoids FIFO's half-rate cliff — miss 144 Hz by one millisecond under strict FIFO and you are pinned to 72. - Mailbox only lowers latency when the renderer sustains above the refresh rate; below it there is never a second frame to replace the queued one, so it degenerates to FIFO latency at Mailbox power draw. Per §5.5 this build will not sustain above 144 Hz on typical hardware today, which makes Mailbox an opt-in for players with headroom, not a default. Defaulting to it would be a thermal regression for most players in exchange for nothing.
FPS cap: derived from the display, not a fixed list. Query DisplayServer.screen_get_refresh_rate(DisplayServer.window_get_current_screen()) and offer "Match display" (default), the integer divisors of that rate, then Unlimited — 144 Hz → 144/72/48, 165 Hz → 165/82/55, 240 Hz → 240/120/80/60.
Non-divisor caps beat against scanout. A fixed 60/75/90/…/360 list is wrong on every panel that is not 60 or 120 Hz. Cap at 100 on a 144 Hz display and
gcd(100,144) = 4: the pattern repeats every 25 frames across 36 refreshes, with frames held for one or two intervals in an irregular sequence — visible micro-stutter. 120 on a 165 Hz panel is 8 frames per 11 refreshes, same failure. Offer the free-form list only behind an Advanced toggle with a warning.
Three implementation constraints, all of which an earlier draft got wrong:
Engine.max_fpsis a throttle, not a pacer. It pads each frame with a post-frame sleep to hit1/max_fps; it has no knowledge of scanout and never phase-locks to a vblank. (Sleep-granularity jitter of roughly ±0.5–1 ms is inferred, not measured — verify on target platforms. The absence of phase locking is structural.)- Grey out the FPS cap whenever VSync is not Disabled. With both active, FIFO clamps presents to vblanks while
max_fpspushes some frames past the next one and not others — frame pacing worse than either setting alone. The menu must not permit the combination. - Godot cannot report the negotiated present mode.
DisplayServer.window_get_vsync_mode()echoes back the mode you stored, not theVkPresentModeKHRthe driver granted, and there is no GDScript API that exposes the latter. An earlier draft's "report what was actually applied" is not implementable, and neither is an in-engine present-latency measurement (that needs LDAT or a high-speed camera). Instead put a livePerformance.get_monitor(Performance.TIME_FPS)readout next to the dropdown: whether the player is above or below their refresh rate is the fact every one of these settings depends on.
The renderer is Forward+ (project.godot:21, config/features=PackedStringArray("4.7", "Forward Plus")), so the usual "Mailbox is unavailable on Compatibility" caveat does not apply as written — but rendering/renderer/rendering_method is not pinned in project.godot, so a --rendering-method gl_compatibility launch or a driver fallback loses it silently. Mailbox is also commonly unavailable on macOS/MoltenVK. (Needs empirical verification on target OS versions.)
5.5 Can this build produce frames at all?
§5.4's table describes a machine this project is not. Nothing in the repo has ever been profiled, and the render configuration is a showcase build, not a competitive one. Every item below is on by default and none is reachable from video_settings.gd, which persists exactly three values (:14-16: aa_mode, glow_scale, brightness).
From scenes/arena_base.tscn, the Environment every arena inherits:
arena_base.tscn |
Setting | Note |
|---|---|---|
:47-50 |
sdfgi_enabled, sdfgi_use_occlusion, sdfgi_bounce_feedback = 0.5 |
Godot 4's most expensive GI path; cascades re-voxelise as the camera moves, and this camera never stops (ship_camera.gd:126,137,156) |
:42-46 |
ssil_enabled, ssil_radius = 4.0 |
A full-resolution screen-space pass on top of SSAO |
:34-41 |
ssao_enabled, ssao_radius = 2.5, ssao_detail = 0.75 |
|
:18-29 |
glow_enabled, 5 levels |
Mip pyramid built and resolved every frame |
:61, 78, 87, 96, 105 |
1 directional + 4 shadow-casting OmniLight3Ds |
Omni shadows are cubemaps: 24 shadow-map faces per frame before the directional |
Plus project.godot [rendering]: msaa_3d=2 (4×) and screen_space_aa=1 (FXAA) and use_debanding=true — mirrored by video_settings.gd:14 defaulting to MSAA_FXAA. Stacking FXAA on resolved MSAA is redundant blur, and the menu (settings_menu.gd) offers no 2× rung between "off" and "4×".
Plus shaders/post_process.gdshader:4, uniform sampler2D screen_texture : hint_screen_texture — a full-screen backbuffer copy every frame, unconditionally. The shader's comment notes that non-turbo frames skip two texture taps, but the copy and the full-screen pass happen regardless because vignette_strength never reaches zero (ship_camera.gd:187 writes 0.22 + …, :243 restores 0.22).
What is not the problem, so nobody optimises the wrong thing:
- The 168 colliders (§1.4) cost zero frame time. They are
CollisionShape3Ds on aStaticBody3D— no draw calls, no vertices. The count is confirmed correct (168 generated + 4 authored slabs = 172 inobjects/arena_boundary.tscn). - The scene is not geometry- or draw-call-bound.
arena_boundary.gd's visual shell is ~1450 triangles in two surfaces of oneMeshInstance3D; the whole match is on the order of 100–150 draw calls and well under 50k vertices. That is nothing.
The project is bound entirely by full-screen passes the player cannot switch off. That inverts §5.4's conclusion about where the leverage is: the largest win per line of code is not a vsync dropdown, it is a graphics preset that gates SDFGI/SSIL/SSAO/omni shadows. Task 0.15b blocks 0.16 and 0.17 for exactly this reason — every number in §5.4 is a priori, and the first measurement may invalidate the fps list entirely.
One mitigating subtlety, which cuts both ways: project.godot [display] sets window/stretch/mode="viewport" with a 1920×1080 base and aspect="expand", so the 3D renders at a fixed ~1080p and is blitted to the window. A 1440p or 4K player therefore does not pay more for any of the above — but also cannot render at native resolution, and a 1080p player cannot render lower. Task 0.17c owns that decision; it interacts directly with render scaling (0.17b) and cannot be left implicit.
5.5.1 Measured (task 0.15b, 2026-08-18)
6-ship Match, 1080p, non-headless. Hardware: Apple M4 (Metal), 10-core — a development laptop, not a dedicated gaming reference machine; treat absolute fps as directional, not a promise to players on other hardware.
| p50 | p99 | fps (p50 / p99) | |
|---|---|---|---|
| All effects on (project defaults) | 17.93 ms | 20.39 ms | 55.8 / 49.0 |
| All effects off | ~17.2 ms | — | ~58 |
This invalidates the a priori §5.4/§5.5 fps list exactly as flagged. Default settings cannot sustain even 60 fps on this hardware, let alone 144 — and the surprising part is why: turning every toggleable effect off (SDFGI, SSIL, SSAO, glow, all 5 shadow casters, MSAA, FXAA, PostFX) only recovers the difference between ~56 and ~58 fps. The ~17 ms floor is not made of the full-screen passes this section blamed — something else (base forward-clustered shading, the ~150 draw calls, per-ship VFX materials, or fixed engine/CPU overhead at 6 ships) dominates, and 5.4's framing ("the project is bound entirely by full-screen passes") is wrong as measured on this hardware.
Per-effect isolated cost (each toggled off individually against a fixed baseline sample), for reference — treat these as low-confidence: they cluster tightly at 2.9–3.8 ms each with no clear outlier, which is consistent with most of that spread being sampling noise from a ~1 ms-jittery baseline rather than real per-effect attribution:
| Setting | Cost (ms) |
|---|---|
| SSAO | 3.77 |
| PostFX | 3.82 |
| Omni shadows (×4) | 3.69 |
| SSIL | 3.44 |
| FXAA | 3.37 |
| Directional shadow | 3.30 |
| SDFGI | 3.24 |
| MSAA 4× | 3.12 |
| Glow | 2.89 |
Consequence for 0.17/0.26/0.28: a graphics preset alone will not reach a 144 fps target on hardware in this class — Low-preset gets to only ~58 fps by this measurement, not the 2×+ jump §5.4 assumed. 0.26 (bake GI) and 0.28 (separate physics thread) need to re-justify their expected win against this floor before implementation.
Root-cause follow-up, attempted and inconclusive (2026-08-18). Three further remote-automated profiling passes (via godot-mcp game_eval sampling Performance.get_monitor() against a live instance, no human at the editor) were run to find what the ~17 ms floor actually is. They did not converge:
| Pass | Setup | Result |
|---|---|---|
| 1 (above) | 6-ship 3v3, sustained | 17.93 / 20.39 ms (p50/p99), all-off floor ~17.2 ms |
| 2 | Reportedly 6-ship, actually 1v1 (misconfigured) | CPU 17.64 ms + frame 10.75 ms — internally inconsistent (CPU time exceeding frame time from non-atomic sampling); agent also reported the game becoming unresponsive mid-run |
| 3 | 6-ship 3v3, atomic single-eval sampling, retried after pass 2's failures |
8.7–10.2 ms (98–115 fps), reported CPU time 0.013 ms — implausibly low for a frame running Jolt physics + GDScript bot inference across 6 ships, so not trusted either |
Passes 1 and 3 supposedly measured the same scenario and differ by ~2×. The likely explanation is the measurement method itself, not the game: each game_eval round-trip through the MCP bridge has its own latency and can perturb the very frame timing it's sampling, and nothing here confirms the scene state (ship count, bot activity, camera framing) was identical across passes. Read the specific numbers in this subsection as evidence a floor well under 144 fps exists, not as an attributed cause — the SSAO on/off screenshot check in pass 3 did confirm effect toggles are visually real (ruling out "the toggles are no-ops" as an explanation), which is the one finding that survived across passes.
What this needs next, and why an agent can't finish it remotely: a proper frame-time attribution needs either a human at the Godot editor reading the Debugger's built-in Monitors/Visual Profiler (which breaks GPU time down by pass — opaque, shadow, post-process, etc. — instead of one aggregate number), or an external GPU profiler (RenderDoc, Xcode GPU capture on this hardware). Both require eyes on a live UI, not remote eval polling. This is now the concrete blocker for 0.26/0.28, not further scripted measurement passes. 0.15b's original acceptance criterion (write a max-frame-rate number into §5.5) is still met by pass 1 — the floor is real and under both 60 and 144 fps — but the deeper "why" is open and parked here rather than guessed at.
Root cause of the pass-to-pass inconsistency, found (2026-08-18): a Godot editor and an orphaned headless training process had both been running on the profiling machine, untouched, for 11 days (since 2026-08-08) — leftover from earlier local work, unrelated to this investigation. godot-mcp's automated launches were plausibly contending with that stale editor instance rather than getting a clean process every pass, which is a much better explanation for a ~2× swing between "identical" scenarios than genuine frame-time variance. Both processes were killed and a clean re-check was run.
Is it just that we're on a Mac? Partly, but not via the mechanism first suspected. HiDPI/Retina resolution inflation was checked directly and ruled out: the live viewport renders at 2036×1080 against a target of 1920×1080 — about 6% more pixels, non-uniformly (width only; the 2× multiplier a true Retina backbuffer would apply is not happening, display/window/dpi/allow_hidpi=true notwithstanding). A 6% pixel-count difference cannot produce the ~2× frame-time swings seen above, so resolution is not the explanation for this session's inconsistency — that was the stale-process contention above. It's still worth a one-line fix later (0.17c owns display/stretch decisions) since 2036×1080 is a mildly wasteful, non-native render target.
What Mac hardware does plausibly bias is the shape of the result, not the run-to-run noise: Apple Silicon GPUs are tile-based deferred renderers (TBDR), architecturally unlike the immediate-mode AMD/Nvidia GPUs the target "reference hardware" (a Windows/Linux gaming PC) uses. TBDR keeps a frame in on-chip tile memory and is comparatively cheap at MSAA resolve, but any pass needing to read arbitrary neighbouring pixels across the whole frame — SSAO, SSIL, the glow downsample/upsample chain, the PostFX shader's screen_texture read — forces a break out of tile memory into a full system-memory resolve, an overhead that is largely constant per pass rather than proportional to what the pass computes. That lines up with pass 1's finding that SDFGI/SSIL/SSAO/MSAA/FXAA/shadows/PostFX all cost within a tight 2.9–3.8 ms band regardless of what each one actually does — consistent with a shared TBDR resolve tax dominating over each effect's real cost. Numbers measured on this machine should be treated as informative about relative ordering at best, not as a stand-in for target-platform (desktop GPU) behaviour — confirmed below.
5.5.2 Measured on real reference hardware — RTX 3090, Linux (2026-08-19)
Same 6-ship 3v3 Match, 1080p, via a purpose-built harness (Game/tools/gpu_profile_harness.gd) run directly against a real GPU-bound X session (not Xvfb — an earlier attempt through Xvfb silently fell back to Mesa's llvmpipe software rasterizer, ~35x slower and completely unrepresentative; caught via the harness's own adapter-name check, not assumed). This is the number that matters — an actual discrete immediate-mode GPU, the architecture players will actually have:
| p50 | p99 | fps (p50) | |
|---|---|---|---|
| All effects on (project defaults) | 1.85 ms | 2.98 ms | 540 |
| All effects off | 0.53 ms | 1.53 ms | 1883 |
This overturns §5.5.1's conclusion, not just its numbers. On real hardware, disabling every effect gives a 3.5× speedup — the opposite of the Mac's ~1.03× — and the per-effect breakdown finally makes physical sense instead of clustering suspiciously:
| Setting off | Frame time | Implied cost |
|---|---|---|
| (baseline, all on) | 1.85 ms | — |
| SDFGI | 1.49 ms | 0.36 ms |
| SSIL | 1.60 ms | 0.25 ms |
| Glow | 1.75 ms | 0.10 ms |
| Shadows (all 5 casters) | 1.76 ms | 0.09 ms |
| SSAO | 1.82 ms | 0.03 ms |
| MSAA 4×, FXAA, PostFX | 1.87–2.12 ms | noise-level (see below) |
SDFGI and SSIL alone account for over half of the effects' total cost, matching §5.4's original expectation (voxel cone tracing and a full-res screen-space GI pass being the expensive ones) — the Mac's flat, undifferentiated cost profile was the anomaly, not this one. MSAA/FXAA/PostFX show negative "costs" (disabling FXAA measured as slightly slower than leaving it on) — at ~1-2 ms absolute frame times, OS scheduling jitter is larger than the real signal for cheap passes; those three need a longer sampling window or a proper GPU profiler to resolve, not this harness's coarse get_process_delta_time() sampling. Note also that all-off (0.53 ms) is faster than baseline-minus-sum-of-individual-savings (1.85 − 0.36 − 0.25 − 0.10 − 0.09 − 0.03 ≈ 1.02 ms) — the combined removal saves more than the parts, consistent with each full-screen pass carrying some fixed per-pass overhead (pipeline barriers, render-target switches) on top of its own work, which compounds when several stack.
Consequence for 0.17/0.26/0.28, revised: at 540 fps p50 with every effect enabled, this scene is nowhere near GPU-bound on reference-class hardware — the entire "must hit 144 fps" framing in §5.4/§5.5 was solving a problem that doesn't exist on the hardware tier it was written for. That reframes the two gated tasks rather than clearing them outright:
- 0.26 (bake GI, retire SDFGI) — the relative win is real and correctly targeted (SDFGI is the single largest line item, ~19% of the effects-on budget), and the preset design already bets on this being right (Low/Medium turn SDFGI+SSIL off first, matching exactly what this data says to cut). But "largest frame-time reduction of any task here" (its acceptance bar) oversells it on a 3090 — 0.36 ms off an already-tiny budget is not the headline win §5.7 implied. The task is worth doing for lower-end/integrated GPUs, where the same relative cost almost certainly scales to something that matters — but that's now the open question, unmeasured on this pass.
- 0.28 (physics/3d/run_on_separate_thread) — its whole motivation is smoothing frame-time variance caused by the physics tick sharing the render thread; at a 1.85 ms p50 / 2.98 ms p99 baseline (both far under even a 240 Hz frame budget), there's no variance problem to fix on this hardware. Deprioritize below 0.26 unless a lower-end-hardware pass shows otherwise.
- The preset ladder itself (task 0.17, done) needs no changes — its bundle choices (drop SDFGI/SSIL first) are now empirically justified rather than just plausible-sounding.
Still open: no low/mid-tier GPU has been profiled. The 3090 result rules out "the game is GPU-bound on reasonable hardware" as a near-term concern, but says nothing about a GTX 1660 or an integrated Iris/Vega part, which is where a real preset ladder earns its keep. Re-run gpu_profile_harness.tscn on weaker hardware before spending more effort on 0.26/0.28.
5.6 Closing the gap to the reference — without lowering settings
§5.2 lands at ≈174 ms against a ~90–110 ms reference band. The instinct is that reaching it means trading visual quality for frames. It does not. Decompose the 174:
At 60 ms RTT, 60 ms is transit and irreducible in code. That leaves 114 ms of local overhead, of which frame rate governs only two terms — input freshness (10) and present (25) — and quality settings govern neither directly. Present latency is a function of vsync mode and swapchain depth, not of how many effects are enabled; a 60 fps client with a shallow present queue beats a 240 fps client with a deep one. The entire 60 → 240 fps range is worth ~12 ms once a low-latency present is in place (§5.4). The other ~100 ms is netcode time model and simulation rate.
Four levers, none of which touches a graphics setting:
| Lever | Saves | Risk | |
|---|---|---|---|
| L1 | Extrapolate remote visuals to present time instead of interpolating the past | −30 | Mis-prediction pops |
| L2 | 120 Hz simulation | −21 | Bot retrain, ½ server density, 2× bandwidth |
| L3 | Adaptive jitter-buffer depth, 0 on clean links | −8 | Starvation on jittery links |
| L4 | Shallow present queue + Adaptive vsync | −17 | Throughput loss if GPU-bound |
L1 is the big one, and it is nearly free
§4.1 already computes remote entities' present-time state — that was the fatal correction that put the collider at server_time_est. $Visual is then deliberately rendered ~68 ms in the past for smoothness. Render it at present time too and the whole 37.5 ms interpolation buffer disappears, leaving only a residual for error smoothing.
The reason this is safe here is that ships have bounded acceleration and the hull is large. Extrapolating with known velocity, error is ½·a·t² over the full 68 ms horizon:
| max accel | error @ 38 ms | error @ 68 ms | |
|---|---|---|---|
| position, cruise | 30 m/s² (thrust_power 150 / mass 5) |
0.022 m | 0.069 m |
| position, turbo | 75 m/s² (turbo_multiplier 2.5) |
0.054 m | 0.173 m |
| yaw | 20 rad/s² (rotation_power 20 / inertia.y 1) |
0.8° | 2.6° |
| pitch / roll | 2.9 rad/s² (inertia.x/z 7) |
0.1° | 0.4° |
0.17 m and 2.6° worst case, against a 4 m hull. That is well under the width of the ship and an order of magnitude smaller than the 3.5 m staleness §4.1 was written to eliminate. Feed the residual through the same soft-correct pipeline already specified for the local ship (§4.4) and remote ships are visually at present time with a sub-decimetre wobble.
Two bonuses: it collapses §4.1's dual clock back into one — collider and visual both at server_time_est, so §5.4b's _process/_physics_process split and the two-regimes-for-one-node-name hazard both go away — and it applies to the ball, which is near-ballistic between contacts and therefore extrapolates better than ships do.
The cost is real but narrow: a remote ship that reverses input at the moment you sample it mispredicts by the numbers above and then visibly corrects. Interpolation never mispredicts; it is just always late. This is the genuine trade, and it is the one the reference class makes.
The reachable budget
| Term | today | L1 + L4 (v1) | + L2 + L3 | at 144 fps |
|---|---|---|---|---|
| input freshness | 10 | 10 | 10 | 5.5 |
| wait for next tick | 8.3 | 8.3 | 4.2 | 4.2 |
| client → server | 30 | 30 | 30 | 30 |
| jitter buffer | 16.7 | 16.7 | 4.2 | 4.2 |
| server tick + flush | 8 | 8 | 4 | 4 |
| server → client | 30 | 30 | 30 | 30 |
| interp buffer → extrapolation residual | 37.5 | 8 | 8 | 8 |
| client physics interpolation | 8.3 | 8.3 | 4.2 | 4.2 |
| present | 25 | 8.3 | 8.3 | 3.5 |
| World response | ≈174 | ≈127 | ≈103 | ≈94 |
≈103 ms at 60 fps with every effect enabled, and ≈94 at 144 fps. That is inside the reference band, reached without disabling SDFGI, SSIL, SSAO or shadows. Even a client struggling at 30 fps on maximum settings lands near ≈120 ms.
Sequencing follows ms-per-unit-of-risk: L4 then L1 for v1 (≈127 ms, no bot retrain, no protocol change); L2 and L3 after, when a retrain is affordable. §5.5's preset system remains worth building — but for frame rate and thermals, which is what it actually buys, not for latency.
The largest lever is not on this list. All of the above assumes 60 ms RTT. Regional server siting that puts most players on a 30 ms RTT takes ≈127 to ≈97 and ≈103 to ≈73 with no code at all. Phase 6 owns it, and it should be argued against these numbers.
Perspective on where this matters. Own ship and ball are already at ≈52 ms and are unaffected by every lever here — they are predicted locally. World response governs opponent ships. In a game whose subject is a ball, that ordering is favourable: the two objects a player tracks most closely are the two already at single-digit-tick latency.
5.7 The next tier — and where it stops paying
§5.5 and §5.6 are the first-order work. This section is what remains after them, and it is deliberately honest about the point where further effort stops being worth it.
Frame rate: SDFGI is the wrong tool for this arena
The single largest available win, and it costs no visual quality. arena.gd and goal.gd have no _process, no _physics_process, no AnimationPlayer and no Tween — the floor, walls, ceiling, goals and every light are static for the entire match. The only things that move are 6 ships and a ball, all small and all self-lit.
SDFGI exists to light dynamic worlds, and it pays for that by re-voxelising cascades as the camera moves — and this camera never stops moving (ship_camera.gd:126, 137, 156). It is the most expensive thing in the frame, doing continuous work to solve a problem this project does not have.
- Replace
sdfgi_enabledwith baked GI —LightmapGIfor the static shell, orVoxelGIif bounce onto moving ships matters. Bake cost is offline; runtime cost is a texture fetch. The look is preserved or improved (baked bounce is higher quality than SDFGI's cascades), and it survives on the High preset rather than being the first thing a preset has to switch off. ssil_enabledbecomes largely redundant once bounce is baked. It is a full-resolution screen-space pass duplicating information the lightmap already has.
This is the answer to "lowest lag and highest fps without lowering settings": the expensive setting was solving the wrong problem.
Frame rate: expensive defaults that project.godot never overrides
[rendering] contains exactly three keys (msaa_3d, screen_space_aa, use_debanding). Everything else runs at engine defaults, including:
| Setting | Default | Note |
|---|---|---|
lights_and_shadows/positional_shadow/atlas_size |
4096 | Shared by all shadowed positional lights; 2048 is usually indistinguishable here |
lights_and_shadows/directional_shadow/size |
4096 | |
lights_and_shadows/directional_shadow/soft_shadow_filter_quality |
high | |
occlusion_culling/use_occlusion_culling |
off | Low value in an enclosed arena — measure before adding bake time |
mesh_lod/lod_change/threshold |
— | Irrelevant: the scene is ~1450 triangles of arena plus low-poly ships (§5.5) |
Also worth counting: _build_movement_vfx creates two OmniLight3Ds per ship (ship.gd:270-278), so a 3v3 has 12 dynamic lights on top of the arena's 5. They are correctly shadow_enabled = false and omni_range = 3.5, so they are cheap — noted so nobody "discovers" them and disables engine glow for nothing.
Frame rate: the CPU side, which §5.5 does not cover
§5.5 establishes the project is GPU-bound on full-screen passes. Once those are fixed it becomes CPU-bound, and §5.4's frame-time variance becomes the ceiling. Three levers:
physics/3d/run_on_separate_thread(not set; defaults off). This decouples the physics step from the render thread and directly attacks "one frame in four carries the whole tick." It is the highest-leverage item here and the riskiest — it changes when_integrate_forcesruns relative to script code, and this project puts real logic there (ship.gd:346-357) plus an RL training path. Prototype and measure; do not enable on faith.ArenaBoundary.get_surface_pullhas no early-out. It runs ato_local()plus five_falloffcalls for every dynamic body every tick, including for a ball sitting in the middle of the arena where every term is zero. A single bounds check againstwall_range/ceiling_rangeskips almost all of it in open play — 7 bodies × 120 Hz once L2 lands.- Bot inference is ~6.5k GDScript multiply-accumulates per bot (
policy_network.gd). Task 0.8 staggers them; beyond that, the lever is network width, which is a training decision, not a rendering one.
Latency: what is actually left
After L1–L4 and 120 Hz simulation, at 144 fps, the budget is ≈94 ms — and 60 of that is RTT. The remaining 34 ms of local overhead breaks down as input freshness 5.5, tick wait 4.2, jitter 4.2, server 4, extrapolation residual 8, physics interpolation 4.2, present 3.5. Every one of those is at or near a floor set by physics rate or hardware.
Two code ideas remain, both small and both with a cost:
- Forward-extrapolate the local
$Visualinstead of interpolating between the last two ticks — render the predicted ship at present time rather than up to one tick behind. Worth ~4 ms. Risk: overshoot at the moment of a collision, which is the most visually sensitive moment in the game. - Tighten the extrapolation-error smoothing (§5.6's 8 ms residual). Worth ~4 ms, paid for in more visible correction pops.
That is the whole remaining code budget: ~8 ms, both items trading visual stability for it. Meanwhile:
- Regional server siting takes a 60 ms RTT to 30 for most players: −30 ms, four times the remaining code budget, no code at all.
- Ping-weighted matchmaking and a server browser sorted by measured ping convert that into something players actually experience rather than something that is true on average.
- Steam Datagram Relay (Phase 7) is planned for NAT traversal and DDoS protection, but Valve's backbone frequently routes better than raw BGP paths — for some player pairs SDR is a latency reduction, not a tax. Measure it both ways rather than assuming it costs.
Where this stops paying
Two limits worth writing down before someone spends a month on the last 5 ms:
- Past ~100 ms, you are optimising 3–4 ms at a time against a 60 ms constant. The ratio of engineering effort to felt improvement collapses. Server siting and matchmaking dominate everything else from that point on.
- "Lowest lag" and "best feel" diverge at the end. Both remaining code levers, and L1 itself, buy milliseconds by predicting further ahead and correcting harder. Past a point that makes the game feel worse — twitchier, less stable, more prone to visible snapping — while the latency number keeps improving. The number is a proxy, not the goal. Task 4.7's tuning pass, with a human in the seat, is the authority; the budget table is not.
6. Match lifecycle
6.1 State machine
LOBBY -> LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP -> ...
-> FULL_TIME -> OVERTIME_WARMUP -> OVERTIME -> ...
-> RESULTS -> LOBBY
Broadcast as the match_state byte in every snapshot, and on transition via state_change(state, at_tick).
6.2 Sequence
- Connect. Client sends
hello(protocol_version, physics_ticks_per_second, display_name, auth_ticket). Server rejects a mismatch on either version or tick rate, with a reason string, thendisconnect_peer. (A client at 30 Hz advances its sequence numbers at half rate and confuses every control loop.)auth_ticketis an emptyPackedByteArrayuntil Phase 7 — reserve the field now. - Welcome. Server assigns
player_id, balances teams, replieswelcome(player_id, server_info, roster, match_state, server_tick, score, end_tick), broadcastsplayer_joined. - Lobby.
ready_toggle(); start when all ready, or--auto-startafter--min-playersplus a countdown. - Config.
match_config(match_id, arena_path, team_size, match_length_ticks, roster[], seed).roster[i] = {slot, team, spawn_index, player_id, name, is_bot}— slot order here is the snapshot's body order for the whole match. The client validatesarena_pathagainstArenaRegistry.ARENASbeforeload(); a malicious or buggy server must not be able to make a client load an arbitraryres://path. - Load. Both sides load
networked_match.tscn. Each peer loads the arena and spawns the roster in slot order. Client additionally spawns a camera rig on its own ship and addsHUD.tscnin code —networked_match.tscnmust have no HUD child, becauseGameMode._ready()(game_mode.gd:44-45) would pick it up server-side. Client sendsscene_ready(match_id). - Kickoff. Server waits for all
scene_ready(10 s timeout → proceed). Broadcastskickoff(reset_transforms[], countdown_start_tick, reset_gen). Both sides freeze bodies. HUD counts down fromserver_tick, not a localTimer. Atcountdown_start_tick + 180the server unfreezes and broadcastsstate_change(PLAYING). - Play. Inputs up, snapshots down.
- Goal. Server's
Goalsensor fires →_handle_goal_scoreddebounce →goal_scored(scoring_team, score, goal_tick, resume_tick). Bodies freeze. Clients play the cinematic within[goal_tick, resume_tick]. Atresume_tick:kickoff(...). - Clock. Tick-derived:
remaining_ticks = end_tick - current_server_tick.end_tickand arunningflag ship inmatch_configand inclock_state(running, end_tick, at_tick). - Full time / overtime / results.
RESULTSholds, thenstate_change(LOBBY)and both sides loadlobby.tscn. Clients return to the lobby, not the main menu — a community server that empties every 2.5 minutes is dead on arrival.
Every lifecycle message carries absolute ticks, never durations. That is what makes reliable-channel latency harmless: on a lossy link ENet's RTO can stretch a goal_scored → kickoff → state_change burst to ~600 ms. Specify the late-arrival case explicitly: a kickoff that lands after its own resume_tick must apply the reset immediately and skip the countdown, not schedule it into the past.
NetworkedMatch must declare all five signals HUDController duck-types on (HUDController.gd:65, 88, 100, 103, 106) — timer_updated, score_changed, match_ended, kickoff_countdown, overtime_started — and emit them from RPC handlers instead of from local logic. Otherwise the HUD silently omits rows.
6.3 Late joiners and spectators
welcome carries full state, so a late joiner reconstructs immediately.
- Free slot and state is
LOBBY/WARMUP→ join as a player now. - Free slot mid-match → spectate now, take the slot at the next kickoff. Swapping a controller at a kickoff boundary is free; mid-play it is not.
- No free slot → spectator. A spectator receives identical snapshots (the snapshot is already a broadcast — zero extra server work), spawns no ship, and points a camera rig at a chosen ship or the ball. Cap with
--max-spectators.
HUDController._initialize_hud() push_errors and bails when ship is null (HUDController.gd:41-46). Spectators need a path through that.
6.4 Disconnects — no ship is ever despawned
On peer_disconnected the server keeps the ship and swaps its controller:
--fill-bots: replace with anAIShipControlleron the server's configured model.--no-fill-bots(default for public servers, see §1.4): swap to the baseShipController— inert but simulated, exactly the placeholdergame_mode.gd:216already uses.
Set flags.stalled so clients can grey out the nameplate. Reserve the slot for 30 s keyed by identity so a reconnect gets its ship back. If the last human leaves, abort to LOBBY.
Justification is wire-format simplicity, not the bot cache. Fixed slot order means the snapshot needs no add/remove machinery, no MultiplayerSpawner, and no re-indexing. That reason stands on its own.
ai_ship_controller.gd currently caches teammate/opponent lists once with the comment "rosters never change mid-match (no despawn path exists anywhere in this codebase)". Do not let that be the justification — protecting a bot's implementation detail is tail-wagging-dog, and taken as an architectural constraint it permanently forecloses 3v3→2v2 shrink, mid-match rebalancing, and join-onto-a-new-slot. Fix the cache anyway (task 0.9): filter(is_instance_valid) plus a roster_changed signal, ~5 lines of cheap insurance.
7. Phase and task breakdown
[P] parallelisable within its phase · [D:x.y] hard dependency
Phase 0 — Non-networked refactors
Every task lands on master independently, is verifiable in single-player today, and cannot break anything. Near-total parallelism.
| # | Task | Files | Acceptance |
|---|---|---|---|
0.1 [P] |
DONE. Added ShipAction.copy(). Audit: the only get_action() call site (ship.gd:347) reassigns _current_action fresh each tick rather than buffering it, so no aliasing bug exists yet — copy() is a no-op today, ready for Phase 4's prediction ring |
ship_action.gd, player_ship_controller.gd |
Free Play unchanged; copy() returns a distinct object with equal fields |
0.2 [P] |
DONE. Inserted Visual (Node3D) into ship.tscn, reparented Nose/TailFin under it, redirected all four code-driven add_child calls onto $Visual (now a public @onready var visual), resolved _apply_team_color's lookup to "Visual/" + mesh_name |
objects/ship.tscn, scripts/ship.gd |
Child-type assertion holds; ship looks identical in Free Play; team colours still apply on both teams |
0.3 [D:0.2] |
DONE. ship_camera.gd's three target.global_transform reads (ball cam, ship cam ×2) now read target.visual.global_transform |
scripts/ship_camera.gd |
Camera behaviour unchanged in Free Play and Match — visual has identity transform relative to the body until Phase 4 writes an offset, so this is a no-op today |
0.4 [P] |
DONE. can_sleep = false on Ship and Ball |
objects/ship.tscn, objects/ball.tscn |
No behaviour change |
0.5 [P] |
DONE. continuous_cd = true on Ship (Ball already had it) |
objects/ship.tscn |
No tunnelling at max speed into the ball or walls |
0.6 [P] |
DONE. Spawned ships renamed to Ship_T%d_S%d |
game_mode.gd |
Names are (team, spawn_index)-derived, not insertion-order |
0.7 [P] |
DONE. _jittered now uses an owned RandomNumberGenerator, self-randomized in _ready() unless kickoff_rng_seed is set explicitly (a fresh RandomNumberGenerator defaults to a fixed internal state, unlike the global randf_range Godot auto-randomizes at startup — call this out for whoever reads the diff and expects .new() alone to be enough) |
game_mode.gd |
Kickoff jitter unchanged in feel; a fixed seed reproduces kickoffs exactly |
0.8 [P] |
DONE. _ticks_until_decision = randi_range(1, reaction_ticks) at spawn, after load_policy() (which still resets to 0 on later calls, e.g. league opponent swaps — harmless, those land at reset boundaries) |
ai_ship_controller.gd |
Six-bot Spectate shows no periodic frame spike |
0.9 [P] |
DONE. Roster validity checked (Array.any()) once per decision tick, not every physics tick; filter(is_instance_valid) + roster_changed signal only fire on an actual stale reference |
ai_ship_controller.gd |
Bots behave identically; freeing a ship mid-match no longer corrupts observations |
0.10 [D:0.12] [P] |
_owns_goal_logic(), _allows_time_scale_effects(), _goal_pause_seconds(), _owns_world_simulation()_allows_time_scale_effects() dropped: 0.12 deletes Engine.time_scale from the file entirely, so there is nothing left for it to gate. Implemented _owns_goal_logic(), _owns_world_simulation(), _goal_pause_seconds(), all behaviour-preserving (default true/GOAL_CELEBRATION_SECONDS), gating the goal-signal connection and _respawn_escaped_bodies() |
game_mode.gd |
Free Play, Match, Spectate and Training all behave identically — verified no other virtual was load-bearing today; these exist for a future networked-client mode |
0.11 [P] |
DONE. _handle_goal_scored checks is_inside_tree() after each await and bails before touching arena/hud state |
game_mode.gd |
A scene change mid-celebration cannot strand the flag |
0.12 [P] |
Engine.time_scale hit-stop and goal slow-mo with camera-only effectsShipCameraRig's "Impact Punch" group (punch_fov_kick/punch_vignette_kick/punch_chroma_kick/punch_decay, applied additively after _update_speed_feel each tick, decaying via move_toward over real delta) triggered from the existing _on_target_ball_contact; goal moments now rely on the pre-existing begin_goal_cut/end_goal_cut cinematic cut alone, no separate slow-mo effect needed. All Engine.time_scale fields/methods deleted from game_mode.gd (_hit_stop_*, _goal_slowmo_active, _restore_hit_stop, _run_hit_stop, GOAL_SLOWMO_SCALE) |
game_mode.gd, ship_camera.gd |
Goal and impact feel is at least as good; Engine.time_scale is never written — confirmed via grep -rn time_scale scripts/ |
0.13 [P] |
DONE. physics_jitter_fix = 0.0 set. CLAUDE.md's architecture section had stale prose dimensions ("inner x ±12, z ±18, height 12, goal lines z ±17") — corrected to reference the actual named constants (INNER_HALF_X 18, INNER_HALF_Z 27, INNER_HEIGHT 18, GOAL_LINE_Z = INNER_HALF_Z) instead of restating numbers that can drift out of sync again |
project.godot, CLAUDE.md |
Flight feel unchanged; CLAUDE.md matches arena_boundary.gd:8-14 |
0.14 [D:0.2] |
DONE. Added Ship.set_visual_action(thrust_z, turbo), Ball.set_visual_speed(speed) (with a _visual_speed_override field the trail prefers when ≥0), and Ship.net_vel_correction/net_visual_offset fields plus the guarded hook at the top of _integrate_forces (decays net_visual_offset via _tick_scaled, writes it to visual.position) |
ship.gd, ball.gd |
No-op until Phase 4; single-player unchanged — nothing calls any of these yet |
0.15 [P] |
DONE. Ship/Ball gained queue_teleport(to); _integrate_forces applies it via state.transform + zeroed velocities + reset_physics_interpolation(). GameMode._reset_body now calls body.call("queue_teleport", to) (dynamic dispatch — RigidBody3D itself has no such method) instead of set_deferred |
game_mode.gd, ship.gd, ball.gd |
Kickoff resets in Match are visually identical, with no interpolation smear |
| 0.15b | DONE, superseded by §5.5.2 — read that, not the Mac numbers below. First pass measured a live 6-ship Match, 1080p, on an Apple M4 dev laptop (§5.5.1): all-on p50 17.93 ms, all-off floor ~17.2 ms, with per-effect costs clustered suspiciously flat (2.9–3.8 ms each). That data turned out to be a poor stand-in for the target platform — Apple's tile-based GPU architecture, not a real bottleneck — and was superseded by a same-scenario re-run on real reference hardware (RTX 3090, §5.5.2): all-on p50 1.85 ms / all-off 0.53 ms, SDFGI+SSIL clearly dominant as originally expected, everything else cheap. Keep §5.5.1 for the record of what was tried and why it was distrusted, not as a performance reference | scenes/arena_base.tscn, shaders/post_process.gdshader, Game/tools/gpu_profile_harness.gd |
Measured max frame rate written into §5.5.2 from real reference hardware. At 540 fps p50 with everything on, this scene is nowhere near GPU-bound on a 3090-class GPU — the a priori §5.4 fps list was solving for a constraint that doesn't hold at that hardware tier. 0.17 (done) needed no changes: its preset bundle choices are now empirically validated. 0.26 stays open (real but smaller win than assumed); 0.28 closed (no variance problem exists to fix) |
0.16 [D:0.3] |
DONE. Camera rig moved _physics_process → _process; reads target.visual.get_global_transform_interpolated() in both ball-cam and ship-cam; rig itself has physics_interpolation_mode = OFF (it writes its own transform every rendered frame now, so Godot's built-in interpolation would just fight the manual smoothing). target setter primes interpolation (target.visual.reset_physics_interpolation()) and calls the new snap_to_target() so a freshly-assigned target (or a Spectate switch) doesn't lerp in from wherever the rig was previously. Shake re-derivation, implemented differently than drafted: rather than rescale frequency, _apply_shake now quantizes the noise-domain input to whole 60Hz ticks (floori(_shake_time * SHAKE_UPDATE_HZ)) — every render frame within one 1/60s window reuses the identical noise sample, so consecutive distinct samples stay exactly frequency (2.5) domain-units apart at any render frame rate, reproducing 60fps's original jitter character everywhere instead of smoothing out at high fps. snap_to_target() is called from game_mode.gd's reset_ships(), not directly from ship_camera.gd's own kickoff-adjacent code — reset_ships() is now async and awaits one get_tree().physics_frame before snapping, because _reset_body's queue_teleport (task 0.15) defers the actual transform write to the ship's next _integrate_forces; snapping immediately would read the pre-teleport position. Goal-cut shake decay extracted into _decay_shake(), called from the _goal_cut_active branch. Validated: scripts compile, Free Play renders correctly non-headless, reset produces no camera jump, all three headless scenes exit clean |
scripts/ship_camera.gd, scripts/game_mode.gd:reset_ships |
Turbo FOV kick and post-process are smooth at an uncapped frame rate; shake reads the same at 60 and 240 fps; a kickoff cuts the camera rather than lerping it across the arena |
0.17 [D:0.15b] |
DONE. VideoSettings gains Preset (Low/Medium/High/Custom) driving a bundle (sdfgi_enabled, ssil_enabled, ssao_enabled, shadows_enabled, glow_enabled, aa_mode, resolution_scale) via apply_preset(); a settings_changed signal lets an already-loaded arena re-apply live (arena.gd connects in _ready()) rather than only affecting the next arena load — meets "settings persist and apply without a restart" without needing a scene reload. Shadow gating targets the actual Light3D nodes (found once at load via find_children, cached, re-applied on every settings change — deliberately not re-derived from current state each time, since a light this code just turned off would otherwise become indistinguishable from FillLight, which is authored shadow_enabled = false on purpose and must never be turned on by the preset ladder). vsync_mode (Disabled/Enabled/Adaptive, Adaptive default) and fps_cap_divisor (0 = uncapped, else divides the live refresh rate at apply time rather than storing a raw fps number, so the same preference re-derives correctly on a different display) added to the settings menu; FPS cap dropdown is disabled (greyed) unless VSync is Disabled; refresh-rate query ≤0 falls back to "Uncapped" only. Live fps readout via _process reading Performance.TIME_FPS. main_menu.gd's _leave_to_gameplay now calls VideoSettings.apply_fps_cap() instead of hardcoding Engine.max_fps = 0, so the player's cap actually reaches gameplay scenes. Acceptance numbers: not run as a literal Low-vs-High preset A/B, but strongly implied by §5.5.2 — real hardware (RTX 3090) runs the High-equivalent (all effects on) at 540 fps p50 already, so Low (which additionally turns off the two dominant costs, SDFGI+SSIL) clearing "≥2×" is close to guaranteed rather than measured directly; the flat-p99-histogram claim genuinely wasn't tested (gpu_profile_harness.gd measures per-toggle cost, not vsync/cap histograms) |
scripts/video_settings.gd, scripts/settings_menu.gd, scenes/settings.tscn, scripts/arena.gd, scripts/main_menu.gd |
Low preset ≥2× the frame rate of High on the same hardware; settings persist and apply without a restart; every offered cap gives a flat frame-time histogram (p99−p50 < 1 ms) with VSync disabled on a 144 Hz and a 165 Hz display; refresh-rate query returning -1 falls back cleanly |
0.17b [D:0.15b] [P] |
DONE. VideoSettings.resolution_scale (0.5–1.0, default 1.0) drives Viewport.scaling_3d_mode/scaling_3d_scale/fsr_sharpness via apply_resolution_scale() — SCALING_3D_MODE_FSR2 below 1.0 (chosen over bilinear: this project already gave up native resolution at the fixed-1080p blit per 0.17c, so FSR2's sharpening recovers more of that loss than a plain bilinear upscale at the same internal scale), SCALING_3D_MODE_BILINEAR with scale pinned to 1.0 at the top of the range (a no-op scaling mode when the scale is 1:1). Low preset defaults to 0.8. Exposed as a slider in the settings menu; not yet measured against the "0.7 scale gives a large, measurable frame-time drop" bar — same real-hardware caveat as 0.17 |
scripts/video_settings.gd, settings_menu.gd |
0.7 scale gives a large, measurable frame-time drop with acceptable image quality; setting persists |
0.17c [D:0.17b] |
DONE — decided, not changed. Kept stretch/mode="viewport" fixed at 1080p rather than moving to "disabled", documented inline in project.godot [display] with rationale: 0.17b's scaling_3d_scale already covers "render lower than the window" independently of stretch mode (it scales the 3D viewport's internal resolution before this blit, not the window itself), and separately, task 0.15b found an unexplained ~6% non-uniform width scaling on the one machine this was tested on (2036×1080 measured against a 1920×1080 target — see §5.5.1) that needs understanding before stretch mode is touched, not blindly carried into a resolution-dependent change |
project.godot |
The decision and its rationale are written into §5.5; render resolution follows the player's setting |
0.17d [P] |
INVESTIGATED — no such lever exists in Godot 4.7. Searched the full project.godot schema (read_project_settings) for rendering/rendering_device/vsync/frame_queue_size and every variant (frame_queue, swapchain, present, present_queue) — none exist as a project-settable parameter in this engine version; the RenderingDevice backend may manage its own present queue internally but doesn't expose it. Adaptive vsync (task 0.17, done) is the only half of "L4" actually achievable through project settings. The §5.6 ~17 ms figure for a shallow present queue is therefore not obtainable as specced — closing this without a code change is correct here, not a shortfall; reaching it would need engine-level (C++/RenderingDevice) changes out of scope for a project-settings task |
||
0.18 [P] |
DONE, with one discovered GDScript constraint. New scripts/sim_constants.gd (class_name SimConstants, plain const TICK_HZ := 60, not an autoload) is the source of truth for ship.gd's _tick_scaled and training_mode.gd's TICKS_PER_SIM_SECOND — both reference it via const SimConstants = preload("res://scripts/sim_constants.gd") rather than the bare global class_name symbol, because a cross-script const X := f(OtherClass.CONST) initializer needs the reference resolved before the global class table is guaranteed populated. @export_range() upper bounds cannot take even a preloaded reference — export hint arguments must be true literals — so reaction_ticks/bot_*_reaction_ticks (ai_ship_controller.gd, match_mode.gd, spectate_mode.gd ×2) stay at a literal 60; these are editor-inspector slider bounds, not the timing math itself, so this doesn't reopen the bug the task exists to close, but it means the acceptance criterion below is met for tick-rate math and not for export-hint bounds |
ship.gd, training_mode.gd, new scripts/sim_constants.gd |
Tick-rate-derived timing math has no bare 60; changing TICK_HZ changes _tick_scaled and TICKS_PER_SIM_SECOND coherently. reaction_ticks export bounds remain literal by GDScript necessity |
0.19 [P] |
DONE. AAMode gained MSAA_2X, appended (not inserted) so existing user://settings.cfg ordinals keep their meaning; default aa_mode changed to FXAA; settings_menu.gd's AA_OPTIONS now lists five entries |
video_settings.gd, settings_menu.gd |
Five AA options; default is FXAA; existing saved preferences migrate without resetting |
0.20 [P] |
DONE. New autoload scripts/perf_overlay.gd (PerfOverlay), toggled by a new toggle_perf_overlay input action (F3 default). Headless-guarded; builds its own Label in code rather than touching HUD.tscn |
new scripts/perf_overlay.gd, project.godot [input] |
TIME_PROCESS vs total frame time tells the player whether they are CPU- or GPU-bound |
0.21 [P] |
DONE. Shared HudInstrument._throttled_redraw(delta) paces queue_redraw() to ~60/s; value smoothing itself still runs every _process call, only the repaint is throttled |
scripts/hud_instrument.gd, scripts/hud_gauge.gd, scripts/hud_attitude_indicator.gd, scripts/hud_heading_tape.gd |
HUD is visually identical; instrument _draw call count is capped at ~60/s regardless of frame rate |
0.22 [P] |
DONE. Engine.max_physics_steps_per_frame = 4 set in GameMode._ready(), applies to every mode including headless Training |
scripts/game_mode.gd |
A client throttled to 20 fps degrades smoothly instead of compounding |
0.23 [P] |
DONE. New autoload scripts/background_fps.gd (BackgroundFPS) drops to 30 fps on NOTIFICATION_APPLICATION_FOCUS_OUT / restores on focus-in, independent of scene. main_menu.gd/settings_menu.gd each cap to DisplayServer.screen_get_refresh_rate() in _ready() (falling back to uncapped on a -1 query); leaving the main menu for a gameplay scene uncaps again via a new _leave_to_gameplay() helper, since gameplay has no cap of its own yet (0.17) |
new scripts/background_fps.gd, main_menu.gd, settings_menu.gd |
An unfocused window and an idle menu both stop rendering at 900 fps |
0.24 [P] |
DONE. Both guarded with if DisplayServer.get_name() == "headless": return — arena.gd:_ready() skips the whole Environment block, video_settings.gd:_ready() skips apply_aa() |
scripts/arena.gd, scripts/video_settings.gd |
--headless allocates no Environment and no AA state |
0.25 [P] |
DONE. _process still calls to_local() every frame (needed for the comparison itself) but skips set_shader_parameter() — the actual GPU-facing cost — below a 0.05 m movement threshold |
scripts/arena_boundary.gd |
Field shader behaves identically; the expensive call is skipped on most frames |
0.26 [D:0.15b] |
Bake the arena GI and retire SDFGI (§5.7). arena.gd/goal.gd have no _process, no animation — the arena is fully static, and SDFGI is paying continuously to solve a dynamic-world problem this project does not have. Add UV2 to the arena shell, bake LightmapGI (or VoxelGI if bounce onto ships matters), disable sdfgi_enabled and re-evaluate ssil_enabled |
scenes/arena_base.tscn, scenes/arena_0*.tscn, scripts/arena_boundary.gd |
Largest frame-time reduction of any task here, with equal or better image quality; High preset keeps its look; bake is reproducible from a documented step |
0.27 [P] |
DONE. lights_and_shadows/positional_shadow/atlas_size and directional_shadow/size set to 2048 (from the 4096 engine default), soft_shadow_filter_quality=2 |
project.godot |
Measurable frame-time reduction; no visible shadow-quality regression at 1080p |
0.28 [D:0.15b] |
CLOSED, not implemented — the problem it targets doesn't exist. Was: prototype physics/3d/run_on_separate_thread (§5.7) to attack frame-time variance from the physics tick sharing the render thread — the riskiest item in this phase, since it changes when _integrate_forces runs relative to script code, and both ship.gd:346-357 and the RL training path depend on that. §5.5.2's real-hardware measurement (RTX 3090) found a 1.85 ms p50 / 2.98 ms p99 baseline with every graphics effect enabled — both comfortably under even a 240 Hz frame budget, with no meaningful p99-over-p50 variance to explain away. Taking on this task's real risk (reordering _integrate_forces relative to script code, with the RL training path depending on today's ordering) for a variance problem that isn't measurably present is a bad trade. Reopen only if a lower-end-hardware pass (§5.5.2's "still open" item) finds real physics-tick-driven variance that 0.26 and the preset ladder don't already cover |
— | (closed without a code change; see §5.5.2 for the evidence) |
0.29 [P] |
DONE. Bounds check against wall_range/ceiling_range at the top of get_surface_pull, returning Vector3.ZERO before to_local() and the five _falloff calls whenever every term would be zero mid-arena |
scripts/arena_boundary.gd |
Identical flight feel and identical RL observations; measurable tick-time reduction with 7 bodies |
These tasks exist because of the high-refresh-rate mandate, and their order matters. 0.15b blocked everything else, and did invalidate the a priori fps list — but not in the direction first assumed (see §5.5.1 vs §5.5.2): on the Mac the game looked GPU-bound and undifferentiated; on real reference hardware (RTX 3090, §5.5.2) it runs at 540 fps p50 with everything on, nowhere near bound by anything. 0.17/0.17b/0.19 (done) are still the right frame-rate levers — SDFGI/SSIL genuinely dominate the optional-effects cost, exactly as originally assumed, just at a much smaller absolute scale than feared on this hardware tier. 0.16 and 0.20–0.25 are the per-frame hygiene that makes a high frame rate worth having. 0.18 buys nothing today — it is what keeps a future 120 Hz simulation a config change plus a retrain rather than a protocol rewrite. 0.28 closed without a code change (§5.5.2) — the frame-time variance it targeted isn't measurably present on reference hardware.
0.19–0.29 are all pure single-player wins with no netcode content. If the multiplayer effort is ever paused, they should still land. Within them, 0.26 (bake the GI) is the largest single frame-time win in the document and costs no image quality — the arena is fully static, so SDFGI is paying continuously for a problem this project does not have (§5.7). 0.28 is the riskiest; it is the only Phase 0 task that can plausibly need reverting.
Correction — task 0.2 is wider than an earlier draft claimed. That draft argued the refactor was "narrow" because
_build_merged_hulland_build_movement_vfx"onlyadd_child()". That is exactly the problem: theyadd_child()ontoself, theRigidBody3D—ship.gd:208(MergedHull: Hull, Canopy, EngineGlowL/R),:241(engine cores),:268(flames),:278(lights). Leave those and §4.4's soft correct offsets onlyNoseandTailFinwhile the hull, canopy, glows, flames and lights stay welded to the corrected collider — every correction visibly tears the ship in half. The old acceptance criterion ("looks identical in Free Play") passes either way, which is why the criterion is now a child-type assertion.ship.gd:218's controlleradd_childcorrectly stays on the body;CollisionShape3Dstays on the body.Still true from that draft, and re-verified:
ship.gd:44-47documents whyNose/TailFinremain separateMeshInstance3Ds, and the RL path is untouched —ship_observations.gdreads onlyglobal_position, basis, velocities andPhysicsServer3Dcontacts, andtraining_mode.gd's onlyget_nodeisarena.get_node("Boundary").
Phase gate: the game plays identically to master in Free Play, Match, Spectate, and headless Training, with Engine.time_scale never written — and additionally: §5.5 contains a real measured frame-time table (0.15b), the Low preset roughly doubles the frame rate of High (0.17), and the game looks correct uncapped on a high-refresh display with no 60 Hz stepping in FOV, shake or post-process.
Phase 1 — Transport, connection, lobby
| # | Task | Acceptance |
|---|---|---|
| 1.0 | DONE, two real bugs found and fixed after adversarial review. tests/test_runner.tscn + test_runner.gd: discovers every *.gd under tests/cases/, instances it, calls every test_*() method via get_method_list(), aggregates failures, get_tree().quit(1 if failed else 0). tests/test_case.gd is the assertion base (assert_true/assert_eq/assert_almost_eq); case scripts use extends "res://tests/test_case.gd" (path-based) and the runner uses preload(), not a bare class_name reference — the global script-class cache isn't guaranteed populated on a fresh headless run (same class of issue as task 0.18's SimConstants). tests/cases/test_smoke.gd proves discovery/dispatch/aggregation and is the first real case file. An Opus subagent's adversarial review found: (1) GDScript has no exceptions, so a test that hit a runtime error before its first assert_* call left failures empty — exactly like every assertion passing — and was silently counted as a PASS. Fixed: TestCase now tracks assertions_made, incremented by every assert_*; the runner treats zero assertions as a failure in its own right ("made no assertions"). (2) A case file with a parse/compile error hung the whole runner forever — load() on a broken script does not return null here, it returns a non-null but uninstantiable GDScript resource, so a plain null check doesn't catch it; calling .new() on it threw an error severe enough to abort _ready() before ever reaching quit(). Fixed with Script.can_instantiate() as the real guard |
godot --headless --path Game res://tests/test_runner.tscn runs and exits 0; verified exit 1 with a deliberately-failing assertion, then removed. Re-verified both fixes with scratch case files (not committed): a test that null-derefs before asserting now correctly fails with "made no assertions" (exit 1, not a false pass); an uncompilable case file now fails loudly and promptly (exit 1, not a 124-timeout hang) while the other valid case files in the same run still execute normally |
1.1 [D:1.0] [D:0.18] |
DONE. scripts/net_codec.gd: protocol constants, PacketType enum, channel ids, i16/i8/thrust-z-bin quantisers, pack_input/unpack_input, pack_snapshot_body_segment/pack_snapshot_client_header/pack_snapshot/unpack_snapshot. New scripts/net_body_state.gd is the plain per-body data holder the snapshot functions read/write (not Ship/Ball themselves, so the codec stays callable with no scene tree). NetCodec.TICK_HZ derives from SimConstants.TICK_HZ via preload() (same cache-timing reason as 0.18); ring sizes / seq windows / INTERP_DELAY / timeouts don't exist as constants yet — they land with the tasks that consume them (3.1+), so "derives from TICK_HZ" is satisfied for what exists today |
scripts/net_codec.gd, scripts/net_body_state.gd, tests/cases/test_net_codec.gd |
1.2 [D:1.1] |
DONE, strengthened after adversarial review. scripts/network_manager.gd autoload (NetworkManager in project.godot [autoload]): host(port, max_clients)/join(address, port)/shutdown(), client_connected/client_disconnected/connected_to_server/connection_failed/disconnected_from_server signals forwarded from multiplayer's own, server_relay = false set the moment a peer exists, is_server/is_client state. Gained a shutting_down() signal, emitted at the top of every shutdown() regardless of role or reason — see task 1.4's row for why |
An Opus subagent's adversarial review (independently verified by the primary session before applying fixes) found the original tests/net_smoke.gd only proved each process exits cleanly on its own initiative, never that the OTHER peer actually observes the disconnect. Rewrote it: the host now waits for both client_connected and client_disconnected before passing; the client explicitly calls shutdown() mid-test (not just on process exit) and gives it a beat before quitting, same reasoning as §9 gotcha 26 for connects — a clean disconnect notice still needs a few poll() cycles to reach the wire, or the other side falls back to its ~5s peer timeout (gotcha 11) instead of a prompt one. Re-verified passing with both directions actually observed |
1.3 [D:1.2] |
DONE for what exists today. NetworkManager._ready() calls get_tree().set_multiplayer_poll_enabled(false) (Godot 4.7's actual method name — the doc's set_multiplayer_poll(false) was shorthand) and exposes NetworkManager.poll() as the one entry point every caller uses instead. Verified against tests/net_smoke.gd, updated to poll from both _process and _physics_process every frame — connect/disconnect still works cleanly under manual-only polling (§9 gotcha 26 still applies: give a beat after a connect signal before shutdown). The per-call-site placement this task specifies (client: end-of-physics-tick flush after input send, top-of-frame receive; server: tick-start drain, tick-end flush) has no real per-tick caller yet — there is no input/snapshot traffic until tasks 1.4+/Phase 2 exist to send any, so there's nothing to place a flush after. That placement, and the RTT/staleness measurement below, land with the input pipeline, not as a separate task |
godot --headless two-process test still connects/disconnects cleanly with automatic polling off (verified). RTT/staleness improvement not yet measured — deferred until Phase 2/3's real per-tick traffic exists to measure against, same honesty as task 1.1's "constants that don't fully exist yet" |
1.4 [D:1.2] |
DONE. scripts/match_net.gd autoload (MatchNet): _hello/_welcome/_player_joined/_player_left/_rejected RPCs, protocol_version (NetCodec.PROTOCOL_VERSION) and physics_ticks_per_second (SimConstants.TICK_HZ) checked on the server before a peer is added to roster; on mismatch, server sends _rejected with a readable string then disconnect_peer()s after a 0.3s beat (§9 gotcha 26 applies here too — a bare RPC then immediate disconnect would drop the rejection message). roster: Dictionary[int, PlayerInfo] never contains peer 1 (§1.1 decision 2). A new peer is told about the existing roster via targeted RPCs before the broadcast that tells everyone (including itself) about the new peer, so no client ever observes an unexplained peer_id |
Verified with a real two/three-process test (tests/match_net_smoke.gd/.tscn): matched client → both sides see player_joined/welcomed; deliberately wrong protocol version → client receives rejected("protocol version mismatch: server=1 client=100") and is disconnected. Caught and fixed one real bug in the process: the server's own roster update in _hello() didn't locally emit player_joined (the broadcast RPC is call_remote, never loops back to the sender) |
| — | Two more real bugs found by an Opus subagent's adversarial review, both confirmed independently and fixed. (1) _hello's player_name was completely unvalidated and broadcast verbatim to every peer — a demonstrated DoS: a multi-MB name relayed to all peers head-of-line-blocked the reliable control channel hard enough that a concurrently-joining client's own _welcome never arrived. Fixed with a hard MAX_INPUT_LENGTH = 256 reject (any legitimate client only ever sends local_player_name, which the UI already keeps short — anything past this is a bug or an attacker, not a name to politely truncate) followed by _sanitize_player_name(): strips control/formatting characters, clamps to MAX_PLAYER_NAME_LENGTH = 24, falls back to "Player" if empty. (2) MatchNet.roster was never cleared when a HOST stopped hosting — only the client-side disconnect path cleared it, so Host → Lobby → Leave → Host again left a phantom player in roster permanently, mis-balancing teams and getting broadcast to every future joiner. Fixed via NetworkManager's new shutting_down() signal (task 1.2), which MatchNet now clears roster on unconditionally, regardless of role or reason |
_sanitize_player_name is static (pure function of its argument) with 5 dedicated unit tests in tests/cases/test_match_net.gd, plus a live rejection test (match_net_smoke.gd --role=client-longname, a 500 KB name, confirmed rejected before ever reaching a broadcast). New regression test match_net_smoke.gd --role=host_recycle: host, client joins (roster.size()==1), host leaves and re-hosts, confirms roster.is_empty() before any new connection — reproduced the bug pre-fix, confirmed fixed post-fix |
1.5 [D:1.4] |
DONE, strengthened after adversarial review. scenes/lobby.tscn + scripts/lobby.gd: roster split into two team columns (dynamically rebuilt Label rows on MatchNet.player_joined/player_left/player_state_changed/welcomed), Switch Team + Ready CheckButton (server process gets a read-only view — never a roster member, §1.1 decision 2), Leave. MatchNet grew team/ready fields on PlayerInfo, a balanced-team auto-assign on join (_pick_balanced_team), and request_set_team/request_set_ready + their server-authoritative RPCs, broadcasting _state_changed the same way _player_joined already did |
Verified with a real two-process test (tests/lobby_smoke.gd/.tscn) that loads lobby.tscn via change_scene_to_file exactly as main_menu.gd's Host/Join flow (task 1.7) does, then presses the real %SwitchTeamButton/%ReadyButton nodes via a persistent test-only helper (tests/lobby_test_hooks.gd, not a project autoload — parented under get_tree().root so it survives the scene swap, never referenced by production code). An Opus subagent's adversarial review found the original test's host role never actually loaded lobby.tscn at all — it only hosted and waited, so lobby.gd's is_server branch (the read-only view a self-hosting player reaches via main_menu.gd's own Host button — a real, production-reachable path, not a hypothetical) had never run under this task's own suite. Fixed: the host role now loads lobby.tscn too and a new run_host_test() in the shared test helper verifies %ControlsRow is hidden, the roster row renders, and the status text is correct, holding the connection open long enough (MIN_HOST_LIFETIME_SECONDS) for the client's own longer flow to finish against it. Confirmed: roster renders correctly server- and client-side (now genuinely, not just asserted), team switch moves the row to the other column, ready toggle updates the checkbox and the label's ✓ marker, row count matches roster size on both peers |
1.6 [D:1.4] [P] |
DONE. scenes/server_boot.tscn + scripts/server_boot.gd: --port=/--max-clients=/--log-level= from OS.get_cmdline_user_args(), Engine.max_fps = 60, structured [elapsed] LEVEL event key=value… log lines for server_started/peer_connected/player_joined/player_left/peer_disconnected, and a physics-overrun watchdog comparing Engine.get_physics_frames() deltas frame-to-frame. Does not spawn a match yet — that's Phase 2's networked_match.gd — this is just the process shell: listen, log, idle cheaply. Two real bugs caught and fixed while verifying, both in the watchdog: (1) the very first _process() after boot compared against a pre-_ready() baseline and logged a spurious one-time steps=5; skip the first measurement. (2) the initial steps > 1 threshold fired continuously (every 30–100ms) on a perfectly idle, healthy server — because §9 gotcha 6 means frames legitimately alternate between 0 and 2 physics ticks under physics_jitter_fix = 0.0, not a flat 1/frame; that's quantisation, not backlog. Raised the threshold to steps > 2 (3+ ticks = the accumulator actually failing to drain), which produced zero false positives over a 4.8s idle run |
Verified with real headless runs: idle CPU measured via ps -o %cpu at 0.0% (bar is <5%); a real client connect/disconnect via tests/net_smoke.gd --port= produces exactly the expected 4-line log sequence with no spurious warnings |
1.7 [D:1.5] [P] |
DONE. main_menu.tscn gained a Multiplayer section (Host button; Join row with an IP LineEdit, default 127.0.0.1; inline error label) and a full-screen ConnectingOverlay (status label + Cancel). main_menu.gd: _on_host_pressed calls NetworkManager.host() then goes straight to lobby.tscn (synchronous — no overlay needed); _start_join calls NetworkManager.join(), shows the overlay, and starts an app-level CONNECT_TIMEOUT_SECONDS = 6.0 timer; _on_connected_to_server/_on_connection_failed/Cancel/timeout each resolve to the overlay hiding and either lobby.tscn or a visible error, gated by a token counter so a late/stray signal after the attempt was already resolved is a no-op |
Verified with real multi-process runs of scenes/main_menu.tscn itself (not a wrapper — driven by a temporary-autoload test helper, tests/main_menu_test_hooks.gd, pressing the real HostButton/JoinButton/ConnectingCancelButton) across all four paths: Host → lobby.tscn; Join → connects → lobby.tscn; Join with nothing listening → times out → error shown, stays on menu; Join → Cancel → overlay hidden, stays on menu, is_client false. Two real bugs found and fixed in the process, both pre-existing from earlier Phase 1 tasks, not new to 1.7: (1) NetworkManager's clock ping (task 1.8) gated only on is_client, which turns true the instant join() is called — a slow or refused connect attempt spammed "Trying to call an RPC via a multiplayer peer which is not connected" every frame; fixed by also requiring _peer.get_connection_status() == CONNECTION_CONNECTED. (2) ENet's own connection_failed proved unbounded in practice — verified empirically against a genuinely refused loopback connection, it hadn't fired even 14s in — which would have left a player staring at "Connecting…" indefinitely; task 1.7's own CONNECT_TIMEOUT_SECONDS is what actually satisfies "connection-refused reaches a sane UI state", not the built-in signal alone |
1.8 [D:1.2] [P] |
DONE, strengthened after adversarial review. Folded into network_manager.gd: client pings the server once a second (_ping/_pong RPCs, reliable, channel 0); clock_offset_ms is the min-RTT sample in a rolling 5s window (_clock_samples, pruned by wall time); get_server_time_estimate_ms() is the public API later phases (INTERP_DELAY, tick_offset seeding) will actually call; clock_updated(rtt_ms, offset_ms) signal for observers. New scripts/net_debug_overlay.gd autoload (F4, toggle_net_overlay input action) mirrors perf_overlay.gd's headless-guarded pattern, shows RTT + offset client-side or peer count server-side |
Verified with a real two-process test (tests/clock_smoke.gd/.tscn) on localhost: first sample at t=0.95s, offset converged to 1534.50ms by t=2.0s (well inside the 2s bar), and stayed within 1.5ms of that value through t=3.96s — comfortably under the ±1 tick (16.67ms) bar. An Opus subagent's adversarial review correctly pointed out this self-consistency check couldn't have caught a systematically-wrong-but-stable offset (e.g. a missing /2 on RTT, or a sign flip — it would converge just as cleanly). Fixed by adding an independent ground-truth cross-check: both host and client compute Time.get_unix_time_from_system()*1000.0 - Time.get_ticks_msec() (each process's own offset from the shared OS wall clock — the same real clock on both, since they're on the same machine), exchanged via a shared temp file written by the host, purely for test orchestration and touching no production code. The true required offset is just the difference of those two numbers; re-run measured the converged offset against it and found 0.99ms of error, comfortably inside a deliberately loose 250ms tolerance (OS wall-clock read resolution and sampling-instant skew, not NetworkManager's own precision, is what sets the tolerance floor here). Note the converged offset value itself is large and arbitrary (~1.5s) because Time.get_ticks_msec() counts from each process's own start, not a shared epoch — expected, and exactly what clock_offset_ms exists to absorb |
main_menu.gdgains its first async flow. Every existing handler isGameSettings.x = y; change_scene_to_file(...)— there is no loading screen, no error state, and no back-navigation state machine to extend. Budget for that.
Phase gate: two clients connect to a headless server, appear in a shared lobby, ready up, and disconnect cleanly.
Phase 2 — Server-authoritative simulation, dumb client
No own-ship prediction yet: the client renders everything, including its own ship, from the interpolation buffer. Unplayable over the internet, fine on LAN, and it proves the whole state pipeline before prediction complicates the picture.
This phase is load-bearing, not throwaway — the codec, slot mapping, snapshot pipeline, interpolator and HUD signal surface all survive into Phase 4. Roughly ten lines get discarded.
| # | Task | Acceptance |
|---|---|---|
2.1 [D:1.4] |
DONE. New MatchSim autoload (scripts/match_sim.gd) carries all Phase 2 hot-path RPCs (match_config, input, snapshot, score_update) per §1.1's "hot RPCs live on autoloads" decision — NetworkedMatch itself (scripts/networked_match.gd + scenes/networked_match.tscn, no HUD child) stays a plain scene node with no networking identity of its own. Server builds deterministic team/spawn-index slots by iterating MatchNet.roster.keys() sorted, loads a random arena via ArenaRegistry.random_path(), spawns ball/ships, then send_match_config()s. Client validates the received arena_path against ArenaRegistry.ARENAS before loading it |
Both peers spawn an identical tree in real two-process runs (tests/networked_match_smoke.gd/.tscn); an invalid arena path is refused before load |
2.2 [D:2.1] |
DONE. Server reuses RLShipController as the remote-input controller exactly as the architecture doc anticipated — each connected peer's real Ship is driven by one, fed by MatchSim.input_received. _broadcast_snapshot() runs every physics tick (60 Hz), packing NetBodyState for every ship + ball via NetCodec.pack_snapshot_body_segment and sending per-slot, filtered through multiplayer.get_peers() so a disconnected peer doesn't get an RPC send attempt |
Server-side snapshot cadence confirmed stable at 60 Hz across multiple two-process runs; no "unknown peer ID" spam after the get_peers() filter fix (found via a real disconnect-mid-test case) |
2.3 [D:2.2] |
DONE. New scripts/net_interpolator.gd (class_name NetInterpolator, RefCounted) buffers up to MAX_SAMPLES=16 timestamped NetBodyStates per remote body and produces interpolated (or clamped-extrapolated, MAX_EXTRAPOLATION_MS=150) states at any fractional server tick via sample_at(). Client-side _on_snapshot_received feeds each body's decoded state into its interpolator; ships/ball spawn FREEZE_MODE_KINEMATIC so they never call _integrate_forces/get_action() |
Client observed 31.43 m of real, physics-verified movement over a 2s held-thrust drive purely from interpolated snapshots, no local simulation |
2.4 [D:2.3] |
DONE — dual-time remote entities (§4.1). Collider updates happen in _physics_process at server_time_est (present-time, correct contact resolution); $Visual updates happen separately in _process at server_time_est - INTERP_DELAY (physics_interpolation_mode = OFF, since the node's transform is overwritten every rendered frame). _current_interp_delay_ms() computes a simplified INTERP_DELAY (one_way + interval*1.5, clamped [25,200] ms) — no jitter term yet, that lands with Phase 3's jitter buffer |
Verified via the smoke test's separate collider/visual checks; Engine.get_physics_frames()/Time.get_ticks_msec() epoch correlation (NetInterpolator.to_tick()) confirmed working with no extra sync handshake needed |
2.5 [D:2.3] [P] |
DONE. _send_local_input() samples via a stateless, never-added-to-tree PlayerShipController instance (reading real Input state) and sends the resulting ShipAction every physics tick, no redundancy/buffering yet (Phase 3) |
Input reaches the server and visibly moves the ship — confirmed via a real held move_forward keypress driving 31.43 m of server-authoritative movement |
2.6 [D:2.3] [P] |
DONE, and empirically verified, not just inferred — turned out to already be satisfied as a natural consequence of 2.1–2.5's implementation (_apply_ship_visual_state already calls set_visual_action for remote ships; _process's ball branch already calls set_visual_speed) |
Smoke test explicitly reads interpolator.latest().thrust_z mid-drive and asserts >0.5 while move_forward is held (not inferred from movement alone) — measured thrust_z=1.00 |
2.7 [D:2.3] [P] |
DONE, also a natural consequence of the above — spawn_camera_rig(_my_slot.ship) and _spawn_hud() are called once the client's own ship is identified in _on_match_config_received |
Smoke test asserts _camera_rig and hud both is_instance_valid() on the client; confirmed true in every clean run |
2.8 [D:1.1] [P] |
DONE. New NetSim autoload (scripts/net_sim.gd): seeded (--net-sim-seed=, fixed default so a bad run reproduces), CLI-driven (--net-sim-latency=/--net-sim-jitter=/--net-sim-loss=/--net-sim-dup=), a pure passthrough (send() calls the dispatch immediately) unless at least one flag is non-zero — confirmed byte-for-byte inert against every Phase 1/2 regression test with no flags set. Wraps MatchSim.send_input/send_snapshot per this row's original scope, plus NetworkManager's _ping/_pong dispatch — a deliberate scope addition, since that's the only RTT measurement that already exists and is already tested (task 1.8), so it's what makes this task's own acceptance criterion checkable today without waiting on Phase 3's per-peer snapshot echo. "Asymmetric-capable" needs no special-case code: each process reads only its own CLI args and delays only its own outgoing sends, so hosting and joining with different flags is already asymmetric. One correctness subtlety, caught before it shipped: callers that embed a timestamp in a wrapped RPC (_ping/_pong) must capture Time.get_ticks_msec() before calling NetSim.send(), not inside the wrapped Callable — capturing it inside would silently absorb that process's own added delay out of the round-trip measurement instead of adding to it, since the timestamp would then reflect "after my delay" rather than "when I actually tried to send". A second real bug, found by actually running Phase 2's own milestone gate (a full networked_match_smoke run under --net-sim-latency=80 --net-sim-jitter=20, not just the isolated ping/pong test above): a delayed send can outlive the window its target was valid in — the host hit "Attempt to call RPC with unknown peer ID" (the client had disconnected during the ~80-100ms hold, after _broadcast_snapshot's existing get_peers() filter had already passed at schedule time) and the client hit "'_recv_input' on yourself is not allowed by selected mode" (its own shutdown() had already reset multiplayer_peer to a fresh OfflineMultiplayerPeer before a still-pending delayed send fired, so peer id 1 now meant itself). Fixed by having send() accept an optional target_peer_id and re-validating it — plus that this process still has a real (non-Offline) peer at all — at fire time inside a new _fire(), not just at schedule time; the synchronous/inactive path is deliberately left unvalidated so NetSim stays a true no-op when idle |
Verified with a real two-process test (tests/net_sim_smoke.gd/.tscn): baseline (no flags) observed rtt_ms=7.00 on loopback; --net-sim-latency=80 on the host alone raised the client's observed rtt_ms to 83.00 (want ≥70, confirmed measurably higher than baseline); --net-sim-loss=1.0 on the host produced zero pong samples over 7s (rtt_ms stayed -1, confirmed the drop path actually drops rather than relabels). Phase 2's own milestone gate re-run and passing: networked_match_smoke under --net-sim-latency=80 --net-sim-jitter=20 on both peers — client still observed 26.08m of clean server-authoritative movement via interpolation, thrust_z=1.00 confirmed mid-drive, zero RPC errors (the fire-time-revalidation fix above). Re-ran the full Phase 1 + Phase 2 regression suite (test_runner, net_smoke, match_net_smoke, clock_smoke, lobby_smoke, server_boot, networked_match_smoke) with NetSim present but inactive — all still pass with unchanged behaviour (clock offset converged to the same value, networked_match_smoke still showed clean server-authoritative movement) |
| — | An Opus subagent's adversarial review of all of Phase 2 found real, verified bugs the smoke tests couldn't catch, since constant-velocity dead reckoning still moves a ship far enough to pass a moved > 1.0 check. Fixed, all independently re-verified with real two-process runs and temporary instrumentation (removed after confirming):
(1) The interpolator never actually interpolated — every sample_at() call took the extrapolation branch, 100% of the time, LAN or under simulated latency alike. NetInterpolator.to_tick() assumes Time.get_ticks_msec() == physics_frame * TICK_MS on the server; real engine/autoload startup work before the first physics step (plus any dropped tick, which only ever widens it) breaks that by a steady +45-55ms in practice. networked_match.gd now tracks one shared _tick_bias_ms estimate (_update_tick_bias, called from _on_snapshot_received) — the minimum to_tick(server_time_est) - server_tick over a rolling 5s window, same rationale as NetworkManager's own min-RTT filtering: the least-delayed sample best isolates the constant bias from per-packet transit noise, and a rolling (not all-time) window still tracks a real future increase. _estimated_tick() subtracts it before every to_tick() call. Verified: bias converged to ~50-56ms (matching the bug's own measured magnitude exactly) and real interpolation rose from 0% to ~70% of calls (interp=436 extrap=182 out of 618, up from interp=0 extrap=617). A first attempt at this fix was itself broken and made the lead ~30x worse (90+ ticks, ~1.5s): early snapshots arrive before NetworkManager's first clock pong lands (rtt_ms < 0, clock_offset_ms still 0.0), so server_time_est briefly means "my own raw local uptime" — a wildly wrong bias sample that the 5s rolling-min then locked onto for a whole short test, since 5 real seconds never fully elapsed before the test ended. Fixed by skipping bias recording entirely while rtt_ms < 0.
(2) Goals caused a ~27m visual slide. _on_goal_scored bumped _reset_gen immediately, but reset_ball()/reset_ships() only queue teleports (task 0.15, applied on each body's next _integrate_forces) — so the broadcast that same tick carried the NEW gen with the OLD (still-in-goal) position, and the client's buffer-clear-on-reset kept exactly that stale sample and lerped a full-arena slide to the next, genuinely-reset one. The first fix attempt (defer the bump to "the next _physics_process" via a plain boolean) didn't work either — emperically, the goal Area's body_entered signal fires as part of physics tick N's own step, before tick N's _physics_process callback, so a flag set in the handler is already true by the time that same tick checks it: no delay was actually introduced. Fixed by recording the tick the goal was detected on (_pending_reset_gen_bump_tick) and only bumping once Engine.get_physics_frames() > _pending_reset_gen_bump_tick — i.e. strictly on a later tick, which guarantees the queued teleport's _integrate_forces has already run. Verified by forcibly teleporting the ball into a goal mid-test and logging the server's own broadcast stream tick-by-tick: gen change and the already-reset position now land in the identical broadcast, every time.
(3) _local_input_sampler (a PlayerShipController, i.e. a plain Node) was created but never added to the tree and never freed — this was the unexplained "3 resources still in use at exit" warning on every prior Phase 2 test run, confirmed by --verbose naming the exact leaked script chain and by the warning disappearing once a _exit_tree() cleanup was added. Also leaked on the server despite its "client only" comment, since the field initializer is unconditional.
(4) Ball angular velocity decoded 8x too small — NetCodec.rescale_avel() exists specifically to correct a ball's decoded angular_velocity from the ship-range assumption unpack_snapshot() decodes every body with, and was never called. Dormant today (nothing read decoded angular_velocity yet) but silently wrong the moment ball-spin VFX or Phase 4 prediction reads it; now called in _on_snapshot_received.
(5) get_server_time_estimate_ms() was used unguarded before the clock had synced, contradicting its own doc comment — against a long-running dedicated server this freezes every remote body at the oldest buffered pose for the whole first second of every match (clock_offset_ms == 0.0 compares this process's own short uptime against the server's much larger tick count). Both _physics_process and _process now skip their collider/visual update entirely while NetworkManager.rtt_ms < 0.0.
Smaller fixes, all confirmed via the regression suite: net_sim.gd's delayed-send timer now uses process_always = true (a simulated wire shouldn't stop just because the local game pauses) and _fire() also checks get_connection_status() == CONNECTION_CONNECTED, not just non-Offline, before dispatching (a known, accepted residual gap remains: a shutdown-then-reconnect inside one delayed send's hold window isn't fully closed, judged disproportionate to fix for debug-only tooling); _broadcast_snapshot() now appends one body per slot unconditionally (a zeroed placeholder for a momentarily-invalid ship) so the ball's fixed index assumption can't silently break if "no ship is ever despawned" (§6.4) ever stops holding; networked_match.gd now only declares score_changed (the one signal it actually emits) instead of also declaring timer_updated/match_ended/kickoff_countdown/overtime_started, which — despite never being emitted — made HUDController show a permanently frozen timer widget purely because has_signal("timer_updated") was true.
Confirmed fine, not just assumed, via a real hostile-client stress test and a real 3-process multi-client run: a malformed/garbage/oversized _recv_input payload cannot crash the server (Godot's StreamPeerBuffer silently zero-fills past EOF; count is a bounded u8); NetworkedMatch skipping GameMode._ready()'s super() call drops nothing load-bearing; deterministic team/spawn-index slot assignment is correct with 2 simultaneous clients (verified with a real 3-process host+2-client run); RPC authority enforcement on _match_config/_score_update/_snapshot genuinely rejects a forging client server-side | Full Phase 1 + Phase 2 regression suite (test_runner, net_smoke, match_net_smoke, clock_smoke, lobby_smoke, networked_match_smoke baseline and under the --net-sim-latency 80 --net-sim-jitter 20 milestone gate, net_sim_smoke) re-run clean after every fix |
net_sim.gdbelongs in this phase, not Phase 3. A LAN-only phase gate passes even with §4.1's flaw fully present, because LANINTERP_DELAYsits at the clamp floor and closing-speed error is small. Phases 2 and 3 would both go green and Phase 4 would discover the architecture is wrong.
Phase gate — MILESTONE: a real 1v1 at --net-sim-latency 80 --net-sim-jitter 20, not just on LAN. Ships fly, the ball moves, goals detect server-side.
Phase 3 — Input pipeline hardening
| # | Task | Acceptance |
|---|---|---|
3.1 [D:2.5] |
DONE. Client sends the last NetCodec.MAX_REDUNDANCY (4) ticks' actions per packet, newest-first (the wire format already supported this from Phase 1 — Phase 2 just wasn't using it). Server gains a real per-slot ring buffer, new standalone scripts/input_jitter_buffer.gd (InputJitterBuffer, RefCounted, no scene dependency — same reason net_codec.gd/net_interpolator.gd are pure classes), consuming exactly one sequence number per physics tick |
Verified both by unit test (test_redundancy_survives_3_packet_burst_loss) and live: 25% random simulated input loss produced zero observed starvation ticks; 100% loss correctly produced zero seeding/consumption (no crash, ship simply never receives a command) |
3.2 [D:3.1] |
DONE. InputJitterBuffer.consume(): repeat-last on starve, zero + stalled=true only after STARVE_ZERO_TICKS (30 = 500ms). input_buffer_depth/last_input_seq/echo_client_send_ms are now genuinely per-peer in every snapshot (_broadcast_snapshot builds them from each slot's own InputJitterBuffer), replacing Phase 2's hardcoded zeros |
One real bug found wiring this into a live match: the server's ring buffer started counting its own "expected sequence" from a local 0 the instant a slot was created — well before that player's first real packet could possibly arrive (connection/spawn setup takes real time) — so the two numberings never converged and the ship silently never moved. Fixed by seeding last_applied_seq from the client's own numbering on first real ingest(), not assuming a shared from-zero baseline. Verified with real two-process runs before and after the fix |
3.3 [D:3.2] [P] |
DONE. New standalone scripts/input_lead_controller.gd (InputLeadController, unit-tested like InputJitterBuffer): clamp [1,12], fast attack (+3, debounced to once per 30 ticks) on any server-reported starve, slow release (−1 per 60 ticks) gated behind a one-time 2s clean-surplus bar. A lead change is realized as extra distance between the client's own outgoing seq and what the server has consumed — attack skips extra seq numbers, release duplicates (re-sends) the current one; the server's ring buffer needs no special handling for either, since a skip is an ordinary drop and a duplicate is a same-seq resend already discarded |
Verified live: on a clean LAN, one early attack (a momentary connection-setup hiccup) recovered via two releases within ~4s, settling back near minimum; under sustained 30% simulated loss, lead climbed to 7 via repeated attacks and never released while genuine loss continued — confirming debounce, attack, and release gates all fire correctly on real conditions |
3.4 [D:3.1] [P] |
DONE. MatchSim._recv_input validates before decoding: per-peer rolling-1s rate limit (packet count AND byte budget, §3.1's own numbers), disconnect after 3 consecutive over-budget seconds; framing (redundancy count + payload size checked against NetCodec's own layout, since StreamPeerBuffer silently zero-fills past EOF instead of erroring — a Phase 2 adversarial-review finding), disconnect after 20 malformed packets. networked_match.gd additionally rejects seq > server_tick + 20 and counts (rather than silently ignoring) input from a peer with no slot. Server-side input_lead enforcement from arrival times was scoped down to observability rather than active enforcement — see the note below the table |
Two new permanent regression tests (networked_match_smoke.gd --role=client-abuse-malformed / client-abuse-flood) call MatchSim._recv_input directly with garbage bytes and a legitimate-but-too-frequent flood respectively — bypassing the honest client encoder entirely, i.e. exactly what a hostile custom client sending raw ENet packets looks like. Both confirm a real disconnect, not just tolerance. Found and fixed two smaller bugs getting these to pass cleanly: a GDScript lambda-captures-by-value mistake in the tests themselves (fixed by capturing a single-element Array instead of a plain bool), and a real race where NetworkManager's own ping/pong reply could target a peer a concurrent abuse-disconnect had just removed from the same poll() batch (now guarded); the post-shutdown physics path now also uses the safe NetworkManager lifecycle flag and stays error-free |
3.5 [D:3.2] [P] |
DONE. tests/cases/test_input_jitter_buffer.gd and test_input_lead_controller.gd: sequential consumption, redundancy surviving a 3-packet burst loss (3.1's own acceptance text, verbatim), starvation repeat-then-zero timing, stale/reordered-packet handling, buffered-depth reporting, ring-wraparound slot-tagging safety, and the full attack/debounce/release state machine including a starve mid-release-window forcing a fresh clean-surplus wait |
14 new tests, all passing (test_runner.tscn: 33 total, 0 failed) |
3.6 [D:2.8] |
DONE, with one honest scope note. networked_match.gd's client can swap its input sampler for a real AIShipController (--test-bot, optionally --test-bot-model=) instead of PlayerShipController — parented onto the client's own ship via Ship.set_controller() since (unlike the human sampler) it needs real scene context. Known limitation, documented in code: this client's ships are all FREEZE_MODE_KINEMATIC, driven purely by transform writes, so nothing ever writes linear_velocity/angular_velocity onto them — the bot's observations always see every ship as stationary. It still produces well-formed, bounded actions from that degraded input (the policy network's output layer is bounded regardless of input quality), sufficient for this task's actual job (CI traffic generation, not bot skill). New CI driver tests/networked_match_ci.gd/.tscn: headless server + two headless --test-bot clients. This task's own original acceptance text names "p95/p99 prediction error" and "snap count" — both Phase 4 concepts that don't exist yet (no client-side prediction or hard-snap threshold exists before Phase 4); asserting on data that doesn't exist would be fabricated, so those two are explicitly not checked, with the gap called out in the driver's own header comment rather than silently dropped |
Real 3-process runs: both bots' independently-written final scores agreed after a deterministically forced goal (bot-vs-bot scoring isn't reliable enough within a short run to gate on), both saw 500+ snapshots over an 8s run (well above the 60Hz-scaled floor), all three processes exited 0. "Clean stderr" is the external invocation's job (grep the captured output), same as every other smoke test in this project — verified manually, not self-asserted by the script |
3.7 [D:2.8] [P] |
DONE, with prediction error deliberately omitted (documented, not silently dropped — same Phase 4 gap as 3.6). Extends net_debug_overlay.gd with jitter (new RFC3550-style EWMA in NetworkManager, from raw per-sample RTT — Phase 1's rtt_ms is a min-filtered sample, deliberately jitter-insensitive by design, so it can't answer this on its own), snapshot loss (new EWMA in networked_match.gd over each received snapshot's own server_tick gap — snapshots go out at a steady one-tick cadence, so a gap is direct evidence of a drop or reorder), snapshot age (computed on demand from the same bias-corrected tick estimate the interpolator itself uses), input buffer depth and input_lead (both already tracked client-side for 3.3), and bandwidth (new rolling per-second byte counters in MatchSim, the two 60Hz hot-path channels only) |
Verified values are live and plausible, not just present, by calling get_net_debug_stats() directly in a real two-process test: bandwidth matched the wire format's own byte math almost exactly (measured ≈2400 B/s sent against a computed 40B×60Hz, ≈3540 B/s received against 59B×60Hz for a 1v1), and buffer depth/lead/loss all moved in the correct direction between a clean LAN run and one under simulated 60ms latency + 10% loss |
AIShipControllerruns a policy in pure GDScript with no Python or ONNX dependency, so 3.6 gets a competent automated player for free.
Server-side
input_leadenforcement from arrival times (§3.3's closing paragraph) was scoped down to observability, not built as active enforcement. The concrete, mechanically well-specified parts of task 3.4 (rate limiting, malformed-packet counting, seq-range rejection, disconnect policy) fully close the load-bearing security gaps; the advantage a client gains from claiming a dishonestly lowinput_leadis explicitly described in the doc itself as "small" (reduced apply latency, not an outright cheat — there's no prediction/reconciliation yet for a bad lead to actually corrupt), and building real arrival-jitter-derived enforcement well — without risking a third, subtly-interacting control loop on top of the two §3.3 already warns against — is a genuine design task in its own right, not a mechanical one. Revisit if Phase 4's prediction work turns "slightly lower latency" into a sharper edge.
Phase gate — MET. Both networked_match_smoke and the CI driver (task 3.6) re-run under the gate's own exact condition, --net-sim-latency 80 --net-sim-loss 0.05, on every peer: the human-driven smoke test still shows clean server-authoritative movement (22.61m over the usual 2s drive), and the two-bot CI run still shows both clients independently agreeing on the final score after a forced goal, 495+/504 snapshots received each, all three processes exiting 0 with clean stderr.
| — | A second adversarial review of the fix commit above found that two of its nine fixes silently cancelled each other out, re-creating the original critical bug at a lower failure threshold — plus four smaller real issues, all re-verified with real two- and three-process runs.
CRITICAL — the seq-range guard fix (round 1's MEDIUM item, gotcha 42) made the ring-overflow resync fix (round 1's CRITICAL item, gotcha 39) unreachable in production. The guard bounded every accepted seq at last_applied_seq + RING_SIZE — the consumer's position — which in turn caps InputJitterBuffer's own highest_ingested_seq at that same ceiling, since nothing above the bound is ever allowed to reach ingest() at all. But consume()'s resync condition needs highest_ingested_seq to reach expected + RING_SIZE, one full ring past that same ceiling — arithmetically impossible on the only call path that exists. The two fixes read as independent (one in the jitter buffer, one in the caller) but shared a variable and quietly defeated each other; the round-1 commit's own new unit test for the resync never caught it because it called ingest() directly, bypassing the guard entirely — the exact composition the bug lived in. Verified failing on the committed code: a 0.6s SIGSTOP host freeze reproduced the original 0.00m death, at a lower threshold than the pre-round-1 bug (~0.6s vs ~0.7s), reachable via ordinary server tick loss with no external trigger at all (Engine.max_physics_steps_per_frame = 4 means a server that falls behind wall-clock time during any stall never catches back up on its own). Fixed by rebinding the guard to highest_ingested_seq (now a public field, matching last_applied_seq's own convention) instead of last_applied_seq — the client's actual send epoch, which ingest() updates once per accepted packet regardless of how far the consumer has fallen behind, rather than the consumer's own lagging position. Re-verified against a real 2-bot CI match with a 1.5s host SIGSTOP freeze injected mid-run (well past the 0.6s failure threshold): both peers kept moving (47.46m / 10.19m and, on a repeat run, 25.62m / 17.96m), stalled=false, sampled while genuinely still connected.
HIGH — the InputLeadController release fix (round 1's HIGH item, gotcha 40) was itself incomplete. Round 1 added a real depth check (input_buffer_depth > TARGET_DEPTH) but left the original gate, and lead > LEAD_MIN, still ANDed onto the same final condition — so a backlog the controller never caused (lead pinned at its own floor) still could never release, since that old clause always failed regardless of what the new depth check found. Confirmed by the round-1 commit's own new unit test, whose assertion text literally read "lead cannot release below its own floor even under large surplus" as if that were the intended behaviour. Fixed by splitting the one gate into two independent decisions: whether to duplicate this tick's seq (the only thing that actually narrows real buffered depth) now follows the depth signal alone; whether to keep decrementing lead's own bookkeeping below its documented [LEAD_MIN, LEAD_MAX] floor is a separate, purely cosmetic choice made inside that branch.
MEDIUM — task 3.6's CI gate (round 1's own fix for gotcha 43) still sampled after both bots had legitimately disconnected. The fix used a run_seconds - 0.5 margin, narrower than the original bug (sampling after the full run) but still not enough: multiplayer.get_peers() at sample time was already empty, and the check was only passing on STARVE_ZERO_TICKS's own ~200ms of residual starvation grace, not because it was genuinely still connected as its own print claimed. Widened the margin to run_seconds - 2.0 and added an explicit slot.peer_id in multiplayer.get_peers() assertion at sample time, so a future regression in either direction fails loudly here instead of silently passing on residual grace.
LOW — the human smoke test's movement bar was beatable by gravity alone. moved > 1.0 measured full 3D distance; a 1.2s window of completely dead input still registered ~1.07m from pure vertical settling (spawn height dropping to the floor) — above the bar, with only the separate thrust_z_ok check actually catching the failure. Forward thrust is a horizontal force, so switched to XZ-only displacement, which gravity alone cannot satisfy regardless of spawn height or timing.
LOW — "clean stderr" wasn't actually clean. Every disconnect logged ERROR: Unable to send packet on channel 0, max channels: 0 from match_net.gd's _remove_player, which broadcasts _player_left to every peer in multiplayer.get_peers() — including, transiently, the peer that just disconnected (whose own ENet connection can still be momentarily present in that set with its channels already torn down), and — found only after the first fix still left an error in the 2-bot CI scenario specifically — including a second still-connecting peer when two clients disconnect within the same poll() batch, since get_peers() hadn't yet been updated for the one not currently being handled. Fixed by deferring the whole notification (call_deferred) to the next idle frame, by which point poll() has fully returned and every disconnect event in the batch has actually settled, then explicitly excluding the peer that left. Re-verified clean (grep for ERROR) across both the basic 2-process smoke test and a real 2-bot CI run with a mid-match host freeze injected.
Noted, not fixed — a related but distinct stderr source in _broadcast_snapshot. The deliberately-adversarial client-abuse-malformed smoke test still logs one Unable to send packet from networked_match.gd's snapshot broadcast, racing a host-forced disconnect_peer() in match_sim.gd's abuse-disconnect path against the same tick's connected_peers.has(slot.peer_id) snapshot — a different call site than the one just fixed, only reachable via the abuse-detection disconnect path rather than a normal client-initiated one, and out of scope for this pass. Left for a dedicated look rather than a rushed fix under this round's time pressure.
Confirmed fully correct, not just re-asserted: the _input_history fix (round 1's LOW-MEDIUM item) was re-verified via a synthetic-marker harness stamping a computable value into every outgoing action and checking it through a real 3-process match under 60ms+40ms jitter+18% loss — 1080 marker checks, 0 mismatches, including real attacks and releases; the leaky-bucket rate limiter (round 1's MEDIUM-HIGH item) cannot false-positive on honest traffic (~1.8x measured margin under real impairment); the resync boundary arithmetic itself is correct under packet reordering and duplication. The lesson that mattered most this round wasn't any single fix — it was that two fixes landed in the same commit, each individually correct in isolation, that silently cancelled each other out (see gotcha 45) | Full regression suite (35 unit tests, the basic 2-process smoke test, the malformed/rate-limit/duty-cycle abuse roles, and a real 2-bot CI run with a 1.5s host SIGSTOP freeze injected mid-match) re-run clean after every fix in this round
| — | An Opus subagent's adversarial review of all of Phase 3 found a critical, silent, permanent bug plus eight smaller real issues — all empirically verified with real two- and three-process runs, not just code reading.
CRITICAL — InputJitterBuffer's 32-entry ring permanently bricked a player's input on any backlog bigger than the ring. consume() advanced last_applied_seq by exactly 1 per tick with no resync; once the un-consumed backlog exceeded RING_SIZE, a fresh arrival would land in the exact slot consume() was still waiting on, and since both counters only ever advance, the gap never closed — the affected player's ship silently went to zero thrust for the rest of the match. The reviewer reproduced this with a real SIGSTOP/SIGCONT host freeze (a faithful stand-in for a GC/IO/scheduler hitch on a listen-server host): client movement dropped from ~26m to a flat 0.00m at ~0.7s of freeze, reproducible 4/4 times, and found the cliff got worse under real network conditions (a lossy link that had already pushed input_lead up lowered the fatal threshold to ~400ms) and could be reached with no external trigger at all via ordinary client/server clock drift (~1.7% faster client death-spiraled within ~60s). Fixed with a real resync mechanism: ingest() now tracks the highest seq ever seen regardless of ring capacity, and consume() detects when the gap to that value exceeds RING_SIZE and jumps directly to what the ring can still actually provide, instead of starving through an unrecoverable span. Re-verified with the reviewer's own reproduction: a 3-second SIGSTOP freeze mid-drive now fully recovers (27m+ movement), both via the human smoke test and a real 2-bot CI match. New unit test test_ring_overflow_resyncs_to_fresh_data_instead_of_starving_forever covers the exact under-tested direction the reviewer flagged (the original suite only exercised the under-full ring case).
HIGH — InputLeadController's release logic couldn't drain a backlog it didn't itself create. Release was gated on lead > LEAD_MIN — this controller's own memory of past attacks — so a backlog from an external cause (a server hitch, persistent clock drift) left input_buffer_depth elevated indefinitely while lead (and the release gate) never moved, since the controller never itself attacked. Fixed by gating release on the actual server-reported input_buffer_depth > TARGET_DEPTH (§3.3's own target_depth = 1), not on self-tracked state. New unit test test_release_drains_a_backlog_it_never_caused_itself reproduces the scenario directly.
MEDIUM-HIGH — the rate limiter was trivially evaded by a duty-cycled flood. The original design tracked "N consecutive over-budget seconds" and hard-reset that streak to 0 on any single clean window, so a burst-then-idle attacker (flood hard, one clean window, repeat) evaded it indefinitely — the reviewer sustained ~33x the packet budget for 28.5s with zero disconnect warnings against the real MatchSim._recv_input. Replaced with a leaky-bucket excess accumulator (grows by each window's actual total, drains by exactly one window's worth of budget every window, regardless of how the excess is distributed in time) — immune to the same evasion by construction. New permanent regression test client-abuse-flood-dutycycle reproduces the reviewer's exact attack shape (0.35s burst / 3.0s cycle) and confirms it now disconnects.
MEDIUM — the seq > server_tick + 20 guard compared two unrelated epochs. Engine.get_physics_frames() counts from the SERVER PROCESS's own start; a client's _input_seq starts at 0 when ITS match scene loads — input_jitter_buffer.gd's own seeding logic exists specifically because these share no baseline. Bounding against server uptime meant the guard could never fire on a long-running dedicated server (no real protection, despite the comment's claim) and could silently drop an honest client's input forever once enough accumulated server tick loss closed whatever accidental head-start margin existed. Fixed by bounding against the slot's own last_applied_seq + RING_SIZE — the client's actual epoch, using the same capacity the ring-overflow fix itself treats as "unrecoverably far ahead."
MEDIUM — InputJitterBuffer.stalled was computed but never reached the wire. _ship_to_net_body_state never set NetBodyState.stalled even though NetCodec already packed/unpacked the bit — the one signal that would have made the ring-overflow bug visible to the client, the debug overlay, and the CI gate was silently dropped between the buffer and the snapshot builder. Now wired through.
MEDIUM — task 3.6's own CI gate passed with a completely dead input pipeline. Its assertions (snapshot count, a server-forced goal's score agreement) don't depend on client input reaching the server at all; the reviewer confirmed it kept reporting SMOKE PASS with the ring-overflow bug actively triggered mid-run. Fixed by recording each bot's ship position before the run and asserting real server-side movement plus a non-stalled jitter buffer — sampled while clients are still actively connected, not after (an early attempt sampled too late and caught each bot's own legitimate end-of-match disconnect instead of the bug, since a departed peer's buffer starves too — that's correct behaviour, not a regression, just the wrong moment to check it). Re-verified: the fixed CI gate still passes cleanly under a real mid-match host freeze now that the underlying bug is fixed, and (checked by inspection during the fix) would have caught the original bug had it still been present.
LOW-MEDIUM — a lead change silently mislabelled the redundancy history. _input_history was always push_front'd regardless of the seq delta, but the wire format has no per-entry seq field (actions[i] is implicitly seq - i) — a duplicated tick (release) shifted older entries under a label that no longer matched what was actually there, and a skip-ahead (attack) left the whole history discontiguous with the new seq, so the server could replay already-applied input or apply the wrong redundant copy. The original code comment's claim that this "only ever degrades a backup copy, never the real per-tick record" was itself wrong. Fixed by handling each delta case on its own terms: ordinary ticks still push; a release replaces the front entry in place instead of shifting everything back; an attack resets the window to just the current sample, which rebuilds naturally over the next few ticks (the same way it does at connection start).
LOW — bandwidth and snapshot-loss overlay metrics froze at their last value instead of decaying during a total outage — exactly when they matter most. MatchSim.bytes_sent_per_sec/bytes_received_per_sec are now read through get_bytes_sent_per_sec()/get_bytes_received_per_sec(), which report 0 once meaningfully more than one window has passed with nothing tracked; get_net_debug_stats()'s snapshot_loss_pct now reports 100% once more than SNAPSHOT_STALE_MS has passed since the last actual snapshot receipt. Verified live: all three read their honest post-outage values (0, 0, 100%) after a real ~2.5s gap in traffic, not the frozen pre-outage numbers.
LOW — a guard comment on NetworkManager._ping misdescribed what the code actually does, claiming disconnect_peer(..., now=true) when the real call uses the default force=false (an earlier attempt at force=true, tried and reverted elsewhere this session, made Godot's own peer bookkeeping more inconsistent, not less). Comment corrected to match reality.
Confirmed fine, not just assumed: the jitter metric's magnitude is honest (cross-checked against real 60Hz snapshot-stream jitter on the same impaired link, same order of magnitude), just slow to converge from a 1Hz sampling cadence — worth documenting as "steady-state link quality" rather than a live indicator, not worth rebuilding; --test-bot/AIShipController wiring on a frozen kinematic ship is fully safe, no NaN/Inf even under the bot's permanently-zero-velocity observations; both original abuse-detection regression tests are genuine, confirmed via a working control (a continuous flood still disconnects in ~4.5s); redundancy + adaptive lead + real loss alone (no hitch) is solid over long runs | Full regression suite — including the net-sim-latency milestone gate, all three abuse roles, the CI driver, and the reviewer's own SIGSTOP/SIGCONT reproduction at 3s (well past the original ~0.7s failure threshold) — re-run clean after every fix |
Phase 4 — Prediction and reconciliation, ship and ball
| # | Task | Acceptance |
|---|---|---|
4.1 [D:3.1] |
DONE. Immediate local input with immutable sequence/redundancy bookkeeping; server/training action semantics unchanged | 60 unit tests and 60s LAN/jitter/loss runs pass |
4.2 [D:4.1] |
DONE. 128-entry sequence-tagged prediction history and snapshot matching | Same-sequence free-flight samples resolve in all 60s runs |
4.3 [D:4.2, 0.14] |
DONE. Atomic staged reconciliation, delta rebase, epoch/reset and missing-history recovery | No free-flight hard snaps across LAN, 80±20ms, or 5% loss 60s runs |
4.4 [D:4.3, 0.2] |
DONE. Client-only bounded position and rotation visual offsets/decay; interpolation reset | Free-flight p99 raw residual ≤0.207m; exposed visual p99 0m in final matrix |
4.5 [D:4.3] [P] |
REJECTED / SUPERSEDED. Analytic one-body action replay was removed in favour of same-sequence delta transport | Jolt/contact nondeterminism makes replay unsuitable; see §4.4 |
4.6 [D:4.3] |
DONE. Client-only dynamic proxy, authoritative shadow, RTT-limited reveal, 150 ms blend and 3 m handoff | Final contact run: same-frame reveal, 4 blends, max 152ms, no hard handoff |
4.7 [D:4.4] [P] |
DONE. Client-only debug keys tune thresholds, decay, visual offset and remote-present A/B | Defaults remain 2m/60°/0.4m and render-only tuning never reaches server/training |
4.8 [D:4.4] [P] |
DONE. p50/p95/p99 residual telemetry, elapsed-time snap rate, reason/cohort counters | Final free-flight p99: LAN .150m; 80±20ms .149m; 5% loss .207m; zero hard snaps |
4.9 [D:4.4] |
DONE. Present-time remote visual extrapolation, angular integration, and render-only residual correction; delayed interpolation remains an A/B debug mode | Final two-bot present-time p99 ≤.208m / 3.146°, below .3m / 5° gate |
4.10 [D:4.9] [P] |
DONE. Signed starvation sentinel and client hysteresis/cooldown; headless --test-bot remains target depth 1 |
Jitter run observed starvation fallback; stable runs preserve safe target behavior |
Ball prediction is not optional and not deferrable to a later phase. With §4.1 in place the touch registers correctly on the server, but the ball still renders a third of a beat late — your ship visibly passes through it before it moves. In a game whose entire point is hitting a ball, that is the difference between "networked" and "broken", and it is the same machinery as own-ship prediction applied to one more body. Do it while the prediction code is warm. Buffering server ball state into a shadow copy (rather than discarding it) is what lets you measure disagreement continuously instead of discovering a 3 m error at window end.
| 4.11 [D:4.2] | DONE. Prediction history is filed under the issuing sequence, and a forced-input-transition trace gates the label | Marker mismatch 0.00–1.3% (was 9.3% LAN / 24% at 80±20ms); control run at the old label fails the same gate at 50% |
| 4.12 [D:4.11] | DONE. Issued-but-unsimulated (attack-gap) sequences are recorded and skipped rather than diagnosed as history loss; the release path no longer re-files an already-issued sequence | Free-flight hard snaps 0 across all three 60 s conditions, down from 25/8/4 missing_not_recorded |
| 4.13 [D:4.12] | DONE — two server-side input-death bugs found by adversarial review, both reproduced and fixed with controls. A starve no longer advances past a sequence the client has not sent; the seq-range guard can no longer latch shut permanently | Marker 0.00% in all three conditions (was 1.7–2.5%); 2.0 s and 3.5 s host freezes now recover; control runs with each fix reverted fail the gate |
| 4.14 [D:4.3,4.8] | DONE. Prediction startup distinguishes the server's pre-history sequence-0 acknowledgement from genuine missing/evicted history, so warm-up cannot arm hard-snap recovery | 143 Godot tests pass; two-process ENet match passes 173 prediction samples with 0 hard snaps, 0% snapshot loss and authoritative movement; the 80±20 ms impaired-link run passes the near-surface gate with p95 0.682 m / p99 0.717 m and no free-flight hard snap; the 5% loss run passes with 222 samples, 7.1% observed snapshot loss, p99 0.716 m and 0 hard snaps |
Phase gate — correctness gates MET; the milestone's felt-quality half remains untested. The action-sequence-correctness gap is closed and permanently gated (4.11), the two seq-delta paths it exposed are fixed (4.12), and an adversarial review's two server-side input-death bugs are fixed with controls (4.13). What has not happened is the original milestone's actual subject: nobody has played this with hands on a controller at ~100 ms RTT to judge whether ship and ball feel local and whether contact corrections read as bumps. Numbers cannot answer that, and the contact cohort is where the remaining known weakness lives (see the shadow-world note below). Sign off after a human playtest, not before — item A of §0.
Read 4.13 before trusting any earlier Phase 4 evidence. Until this session the server was silently discarding a connected player's input for ~30 ticks roughly every 6.5 seconds on a clean LAN, and permanently after any ~2 s host hitch. Every Phase 4 number recorded before 4.13 was measured through that, and the gates reported green throughout — for the same reason they missed the label bug in 4.11: a steady input cannot distinguish "the server repeated my last action" from "the server applied my real action".
The mislabelled prediction history, and why every earlier gate missed it. _send_local_input filed each post-step predicted state under _local_net_controller.last_applied_seq — the timeline's estimate of the sequence the server would consume this tick, which trails issuance by input_lead. The body had actually integrated the current raw intent, issued under _input_seq. So predicted[S] held "state after integrating the intent from now" while the server's authority for S is "state after integrating action(S)", sampled input_lead ticks earlier. The two agree only while the commanded action is constant — and every Phase 4 acceptance trace held its input steady (move_forward held, or the free-flight hover alternating on a 0.7 s/0.3 s period). A steady input cannot falsify a sequence label: the marker reads 0/N under a correct and an incorrect label alike. The 60-second free-flight runs genuinely reported marker=0/3784; the instrument was fine, the trace was blind.
Filing the state under _input_seq fixes it and costs nothing. The code comment that had rejected this ("avoids turning client prediction into an input-delay queue") conflated which action the ship uses — decided in LocalNetShipController.get_action(), still the raw current intent, still immediate, untouched by this change — with which sequence its resulting state is filed under. Measured with --exercise-input-transitions (below):
| condition | input_lead |
old label | filed under _input_seq |
|---|---|---|---|
| LAN | 1 | 35/376 (9.3%) | 0–6/456–582 (0–1.3%) |
| LAN, adversarial toggle phase | 1 | 289/576 (50.2%) | — |
| 80±20 ms | 3 | 97/404 (24%) | 0/424 (0%) |
Mismatch scales with input_lead, exactly as the mechanism predicts. It also cut pre-existing missing_not_recorded hard snaps 4× on the 60 s LAN free-flight run (25 → 6, 24.8/min → 6.0/min): the old label's per-tick +1 cursor was an estimate that could drift off the sequence the server actually acknowledged, while an issued sequence is by construction the thing the server acknowledges.
Task 4.12 — the two seq-delta paths, and what is left. Relabelling exposed two further places where the history disagreed with the wire, both now fixed:
- Attack gaps (
delta > 1). The lead controller skips sequence numbers to buy server buffer margin. Those sequences are filled with repeat-last actions and genuinely sent, and the server genuinely acknowledges them — but the client took exactly one physics step that tick, so no post-step state exists for them. They were simply absent from the ring, whichcompare_authoritativecould only report asmissing_not_recorded: indistinguishable from real ring loss, and therefore a hard snap, a full authority teleport, and armed resync suppression several times a minute during ordinary play. They are now recorded stateless viarecord_unsimulated()and report their ownunsimulated_gapstatus, whichNetShipPredictor.decide()answers with a new"skip"mode — no correction, no teleport, no suppression, no snap counted, its own metrics cohort. The next simulated sequence (a tick or two later) reconciles normally. Result: free-flight hard snaps went from 25 / 8 / 4 to 0 / 0 / 0 across the LAN, 80±20 ms and 5%-loss 60-second runs; the doc's own long-standing target was <1/min and LAN was measuring 24.8/min. - Release (
delta == 0)._send_local_inputre-recorded at the unchanged_input_seq, filing the current intent under a sequence that had already gone out carrying a different action.LocalInputTimeline.issue()deliberately refuses to mutate an already-issued sequence ("may be in flight or consumed"), so the ring was contradicting the wire outright. Recording is now skipped entirely on a release tick; the existingpredicted[S]is already correct, and the extra unlabelled local step is precisely the tick of latency the release exists to recover.
The residual is solved — it was not a prediction bug at all. An adversarial review intersected every sequence the server starved on against every sequence the marker flagged, across five two-process runs: 151 of 151 mismatches were the server repeating a stale action on a starve, zero unexplained. When the server starves on seq S it repeats action(S-k) but still acks S, so the snapshot's thrust_z honestly describes a different action than predicted[S] — the marker was correctly reporting a real client/server disagreement that prediction did not cause and could not fix. The apparent correlation with input_lead was a confound: the conditions that raise the lead are the conditions that produce starves. Fixing the starvation cause (task 4.13 below) took the marker to 0.00% in all three conditions, including 80±20 ms and 5% loss where it had been 1.7–2.5%.
Two sub-findings from that investigation, recorded because both are counter-intuitive: dequantize_thrust_z_bin(quantize_thrust_z_bin(0.0)) returns 0.142857, not 0.0 (7 bins over [-1,1], roundi(3.5) == 4), so a server-reported thrust_z of 0.14 literally means "exactly zero" — the 0.26 threshold absorbs it, as designed. And _pending_local_reconciliation keeps only the newest snapshot, so acks are dropped whenever two snapshots land in one physics tick: the marker under-samples, and the true action-disagreement rate is higher than it reports.
The client-only shadow Jolt world is still the open question (item F of §0), but it is now scoped to the contact cohort alone. Even perfectly labelled, the client predicts contacts against remote ships and the ball sitting at interpolated-delayed positions, so a contact-cohort prediction cannot be sequence-correct in the live world — no amount of bookkeeping fixes that, and a shadow world is the only thing that does. It is a large subsystem and effectively the whole-world rollback §1's locked decisions set out to avoid, so do not build it before a playtest says the contact cohort actually reads badly to a human. Free flight no longer needs it.
New smoke role — --exercise-input-transitions. Toggles forward thrust every 6 physics ticks (~100 ms) with alternating yaw, and asserts the action marker stays under 5% mismatch over ≥200 samples. This is the only gate here that can catch a sequence-label regression, for the reason above, so it must not be folded into the steady-input free-flight run:
godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=host --drive-seconds=8
godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=client --drive-seconds=8 --exercise-input-transitions
Verified against a working control: reverting the one-line label makes this gate fail at 50.2% mismatch, so it is not vacuous.
Task 4.13 — two server-side input-death bugs the Phase 4 gates could not see
Both are Phase 3 code, both predate this session, and both were found by an adversarial review of the Phase 4 changes rather than by any gate. Neither was caused by 4.11/4.12; both are squarely in the way of Phase 4's feel milestone, so they are fixed here.
(a) A starve stranded the input stream one sequence ahead of arrivals — permanently. InputJitterBuffer.consume() set last_applied_seq = expected on every tick, including a starve. Because ingest() discards anything seq <= last_applied_seq, a single starve on a sequence the client had not sent yet left the server permanently one ahead: both sides then advance one per tick, the gap never closes, and every honest packet is discarded on arrival. The client's own input_lead RELEASE (delta == 0, which deliberately issues no new sequence for one tick) is sufficient to trigger it — so this fired roughly every 6.5 seconds of ordinary play on a clean LAN, blacking out input for 30 ticks until the lead controller's MIN_CHANGE_INTERVAL_TICKS debounce permitted a +3 attack to jump the client clear. The reviewer measured 2 blackouts in a 23 s run and 2 in a 30 s run, with the host applying the same repeated action for 30 consecutive ticks while the wire carried fresh input every one of them. Fixed by only giving up on expected when strictly newer data has arrived, which proves it lost rather than merely late. Both escape paths are untouched: a silent client still zeroes and stalls on STARVE_ZERO_TICKS, and a far-behind consumer still hits the ring-overflow resync.
(b) The seq-range guard was a one-way door. _on_input_received bounded incoming seq against jb.highest_ingested_seq + RING_SIZE — but highest_ingested_seq only ever advances inside ingest(), which that same guard gates. Once a client's live sequence got more than 32 ahead (a host stall drops the intervening packets wholesale, since input is unreliable), every subsequent packet was rejected, the bound could never move again, and that player's input was dead for the rest of the match with no diagnostic. Reproduced with a 2 s SIGSTOP host freeze: 600+ consecutive rejections, the server applying zero thrust across 1300 sequences while the client's wire carried full thrust throughout. This is the third iteration of this guard, and the structural lesson is that each previous version bounded against a value only the accepted path could advance. Fixed by keeping the bound but adding an escape: after SEQ_REJECT_RESYNC_LIMIT (10) consecutive rejections, accept and let the existing resync machinery re-establish the baseline. This grants an attacker nothing — walking the epoch forward by sustained rejection costs the same packets as walking it forward by acceptance, and §3.4's rate limiter already bounds that rate.
(c) The gate printed PASS while input was permanently dead. The --exercise-input-transitions gate reported SMOKE PASS at 3.76% mismatch on a run where input was completely dead, because suppressed reconciliation stops calling _record_metrics — so the worse the outage, the fewer marker samples and the lower the reported mismatch rate. Every other assertion in that path (local_prediction_ok, moved > 1.0) reads the client's own action and position, which a client flying purely on prediction satisfies perfectly. Fixed by scaling the required sample count with run length (max(200, drive_seconds * 30), half of nominal 60 Hz) and asserting the wire's server_stalled bit. Verified non-vacuous: reverting both fixes and re-running the 3.5 s freeze fails at samples 292/600 with server_stalled=true and input_lead=12 (LEAD_MAX) — while reporting marker=1/292 = 0.34%, which the old gate would have passed.
QA matrix, re-run in full after 4.11 + 4.12 + 4.13 (all green): 72 unit tests; 60 s free-flight at LAN / 80±20 ms / 5% loss — p99 raw 0.141 / 0.168 / 0.154 m, exposed visual p99 0.000 m, 0 hard snaps in every condition, marker 0/3484, 0/3049, 0/3397; forced-input-transition gate at LAN, 80±20 ms and 5% loss, all 0.00%; 2.0 s and 3.5 s SIGSTOP host-freeze recovery; ball contact ×5; two-bot CI ×3; all three abuse roles; net_smoke, match_net_smoke (incl. host_recycle), clock_smoke, lobby_smoke.
Note the free-flight p99 improved (0.170/0.176/0.184 → 0.141/0.168/0.154) and input_lead now sits at 1 on LAN instead of oscillating to 3–4. Both are downstream of 4.13(a): the periodic blackouts were degrading prediction accuracy and driving the lead controller.
Two test defects fixed alongside, both pre-existing and both surfaced by 4.13(a):
- Ball-contact gate flaked 2 in 5.
ball_proxy_moved_before_authority_countrequires the predicted proxy to have visibly moved before the next authoritative ball state arrives — but snapshots land every ~16.7 ms at 60 Hz, so on a loopback LAN the entire pre-authority window is about one physics tick and observing it is a coin flip. It is also the least interesting case: the counter measures RTT-masking, and LAN has no RTT to mask. Measured 5/5 passes (count 2–3) at--net-sim-latency=80. Now asserted only whenNetworkManager.rtt_ms >= 20, with the same-frame reveal — the test's real claim, correct in every run either way — carrying the gate on LAN. Runs asserting the masking behaviour should pass--net-sim-latency. - Two-bot CI compared scores across a 3–5 s window. The host checked each client's recorded score against its own score at read time, but clients write theirs several seconds earlier; any goal in between failed the run with both bots agreeing perfectly with each other. Latent until 4.13(a) made the bots effective enough to reliably score a second goal — then it failed 2 of 3 runs, every failure
server=2vsboth clients=1. The host now polls and records every score it actually holds, and asserts both clients agree with each other and that what they saw is a state the server genuinely passed through. 3/3 green, including a run ending 1–1 where the clients had recorded 0–1. (Polling, notscore_changed: that signal is emitted only in_on_score_update_received, the client path — the server mutatesscoredirectly in_record_goaland never emits. Connecting to it recorded nothing but the initial 0–0.)
Follow-up, not done:
LocalNetShipController.last_applied_seqis now write-only andLocalInputTimeline.consume()is vestigial to the reconciler (still unit-tested, still advancing_last_applied_action, but nothing reads the result). Left in place rather than removed as unreviewed scope — but it now looks load-bearing and is not.
Phase 5 — Match lifecycle
| # | Task | Acceptance |
|---|---|---|
5.1 [D:2.1] |
DONE. scripts/match_state.gd (enum + validated transition table, pure/unit-testable), server-driven machine in NetworkedMatch, state_change RPC on reliable channel 0 carrying an absolute at_tick, and the snapshot match_state byte populated for real |
Client observed LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP with monotonic ticks in a real two-process run; every consecutive pair legal; wire byte asserted independently of the RPC |
5.2 [D:5.1] |
DONE. _end_tick/_clock_running, clock_state RPC, timer_updated emitted from absolute ticks on both peers; goal pause shifts end_tick rather than pausing anything |
No Timer and no _process polling remain in the networked path; both peers derive remaining = end_tick - now from the same server-tick estimate |
5.3 [D:5.1] |
DONE. kickoff RPC carrying resulting transforms (never a seed, per §1), deferred freeze, reset_gen bump, countdown from server_tick, late-arrival skip |
Real two-process run: LOADING -> WARMUP -> PLAYING, countdown ticks match WARMUP_TICKS exactly; a kickoff past its own resume tick unfreezes immediately and emits 0 |
5.4 [D:5.1] |
DONE. goal_scored(scoring_team, score, goal_tick, resume_tick), freeze on the goal tick, reset moved out of the sensor path into the kickoff at resume_tick; cinematic is presentation-only |
PLAYING -> GOAL_PAUSE -> WARMUP -> PLAYING observed on the client; bodies stay where the goal left them for the whole window; Engine.time_scale untouched |
5.5 [D:5.1] [P] |
DONE. Clock expiry -> FULL_TIME -> sudden death on a draw or RESULTS, golden goal in overtime, then LOBBY on both peers. get_tree().paused is never used in the networked path |
Full run observed end to end: LOADING -> WARMUP -> PLAYING -> FULL_TIME -> OVERTIME_WARMUP -> OVERTIME -> GOAL_PAUSE -> RESULTS -> LOBBY, both peers returning to the lobby scene |
5.6 [D:5.1] [P] |
DONE. Controller swap on disconnect (ship never despawned), 30 s identity-keyed reservation, reclaim on reconnect, --fill-bots/--no-fill-bots, stalled set immediately for the nameplate |
Real 3-process run: ship survived, controller valid, slot reserved, reclaimed by name, same ship instance |
5.7 [D:5.6] |
DONE. _swap_slot_controller() rebinds in the same transaction; slot.controller retyped to the base ShipController; every use is_instance_valid-guarded |
The disconnect test caught the real bug: the narrower RLShipController type made the swap assignment fail, leaving a freed reference |
5.8 [D:5.1] [P] |
DONE. A slotless peer spectates (no ship spawned, same snapshot stream), HUDController.spectator_mode keeps clock/score/celebration and hides only the ship cluster, camera cycles ships then ball, --max-spectators cap. §6.3's "take the slot at the next kickoff" is now implemented, not just printed — the line claiming it was there from the start while _is_spectator was assigned once and never revisited (see the Phase 5 note below) |
Spectator path exercised by the mid-match joiner; HUD no longer push_errors and bails with a dead HUD; promotion verified 4/4 from both sides, with a control proving §6.4's reservation outranks the queue |
5.9 [D:5.3] [P] |
DONE. New GameMode._on_bodies_respawned() virtual; NetworkedMatch bumps reset_gen through Phase 2's deferred path so the bump and the respawned pose land in the same broadcast |
Single-player modes unaffected (base is a no-op) |
5.10 [D:5.1] [P] |
DONE. scripts/replay_log.gd, --replay-log=<path>, storing wire bytes verbatim in both directions — plus, after a review found three recording gaps, REJECTED packets with their reason in the kind byte (capped per window so the log cannot become a remote disk-fill amplifier), a failed write that ends the log instead of desyncing its framing, an explicit close() with a summary, and tools/replay_dump.gd to read one back. The reject recording immediately found a real bug: the server was rate-limiting a stall backlog it had caused itself, losing 8.88% of a player's input |
Live 6 s match recorded 1115 records (557 inputs / 558 snapshots); a stored snapshot decodes back to server_tick=100 match_state=WARMUP bodies=2; 6 unit tests incl. truncation and foreign-file rejection |
Ship.set_controller(ship.gd:213-218) callsqueue_free()on the outgoing controller. Task 5.7 exists because the takeover path in 5.6 otherwise leavesMatchNetholding a freed reference — the exact class of bug that surfaces as a random server crash weeks later.
Task 5.10 is the highest-value debuggability investment here. The packets are already flat bytes, so it is ~50 lines. Without it, "my ship snapped" is permanently unreproducible from a field report — the CI gate catches regressions, but it cannot debug a player's bad night.
Task 5.1 notes
scripts/match_state.gd holds the enum and the §6.1 transition table as pure data with no scene/RPC dependency — the same reason net_codec.gd and input_jitter_buffer.gd are standalone — so the table is checked exhaustively (every state reachable, every state has an exit, no self-transitions, abort-to-LOBBY from anywhere, illegal shortcuts rejected) rather than by example. The enum's integer values are the wire format, pinned by a test: match_state has been a u8 in the snapshot header since §2.4, so renumbering an existing state silently reinterprets packets from an older peer. Only append.
The server validates every transition and push_errors an illegal one rather than following it, because the symptom otherwise — clients faithfully following into a state the server's own code never meant to reach — is near-impossible to diagnose from a field report.
Two channels carry the state, deliberately. state_change (reliable, channel 0) is prompt and carries the absolute at_tick; the snapshot's match_state byte is the catch-up path for a client that has not been sent a transition yet — a late joiner (§6.3), or the window between scene load and the first RPC. The byte needs a tick guard: snapshots are unreliable_ordered on channel 2 and ordering holds only within a channel, so a state_change for tick N routinely arrives before an in-flight snapshot from tick N-2. Without the guard the client applies the new state and is immediately dragged back by the older byte, oscillating on every transition — observed directly (LOADING -> WARMUP -> LOBBY -> PLAYING -> LOBBY -> ...) while running a deliberately-broken-byte control. Only a byte at least as new as match_state_since_tick is accepted.
The client deliberately does not enforce the transition table — authoritative state must be accepted, and a late joiner legitimately jumps straight to PLAYING. The table is a server-side invariant. The smoke test asserts legality of what the client observes, seeding its first sample from whatever state the client converged to rather than counting that as a transition, so late-loading clients (seen seeding at WARMUP rather than LOADING) still pass.
5.1 does not gate physics, freezing or input on state. Tasks 5.3 and 5.4 own freeze/unfreeze at kickoff and goal; doing it here would both duplicate that work and change the conditions every Phase 4 prediction gate was measured under. MatchState.is_live() exists for them to use. WARMUP_TICKS/GOAL_PAUSE_TICKS are honest placeholders so 5.1 drives real transitions to verify against — 5.3 replaces the first with the broadcast kickoff (reset transforms + countdown from server_tick), 5.4 the second with _goal_pause_seconds() and the client-cinematic split. The server also leaves LOADING immediately rather than waiting for scene_ready, which does not exist yet (5.3).
New smoke flag --exercise-match-state (pass to both roles — the host forces a goal to drive a GOAL_PAUSE cycle, the client records and validates the sequence):
godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=host --drive-seconds=6 --exercise-match-state
godot --headless --path Game res://tests/networked_match_smoke.tscn -- --role=client --drive-seconds=6 --exercise-match-state
Verified against a control: hardcoding the snapshot byte back to 0 fails both the byte assertion and the transition-legality assertion. That control is why the gate asserts the wire byte separately from the RPC at all — everything else in the check is RPC-driven and would pass identically with a dead byte, which is exactly how Phase 4's mislabelled history survived every gate (gotcha 47).
Phase 5 notes
Task ordering caught three ordering bugs of the same shape, all found by a failing run rather than by review, and all worth remembering as a class: a value consumed by one per-tick updater and cleared by another is order-dependent. _update_kickoff_countdown() clears the _kickoff_resume_tick that _update_match_state() reads to leave WARMUP (match froze forever); _apply_match_state() resets _state_deadline_tick on every transition, so a GOAL_PAUSE deadline assigned before _set_match_state was wiped (match never resumed); and a set_deferred("freeze", true) landed before the queued kickoff teleport could apply, stranding every body where the goal left it.
Freezing is asymmetric between server and client, and this is not optional. On the server every body is a real dynamic simulation and all of them freeze. On a client, freeze is already load-bearing for something else: remote ships and the ball are permanently FREEZE_MODE_KINEMATIC and driven by transform writes, with only the local ship unfrozen for prediction. Freezing "all bodies" on a client therefore unfreezes the remote ones on the way back out — they fall under gravity while the interpolator fights them for the transform. Measured: 210 hard snaps and an infinite p99. A client freezes only the one body it actually simulates.
Prediction is suspended while the match is not live. During a countdown or goal pause the local ship is frozen on both peers, so there is nothing to predict — but the reconciler still ran delta transport and visual-offset maths over those frozen states and produced a p95 position error of 2.4e10 m while the instantaneous error stayed small. Input keeps flowing so the server's jitter buffer does not starve into stalled.
§6.4's two rules conflict and the reservation has to win. "Reserve a departed player's slot for 30 s" and "abort to the lobby once the last human leaves" applied naively means the abort fires instantly in a 1v1 — the moment the only player drops, the match is torn down and the reservation can never be redeemed, making the reconnect path unreachable exactly when it matters (one player whose connection blipped). Abort now waits until nobody is connected and no reservation is outstanding.
Task 5.7's bug was real and the test found it. SlotInfo.controller was declared RLShipController, but §6.4's takeover swaps in an AIShipController or the base controller — a narrower declared type makes that assignment fail its type check, leaving the field pointing at the controller set_controller() just queue_free()d. It surfaced as controller_valid=false on the first disconnect run. The per-tick slot.controller.action write is now also gated on is RLShipController: a disconnected slot's bot drives itself, and overwriting its action from a permanently-starving buffer would pin it to the departed player's last input.
--check-only --script is the only thing that catches a parse error in networked_match.gd. The unit runner never loads it, so bot_model_path being undefined (and later ReplayLog being unregistered) both passed 81/87 unit tests while breaking every two-process run. Validate touched scripts directly. A newly added class_name also needs godot --headless --path Game --import before anything can resolve it — and the same --import is the fix when a previously working class_name stops resolving, which happens on its own: .godot/global_script_class_cache.cfg silently lost MatchState between sessions, and every two-process run then died with Cannot infer the type of "live" variable at the MatchState.is_live() call, with nothing in git status to explain it. Read that error as "the class cache is stale", not "the code is wrong".
The reviewer's p95 0.688 was real, and the three-process framing was a red herring — mine as much as the reviewer's. The report was "a 3-process run failed the free-flight gate at p95 0.688 (bar 0.5) with roughly a third of snapshots missing", so the first investigation compared process counts: two-process p95/p99 0.084/0.098, idle third process 0.084/0.094, spectator 0.084–0.098 / 0.094–0.146 over four runs, and 0.0% snapshot loss even under deliberate 2x CPU oversubscription (20 spinners on 10 cores, where only snapshot_age moved, 14ms → 32.3ms). Every one of those runs passed, so the conclusion recorded here was "not reproducible". That conclusion was wrong, and it was wrong because every probe used --exercise-free-flight — the one mode the 0.5 bound was calibrated on.
It reproduces on two processes, on an idle machine, with 0.0% snapshot loss: the plain --role=client drive fails the free-flight gate roughly a third of the time. Eight plain-role runs measured a free-flight cohort of 12–257 samples with p95 0.275–0.726, failing the 0.5 bound in 3 of 8. The harness's own _run_free_flight_trace comment had already said why — "a straight forward trace reaches the goal/wall in seconds and turns the supposed free-flight QA run into a contact test" — but the plain role went on asserting the open-volume bound against whatever free-flight samples that contact-heavy drive happened to leave behind, sometimes as few as 12.
The underlying difference is not noise. Prediction error near the arena's surface-pull field is genuinely several times higher than in open air: the same build measures 0.084–0.111 under --exercise-free-flight and 0.275–0.726 on the plain drive. Both are honest numbers about different flight profiles, and one bound cannot serve both. --exercise-free-flight keeps the calibrated 0.5/2.0 gate (~5x margin). The plain role now asserts the all-cohort percentiles instead — always well-sampled (545–696, versus a free-flight cohort that can collapse to 12) and much tighter in spread (raw_p95 0.354–0.609, raw_p99 0.362–0.742) — at 1.2/2.0, ~2x above the worst observed, and prints the free-flight numbers explicitly marked reported, not asserted. free_flight_hard_snaps == 0 is still asserted in both modes, and anything past 2.0m is a hard snap by definition, so a genuine free-flight regression cannot hide behind the looser bound. Verified: 6/6 plain-role runs pass where 3/7 previously failed, all four other modes (free-flight, 80±20ms latency, input transitions, ball contact, match state) still pass, and tightening the new bound to 0.3 makes it fail — the gate is evaluated, not skipped.
The other durable improvement from the first investigation still stands: a percentile alone cannot distinguish "the predictor regressed" from "the client never received the data", so the client gate prints snapshot_loss / snapshot_age / rtt on every run and, on a quality failure with >20% loss, says explicitly that the run was transport-starved — without converting the failure into a pass. Both directions verified non-vacuously. It is also what proved the 0.688 was not transport: every reproduction reported 0.0% loss.
Lesson worth more than the fix: probing only with the purpose-built mode is how a flaky gate stays invisible. The first pass ran eight variations of process count and CPU load and never once ran the plain role that the reviewer had actually run.
Task 5.10's three recording gaps, and the real bug closing them found. The review flagged that the replay log ignored store_* failures, never recorded the packets the server rejected, and had no caller for close(). All three are fixed: a failed write now ends the log permanently rather than desyncing every later record's framing (write_failed, checked via FileAccess.get_error() once per record); close() is called from _exit_tree with a summary line, because letting the RefCounted's destructor do it implicitly never tells anyone whether the log is complete; and rejected packets are recorded with their reason in the kind byte (REJECTED_MALFORMED / REJECTED_RATE_LIMIT / REJECTED_SEQ_GUARD, framing unchanged, FORMAT_VERSION 2 so "no rejects" can be told from "this build never recorded them"). Recording is capped at 8 per peer per rate-limit window — without that cap the diagnostic is a remote disk-fill amplifier, since the attacker chooses the packet rate. Verified end to end: an honest client logs 0 rejects; client-abuse-malformed sends 25 and logs exactly 8; client-abuse-flood sustains ~2400 packets/s and logs exactly 8. Uncapped totals are kept separately (MatchSim.get_reject_totals()) and survive the peer's disconnect — the first version stored them on _PeerInputState, which is erased on disconnect, so every summary printed an empty dictionary.
And the bug the recording immediately found: the server rate-limited a backlog it caused itself. A 2s host stall (SIGSTOP, standing in for a GC/IO/scheduler hitch) has the client sending at 60Hz throughout, and ENet delivers the whole backlog in the first window after resume — 70 of an honest client's input packets rejected as "rate limit exceeded", against a limit that client never came close to violating. Redundancy does not cover it, and that was the assumption worth checking rather than asserting: the dropped packets are contiguous, so each one's redundancy window falls inside the same dropped run. Measured with the new log: 0 of 70 rescued, and 82 of 923 sequences (8.88%, ~1.4s of that player's input) never reached the server at all, versus 0.00% on an otherwise identical run with no stall. Every prediction gate still passed — this is the same class as the Phase 3/4 input-death bugs, invisible to every gate that reads only the client's own state.
Fixed by not policing a backlog the server caused: MatchSim._physics_process watches for a wall-clock gap over STALL_DETECT_MS (a stalled process doesn't run that callback either, so the first frame after the stall sees the whole gap, which is exactly the size of the backlog about to arrive) and grants each already-tracked peer a capped, two-window packet grace. The leaky bucket drains against the same graced budget, or a stall would still accumulate excess toward a disconnect for traffic the server just explicitly allowed. Results: 2s stall, rate-limit rejects 70 → 0, sequences missing 8.88% → 0.00%, and REJECTED_SEQ_GUARD 9 → 0 as a second-order confirmation (the guard was firing partly because the dropped backlog let the client's epoch run away). Across eight stall runs on the fixed build, 7 measured 0.00% missing; the eighth measured 23.54% with zero rate-limit rejects and the seq-guard resync visibly doing its job — a separate, occasional transport-level loss during the stall that this change does not address and does not make worse (item D of §0). The three control runs on the unfixed build lost 4.34%, 7.52% and 7.86%, every time.
Abuse detection is unweakened and this was checked rather than argued: all three abuse roles still disconnect, and no flood induced a server stall in any run, so the grace cannot be farmed by flooding. An attacker who can induce server stalls to earn budget already has a strictly worse capability than sending extra input packets.
§6.3's "spectate now, take the slot at the next kickoff" was a print statement, not a feature. The server logged "joined mid-match; spectating until the next kickoff" and then never did anything about it; on the client, _is_spectator was assigned once during _on_match_config_received and never revisited — and that handler returns early whenever _slots is non-empty, so no rebroadcast could ever promote an in-match spectator. The reconnect path only worked because a returning player is a fresh process that runs _on_match_config_received from scratch.
Implemented on both sides. The server queues late joiners in arrival order and drains the queue from _begin_kickoff() — before the reset transforms are read, so a promoted player's ship is placed by that same kickoff instead of being left wherever its previous owner abandoned it, and the controller swap lands on an already-frozen body, which is the entire reason §6.3 puts this at a kickoff boundary. A slot only becomes available once its player has gone and their 30s reservation has lapsed: §6.4 outranks §6.3, because taking a still-reserved slot would quietly break the reconnect promise. _abort_if_abandoned now counts a waiting spectator as somebody still present, for the same reason it already counts an outstanding reservation — otherwise the one person queued for the slot that just opened gets dumped to the lobby at the exact moment they were about to receive it.
The client gets a new broadcast slot_assigned (reliable, channel 0). Broadcast rather than addressed to the new owner, because every client holds its own slot list and one that names the wrong peer keeps flying somebody else's ship as a remote body; reliable, because unlike match_state there is no per-snapshot field that would re-converge a client that missed it. The promoted client undoes everything that made that body remote — fresh interpolator (its buffered samples describe the previous owner's flight), Godot's own physics interpolation switched back on, visual offsets cleared — and then deliberately does not unfreeze: it clears _local_prediction_ready so the next snapshot teleports it to a genuine authoritative pose and starts prediction there, exactly as a fresh client does. The controller-attach block was factored out of _on_match_config_received into _take_local_ownership() rather than copied, since a copy is a copy that drifts.
New --role=host-latejoin / --role=client-latejoin and --slot-reservation-seconds= (a server-side override in the same shape as --match-length, because the interesting moment is otherwise 30 real seconds away). Verified 4/4 from both sides: the joiner is queued, is not promoted merely because the reservation lapsed, takes the slot at the forced goal's kickoff, keeps the same ship instance, and both peers independently measure ~45.7m of movement under its input — the client's own number and the server's agree, so the promoted seat is real rather than relabelled. Control with a 90s reservation: the kickoff fires and nothing is promoted, the slot still reads the departed player's name, and the joiner stays a spectator. The existing spectator test is a second control — a spectator with no free slot is never promoted.
Two test-side races were fixed while getting there, both worth remembering because they produced confident false failures: sampling predicting at an arbitrary frame reported false for a client that then flew 45m, because unfreezing is queued and applied on the body's next _integrate_forces (task 0.15), so there is a real window where the state is PLAYING and _local_prediction_ready is set but ship.freeze has not flipped yet. Poll the whole condition with a deadline, never a proxy signal, and never one instant.
§6.4's reconnect was only ever graded from the server's side, and the client's side was failing the whole time. run_disconnect_host_check ticked 60 physics frames (1.0s) past the reclaim and then shut the server down — so the reconnecting client, whose wiring check waits a 2.0s settle before it looks at anything, had its peer torn out from under it every single run and reported current_scene is not NetworkedMatch after 2.0s. The host printed PASS throughout, and the host was the side anyone read. The hold is now a real window (default 8s), and the host additionally asserts that the reconnected player's input reaches the server and moves the ship the server owns — every other assertion there is slot bookkeeping that would hold identically for a client whose input pipeline came back dead, which is the exact failure the reservation exists to prevent. Both the position and the connection state are sampled while the peer is still connected, not once at the end of the hold: the client leaves on its own schedule, and an end-of-hold sample reported still_connected=false for a perfectly good run — the same mis-timed sampling a Phase 3 review caught in the CI gate.
New --role=client-reconnect grades the returning player: not a spectator, owns a slot whose peer_id is its own, has a real ship, rejoined a live match with the clock already known (_end_tick >= 0 — §6.2 step 2's bootstrap, since a player who must wait for the next goal to learn the score has not really rejoined), and its input still moves its ship. That set is chosen because a stale _last_match_config once made a reconnecting player a spectator, and that bug was visible in this scenario's own logs while it reported PASS. Verified 3/3 both sides, with a control that rejoins while the slot is still occupied and correctly fails on is_player=false. The first version of that control failed with the generic "lost its ship mid-drive", so the spectator case is now reported before the drive rather than after.
tools/replay_dump.gd reads a log back — record counts by kind, plus how much of the input sequence stream actually reached the server once redundancy is counted. It is committed rather than left in a scratch directory because it is what turned "the server dropped some input" into the numbers above, and a log nobody can read is half a feature.
New/changed test surface: --exercise-match-state (both roles; host forces a goal, client validates the whole observed sequence and the wire byte), --role=host-disconnect for the 5.6/5.7 three-process scenario (paired with --role=client-reconnect, which grades the returning player), --role=host-latejoin/--role=client-latejoin plus --slot-reservation-seconds=<s> for §6.3's kickoff promotion, --match-length=<s> to reach FULL_TIME in a short run, --replay-log=<path>, --fill-bots/--no-fill-bots, --max-spectators=<n>. The ball-contact scenario now steers at the ball with closed-loop real input instead of a hand-tuned fixed-heading burst, which 5.3 broke by adding KICKOFF_YAW_JITTER (0 contacts in 3/3 runs); it thrusts while turning rather than hovering to aim, which took it from 2/3 to 5/5.
Phase gate: a full 3v3 start-to-finish including a mid-match disconnect and a late joiner. Not yet run — every scenario above was verified at 1v1 (plus a two-bot CI match). The 3v3 gate needs a real multi-client session; it is item B of §0, alongside Phase 4's un-run human playtest (item A).
Phase 6 — Dedicated server productionisation
| # | Task | Acceptance |
|---|---|---|
6.1 [P] |
DONE. Export preset (dedicated_server=true, custom_features="dedicated_server") and run/main_scene.dedicated_server, mirroring the existing run/main_scene.training mechanism |
Linux Dedicated Server builds |
6.2 [D:6.1] |
DONE. Verify the stripped export boots and scores a goal | Docker smoke runs two exported-server matches and observes server-owned goals from two headless clients |
6.3 [P] |
DONE. Full CLI surface plus a config-file fallback | Unit tests cover precedence, validation, and --help |
6.4 [P] |
DONE. Structured logging (join, leave, goal, kick, rate-limit, tick overrun) with --log-level |
Greppable stdout/stderr events exercised in the smoke |
6.5 [P] |
DONE. Arena rotation between matches; --max-matches N drain-and-exit |
Smoke asserts two different arenas and server_draining |
6.6 [P] |
DONE. systemd unit, Dockerfile, SERVER.md (ports, firewall, sizing per §1.4, and the SIGTERM caveat) |
A third party can host from the docs alone |
6.7 [D:3.6] [P] |
DONE. CI builds the server export and runs the smoke test against the exported binary, not source | .github/workflows/dedicated-server-smoke.yml runs make verify-phase6 on clean checkout |
dedicated_server=trueenables Godot's strip-visuals export mode, which replaces meshes and textures with placeholders per resource. Every relevant site is already headless-guarded —ship.gd:167,ball.gd:25,goal.gd,arena_boundary.gd— so the code should be safe. Verify it against a real stripped build anyway; this is the kind of thing that fails silently.
Docker/VPS is the primary v1 deployment path. Raw ENet self-hosting needs port forwarding, and SDR is Phase 7 — so Phases 1–6 ship something that works on LAN or a VPS and nowhere else. That is fine, but say it out loud rather than letting a player discover it.
Godot 4 gives GDScript no SIGTERM hook.
SIGTERM/Ctrl-Ckills the process immediately and clients see an ENet timeout (~5 s). Acceptable — but document it rather than letting it be discovered.--max-matches Nunder a process supervisor covers planned drains.
Rcon is deferred past v1. An authenticated remote command channel is a real security surface, and
--max-matchesplus a supervisor covers most of the need with none of it.
Phase gate: docker run a server, connect from another machine over the internet, play a full match. Precondition, not a footnote: §0 item C — slot reservations keyed on display name alone — is fixed by task 7.4, so exposing this build to strangers is gated on that, not on this phase.
Phase 7 — Steam transport, browser, identity
| # | Task | Acceptance |
|---|---|---|
7.1 [D:1.2] |
IN PROGRESS. GodotSteam integration and custom export templates — client and headless server | Pinned build inputs and the reproducible validation command are documented; awaiting the custom binaries/SDK access |
7.2 [D:7.1] |
IN PROGRESS. NetTransport boundary extracted with ENet and feature-gated steam_transport.gd (SteamMultiplayerPeer, SDR); advertising waits for ISteamGameServer work |
NetworkManager.host/join(..., transport) selects explicitly; stock builds reject Steam without ENet fallback |
7.3 [D:7.2] [P] |
IN PROGRESS. Server-browser UI and ISteamMatchmakingServers adapter remain intentionally unimplemented until the pinned GodotSteam client API is available; ENet direct-IP remains the supported browser-free path |
No server_browser.tscn or fake Steam API has been added; implementation must wait for real Steam SDK/API access so Internet/LAN/favourites/history behavior can be verified against the actual service |
7.4 [D:7.2] [P] |
IN PROGRESS. TicketVerifier now supports a synchronized backend ban decision before single-use ticket consumption; auth tickets in hello → BeginAuthSession, Steam identity in the roster and persistent ban list remain |
server/domain/auth.go and adversarial tests reject banned identities without consuming their ticket and allow a later verification after unban; GodotSteam auth integration, server-side VAC state and durable ban storage remain |
7.5 [D:7.2] [P] |
IN PROGRESS. SteamBootstrap gates initialization on the steam feature, SteamMultiplayerPeer class and Steam singleton; explicit Steam selection fails closed, while ENet remains the default and never becomes an implicit fallback |
test_net_transport.gd proves stock builds keep ENet available and reject unavailable Steam requests without returning an ENet peer; custom Steam client/server export smoke remains blocked on pinned GodotSteam binaries, and full ENet runtime verification remains blocked on the absent Godot executable |
7.6 [D:7.4] |
IN PROGRESS. Pure Go AuthCoordinator models pending/accepted/rejected/cancelled backend auth sessions around the single-use verifier, including maintenance expiry sweeping; store.PostgresSessions persists only token digests; the authenticated API issues sessions only from an injected verified-identity provider; Godot ControlPlaneClient.login_steam() now submits only the Web API ticket, validates the opaque response and stores the session in memory |
server/domain/auth.go, server/store/session_sql.go, server/api/service.go, control_plane_client.gd and adversarial tests keep identity unavailable while pending, reject cancellation/expiry/wrong-attempt/replay, expire abandoned attempts at the boundary, release identity only after verifier success, reject invalid session inputs/extra identity fields, prove API delegation, validate ticket/session header boundaries and issue opaque sessions; real Steam BeginAuthSession/EndAuthSession adapter, login UI and live PostgreSQL/session integration remain |
7.7 [D:7.1] [P] |
Obtain the production App ID, publisher key, SDR coordinator SDK/signing approval, certificates and hosted-data-centre support from Valve | Production prerequisites and rotation owners are recorded; no Spacewar credential or development certificate can reach a release build |
7.8 [D:7.6,7.7] |
Ticketed Hosted Dedicated Server SDR: routing registration, coordinator-issued player→server relay tickets, client ticket installation, reconnect and expiry | Two real accounts complete and reconnect to an assigned dedicated match through SDR; server/player IPs are not exposed; ENet gates remain green |
The transport interface is written here, not in Phase 1. Eight virtual methods (
begin_auth,advertise,get_identity,supports_server_browser…) designed against an API nobody on the project has used will be wrong. WriteNetworkManager._make_peer()concretely in Phase 1 and extract the boundary once there are two real implementations. Locked decision 3 guarantees the ENet path is never deleted, so there is no migration risk in waiting.
GodotSteam requires custom engine builds and export templates — including for the headless server. That is the part people discover three weeks in. Budget for it.
Phase 8 — Matchmaking, ranked ladder, per-match server autoscaling
1.0 launch blocker. Full design and reasoning: docs/MATCHMAKING.md.
The local control-plane, durable-store, allocated-server, and verification
paths are substantially implemented; the per-row status below distinguishes
that evidence from the remaining live Steam, PostgreSQL/Redis, Agones, release,
and human-playtest gates. Unlike Phases 0–7 this phase adds a component outside
the Godot project — a backend service — and that is the largest architectural
departure in the project's history, so read the design doc before picking up
any task below.
This inverts the server model. Phases 1–7 build a community server: it
runs forever, waits for --min-players, plays a match, rotates arena, repeats,
and players find it by IP or (7.3) the server browser. Matchmaking makes the
player durable instead — queue, get grouped by rating, and a server is
allocated for that one match and destroyed after. Both models ship; they
are different playlists, not a replacement.
Hard dependency on 7.6 and 7.8. Slot reclaim is keyed by display name today. A rating attached to a spoofable identity is farmed trivially, so no queue ships before single-use verified identity lands. Production allocation also depends on the ticketed Hosted Dedicated Server SDR route; ENet remains the local/CI/community transport, not a silent production fallback.
8A — Architecture, contracts and data
| # | Task | Acceptance |
|---|---|---|
| 8.1 | DONE. Add an ADR locking Go + PostgreSQL + Redis, provider-portable Kubernetes, Agones, ticketed Hosted Dedicated Server SDR, EU/NA fleets and independently runnable API, matcher, allocator and maintenance roles; keep README.md/docs/TECH_STACK.md consistent |
docs/ADR-001-matchmaking-platform.md names boundaries/rejected alternatives; current docs name the locked stack and replaceable provider; no application code calls a provider allocation API |
8.2 [D:8.1] |
DONE. Encode the launch SLOs from docs/MATCHMAKING.md: RTT, allocation/connect latency, 99.9% allocation/result success, API latency and tick health |
docs/MATCHMAKING-SLOs.md defines each metric, denominator, percentile/window, owner, alert threshold and release evidence |
8.3 [D:8.1] |
DONE. Publish versioned OpenAPI + WebSocket contracts for Steam login/session, profile/rating, queue create/heartbeat/cancel/resume, proposal accept/decline, assignment/status, server registration/roster/result/shutdown | server/contracts/v1/ contains machine-readable REST/events contracts and dependency-free structural tests; REST resync is specified by the contract; server/api/service.go also exposes the documented /api/v1 route names (including server-assigned idempotent queue ticket IDs and DELETE cancellation) alongside the existing client /v1 routes, covered by TestDocumentedContractRoutesAdaptToServiceAPI |
8.4 [D:8.3] |
DONE. Define opaque IDs, legal queue/match state transitions, revisions and idempotency keys | server/contracts/v1/state-transitions.json locks terminal states, legal edges, stale-revision handling and same-key replay/conflict behavior; contract tests cover the invariants |
8.5 [D:8.4] |
IN PROGRESS. Initial PostgreSQL migration now defines durable idempotency keys, queue ownership/active-participation fencing, identities, sessions/revocations, ranked seasons, ratings/events, matches/participants, penalties, results, audits and outbox; follow-up migrations persist server-derived queue probe RTT metadata, the allocator GameServer/allocation registry, matcher-selected proposal region/protocol/team/slot plans, leased allocating-match claims, and optional shared regional allocation quotas | server/migrations/0001_initial.sql, 0003_queue_probe_metadata.sql, 0004_allocator_registry.sql, 0005_proposal_match_plans.sql, 0006_match_allocation_claims.sql, 0007_allocation_quotas.sql, migrations/runner.go, cmd/migrate and static checks cover the durable tables, uniqueness/check constraints, Redis-as-cache boundary and serialized forward migration recording; opt-in scripts/run_postgres_integration.sh now runs the runner and real queue/assignment ownership, idempotency, revision and expiry checks through pgx. migrations.Rollback now reverses N most-applied migrations via migrations/down/<version>.sql files (one per existing migration, dropping in FK-safe reverse order), wired into cmd/migrate --rollback=N, verified live: roll back to empty and reapply reaches the same schema; remaining serializable adapters and cache-loss repair remain |
8.6 [D:8.3,8.4] |
IN PROGRESS. Add allocated-mode ServerConfig compatibility fields as opt-in defaults |
ServerConfig now validates allocation mode, match/server IDs, playlist version, client build, future assignment expiry, image digest, transport and EU/NA region; server_boot.gd fails closed for the not-yet-wired Steam SDR transport, constrains allocated processes to one match, and emits allocation identity/transport in server_started; signed-authorisation admission, dynamic endpoint wiring and full manifest/runtime tests remain |
8B — Authentication and secure control plane
| # | Task | Acceptance |
|---|---|---|
8.7 [D:7.6,8.3] |
IN PROGRESS. Pure Go ticket policy binds the expected App ID and verified identity, rejects expiry/replay/wrong app/malformed tickets, and consumes each ticket once. cmd/control-plane now wires SessionIssuer: store.PostgresSessions{DB: db} (same discovery/fix pattern as §8.10's ResultSubmitter: the adapter already correctly implemented Issue, just wasn't wired, so /v1/session/steam 503'd even before considering whether SteamLogin — the real, still-correctly-unwired Steam blocker — was available) |
server/domain/auth.go covers single-use and binding invariants; real AuthenticateUserTicket backend adapter, bans, publisher secret store and Steam verification remain. A separate test-only server/cmd/testkit-api binary (never referenced by any Dockerfile/K8s manifest) substitutes a fake Steam login accepting any non-empty ticket, enabling §8.40's real Go+Postgres+Godot integration test |
8.8 [D:8.7] |
IN PROGRESS. Pure Go session policy issues opaque short-lived tokens, stores only digests, authenticates by verified player identity and supports revocation; the production control-plane uses bounded atomic account+IP request limits | server/domain/auth.go covers wrong-token/expiry/revocation behavior; server/store/session_sql.go provides durable digest/revocation persistence and server/api/rate_limit.go plus cmd/control-plane provide per-replica request limiting; distributed revocation coordination and live Steam/session integration remain |
8.9 [D:8.4,8.7] |
IN PROGRESS. Pure Go join policy binds SteamID/player/match/server/team/slot/protocol/expiry, rejects duplicate roster slots, permits same-identity reclaim through backend loss and fences prior server-owned generations | server/domain/reconnect.go covers SteamID/server/slot binding, expiry, repeated reclaim, grace boundary and old-generation fencing; signed token issuance/verification, persistent leases, Godot hello transport and production integration remain |
8.10 [D:8.5,8.31] |
IN PROGRESS. Dependency-free workload credential policy validates an adapter-verified signature, issuer/audience/time bounds, namespace/service account, pod UID, GameServer UID, allocation ID, match ID and server ID before result submission; server/workload now parses projected compact JWTs, rejects unsigned/malformed/ambiguous tokens, verifies signing input through an injected trust callback, and delegates exact binding to the domain policy; the API exposes the workload-authenticated server result route. WorkloadVerify is now wired for real -- the blocker this row previously named (cmd/control-plane/main.go never wiring it, so /register and /result 503'd on every real request) is closed by taking a different, equally-valid approach instead of the Kubernetes-JWT one: serverMutation only ever compares WorkloadBinding.ServerID/.MatchID (and AdvanceServerRegistration only additionally needs .AllocationID) -- nothing downstream requires Namespace/ServiceAcct/PodUID/GameServerUID, so a Kubernetes-issued token was never actually required, just the assumption that it was. server/workload/signed_token.go mints and verifies a short-lived HMAC-signed token with a secret only the control plane holds (the same trust model domain.SessionStore already uses for player sessions), needing no cluster JWKS/TokenReview to validate -- signature and expiry are fully self-contained. The token binds only allocation_id (deliberately, not match_id/server_id too: it must be requestable in the same Agones request that asks for a server, before Agones has picked one) -- store.AllocationBindingByAllocationID resolves match_id/server_id durably from the allocations table at verify time, so a token can never claim a pairing that wasn't actually, durably allocated. api.WorkloadVerifierFromSignedToken combines both and is wired into cmd/control-plane (--workload-secret/COSMIC_CLASH_WORKLOAD_SECRET; a startup warning fires and the route stays 503 if it's left unset) and cmd/testkit-api (fixed test secret). The delivery channel is now wired too: agones.Client gains WorkloadSecret/WorkloadTokenTTL -- when set, Allocate mints a token for the allocation and requests it as a third cosmic-clash.io/workload-token annotation alongside match-id/allocation-id, wired from cmd/allocator's own --workload-secret/COSMIC_CLASH_WORKLOAD_SECRET (must match the control plane's); supervisor.Supervisor.workloadToken() resolves the bearer credential with an explicit --workload-token-path (kept for a possible future Kubernetes-JWT path) always winning, otherwise falling back to that same annotation -- the same fallback pattern matchID() already used for cosmic-clash.io/match-id. WorkloadTokenPath is accordingly no longer required at construction time |
server/domain/workload.go, server/workload/jwt.go, server/api/service.go and adversarial tests reject every binding mutation, missing/unverified signature, none/malformed JWT, ambiguous audience, server/match mismatch and time boundary; server/testkit/pipeline_test.go carries allocation identity through the offline result path; cmd/control-plane now wires ResultSubmitter: store.PostgresResults{DB: db} (a ready-made adapter that had been referenced from nowhere at all, not even a test); server/workload/signed_token_test.go covers round-trip, tampered payload, wrong secret, malformed input and the exact expiry boundary; server/api/workload_verifier_integration_test.go (opt-in, real PostgreSQL) covers acceptance against a real allocation row, rejection of an unrecorded allocation, that two distinct real allocations each resolve to their own and only their own match/server pairing, and the previously-503 Service.WorkloadVerify field itself now succeeding -- all verified clean with -race across multiple runs; cmd/control-plane/main_test.go's TestServerRoutesRequireWorkloadVerifyToBeWired now documents and pins the misconfigured-secret case specifically, not the "always unwired" case; server/agones/allocation_test.go's TestAllocateRequestsAWorkloadTokenAnnotationWhenConfigured and server/supervisor/supervisor_test.go's new annotation-fallback/fails-closed pair cover the delivery channel end to end (mint -> annotation -> supervisor read -> Authorization header) short of a live cluster. What's still missing: this has never run against a real Agones cluster (only HTTP-level fakes), so the exact object_meta JSON casing Agones actually returns remains unverified from this sandbox (see GameServer.ObjectMeta's existing caveat); live duplicate/conflict alerting also remains |
8.11 [D:8.1] |
DONE. Write the threat model: forged clients/replay/queues/results, floods/bots, pod/insider compromise, gameplay and API DDoS, SDR signing-key theft, dependencies and denial-of-wallet | docs/THREAT-MODEL.md records prevention, detection/response, owner and residual risk for every threat; it separates offline CA/online signer, client/game-server/PostgreSQL/Redis trust boundaries and recovery behavior |
8.12 [D:8.11] |
IN PROGRESS. Provider-neutral Kubernetes baseline enforces restricted namespace admission, non-root/read-only/no-capability workloads, separate service accounts, default-deny networking and explicit edge/data/DNS/Agones/metrics flows; application manifests consume externally populated Secret objects; the Go API and allocator now have bounded per-replica account+IP rate/quota boundaries and metrics endpoints, plus an atomic operator-controlled degraded-mode gate that sheds new login/queue/proposal work while preserving live-match paths | deploy/k8s/base/ includes hardened control-plane, allocator, and game-server resources; the control-plane now has health probes, a disruption budget, explicit zero-unavailable/one-surge rolling updates with graceful termination, failure-domain spreading/anti-affinity, and Secret-backed COSMIC_CLASH_POSTGRES_DSN/COSMIC_CLASH_WORKLOAD_SECRET runtime wiring; the authenticated WebSocket now requires RFC 6455 version 13, enforces a bounded 64 KiB frame size, two-minute idle deadline, 120-message/minute inbound budget, and bounded per-player fan-out; server/security/test_kubernetes_policies.py, server/api/rate_limit.go, server/api/admission.go, allocator metrics tests and adversarial checks cover static hardening, secret-reference invariants, fixed-window limits, bounded labels/key memory, atomic account+IP charging, route classification, concurrent toggling and degraded responses; production cmd/control-plane now wires the bounded limiter by default with --rate-limit, --rate-limit-window, and --rate-limit-max-keys, and the base Deployment declares those defaults explicitly; start with control-plane --degraded, enable with SIGUSR1, disable with SIGUSR2; private-store provisioning, distributed/global quotas, edge DDoS/WAF/origin shielding, encrypted backups and live policy/load tests remain |
8.13 [D:8.12] |
IN PROGRESS. Docker/Kubernetes references are digest-pinned, a dependency-free checker rejects mutable tags/plaintext credentials and concrete release overlays can reject template digests; release documentation defines SBOM, dependency/image scanning, signing, admission verification and a 24-hour critical-fix SLA | scripts/verify_supply_chain.py, server/security/test_supply_chain.py, docs/SUPPLY-CHAIN.md and .github/workflows/supply-chain.yml cover repository policy and provenance requirements; registry SBOM/scan/sign/admission execution and a concrete production overlay remain |
8C — Queueing, matchmaking, playlists and rating
| # | Task | Acceptance |
|---|---|---|
8.14 [D:8.4,8.5,8.8] |
IN PROGRESS. Pure Go queue domain enforces one active ticket per verified player under concurrent mutation, 10 s heartbeat/30 s expiry, retry-safe create/heartbeat/cancel, owner-only recovery reads and deterministic candidate projection; store layer adds a rebuildable candidate-cache boundary and authenticated HTTP queue adapter with playlist/build/protocol compatibility metadata | server/domain/queue.go, server/store/candidates.go, server/store/queue_sql.go, server/store/redis_candidates.go and server/api/service.go cover ownership/expiry/idempotency, concurrent create fencing, candidate/player ownership binding, owner/revision-scoped SQL heartbeat/cancel/recovery, injectable PostgreSQL queue backend selected by the HTTP service, authoritative queue-to-cache rebuild, expired recovery as a terminal error, owner-scoped SQL recovery with authoritative expiry handling, server-owned candidate resolution, strict compatibility metadata, TTL-bound Redis upsert/remove/snapshot, API create/heartbeat/cancel projection hooks, optional control-plane Redis configuration and atomic durable-source repair on partial/malformed cache state; opt-in real PostgreSQL execution now covers create/replay/active-player fencing, owner recovery, revision-fenced heartbeat/cancel and expiry, while miniredis covers Redis behavior and repair-source failure — this suite had never actually been run clean against a live database before: doing so once found CreateQueueTicket passing one extra unbound argument to its insert, which failed every real ticket creation with a param-count mismatch (fixed, re-verified against a real postgres:17-alpine container). A separate opt-in real-Redis suite (server/store/redis_integration_test.go, scripts/run_redis_integration.sh, COSMIC_CLASH_REDIS_ADDR-gated) now covers upsert/snapshot/remove, a real TTL actually waited out, and the "lost keyspace" repair path against a genuine FLUSHALL — including that the repair persists back to Redis, not just returned an in-memory answer. TestPostgreSQLConcurrentQueueHeartbeatIsRevisionFencedUnderRealRace races 5 concurrent same-revision heartbeats against real PostgreSQL: exactly one wins, the durable revision lands at exactly 1; live Redis failover-under-load and worker integration remain |
8.15 [D:7.8,8.3] |
IN PROGRESS. Pure Go probe validation treats Steam location as opaque, requires nonce/freshness/region and server-computed RTT, and implements discrepancy quarantine/release; authenticated HTTP now accepts only opaque location/nonce input through a server-owned probe provider and records validated RTT into the active player's durable queue ticket; durable queue projections have a server-derived RTT JSON field for matcher reads | server/domain/probes.go, server/migrations/0003_queue_probe_metadata.sql, adversarial fixtures and server/api/service.go/store/queue_sql.go cover stale/wrong/forged evidence, the 25 ms/30% threshold, three-sample quarantine, five-clean release, authenticated provider arguments, rejection of client RTT fields, player/ticket/expiry binding, persistence failure, playlist-scoped candidate reads and bounded metadata decoding; Steam coordinator, regional probe adapters and multi-region probe population remain |
8.16 [D:8.14,8.15] |
IN PROGRESS. Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction; the runnable matcher now supports explicitly enabled ranked six-player polling with durable verified-Steam identity lookup. Fixed a real crash-loop: Worker.Run treated every RunOnce error as fatal to the whole loop, including "no compatible candidates" (FormFromQueue's completely routine answer when currently-queued players share no verified region) — found building a live two-player integration attempt (see below): two real players with no common region crashed the entire matcher process, taking matchmaking down for every other player in the playlist, and would crash-loop again on restart since the same incompatible candidates stay queued. Now only genuine static misconfiguration (ErrWorkerNotConfigured/ErrUnsupportedPlaylist/ErrInvalidMatcherSize) stops the loop; everything else retries next interval |
server/domain/matcher.go, teams.go, server/matcher/worker.go, server/store/queue_sql.go and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure, queue-backed oldest-anchor formation and incomplete ranked identity metadata; two new tests cover Run (not just RunOnce) surviving a per-pass error via a real concurrent goroutine, and still stopping immediately on a real configuration error, both clean across repeated -race runs. A live two-player Godot proposal integration attempt is on disk but not committed: Game/tests/control_plane_proposal_smoke.gd/.tscn and scripts/verify_control_plane_proposal_integration.sh exist and found the bug above, but the session paused running further concurrent headless Godot processes after discovering they'd been causing native engine crashes (macOS crash reporter, EXC_BAD_ACCESS/SIGBUS) intermittently all session, confirmed by the user; the two-player script was never itself verified to a clean pass. Arena selection and long-running worker integration remain |
8.17 [D:8.14,8.16] |
IN PROGRESS. Pure Go proposal policy sends a 10-second response window to every selected human, requires unanimous acceptance, applies exact decline/timeout cooldowns and ranked escalation; authenticated API exposes revisioned accept/decline mutations; formed matches now pass through a playlist-aware proposal boundary. Fixed a real severe bug: declining a proposal never requeued anyone's ticket — every participant, decliner included, was left stranded at PROPOSED (invisible to the matcher, still blocking a fresh queue_create, renewable forever by an ordinary heartbeat) with no path back into matchmaking. ProposalDeclineRequeueSQL now requeues every participant to QUEUED with a fresh expiry on decline; the not-yet-built decline cooldown mentioned here can later exempt the decliner specifically, but leaving anyone stuck today wasn't that cooldown, it was just broken. The same bug's timeout sibling is fixed too: a proposal that simply expires (no unanimous response inside the window) hit the identical gap in ProposalExpireSQL/ProposalParticipantExpireSQL, reached from both GetProposal (a client recovering after missing the expiry event) and RespondToProposal (a response arriving after the window); ProposalExpireRequeueSQL mirrors the decline fix, guarded on state = 'EXPIRED' so it's safe to call unconditionally. Closed the remaining responsiveness gap too: cancelling a queue ticket directly while it's part of an OPEN proposal used to leave the other participant waiting out the full window instead of being told immediately; CascadeCancelToOpenProposal now declines and requeues the proposal in the same transaction as the cancel |
server/domain/proposal.go, formation.go and server/api/service.go plus adversarial fixtures cover partial/unanimous response, expiry, replay/conflict, stale API revision, casual lineup preparation and ranked metadata validation; real PostgreSQL integration tests confirm the decline, timeout, and cancel-cascade paths all requeue every participant (decliner/uninvolved participant/cancelling player's partner alike) to QUEUED with a refreshed expiry, visible again to ListQueuedCandidates (the matcher's own read), and that a cancelling player's own ticket correctly stays CANCELLED rather than being swept back up; clean across 5 runs each; queue precedence, allocation integration and the decline cooldown itself remain |
8.18 [D:8.5,8.14,8.17] |
IN PROGRESS. Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using FOR UPDATE SKIP LOCKED plus durable uniqueness/revision fences; proposal creation now persists its matcher-selected region/protocol/team/slot plan alongside proposal/participant claims, queue creation has a durable idempotency/owner-read adapter, participant-scoped proposal recovery now expires OPEN proposals and pending participants transactionally at read time, and proposal accept/decline now uses participant/proposal locks, revision fencing and durable idempotency; final unanimous API acceptance invokes replay-safe promotion of the exact stored ALLOCATING match/team/slot topology and claimed tickets to ACCEPTED; runnable matcher polling supports an optional Redis candidate projection that repairs empty/lost cache state from authoritative PostgreSQL before the durable final claim |
server/store/serializable.go, queue_sql.go, proposal_sql.go, proposal_recovery_sql.go, match_sql.go, redis_candidates.go, server/matcher/worker.go, server/api/service.go and tests cover retry classification, claim-boundary invariants, player/ticket/playlist mapping, durable queue replay/conflict, owner-scoped queue/proposal recovery, expiry at read and mutation boundaries, final-response promotion retry, exact match-promotion replay/conflict, fixed team/slot persistence, zero-row claim aborts, atomic statement ordering, incomplete matcher batches, source failures and empty-index Redis repair; opt-in PostgreSQL execution now covers queue create/replay/fencing, assignment persistence, proposal claim/promotion, accepted-proposal match promotion, participant recovery, unanimous response and rollback of partial claims — actually running this suite live (it had not been before) found ProposalParticipantExpireSQL had no expiry-time condition at all, so every call timed out every pending participant on the spot; the very first accept on any proposal then failed with a false conflict. Fixed with the same expires_at <= gate ProposalExpireSQL already used, re-verified live. A real concurrent-goroutine test now covers the two-matcher race this was missing: two proposals sharing one contested ticket, racing two real Postgres connections under -race, exactly-one-wins/loser-fully-rolls-back including the loser's own uncontested ticket, stable across 8 runs; allocation runtime integration tests remain |
8.19 [D:8.18] |
IN PROGRESS. Pure Go casual lineup requires 2–6 humans with at least one per team, fills missing slots with explicit bots, permits kickoff-only bot-slot backfill and assigns no backfill penalty/rating update; proposal preparation now derives the lineup from formed teams | server/domain/casual.go, formation.go cover both-team minimum, bot shape, live-play rejection, zero-penalty backfill and casual proposal composition; queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties and live integration remain |
8.20 [D:8.18] |
IN PROGRESS. Pure Go ranked admission requires six unique verified solo humans, rejects bots/backfill/parties, and allows only random-enabled non-elevated arenas; proposal preparation requires matching metadata for every formed player | server/domain/ranked.go, formation.go cover count, identity, party, bot/backfill, arena eligibility and formed-player metadata rejection; ArenaRegistry integration, allocation wiring and innocent-ticket restoration remain |
8.21 [D:8.5,8.20] |
IN PROGRESS. Pure Go rating core implements canonical Glicko-2, daily inactivity, ranked 1/3 and casual 1/N human-opponent weights, deterministic opponent ordering, and authoritative draw/overtime/abandon scoring; certified result completion now applies per-player updates inside the durable transaction with lexical row locks and revision increments | server/domain/rating.go, server/store/result_sql.go and tests cover canonical/inactivity/weight/invalid-input, draw/OT/abandon, ordered participant snapshots, lock/value re-read and rating update SQL; live PostgreSQL rating and seasons execution now covered (§8.23), and concurrent result transaction cases are covered: TestPostgreSQLConcurrentIdenticalResultSubmissionAppliesRatingsExactlyOnce races 5 identical submissions and confirms one rating application, while TestPostgreSQLConcurrentConflictingResultSubmissionsKeepOneReceipt races different payloads and confirms exactly one winner, one conflict, one receipt and one completion event; live maintenance/DB execution remains |
8.22 [D:8.21] |
IN PROGRESS. Pure Go ranked profile exposes the first ten games as provisional, derives tiers only through validated backend-owned rating bands, and keeps casual ratings outside the API; authenticated HTTP now returns the authoritative ranked view. "Authoritative ranked view" was durable-adapter-shaped but had no durable adapter: rankedProfile/profile only ever read an in-memory map, so every real GET /v1/profile/ranked 404'd regardless of a player's actual rating. RankedProfileProvider (interface) + store.PostgresRankedProfiles close it, preferred over the map when set so existing tests/literals are unaffected; LastSeasonID/SeasonHistory deliberately left unset (no season pointer on ratings, needs its own query/semantics) |
server/domain/rating.go, tier_test.go and server/api/service.go cover provisional override, exact band boundaries, malformed policy rejection, session authentication and ranked-only response fields; server/store/ranked_profile_sql.go, verified against real PostgreSQL via curl (a fresh identity correctly 404s through the real adapter) and via §8.40's integration test (control_plane_smoke.gd now asserts this exact 404 round-trips before queueing); persisted tier policy, client UI and reconnect transport remain |
8.23 [D:8.21] |
IN PROGRESS. Pure Go ranked-only season policy compresses 25% toward 1500, clamps RD to 200–350, preserves volatility/history, is idempotent by season ID, and defines exact 12-week windows/due detection; migration and Go store now persist a per-player/per-season marker and rating update atomically; cmd/maintenance runs bounded due-season batches with signal-bound shutdown |
server/domain/rating.go, season_test.go, server/migrations/0001_initial.sql and server/store/maintenance_sql.go cover compression, floor/cap, duplicate replay, window boundary, completed-season idempotence, bounded enumeration, row locking and conflict-safe rollover markers; opt-in PostgreSQL execution now covers the durable rating update, marker creation and duplicate replay without a second compression — re-run live for the first time as part of the wider integration-suite verification below, after fixing a test setup gap (a missing seasons row tripped the ranked_season_rollovers foreign key before the rollover logic itself ran); live maintenance/DB execution remains |
8.24 [D:8.9,8.20,8.21] |
IN PROGRESS. Pure Go ranked connection policy binds match/server/player/team/slot/protocol, supports 60 s reclaim with server-owned generations, fences old connections, and applies the rolling 7-day 5 m/15 m/1 h/24 h abandon ladder; canonical signed authorisation issuance/verification now gates admission | server/domain/reconnect.go, join_auth.go and adversarial fixtures cover repeated backend-independent reclaim, all signed claim binding, tampering, failed verification, old-generation fencing, grace boundary and deterministic cooldown audit ordering; persistent lease fencing, join transport and full match/result integration remain |
8.25 [D:8.10,8.24] |
IN PROGRESS. Pure Go result policy binds match/server/workload identity, hashes canonical payloads, makes identical retries idempotent, leaves conflicts inert, separates integrity eligibility, classifies roster/simulation/result/fairness evidence, validates annotation signatures/digests, and exposes 5 m alert/30 m review delivery thresholds; Go store SQL now executes receipt → match lock → certified rating updates → completion → receipt acknowledgment → outbox atomically, and exposes bounded ordered outbox reads plus publish acknowledgements for replayable fan-out; OutboxDispatcher now delivers in order and acknowledges only after successful fan-out; the API validates workload-bound server result submissions and the PostgreSQL adapter repeats domain validation before invoking this durable boundary; production now also runs a filtered match_completed dispatcher that turns each committed result into targeted COMPLETED state events for every durable participant, without acknowledging proposal rows |
server/domain/result.go, workload.go, server/workload/jwt.go, server/api/service.go, server/api/outbox.go, server/store/result_sql.go and outbox.go plus adversarial fixtures cover credential binding, duplicate/conflict, annotation forgery, server/match mismatch, invalid direct-adapter payloads, delivery-outage-versus-integrity classification, commit ordering, idempotent SQL reconciliation, ordered rating locks, certified-update gating, unpublished-event replay/ack boundaries, event-type isolation and delivery-before-ack failure ordering; opt-in PostgreSQL execution now covers result-pending completion, durable receipt/outbox publication, ack removal, identical replay and conflicting replay rejection, and real concurrent identical/conflicting submissions; scripts/run_result_fanout_integration.sh now drives real PostgreSQL → API WebSocket delivery for an authenticated participant; production credential verification, Agones annotation persistence/reconciliation and integrity evidence adapters remain |
8D — Agones, allocation and regional scaling
| # | Task | Acceptance |
|---|---|---|
8.26 [D:8.1,8.6,8.12] |
IN PROGRESS. Provider-neutral Kustomize base now defines a restricted Agones Fleet with region/build/protocol/transport labels and UDP game port, plus distinct EU/NA overlays; the base avoids rewriting cross-namespace Agones RBAC and now wires the digest-pinned supervisor image, control-plane Service, dynamic roster volume, signing/drain secret references and required network flow | deploy/k8s/base/fleet.yaml, control-plane-service.yaml, network-policies.yaml, overlays/eu, overlays/na and server/security/test_fleet_manifests.py cover labels, replica floor, UDP declaration, pod hardening, supervisor/runtime arguments, Service selection, egress policy, overlay distinction, Kustomize rendering and RBAC namespace safety; operator secret/image replacement, second-provider fixtures, edge/DNS and SDR POP/cert/public-UDP overlays remain |
8.27 [D:8.26] |
IN PROGRESS. Go supervisor package provides local-safe Agones REST discovery, validates assigned address/port data, injects dynamic SDR_LISTEN_PORT/SDR_IP, performs explicit process-ready probing and Ready transition; direct mode bypasses Agones |
server/supervisor/ covers allocated/direct startup, invalid endpoint rejection, dynamic endpoint/Ready ordering and authenticated drain; allocated Godot now supplies a loopback readiness/drain control surface and agones_sdk.gd supplies sidecar Health/Ready/Shutdown/annotation REST operations; metadata watch, real Agones annotation/shutdown confirmation and emulator integration remain |
8.28 [D:8.6,8.27] |
IN PROGRESS. Supervisor separates explicit process-ready from Agones Ready and never scrapes stdout; allocated mode refuses to mark Ready without a configured readiness probe. The durable control-plane counterpart now exists: a workload-authenticated POST /v1/servers/{id}/register (and its /api/v1 contract alias) advances a match's ALLOCATING -> PROCESS_READY -> ASSIGNMENT_READY state, and every participant's queue ticket with it, as one idempotent SERIALIZABLE transaction, gated on every participant already holding a live, unexpired assignment. The supervisor now calls it: once Agones Ready succeeds, it POSTs process-ready (assignment_ready=false) using a workload token read fresh from disk each call (matching kubelet's in-place rotation of a projected service-account token), and a registration failure kills the child rather than leaving an Agones-Ready-but-control-plane-unregistered process running; ControlPlaneURL unset (the default) is a total no-op. It then reports assignment-ready too: server_boot.gd already verifies its mounted roster synchronously before /ready is ever exposed (so process-ready implies the roster was valid), and the API's ASSIGNMENT_READY gate checks only durable assignments rows server-side — so no new Godot-side state was needed, correcting an earlier overcautious note here. The supervisor retries assignment-ready (default 5 attempts, 2s apart, configurable) since those durable rows may lag process-ready slightly; a persistent failure there does not kill the child, unlike process-ready. Per-allocation data (currently match-id) now has a real channel to an already-Ready pod: server/agones.Client.Allocate requests cosmic-clash.io/match-id/cosmic-clash.io/allocation-id as GameServerAllocation.spec.metadata.annotations (Agones applies these to the allocated GameServer's own object_meta — the only channel that exists post-allocation, since env vars are fixed at pod creation), and the supervisor reads them back from its existing /gameserver SDK call, falling back to them only when MatchID isn't explicitly configured. The image now exists: a new Dockerfile game-server target packages the supervisor as ENTRYPOINT alongside the same dedicated-server export server produces; the base Fleet now invokes that target with the control-plane URL, server/image Downward API identity, roster/signing/drain material, and exported Godot executable. |
server/supervisor/ tests prove Ready follows the probe and direct mode remains functional; server_control.gd, agones_sdk.gd and process-level smokes prove loopback /ready, /health, bearer-protected /drain, sidecar-shaped Health/Ready calls and drain admission fencing; server/api/service.go, server/store/allocation_match_sql.go and adversarial tests cover the registration route, digest/protocol validation, idempotent replay/conflict and the assignment-count gate; server/supervisor/supervisor_test.go covers opt-in registration, the workload-token/body/idempotency-key shape, the kill-on-failure path, the match-ID annotation fallback (both that it's used and that its absence fails closed before any HTTP call), the full process-ready-then-assignment-ready sequence and body shapes, and a retry test that fails assignment-ready twice with 409 before succeeding, asserting Start() still succeeds and the child is never killed; server/agones/allocation_test.go covers the requested annotations. docker build --target game-server verified for real: both binaries present, correct permissions, supervisor prints its usage; server/store/stalled_allocation_sql.go/_test.go and a live TestPostgreSQLStalledAllocationsAreReclaimedWithoutPenalisingPlayers cover the deadline boundary (a recent match must survive untouched), the no-penalty requeue and refreshed expiry, participant release, and idempotence against a second pass. Fixed in passing: Dockerfile's server stage's ubuntu base digest had gone dead on Docker Hub (docker pull returned "not found", verified independently) — make verify-phase6 was silently broken for a clean build before the re-pin; confirmed fixed with a full make verify-phase6 run (arenas rotated, both goals observed, clean teardown). deploy/cosmic-clash-server now wraps its exec in stdbuf -oL -eL (falling back to unwrapped if unavailable), fixing a real, live-confirmed bug: a genuinely detached (docker run -d) container showed zero docker logs output — not even the startup line — for 20+ seconds while the process ran normally, and docker stop's SIGTERM lost that buffered output permanently rather than delaying it; re-verified fixed against the real launcher script, then a full make verify-phase6 re-run confirmed no regression. Health-reclaim now exists: store.ExpireStalledAllocations reclaims a match stuck in ALLOCATING/PROCESS_READY/ASSIGNMENT_READY past a deadline (server crashed, or was reclaimed by Agones as unhealthy, before ever registering) by failing the match and requeuing every participant to QUEUED with a fresh expiry rather than penalising them — task 8.50's own "infrastructure-caused cases cannot penalise affected players" criterion directly settles the requeue-vs-fail design question this had been blocked on. Wired into cmd/maintenance alongside the season-rollover sweep (--stalled-allocation-deadline default 2m, --stalled-allocation-batch). Superseding the fleet.yaml framing above: §8.10's WorkloadVerify blocker, and its delivery channel, are both now closed — a control-plane-self-issued signed token (not a Kubernetes JWT), minted by cmd/allocator into a cosmic-clash.io/workload-token annotation and read back by the supervisor, exactly the way match-id already worked — see §8.10. /register and /result no longer 503 unconditionally once every --workload-secret (control plane, allocator) is set consistently. The remaining gates are live Agones annotation/shutdown behavior and production cluster readiness; those are covered by §8.49 and remain explicitly open. |
8.29 [D:8.26,8.27] |
IN PROGRESS. Supervisor discovers and validates the Agones endpoint, propagates the actual dynamic --port, and exports SDR_LISTEN_PORT/SDR_IP only for Hosted-SDR while preserving an isolated ENet path |
server/supervisor/ tests cover invalid address/port rejection, dynamic port argument/env propagation and SDR-vs-ENet separation; real Agones dynamic/passthrough mapping, POP/cert/firewall/NAT and multi-match fixture remain |
8.30 [D:8.18,8.26,8.28,8.29] |
IN PROGRESS. Pure Go allocator filters Ready servers by region/build/protocol/transport, atomically claims one with idempotent allocation replay, verifies unanimous accepted-proposal/playlist/participant invariants before provider invocation, and now owns the assignment-publication boundary; PostgreSQL adds durable GameServer registration and compatible SKIP LOCKED claims with request-digest fencing, plus a leased ALLOCATING-match claim that derives the immutable compatibility tuple and fences bind/release by deterministic allocation ID; bind atomically attaches only a recorded provider allocation and advances every accepted participant ticket to ALLOCATING; server/agones strictly projects Ready GameServers from Fleet compatibility labels, submits/validates namespaced GameServerAllocation responses and dynamic endpoints, and can recover an already-Allocated GameServer by allocation ID after an ambiguous write; server/allocator reconciles provider success into durable state before exposing the endpoint; cmd/allocator refreshes that Ready projection before driving the lease → provider → durable-record → match-bind sequence and rebinds a recovered durable provider allocation without a second provider call |
server/domain/allocator.go, server/store/allocator_sql.go, server/store/allocation_match_sql.go, server/store/allocation_match_adapter.go, server/agones/allocation.go, server/allocator/service.go, server/allocator/worker.go, server/cmd/allocator, server/migrations/0004_allocator_registry.sql, 0006_match_allocation_claims.sql and tests cover strict Ready-server projection, stale-projection protection for allocated rows, deterministic compatible selection, exhaustion, conflicting/identical allocation replay, allocation-match lease recovery/bind fencing, atomic participant lifecycle transition, recovered-allocation binding without provider recall, immutable Fleet selector labels, provider ambiguity lease retention, unknown allocations, SQL claim ordering, invalid input, provider error/malformed response/IPv6 endpoint handling, accepted-proposal gating, provider recovery metadata/duplicate detection, durable-reconciliation failure isolation and assignment replay/conflict; TestPostgreSQLAllocatorClaimReplayAndCapacityFence and TestPostgreSQLAllocationMatchClaimLeaseAndBindFence cover the live database paths when the disposable database gate is run; TestPostgreSQLConcurrentAllocationClaimNeverDoubleBooksAReadyServer now races real concurrent claims against fewer Ready servers than requesters and proves no double-booking and no stray errors, stable across repeated -race runs; scripts/run_allocator_integration.sh and TestRealAllocatorWorkerReconcilesAgonesAllocationAndBindsMatch now add a real PostgreSQL + Agones-shaped HTTP provider integration covering Ready projection → worker lease → provider request → durable reconciliation → match/ticket bind; full live unknown-provider-outcome recovery and signed roster metadata/cluster integration remain |
8.31 [D:8.9,8.30] |
IN PROGRESS. Pure Go assignment gate requires Allocated state, exact allocation ID/match/server/region/build/protocol/transport compatibility, non-empty hosted endpoint and verified manifest signature before exposure; allocator publication cannot expose Ready state; durable roster persistence now verifies each canonical join-authorisation signature before publishing player rows; allocator service gates roster publication on allocated state and endpoint presence | server/domain/assignment.go, allocator.go, store/assignment_sql.go plus adversarial fixtures cover early-connect, tampered signature/manifest, wrong compatibility, empty endpoint, unknown allocation, forged roster signature, valid signature, premature publication and post-publication mutation rejection; Agones metadata watch, hosted-address registration, production signer and client-ticket publication remain |
8.32 [D:8.2,8.26,8.30] |
IN PROGRESS. Provider-neutral FleetAutoscaler baseline preserves a two-process Ready buffer, caps warm capacity, and leaves Allocated scale-down independent of the Ready floor; Fleet image references remain digest-pinned for current/rollback pre-pull | deploy/k8s/base/fleet-autoscaler.yaml and manifest tests cover Fleet ownership, Buffer policy and floor/cap invariants; regional on-demand node pools/failure domains, pre-pull rollout, warm-allocation p95/p99 and N+1 certification remain |
8.33 [D:8.26,8.32] |
IN PROGRESS. Fleet scheduling now requires on-demand capacity and spreads Ready processes across zones with skew 1; the autoscaler preserves the two-process Ready floor | deploy/k8s/base/fleet.yaml and manifest tests reject interruptible placement and single-zone concentration structurally; regional node pools, forced node-loss testing and measured N+1 headroom remain |
8.34 [D:8.28,8.29] |
Native x86_64 benchmark of boot-to-process-ready and assignment-ready, p99 CPU/RSS/network and 60 Hz ticks; limits/node cap with 30% headroom | Measurements replace old estimates and certify density with no tick backlog |
8.35 [D:8.17,8.19,8.20,8.30,8.31] |
IN PROGRESS. Pure Go initial-connect policy decides ranked 30 s no-show cancellation with abandon ladder and casual 60 s bot start only when each team has a human; empty-team casual allocations cancel | server/domain/noshow.go covers wait/deadline boundaries, deterministic no-show/innocent ordering, ranked cooldown history and no pre-live rating action; persistent ticket restoration, allocation shutdown, bot spawn and live integration remain |
8.36 [D:8.10,8.25,8.28,8.30] |
IN PROGRESS. Supervisor exposes an authenticated loopback-only drain request boundary that never places the token in command arguments/logs and rejects remote/partial/query-bearing configurations; the base now includes a two-Ready PodDisruptionBudget; Supervisor.Run and cmd/game-server-supervisor now orchestrate signal-bound drain-before-kill with a bounded grace deadline |
server/supervisor/, server/cmd/game-server-supervisor/, server_control.gd and deploy/k8s/base/game-server-pdb.yaml cover bearer-token enforcement, loopback URL validation, secret-safe configuration, missing drain credentials, readiness transitions, Ready-floor disruption protection, graceful child exit after drain and force-kill of an unresponsive child; live 300 s/285 s lifecycle, PDB/Fleet drain and infrastructure-abort classification remain |
8.37 [D:8.5,8.10,8.25,8.26,8.31] |
Horizontally scaled primary control plane + warm standby, EU/NA fleets, RPO <=5 m/RTO <=30 m; signed result annotation/reconciliation preserves delivery outages | Restore/failover meet targets; live simulation continues; valid delayed result commits exactly once after recovery |
8.38 [D:7.7,8.26,8.36,8.37] |
Provider migration after Valve approves both providers' EU/NA POPs/certs and public UDP: restore, validate coordinator trust, switch allocations, drain old | Both geographies complete Hosted-SDR matches on new provider and no old-provider live match is terminated |
8E — Client experience and recovery
| # | Task | Acceptance |
|---|---|---|
8.39 [D:8.3,8.14,8.17] |
IN PROGRESS. MatchmakingState now projects queue → proposal → allocation/process-ready/assignment-ready/connect/live plus terminal failure states; autoload ControlPlaneClient provides authenticated queue create/recovery/heartbeat/cancel and proposal response requests with idempotency/revision headers; matchmaking.tscn/matchmaking.gd expose the state and authoritative actions from the main menu; authenticated proposal recovery now reconciles missed proposal events and expires them at read time; the Go API publishes targeted authenticated revisioned queue/proposal/assignment events and the Godot client consumes state/proposal/assignment events, fetching the authoritative assignment after assignment readiness; the API can now use the durable participant-scoped proposal provider for both recovery and accept/decline mutations |
test_matchmaking_state.gd, test_control_plane_client.gd, test_matchmaking_ui.gd, TestProposalRecoveryIsParticipantScopedAndExpiresAtReadBoundary, TestAuthenticatedWebSocketDeliversOnlyTargetedRevisionedEvents, TestStateChangingAPIActionsPublishTargetedEvents, TestProposalRecoveryUsesDurableBackendAndRemainsParticipantScoped and store proposal SQL tests reject stale/gapped/conflicting updates, validate endpoint/token/payload normalization, preserve idempotent duplicates, and guarantee visible phase/terminal copy; wiring durable allocation events into the outbox, wait/latency explanations and Godot runtime verification remain |
8.40 [D:8.3,8.14] |
IN PROGRESS. Pure Go revisioned replica reducer rejects gaps for REST resync, makes duplicate/out-of-order events idempotent, and resumes from the authoritative snapshot revision; authenticated queue-ticket recovery now has an owner-checked REST read; Godot client projection persists non-secret ticket/proposal state, forces authoritative recovery after restart, and can replay a lost queue-create response with the original ticket/idempotency key; the Go API now exposes an authenticated /v1/events WebSocket with bounded per-player queues, strict upgrade/vocabulary validation and REST-resync-safe slow-client failure; Godot ControlPlaneClient can connect to the stream using Godot 4.7's handshake-header API, consume validated events, automatically reconnect with bounded backoff, and trigger REST recovery on projection gaps and stream return, including a deferred recovery when an event arrives during an in-flight HTTP mutation; OutboxDispatcher now provides ordered at-least-once delivery after durable commit; assignment recovery events carry the durable assignment revision without leaking it into REST JSON; prediction startup now distinguishes the sequence-0 warm-up acknowledgement from genuine missing/evicted history, preventing a false hard-snap during a live match; proposal creation writes one durable targeted proposal_changed outbox event per proposal aggregate revision, while result completion writes match_completed and production cmd/control-plane plus the test-only API harness dispatch both event types through separate filtered consumers |
server/domain/sync.go, server/api/events.go, server/api/outbox.go, server/api/service.go, server/store/outbox.go, server/store/proposal_sql.go, server/cmd/control-plane, server/cmd/testkit-api, service_test.go, outbox_test.go, matchmaking_state.gd, control_plane_client.gd, local_prediction_history.gd, net_ship_predictor.gd and tests cover gap, snapshot, replay, same-revision conflict, owner-only ticket recovery, expired-ticket terminal handling, malformed restart snapshots, API-level duplicate-create replay/conflict, authenticated handshake/key rejection, targeted event delivery, exactly-once slow-subscriber closure, invalid-event rejection, delivery-before-ack failure ordering, assignment event/response revision separation, prediction warm-up/hard-resync separation, deferred proposal recovery and proposal/result outbox filtering and delivery failures; Godot 4.7.1 headless project parse and 150-test unit harness pass with compatibility rendering, and a two-process ENet authoritative match smoke passes spawn, movement, prediction, 0% snapshot loss and 0 hard snaps; cmd/control-plane provides a signal-bound API role, cmd/matcher now supports casual and explicitly enabled ranked roles with durable identity lookup, and cmd/maintenance provides bounded season maintenance. Live multi-process control-plane/game verification now exists: scripts/verify_control_plane_integration.sh runs a real postgres:17-alpine, the real api.Service (via the new test-only server/cmd/testkit-api, wired identically to cmd/control-plane except for a fake Steam login — see §8.7), and a real headless Godot client (control_plane_smoke.gd) round-tripping login → fetch_ranked_profile (expect 404, §8.22) → queue_create → heartbeat → cancel_queue over an actual network connection — the first time this boundary was exercised end to end rather than against a mock on either side. The two-player proposal extension now also passes: scripts/verify_control_plane_proposal_integration.sh starts isolated PostgreSQL, the real matcher and testkit API, launches two headless Godot clients, seeds deterministic RTT projections, observes the real WebSocket OPEN proposal, accepts from both clients, and verifies both durable tickets plus the proposal reach ACCEPTED (including authoritative recovery when the concurrent revision advances). scripts/run_result_fanout_integration.sh additionally verifies a real PostgreSQL-backed authenticated WebSocket receives a completed-match event; allocator and Redis fan-out live verification remain |
8.41 [D:7.8,8.9,8.31,8.40] |
IN PROGRESS. Authenticated GET /v1/assignments/{matchId} now exposes only a validated, player-scoped assignment view including the hosted endpoint; Godot AssignmentState/ControlPlaneClient.fetch_assignment() bind the response to the authenticated player, recheck expiry, validate and retain the endpoint, and connect_to_assignment() now starts only the validated ENet/Steam transport after assignment readiness; the opaque join authorisation is carried in the MatchNet hello payload rather than the endpoint URL; allocated Godot servers now fail closed unless an operator-mounted JSON roster of control-plane signed envelopes and an HMAC-SHA256 key are present, and MatchNet verifies the canonical Go claim bytes, checks exact token membership plus match/server/protocol/slot/expiry claims before admitting a peer, rejects concurrent reuse of an active token, tracks server-owned reconnect generations across the 60-second reclaim window, fences expired reclaims and invalidates old peers; migration 0002 and the Go store adapter now persist/recover the complete player-scoped assignment projection with conflict-safe identical replay; AssignmentProviderFromStore wires that durable projection into the API injection point; SaveAssignments publishes a complete signed roster atomically instead of allowing partial player visibility; SaveVerifiedAssignmentRoster rechecks signed claims before deriving player rows; a workload-authenticated GET /v1/servers/{serverId}/roster now returns the complete signed envelope set only for the bound allocation, and the allocated supervisor atomically materializes it before launching Godot; live authenticated Godot/PostgreSQL assignment verification now exists via scripts/verify_assignment_integration.sh, which seeds a durable player-scoped ASSIGNMENT_READY row behind the real testkit API and verifies the returned endpoint/join authorisation over HTTP/JSON (including the live numeric-decoding boundary); scripts/run_supervisor_integration.sh additionally verifies the allocated-server process-ready → roster fetch/materialization → assignment-ready registration path against real PostgreSQL and the real API, with the workload token recovered from an Agones-shaped GameServer annotation |
server/api/service.go, store_adapters.go, service_test.go, assignment_state.gd, control_plane_client.gd, match_net.gd, server_boot.gd, server_config.gd, test_assignment_state.gd, test_control_plane_client.gd, test_match_net.gd, server/contracts/v1/openapi.json, server/migrations/0002_assignments.sql, server/allocator/allocator_integration_test.go, server/supervisor/supervisor_integration_test.go, server/supervisor/supervisor.go and server/store/assignment_sql.go cover participant/identity/expiry/shape/transport/endpoint boundaries, player-scoped schema keys, expiry-filtered reads, assignment upsert conflict handling, atomic batch validation, signed-claim binding, strict endpoint splitting, allowlisted claim rejection, canonical HMAC interoperability, forged-signature rejection, duplicate active-token rejection, reclaim generation/expiry behavior, workload-authenticated roster delivery, atomic file installation, real allocator reconciliation, real supervisor registration and 150-test Godot compatibility coverage; the three-process disconnect/reclaim smoke passes ship retention, replacement drive and old-peer invalidation; dynamic per-match launch flags, SDR relay-ticket installation and live Agones cluster integration remain |
8.42 [D:8.22,8.23,8.24,8.40] |
IN PROGRESS. RankedProfileState and ControlPlaneClient.fetch_ranked_profile() expose the backend-authoritative rating/RD/volatility/games/tier/provisional/season view; matchmaking UI displays provisional/tier status without client-side rating math; runnable API binaries now wire the backend-owned default tier policy; live authenticated profile verification now exists via scripts/verify_ranked_profile_integration.sh, which seeds a durable rating and verifies the populated profile over real PostgreSQL, Go HTTP, and Godot JSON |
test_control_plane_client.gd validates profile shape, numeric safety and provisional display; the real populated profile path now covers durable rating/games/tier/provisional decoding and caught the missing runtime tier-policy wiring; committed revision after reconnect, abandon status and season countdown remain dependent on live auth/backend events and Godot runtime verification |
8.43 [D:8.39,8.40,8.41] |
IN PROGRESS. Matchmaking client now distinguishes expired queue recovery, session expiry, missing records and retryable control-plane outages; a 401 clears the in-memory token, emits session_expired and disables retry until a new session is configured; terminal messages remain visible and active searches are not falsely failed on transient errors; stale proposal accept/decline conflicts now automatically schedule an authoritative proposal recovery instead of requiring each UI caller to implement 409 handling |
MatchmakingState and ControlPlaneClient tests cover explicit expiry and the existing terminal/retry-safe state paths; the real two-player proposal smoke now exercises the client-owned conflict-recovery path; decline, version mismatch, regional outage retry UI, failed reconnect, duplicate-action recovery beyond proposals and broader live Godot verification remain |
8F — Observability, verification, cost and rollout
| # | Task | Acceptance |
|---|---|---|
8.44 [D:8.3,8.4,8.28,8.31] |
LOCAL COMPLETE; PRODUCTION GATE OPEN. Go observability package encodes queue/proposal/match/server IDs and lifecycle stage while recursively redacting auth/relay tokens and credentials. Service.Log is wired to mutation and read routes at every outcome, and cmd/control-plane writes those events as JSON lines to stderr |
server/observability/ covers correlation fields, nested secret redaction, content-aware credential canaries and unnamed-event rejection; API tests cover lifecycle event wiring without logging error text. A production metrics/traces backend and dashboard/alert routing remain open; the local logger is intentionally stderr-only |
8.45 [D:8.2,8.44] |
LOCAL COMPLETE; PRODUCTION GATE OPEN. Go observability package turns the documented RTT, allocation/connect latency, result-success, API-latency and tick/headroom thresholds into executable window checks; the API exporter emits a bounded cumulative latency histogram suitable for querying the documented p95 API SLO | server/observability/slo.go, metrics.go, deploy/observability/prometheus-rules.yaml and adversarial tests cover healthy/violating/empty windows, fixed operation/status labels, cumulative bucket boundaries, arbitrary-path cardinality safety, and optional API p95/5xx alerts. Production scrape configuration, alert routing, wait/MMR/proposal/flood/cost series and runbooks remain |
8.46 [D:8.5,8.7,8.9,8.10,8.14,8.18,8.21,8.23,8.25] |
IN PROGRESS. Go unit/race coverage spans the current domain/store/supervisor policies, and fuzz targets now exercise queue input, result payload hashing and revision events | server/domain/*_test.go, server/store/*_test.go, server/supervisor/*_test.go, server/migrations/*_test.go and server/domain/fuzz_test.go pass normal/race suites; go test -race ./... passes across API, domain, migrations, observability, store, supervisor and testkit; go vet ./... passes; each of the three declared domain fuzz targets passes a bounded 4-second run; PostgreSQL live migration execution now runs clean (§8.5), and four real-concurrency cases are covered against a live database with -race: §8.14's queue-heartbeat revision race, §8.18's two-matcher contested-ticket race, §8.30's cross-allocator-replica capacity race, and §8.21/§8.25's concurrent identical-result-submission race; the "lost Redis" fixture is covered live against a real server (§8.14: real TTL expiry, repair-after-FLUSHALL; fake Steam/allocator fixtures are §8.47's testkit, already done). Further transaction fixtures (e.g. concurrent proposal-recovery expiry races, live Redis failover mid-write under load) remain |
8.47 [D:8.7,8.30] |
IN PROGRESS. Offline testkit provides deterministic fake Steam verification and fake allocation with forced failure injection; the independent Compose runner drives fake-Steam session issuance, real HTTP queue create/heartbeat/cancel, matcher-backed six-player proposal formation/acceptance, and idempotency-conflict checks | server/testkit/ covers verified identity/replay, unknown identity, wrong App ID, expiry, no capacity, compatibility-key conflict, idempotent allocation replay and cloud-free forced allocation failure; the Compose API/matcher slice is wired into CI, while live exhaustive matrix and production Steam remain |
8.48 [D:8.10,8.14,8.17,8.18,8.27,8.31,8.35,8.47] |
IN PROGRESS. Offline testkit exercises verified queue projection → ranked six-player proposal → ENet allocation → assignment-ready manifest → certified durable result receipt; compose.allocated-smoke.yml independently runs the real testkit API, matcher, allocator, Agones-shaped provider, PostgreSQL, and game-server supervisor with a generated signed roster, verifying queue/proposal/allocation binding, authenticated result, idempotent retry, shutdown acknowledgment, durable receipt/audit rows, and SIGTERM-driven game-process drain |
.github/workflows/allocated-compose.yml runs make verify-allocated-compose. Live Docker evidence from this workspace and legacy fixture non-regression remain open |
8.49 [D:8.25,8.26,8.28,8.29,8.30,8.31,8.35,8.36] |
IN PROGRESS. scripts/verify_kind_agones.sh creates a disposable kind cluster, installs pinned Agones, loads the real game-server image, applies the Fleet in an explicitly separate Agones-only supervisor/UDP readiness mode, and verifies readiness plus allocation of a dynamic UDP endpoint; .github/workflows/agones-integration.yml runs it for infrastructure changes and on demand |
The cloud-free runner is committed and fails clearly when Docker/kind/Helm are unavailable. CI/live evidence for production control-plane registration, roster/no-show, both readiness stages, races, multi-match node, result-pending reconciliation, drain, and rollback remains open |
8.50 [D:8.25,8.37,8.43,8.49] |
IN PROGRESS. make verify-chaos-recovery provides a disposable PostgreSQL + real testkit API + real maintenance flow: it restarts the API, injects a stale allocation, and verifies no-penalty requeue plus a durable participant-targeted lifecycle event |
The API-restart/stalled-allocation slice is implemented and documented; 100 ms RTT/jitter/loss, matcher/client restart, game-pod death, node drain, Redis failover, control-plane loss, and live chaos evidence remain |
8.51 [D:8.17,8.18,8.30,8.31,8.45] |
IN PROGRESS. The opt-in make verify-multiplayer-load gate drives 10,000 real HTTP queue-create requests through the service with 256 in flight and records p95/p99, plus 100 concurrent proposal formations through the real matcher/domain path; the handler and in-process ownership boundary are exercised without weakening normal tests |
Local API load passes at p95 <250 ms in normal and race runs, and the matcher forms 100 unique proposals; PostgreSQL saturation, durable matcher fencing under load, forecast launch concurrency x2, and replica scaling remain live infrastructure gates |
8.52 [D:8.32,8.34,8.45,8.51] |
IN PROGRESS. Allocator supports both an opt-in per-replica fixed-window quota (--allocation-quota / --allocation-quota-window) and an optional PostgreSQL-backed EU/NA quota table consumed inside the serializable allocation transaction before any provider call; idempotent replays do not double-count. The allocator now exposes bounded EU/NA Prometheus counters at an explicit /metrics listener (--metrics-addr), including quota denials; the checked-in rule warns on regional denial activity, and hardened Kubernetes Deployment/Service/ServiceMonitor/PDB/placement resources plus an observability Kustomization provide the provisioning, health, rollout, disruption, discovery, and failure-domain-spreading contract |
Normal/race/vet tests cover local quota exhaustion, window reset, region isolation, invalid input, atomic concurrent consumption, bounded metrics labels, read-only endpoint behavior, and hardened deployment/network policy/lifecycle/PDB/placement invariants; migration/SQL and manifest coverage define the shared quota/metrics boundaries; real image digest/secrets, measured regional cost model, threshold tuning, and denial-of-wallet rehearsal remain |
8.53 [D:7.8,8.13,8.38,8.45,8.46,8.48,8.49,8.50,8.51,8.52] |
IN PROGRESS. scripts/verify_release_gate.py provides a fail-closed promotion check for the ordered development → internal → casual canary → casual → provisional ranked → ranked stages, requiring an evidence report for SLO, security, cost, rollback, EU+NA playtests, and both legacy gates |
Validator and adversarial tests cover skipped stages, unknown stages, missing gates, non-boolean gate values, and blank release IDs; the actual reports, production rollback rehearsal, regional playtests, and live promotion remain open |
Implementation invariants for every task above:
- Matchmade mode is opt-in; every new
ServerConfigdefault preserves the existing community-server path. compose.phase6-smoke.yml,make verify-phase6, andmake verify-enet-integrationare not repurposed or weakened.- Production uses ticketed Hosted Dedicated Server SDR; ENet remains the deterministic local/CI and direct-IP path.
- One process serves one match. Warm processes/nodes absorb startup variance; capacity and cost are determined from 8.34 measurements, not old estimates.
- Implementation evidence is appended under the completed task as in earlier
phases; design changes first update
docs/MATCHMAKING.mdand dependencies.
8. What needs refactoring, not extending
| # | Location | Why extension is insufficient |
|---|---|---|
| 1 | objects/ship.tscn, ship.gd:175-180, 189-208, 241-278 |
No node exists to carry a render-only offset — meshes hang directly off the RigidBody3D. Needs $Visual. |
| 2 | ship_camera.gd:115, 149, 150 |
Camera reads the body's transform, so it would jump the full correction error while the mesh smoothly lags. |
| 3 | match_mode.gd:36, 59-64, 76-82, 93-96, 107-109 |
The Timer + _process clock is frame-rate and time_scale coupled. Must become tick-derived. Five call sites. |
| 4 | match_mode.gd:162-171 |
get_tree().paused = true stops the client's own send loop and snapshot processing, and the return-to-lobby RPC lands in a tree that cannot act on it. |
| 5 | game_mode.gd:95-121, 171-194 |
Engine.time_scale is fundamentally incompatible with a shared tick clock — sequence numbers ride on Engine.get_physics_frames(), so a hit-stop at 0.06 starves the jitter buffer within a few frames. The effects must be reimplemented, not merely disabled. |
| 6 | game_mode.gd:85-92 |
_handle_goal_scored interleaves timing with presentation. On a headless server _play_goal_celebration returns synchronously, so the reset fires on the same frame as the goal — while clients are 1.6 s into a cinematic. |
| 7 | game_mode.gd:248-263 |
_jittered uses global RNG; _reset_body uses set_deferred. Both must become authoritative-broadcast plus a Jolt-correct teleport. |
| 8 | game_mode.gd:54-55, 284-285 |
Unconditional goal-signal connection (an interpolated ball entering a client's local Goal would score locally) and unconditional escape-respawn both write authoritative state on clients. |
| 9 | main_menu.gd (all handlers) |
Every mode launch is a synchronous change_scene_to_file. Connecting is async and can fail — a genuinely new UI state, not another button. |
| 10 | HUDController.gd:41-46 |
Hard-requires a ship; spectators have none. |
| 11 | player_ship_controller.gd |
Single reused ShipAction instance; buffering aliases every history entry. |
| 12 | ship_camera.gd:86 (whole rig) |
Runs in _physics_process, so on a 240 Hz display the FOV kick (:182) and PostFX parameters (:186-187) step at 60 Hz — neither is a transform, so global physics interpolation does not cover them — and the shake noise (:200-212) loses its high-frequency character. Must become _process + get_global_transform_interpolated() (§5.4a, task 0.16). |
| 13 | video_settings.gd:14-16, settings_menu.gd |
Persists AA, glow and brightness only — three values. The three genuinely expensive settings (SDFGI, SSIL, SSAO) and the five shadow-casting lights are unreachable, and neither vsync_mode nor max_fps is set anywhere. A player chasing 240 fps has exactly one lever: turn glow off. Needs a preset system, not another checkbox (§5.5, tasks 0.17/0.17b). |
| 14 | scenes/arena_base.tscn:18-50, 61-105 |
The Environment every arena inherits enables SDFGI + SSIL + SSAO + a 5-level glow pyramid simultaneously, with four shadow-casting OmniLight3Ds (24 cubemap faces/frame). Not tunable per-arena around a preset; the preset must gate the shared base (§5.5). |
| 15 | shaders/post_process.gdshader:4 |
hint_screen_texture forces a full-screen backbuffer copy every frame, not only during turbo — vignette_strength never reaches 0 (ship_camera.gd:187, 243). Either bake the static vignette into Environment.adjustment_* and hide PostProcess when chromatic_aberration is at rest, or drop the screen read for a plain gradient overlay and keep it only for the turbo chroma. |
| 16 | project.godot [display] |
stretch/mode="viewport" + 1920×1080 base + aspect="expand" fixes the 3D render at ~1080p and blits. A 4K player cannot render native; a 1080p player cannot render lower. Blocks any render-scaling setting until decided (task 0.17c). |
On Engine.time_scale: replace hit-stop and goal slow-mo with the camera-based effects in single-player as well (task 0.12), so there is one code path and one game feel to maintain rather than a networked variant that drifts away from the single-player one. ShipCameraRig already has _shake_strength, shake_decay, max_shake_offset and a PostFX ShaderMaterial to build on.
What does not need surgery: the ShipController seam, the Arena/GameMode split, code-driven spawning, group-based discovery, and the dumb Goal sensor all extend cleanly. CLAUDE.md's claim about the three load-bearing seams is accurate — they hold. rl_ship_controller.gd is already the remote-input controller (a public action field that something else writes, pulled each tick), so no new class is needed for it.
9. Godot 4.7 + Jolt gotchas
ENetMultiplayerPeer.server_relaydefaults totrue— clients can RPC each other through your server. Set itfalse.MultiplayerAPI.poll()runs on the idle frame, so anrpc()from_physics_processwaits up to a full frame — andEngine.max_fps = 60on the server is what creates that delay on the return leg. Take manual control (task 1.3). ~16–33 ms of round-trip, for ~10 lines.- Jolt sleeps bodies. A ship corrected to near-zero velocity can sleep and then ignore
state.linear_velocitywrites.can_sleep = falseon Ship and Ball. - Teleporting a rigid body:
state.transforminside_integrate_forcesis the only path with no frame of lag.set_deferred("global_transform", …)lands between frames and interacts badly with Jolt's sleep/wake ordering. reset_physics_interpolation()is not automatic forstate.transformwrites (it is when you setglobal_transformdirectly). Call it explicitly, on the body and on$Visual.physics_jitter_fix = 0.0does not give you "a flat 60 Hz." You still get occasional 0-tick and 2-tick frames, because frame time is never exactly 16.667 ms. The real reason to set it to 0 is that you never want a tick's input delayed by the accumulator smoother. The send path must therefore transmit both ticks' actions on a 2-tick frame — redundancy-4 covers this, but only if you actually send both._integrate_forcesis not called on frozen bodies, so remote ships never pullget_action()— henceset_visual_action. UseFREEZE_MODE_KINEMATIC, notSTATIC, or contact velocity transfer breaks.- Never write
linear_velocityto a frozen body — Godot/Jolt zeroes and holds it. Engine.max_physics_steps_per_framedefaults to 8. If a server tick overruns 16.7 ms the accumulator backs up and the next frame runs multiple ticks, spiking CPU further. Log overruns (task 1.6).- ENet channel indices are offset by Godot's reserved system channels — verify the mapping empirically.
- ENet peer timeout defaults to ~5 s. Tune via
ENetPacketPeer.set_timeout()for faster drop detection. - Jolt is not bit-deterministic across platforms or across differing contact orderings. Never rely on it anywhere, including in "obviously safe" places like a client-side goal check.
dedicated_server=trueexports strip visual resources. Verify against a real stripped build (task 6.2).- MTU: ENet fragments above ~1400 B. At 219 B/snapshot there is ~6× headroom; recheck if per-body cosmetic state is ever added.
- RPC NodePath caching: the first
rpc()to a node sends the full path, later calls send a cached int. Routing hot paths through autoloads warms the cache once at connect and never invalidates it on scene change. - Physics tick rate is 60 for v1 — and must never be a literal. Every policy in
Game/bots/is tick-coupled throughship.gd:450's_tick_scaled(defined at a 60 Hz reference) andai_ship_controller.gd'sreaction_ticks, so raising it toward Rocket League's 120 invalidates every trained model and halves server density. But it is the largest single latency term left (§5.4), so it will be revisited: derive everything fromTICK_HZ(tasks 0.18, 1.1) so that day is a config change plus a retrain. Node3D.get_global_transform_interpolated()is the only correct way to track a physics-interpolated body from_process.global_transformreturns the last physics tick's pose, so a per-frame camera reading it chases a 60 Hz staircase. Per the engine docs the method "creates an interpolation pump… the first time it is called" — call it once before anyreset_physics_interpolation()on that node, or the first hard snap streaks (§4.5).- Physics interpolation covers transforms only.
camera.fov, shader parameters, light energy and anything else written from_physics_processsteps at 60 Hz on a 240 Hz display. Either write them from_processor accept the stepping deliberately. display/window/vsync_modedefaults to enabled (FIFO) andmax_fpsto uncapped. Neither is set inproject.godot. FIFO present latency is 1.5–3 refresh intervals depending on swapchain image count (2 vs 3) and whether the present queue is full — §5's tables use the optimistic 1.5, which assumes the renderer is not GPU-bound. The model does not hold below refresh, where a missed vblank under strict FIFO halves the effective rate and roughly doubles present latency. Prefer Adaptive as the default, not Mailbox (§5.4). (Swapchain image count per platform needs empirical verification.)Engine.max_fpsis a throttle, not a frame pacer. It pads each frame with a post-frame sleep; it has no vblank phase lock. Caps that are not integer divisors of the refresh rate beat against scanout, and combining a cap with an active vsync paces worse than either alone (§5.4). Derive the offered caps fromDisplayServer.screen_get_refresh_rate().DisplayServer.window_get_vsync_mode()echoes your request, not the driver's grant. There is no GDScript API for the negotiatedVkPresentModeKHR, so a UI cannot honestly report what was applied. Show a live fps readout instead and let the player infer it.Engine.max_physics_steps_per_frame = 8is a client problem too, not just a server one (gotcha 9). A client hitching to 20 fps runs 3 ticks per frame, and each of those frames also runs the per-frame camera rig and remote-visual sampling. Set it to 4 client-side (task 0.22). On a multi-tick frame the send path must transmit every tick's action (gotcha 6) — §4.3's_physics_processsampling does this naturally, but nothing else guarantees it.hint_screen_textureforces a full-screen backbuffer copy on every frame the node is drawn, regardless of what the shader then does with it. Branching inside the shader saves taps, not the copy. Hide the node when the effect is at rest.physics_jitter_fixmatters less the higher the frame rate. Its purpose is smoothing when frame rate ≈ tick rate; at 240 fps against 60 Hz physics most frames run zero ticks and the accumulator is never near an edge. Gotcha 6's reasoning for setting it to0.0still holds, but do not expect a visible difference on a high-refresh machine — test that change at 60 fps.MultiplayerAPI.multiplayer_peer's default value is anOfflineMultiplayerPeersentinel, notnull. Resetting it withmultiplayer_peer = null(rather than a freshOfflineMultiplayerPeer.new()) leaves the API in a state distinct from its own default and is a known source of "the server never seespeer_connected,get_peers()stays empty" bugs (godotengine/godot#81540) — confirmed the hard way while building task 1.2'sNetworkManager.shutdown(). Always reset to a realOfflineMultiplayerPeer.- Don't tear down a peer the instant its own connect signal fires.
connected_to_server(client-side) fires once the client's local view of the handshake completes, but the final ACK the server needs to consider its side complete may not have hit the wire yet — closing the peer or quitting the process in the same callback can drop it, and the other side then never seespeer_connected/connected_to_serverat all, even though your own side looked successful. This isn't a corner case: it reproduced on every attempt until fixed, is easy to misdiagnose as a server-side bug (the server-side symptom —get_peers()staying empty — is identical to gotcha 25's), and cost significant debugging time before the actual cause (client-side premature teardown) was found. Give at least one frame — in practicetests/net_smoke.gduses 0.3 s — between a fresh connect signal and callingshutdown()/quit(). Directly relevant to task 5.6's disconnect/reconnect controller swap and any CLI test client that connects, asserts, and exits quickly. change_scene_to_file()must be called on (or from a descendant of) the actualget_tree().current_scene, and never synchronously from_ready(). Both failure modes were hit building task 1.5'slobby.tscn/tests/lobby_smoke.gd: (a) a test harness that instantiatedlobby.tscnas a plain child of a driver node — rather than loading it as the real current scene, the waymain_menu.gd's Host/Join flow will — causedlobby.gd's own (entirely correct, standard-pattern)change_scene_to_file(ScenePaths.MAIN_MENU)disconnect handler to hang the process completely on a real disconnect, with near-zero CPU (blocked, not spinning) and no error output; the fix was to load the scene the way production actually will, not to change the production code. (b) callingchange_scene_to_file()(oradd_child()onget_tree().root) synchronously from inside_ready()throws "Parent node is busy … Consider using.call_deferred()", because the tree is still mid-traversal adding the very node whose_ready()is running;main_menu.gd's real button-press handlers won't hit this (they run outside any_ready()), but anything that needs to trigger a scene change during its own initialization must.call_deferred()it.ENetMultiplayerPeer'sconnection_failedsignal is not bounded to anything a UI should make a player wait for. Verified empirically (task 1.7): against a genuinely refused loopback connection (nothing listening on the target port),connection_failedhad still not fired 14 seconds in. Don't rely on it alone to end a "Connecting…" state — run your own app-level timeout (main_menu.gd'sCONNECT_TIMEOUT_SECONDS = 6.0) that shuts the peer down and shows an error regardless of whether ENet ever gets around to reporting failure itself.- A
MultiplayerPeer's "am I a client" flag (however you track it —NetworkManager.is_clienthere) turns true the instantjoin()/create_client()is called, not once the connection actually completes. Anything gated on that flag alone (task 1.8's clock ping, innetwork_manager.gd's_process) will try torpc_id()on a peer that's stillCONNECTING— or has already failed — during a slow or refused connect attempt, and Godot logs "Trying to call an RPC via a multiplayer peer which is not connected" every single frame until it resolves. Gate on the peer's actualget_connection_status() == MultiplayerPeer.CONNECTION_CONNECTED, not just the higher-level intent flag. load()on a.gdfile with a parse/compile error does not returnnull. Found via adversarial review oftests/test_runner.gd: it returns a non-null but uninstantiableGDScriptresource, soif script == nullsilently fails to catch the failure — and the natural next line,script.new(), throws "Invalid call: Nonexistent function 'new'", severe enough to abort the entire calling function (not just that statement) without ever reaching whatever cleanup/exit code follows. In a loop over multiple files with no per-iteration error boundary, this reads as a hang: the loop that would have moved to the next file, and the code that would have calledquit(), both never run. The real guard isScript.can_instantiate().- An
@rpcmethod named_inputcollides withNode's built-in_input(event: InputEvent)virtual. Found building task 2.1'sMatchSimautoload: naming the client→server input RPC_input(bytes: PackedByteArray)produced a parse error ("function signature doesn't match the parent") — and because this was on an autoload, the error broke the entire autoload from loading, cascading into unrelated failures across every scene that touchedMatchSimat all, none of which mentioned RPCs or_inputin their own error output. Renamed to_recv_input. General lesson: on an autoload especially, treat any bare virtual-sounding method name (_input,_process,_ready,_unhandled_input, …) as reserved regardless of what you intend it to do — a signature mismatch there doesn't fail locally, it fails the whole autoload. - Disabling automatic multiplayer polling (task 1.3) is global, not autoload-scoped — every scene that touches an RPC, not just
NetworkManager-adjacent code, must callNetworkManager.poll()itself every frame it wants traffic to move. Building task 2.1–2.3,networked_match.gd's_physics_process/_processsent and listened for RPCs (MatchSim.request_match_config,send_input, snapshot RPCs) but never calledpoll()— nothing sent viarpc()in this scene ever reached the wire in either direction, silently, with no error in either process's log. Confirmed via debug prints: the client's request fired, but the host's handler print never appeared. The first (wrong) hypothesis was a startup race between the server's broadcast and the client's listener connecting — that fix (a request/response retry pattern, still worth keeping for the genuine late-join case) didn't resolve it alone. The real fix was addingNetworkManager.poll()at the top of both_physics_processand_processin the new scene. If a scene sends or receives RPCs and nothing arrives with no errors at all, check for a missingpoll()before anything else. - A request/response fallback for a one-shot broadcast can double-deliver, and the receiving handler must be idempotent. Once gotcha 32's fix made polling actually work,
_on_match_config_receivedran twice per client — once from the server's original one-shot_match_config.rpc()broadcast (queued the whole time, since it had been sent before polling was fixed) and again from the request/response retry — producing two arenas, two ship sets, two HUDs (_slots.size() == 2instead of 1). Any handler reachable via both an original broadcast and a "resend on request" path needs its own guard (here:if not _slots.is_empty(): returnat the top) rather than assuming "only sent once" from the RPC design alone. - An
Area3D'sbody_enteredsignal fires as part of physics tick N's own step, strictly before tick N's_physics_processcallback — not "on the next frame." Found while fixing the goal-reset-ordering bug above: a boolean "handle this on the next_physics_process" flag set from inside abody_enteredhandler is a no-op, because that same tick's_physics_processhasn't run yet and sees the flag already true — it "defers" to the same tick it was set on, not the next one. If you actually need next-tick-or-later semantics, compareEngine.get_physics_frames()against the tick the flag was set on and require strictly-greater, not just "check a boolean at the top of_physics_process." - A queued
queue_teleport()(task 0.15) can take one tick longer to land than "the very next_integrate_forces" suggests, when the call originates from a signal handler mid-physics-step rather than from a_physics_processcallback. Empirically confirmed by teleporting a body into a goal and logging the server's own per-tick broadcast: the goal was detected on tick N (per gotcha 34, during tick N's own step), but the reset position didn't appear in a broadcast until tick N+1's, one tick later than "queued during N, applied on N+1's_integrate_forces" alone would predict. Don't assume queued-teleport timing without checking a real tick-by-tick log for your specific call site — the exact tick it lands on depends on where in the physics step the queuing call happens, not just "next frame" intuition. NetworkManager.get_server_time_estimate_ms()(and anything derived from it) is not just imprecise before the first clock pong lands — it's actively wrong in a way that can persist far longer than the sync window implies.clock_offset_msis0.0until the first pong, so a value derived fromget_server_time_estimate_ms()during that window means "my own raw process uptime," not a server-synced estimate — and if that value feeds a rolling-window filter (e.g. a min-tracked bias, per the interpolator epoch-bias fix in Phase 2's adversarial review), the bad early sample can dominate the window for the filter's entire configured duration if a short test or a short match doesn't run long enough for real time to age it out. Always gate recording, not just consuming, anything derived from this estimate onrtt_ms >= 0.0.- Anything that deliberately delays an RPC dispatch (task 2.8's
net_sim.gd) must re-validate its target at fire time, not just at the moment it was scheduled. Found by actually running Phase 2's own gate (networked_match_smokeunder--net-sim-latency=80 --net-sim-jitter=20), not the isolated ping/pong test alone:_broadcast_snapshot's existingget_peers()filter (gotcha from task 2.2's own fix) only proves the target was valid when the send was queued — a target that legitimately disconnects during the ~80–100ms hold produces "Attempt to call RPC with unknown peer ID" anyway, and if the delay outlives this process's ownshutdown(),multiplayer_peerhas already been reset to a freshOfflineMultiplayerPeer(§9 gotcha re: never resetting to rawnull), so a stalerpc_id(1, …)now means "call yourself" and Godot rejects it. Any deliberate-delay layer needs to re-check both "do I still have a real peer at all" and "is this specific target still inget_peers()" inside the delayed callback itself, immediately before dispatching — not reuse whatever validation the caller did before the delay was added. - A GDScript lambda captures an enclosing local variable BY VALUE at the moment the lambda is created, not by reference. Bit two separate Phase 3 test scripts the same way:
var disconnected := false; some_signal.connect(func(): disconnected = true)compiles and runs with no error or warning, but the assignment inside the lambda mutates only that lambda's own captured copy — the enclosing function'sdisconnectedstaysfalseforever, even after the signal genuinely fires (confirmed firing via an extra debug print before the real cause was found). The underlying disconnect-detection code was correct the whole time; only the test's own assertion logic was broken. The fix is to capture a container instead of a value —var disconnected := [false]anddisconnected[0] = trueinside the lambda — since capturing anArray/Dictionary/Objectcaptures a reference to the same instance, and mutating its contents from inside the lambda is visible outside it. Relevant anywhere a lambda is used to flip a flag or accumulate a result for a caller to read later (aconnect(func(): ...)one-liner is the single most common place this bites). - A fixed-size ring buffer fed by an unbounded-rate producer needs an explicit resync path, not just "wait for the next expected slot."
InputJitterBuffer's 32-entry ring assumed the consumer (consume(), one call per server physics tick) would never fall more thanRING_SIZEticks behind the producer (ingest(), driven by real wall-clock packet arrival, unrelated to the consumer's own tick rate) — but a server-side stall, or even ordinary client/server clock drift with zero external trigger, breaks that assumption, and once broken, a design that only ever advances its "expected" pointer by exactly one per call can never catch up: newer arrivals silently overwrite the exact slot still being waited on, and the wait never ends. If a ring's producer and consumer rates aren't provably bounded relative to each other, the consumer needs a way to detect "the data I'm waiting for no longer exists in the ring at all" (track the newest value ever seen, independent of ring capacity) and jump directly to what's still available, rather than assuming "keep waiting" is always eventually correct. - A client-owned adaptive control loop must react to the actual ground-truth signal it's regulating, not to its own memory of past decisions.
InputLeadController's release logic was gated onlead > LEAD_MIN— a count of the controller's own past attacks — rather than on the real server-reportedinput_buffer_depthit exists to keep near target. Any elevated depth the controller didn't itself cause (an external stall, drift, a burst redelivery) was invisible to that gate and so never got drained, even while the "real" signal sat well above target the whole time. When a control loop's condition for acting can be satisfied or blocked by state the loop itself controls, rather than by the environment it's meant to respond to, it can silently stop responding to the environment. - A "consecutive N over-budget windows" streak counter that hard-resets to 0 on any single clean window is trivially evaded by a duty-cycled attacker (burst hard, one clean window, repeat) — confirmed sustaining ~33x a stated packet budget indefinitely with zero disconnect warnings. A leaky-bucket accumulator (grows by each window's actual total, drains by exactly one window's worth of budget every window, disconnect once the accumulated excess crosses a threshold) is immune to the same evasion by construction, since it doesn't matter how the excess is distributed in time — only the sustained average matters.
- Two counters that don't share an epoch must never be compared directly, even when both are monotonically increasing integers that "look like" the same kind of thing.
seq > Engine.get_physics_frames() + 20compiled, ran, and looked like a sane bound — butEngine.get_physics_frames()counts from the SERVER PROCESS's own start while a client's_input_seqstarts at 0 when ITS match scene loads, so the check either never fires (on a long-running server, no real protection despite its own comment's claim) or fires wrongly and silently drops an honest client's input forever, depending entirely on how much unrelated head-start or drift has accumulated between the two clocks. Bound a value against another value that shares its own actual epoch (here: the receiving buffer's ownlast_applied_seq), not against a same-typed number from a conceptually different clock. - A regression test that doesn't independently exercise the specific mechanism it claims to gate will pass even when that mechanism is completely broken. Task 3.6's CI driver asserted snapshot throughput and a server-forced goal's score agreement — neither of which depends on client input ever reaching the server — and kept reporting PASS with a real, reproduced bug (§7's ring-overflow) actively zeroing both bots' input for the whole run. A CI gate's assertions should trace back to the specific claim in the task's own acceptance text, not just "the match ran and didn't crash."
- When adding a "is this connection still healthy" check to a test, sample it while the peer is still actively connected — not after its own end-of-run disconnect, which produces symptoms indistinguishable from the bug being checked for. Fixing gotcha 43 first sampled
InputJitterBuffer.stalledand ship movement after the full test run (plus a buffer for score-file writes), which meant both readings came from ~4s after the bot had already legitimately shut down — a departed peer's input naturally starves and goesstalled=truetoo, and that's correct, expected behaviour, not the bug. Move the check to a moment still comfortably inside the peer's own active connection window. - Two fixes landed in the same commit, each individually correct in isolation, can share a variable and silently cancel each other out — and a fix's own unit test can miss it by testing the mechanism in isolation from the thing that defeats it. Gotcha 39's resync fix and gotcha 42's guard rebound were reviewed, tested, and verified independently, each against its own scenario, both passing. Combined, the guard caps the exact variable (
highest_ingested_seq) the resync's own trigger condition depends on, making it permanently unreachable — recreating the original critical bug at a lower failure threshold than before either fix existed. The resync's own new unit test calledInputJitterBuffer.ingest()directly, which is correct in isolation but bypasses the guard entirely, so it could never have caught this regardless of how thorough it was on its own terms. When two fixes in the same round touch the same subsystem, explicitly re-test the combination end-to-end (here: a realSIGSTOPfreeze against the actual production RPC call path, not a direct unit-level call into the class the fix lives in) — passing tests for each fix individually is not evidence the pair composes correctly. - A guard that bounds an incoming value against the consumer's position, rather than against the producer's own epoch, re-introduces exactly the "consumer can never catch up past a stall" failure it's often added specifically to prevent. The seq-range guard bounded
seqagainstlast_applied_seq(advanced only byconsume(), i.e. gated on however fast the physics tick loop is actually running) rather thanhighest_ingested_seq(advanced byingest(), i.e. gated on however fast packets are actually arriving and being processed bypoll()) — during a stall where ticks fall behind but polling keeps pace (the common case: a single-frame hitch, orEngine.max_physics_steps_per_framecapping tick catch-up whilepoll()itself isn't similarly capped), bounding against the lagging consumer rejects the very packets that would let the buffer refill and the resync condition ever trigger. Bound against whichever side of a producer/consumer pair is not the one already known to be falling behind. - A trace that holds its inputs steady cannot falsify anything about which sequence a prediction is filed under — and "we hold thrust for 60 seconds" describes almost every prediction test people write. Phase 4's history was filed under the wrong sequence (the estimated server-consumption seq,
input_leadticks behind issuance, instead of the issuing seq), and the dedicated action-marker instrument built to catch exactly that reported a flawlessmarker=0/3784across 60-second LAN, 80±20 ms and 5%-loss runs. It was not broken: while the commanded action is constant, "the intent from this tick" and "the action the server consumes for seq S" hold the same value, so a right and a wrong label are indistinguishable. Only an input edge separates them, and only for aboutinput_leadticks per edge. The bug then scales withinput_lead— 9.3% mismatch at lead 1, 24% at lead 3 — meaning it was worst precisely on the impaired links the test matrix existed to cover, and invisible in all of them. When a test is meant to validate a label, an index, or a phase relationship rather than a magnitude, the trace has to change that quantity frequently; a steady-state trace validates the magnitude and silently asserts nothing about the label. - A guard whose bound is derived from a value only the ACCEPTED path can advance is a latch, not a guard. The seq-range check has now been written three times — bounded against server uptime, then
last_applied_seq, thenhighest_ingested_seq— and all three could permanently reject an honest client's input, because in every version the quantity being compared against could only move forward via a packet that got through. Once enough drift or loss accumulated, nothing could ever move it again. The property to check when writing a guard like this is not "is the bound correct?" but "if this guard rejects everything from now on, what advances the bound?" If the answer is "an accepted packet", it needs an independent escape path (here: resync after N consecutive rejections) regardless of how well-chosen the bound is. - Advancing a consumer cursor past data that has not arrived is not a lossy shortcut — it is permanent, because the producer-side filter then rejects the very data being waited for.
InputJitterBuffer.consume()advancedlast_applied_seqon a starve, andingest()discardsseq <= last_applied_seq. One starve on a sequence the client had not sent yet therefore stranded the stream one ahead of arrivals forever — both sides advancing in lockstep, the gap never closing, every packet discarded on arrival. The client's own routineinput_leadrelease was enough to trigger it, roughly every 6.5 s on a clean LAN. Only give up on an expected item once strictly newer data proves it lost; "it hasn't arrived yet" and "it will never arrive" are different states and must not share a code path. - A metric that stops sampling during a failure will report that failure as healthy. The action-marker gate printed
SMOKE PASSat 3.76 % on a run where the player's input was permanently dead — because reconciliation suppression stops_record_metricsbeing called, so the worse the outage, the fewer samples and the lower the computed mismatch rate. Every rate-shaped assertion needs a companion assertion on the denominator (here: a sample count scaled to run length), or an outage silently becomes an absence of evidence and then evidence of absence. - An architectural blocker inherited from a previous session is a claim to verify, not a premise to build on. Phase 4 was handed over blocked on approval for a client-only shadow Jolt world — a large subsystem, and effectively the whole-world rollback §1's locked decisions rule out. The actual same-sequence defect turned out to be a one-line mislabel, falsifiable in about an hour with instrumentation that already existed; the shadow world remains genuinely necessary for the contact cohort but nothing else, which is a far smaller commitment than "Phase 4 is blocked on it." Reconstruct the failing invariant from the code and reproduce it against a control before accepting a scope estimate attached to it — especially when the recommendation arrives without the cheaper alternative recorded as tested.
10. Testing
Editor. Debug → Run Multiple Instances, 2–3 instances with per-instance args (-- --server, -- --connect 127.0.0.1:27015) and --position so windows don't stack.
CLI.
godot --headless --path Game res://scenes/server_boot.tscn -- --port 27015 --team-size 1 --auto-start
godot --path Game -- --connect 127.0.0.1:27015 --name Alice
CI smoke test (task 3.6). Headless server plus two headless --test-bot clients, driven by the existing AIShipController. Asserts:
- snapshots received ≥
N * snapshot_hz * 0.9 - own-ship prediction error p95 < 0.5 m, p99 < 2.0 m, hard-snap count < 3
- final score identical on the server and both clients
- no
push_erroremitted (scrape stderr)
Network conditions. net_sim.gd (task 2.8) is first-class: seeded so failures reproduce, works in CI, needs no display, and can be applied asymmetrically — which OS tools make painful. tc netem / Network Link Conditioner / clumsy for a pre-release realism pass only. A real remote host once per phase from Phase 4 onward is the only true test of the jitter buffer's adaptivity.
Unit tests (task 1.0). No test framework exists today, so keep it minimal — a scene that runs pure-function assertions and exits with a code. High-value targets, all zero-engine-state: codec quantise/dequantise round-trip and bounds; quaternion max error; snapshot pack→unpack identity; input packet framing; jitter-buffer policy against scripted arrival traces; ShipAction.copy() non-aliasing. These are exactly where a bug is invisible in play and catastrophic in aggregate.
11. Flagged, not solved
Former item C — display-name slot takeover — RESOLVED locally. _try_reclaim_slot now compares the verified signed PlayerID retained in the server roster and slot; late-join promotion carries the same identity. A changed display name can reconnect, but a same-name peer with a different identity cannot. Direct unauthenticated servers retain a documented display-name fallback for backwards-compatible community hosting. Live public-internet verification still remains a separate Phase 6 gate.
Low-latency present and graphics presets — now specified, see §5.4, §5.5 and tasks 0.17/0.17b. Left here as a pointer because they are the largest wins in the document per line of code changed, and they are video settings rather than netcode.
120 Hz simulation — deliberately deferred, not dismissed. §5.4 and §5.6 record what it would buy (≈21 ms of world response once L1 has taken the interpolation buffer out, plus ≈8 ms of own-ship feel — the difference between ≈127 ms and ≈107 ms), what it costs (a full bot retrain, half the server density, double the bandwidth), and the one rule that keeps the door open: TICK_HZ, never 60.
The latency gap to the reference has a plan but not yet a measurement. §5.2 lands at ≈174 ms as designed; §5.6 routes that to ≈127 ms (tasks 0.17d, 4.9) and ≈103 ms (tasks 4.10 plus 120 Hz simulation), against ~90–110 ms for the reference class at the same RTT. Every figure in §5.6 is arithmetic on the budget, not a measurement — task 4.9's acceptance criterion exists to make it one. Beyond that the residual is RTT, which is a server-siting problem (§6) rather than a code one and is worth more than every remaining code lever combined.
Audio. TODO.md records that there is none. set_visual_action / set_visual_speed (task 0.14) is precisely where remote-ship engine audio will hang, and "ball feel" (task 4.6) is half auditory. Design those hooks with that in mind rather than retrofitting.
Split-screen. Tracked separately in TODO.md; unrelated to this effort, though the camera-outside-the-ship structure that enables it is the same structure this plan relies on.
Item E of §0 — stale snapshot sends after forced disconnect — is now resolved locally. MatchSim.send_snapshot() validates the live peer and NetSim._fire() revalidates delayed targets immediately before dispatch, covering the deliberately adversarial client-abuse-malformed path as well as normal disconnects. A full multi-process abuse smoke remains a useful runtime check, but the stale-target call sites no longer enter Godot's RPC path after peer teardown.
Deployment wiring update (2026-09-01)
The current working implementation now wires deploy/k8s/base/fleet.yaml to the digest-pinned game-server supervisor target, the in-cluster control-plane Service, workload roster materialization, signing/drain secret references, downward-API server/image identity, and the required game-server egress policy. kubectl kustomize deploy/k8s/base and server/security/test_fleet_manifests.py pass. The older 8.28 narrative above still records the pre-wiring state; live Agones, operator secret/image replacement, and real cluster readiness remain explicit gates.
An adversarial Fleet-entrypoint review found that the supervisor invocation had
no executable after --, and that its required supervisor-level protocol flag
was missing. The Fleet now passes the exported Godot server explicitly and
sets --protocol-version=1; the NA overlay's positional patch and manifest
regression test were updated together. This is a local launch-contract fix,
not evidence of live Agones readiness.
The NA overlay now also patches the allocated child’s --region=NA argument, keeping it aligned with the NA Fleet label; rendered EU and NA overlays and the adversarial manifest test verify that regional assignment validation cannot silently remain EU in the NA deployment.
Allocated Godot startup now derives its min-players floor from the verified signed roster size, preventing the direct-server default of one player from starting a partially admitted allocated match. A focused regression test covers six-player, casual two-player, and direct-server behavior; the full Godot harness is currently unavailable because Godot cannot open its shared user:// log and crashes in the macOS renderer before test execution.
The former display-name reclaim weakness (flagged item C) is now closed for allocated matches: the signed PlayerID is retained in the server roster and slot, and both reconnect reclaim and late-join promotion carry that stable identity across peer-id changes. Display-name matching remains only as a legacy fallback for unauthenticated direct servers. A focused adversarial unit test covers changed names, same-name impostors, missing identities, and the direct-server fallback.
Allocated supervisor launch arguments now have a direct regression guard: authoritative match/server/image/assignment-expiry values replace stale child placeholders without mutating the caller’s command slice or disturbing unrelated arguments; dynamic Agones port propagation remains covered by the existing startup test. This closes the local implementation portion of task 8.29; live Agones passthrough/NAT and multi-match validation remain infrastructure gates.
Read-only authenticated queue, proposal, assignment, legacy profile, and ranked-profile routes now emit lifecycle-safe observability events for successful, rejected, and not-found reads. An API regression exercises all five real HTTP routes and verifies the event set; event fields remain free of credentials. This closes the local read-route portion of task 8.44; metrics/traces export, dashboards, and alert routing remain operational work.
Allocated join admission now retains and applies the signed assignment’s authoritative team and global slot: peer order can no longer rebalance a valid allocation, and inconsistent team/slot claims are rejected before roster admission. The server exposes the verified assignment list for allocation-aware startup and uses the per-team spawn index derived from the assigned slot. Go verification is clean; Godot execution remains blocked by the documented macOS pre-test crash.
Allocated boot now also validates the complete signed roster shape before opening the gameplay endpoint: malformed claims, duplicate player identities, duplicate slots, and team/global-slot mismatches fail closed rather than leaving a partially usable server. The Godot --check-only attempt still reaches the known macOS renderer/ZSTD crash before script parsing, so this startup guard remains statically reviewed and covered by the existing signed-claim tests pending a working Godot runtime.
The control plane now mirrors that topology fence at roster publication: signed entries with duplicate players, duplicate slots, or a team inconsistent with the canonical global slot are rejected before durable assignment rows are written. Focused store tests cover forged topology and duplicate entries; normal/race Go suites and vet pass.
The backend roster persistence boundary now enforces the same duplicate-player, duplicate-slot, and team/global-slot invariants as Godot startup. This closes the remaining local consistency gap in task 8.31; production signer/client-ticket publication and live Agones verification remain external gates.
The no-show policy now has an explicit domain translation layer (PlanInitialConnect): WAIT remains non-mutating, ranked no-shows produce a CANCELLED match plan with innocent-player IDs, and eligible casual play produces a LIVE plan plus the complete bot-filled six-slot lineup. Normal/race domain tests cover both branches; applying the plan transactionally to durable tickets/matches and wiring it into the allocated server lifecycle remain task 8.35 work.
The durable no-show boundary is now implemented by ApplyInitialConnectPlan: it locks the match and roster, validates that the plan covers every active participant, records deterministic no-show cooldown penalties, fails no-show tickets, requeues innocent tickets on cancellation or advances connected tickets to LIVE for eligible casual bot start, and emits a replayable state-change outbox event under the same serializable transaction. Idempotency keys reject conflicting retries. Focused store tests, race tests, and vet pass; the real PostgreSQL integration remains an environment-dependent gate.
The maintenance command now invokes a bounded ReconcileInitialConnect sweep for ASSIGNMENT_READY/ASSIGNED/CONNECTING matches, carrying ranked no-show history into the domain ladder and skipping non-actionable WAIT plans. This closes the local control-plane trigger for task 8.35; actual allocated-server bot spawning, shutdown signaling, and live Agones integration remain separate gates.
Allocation registration now writes a participant-targeted, revisioned state_changed outbox event for both PROCESS_READY and ASSIGNMENT_READY transitions. The production and test API binaries run a type-scoped dispatcher with delivery-before-ack semantics, so allocation lifecycle events survive WebSocket outages without competing with proposal or result consumers. Store/API adversarial tests cover event-type isolation, target validation, and revision mismatches; live allocator/Agones delivery remains an integration gate.
The state-event implementation is now complete through the registration boundary: the durable registration SQL returns the authoritative match revision, includes every participant target in the payload, and the dispatcher validates aggregate/revision/state consistency before fan-out. Full Go tests, race checks, and vet pass after an adversarial database-cursor review.
Allocated Godot runtime now applies the same initial-connect policy: ranked allocations cancel and exit after 30 seconds if the signed roster is incomplete; casual allocations wait 60 seconds, cancel when fewer than two humans or one team is absent, and otherwise start with a deterministic six-slot assignment-derived lineup containing explicit bots. The bot branch is opt-in and consumed once, so direct servers and ranked matches cannot inherit it. Godot parse plus the 155-test harness and manifest checks pass; durable no-show penalties/state reconciliation remain owned by the control-plane sweep.
An adversarial transaction review found that cancellation released only no-show participant rows, which would leave innocent players marked active in the cancelled match and trip the active-match uniqueness fence on their next match. ApplyInitialConnectPlan now releases the complete participant roster on cancellation, while retaining cooldown penalties only for no-shows; the full Go suite, race checks, and vet pass.
The documented server_shutdown reliable control message is now implemented in MatchNet, with bounded reason sanitisation and an authority-only receiver signal. Controlled drain broadcasts server_draining; allocated initial-connect cancellation broadcasts its policy reason and waits a transport-flush beat before closing. The 156-test Godot harness covers emission and bounds; full multi-process drain delivery remains a live integration gate.
Clients now consume planned shutdowns: the reason is retained for presentation, an in-match client returns to the lobby after the notice, and the generic disconnect callback is fenced so it cannot overwrite that planned transition. Lobby clients surface the reason directly. The complete Godot harness remains green; real two-process drain delivery is still an external runtime gate.
An adversarial UI review found the lobby’s generic disconnect handler still replaced that message with the main menu immediately afterward. Planned disconnects are now fenced in the lobby, and a lobby reached from an active match restores the retained reason on startup; unplanned disconnects keep the existing main-menu behavior.
The workload-authenticated POST /servers/{serverId}/shutdown contract is now exposed for allocated servers. It validates the bound credential and reason, records an idempotent SERVER_SHUTDOWN audit event under a serializable transaction, and returns a stable acknowledgment on retry; match-state transitions remain owned by the no-show/result transactions. API/store tests cover authorization, validation, idempotency SQL, and audit wiring; live PostgreSQL delivery remains an integration gate.
The allocated supervisor now calls that shutdown acknowledgment during signal-bound controlled drain, using the same workload credential and a deterministic idempotency key after the local drain request succeeds. The lifecycle test verifies the drain-before-ack ordering, credential separation, and bounded graceful child exit; live pod termination and control-plane outage behavior remain deployment gates.
Allocator-selected region, build, protocol, and transport now travel with the allocation as Agones annotations and override stale child launch flags immediately before an allocated process starts. The overlay rejects control characters and preserves direct-server command behavior; focused supervisor/allocator tests cover precedence and annotation payloads, while live Agones passthrough remains an infrastructure gate.
The same allocation path now carries the matcher-selected playlist, preventing a ranked match from inheriting the Fleet’s casual default. Durable allocation claims return the playlist, the worker includes it in Fleet selection metadata, Agones copies it to the allocated GameServer, and the supervisor overrides --playlist before launch; the existing compatibility tests remain green.
The allocator’s durable bind now increments the match revision and writes a participant-targeted state_changed(ALLOCATING) outbox event in the same serializable transaction as the server binding and ticket transitions, so clients can recover allocation progress after a delivery outage.
Ranked maintenance now marks expired seasons with no ranked profiles as rolled over, preventing an empty season from being selected and reconsidered on every maintenance pass; the boundary is covered by the integration-tag regression suite.
Season rollover now computes from the row locked inside its serializable transaction rather than a stale caller snapshot; the PostgreSQL integration regression deliberately passes a 1900 profile against a durable 2000 rating and verifies the 1875 result is preserved.
The production ranked-profile adapter now projects the active ranked season ID from the durable seasons table while keeping rollover history separate; the API prefers that current-season value and retains the legacy in-memory fallback for existing callers.
Allocator quota accounting now charges only fresh provider attempts; recovery of a provider result after an ambiguous durable write does not consume the same regional quota a second time.
Accepted-proposal allocation now binds the request back to the proposal’s playlist, arena, region, and protocol before any provider call; adversarial mismatches fail closed.
Ranked proposal admission no longer trusts the matcher’s --ranked-random-arena boolean. The Go domain now owns a named allowlist for the three floor-goal ArenaRegistry entries, and rejects unknown and elevated IDs before any proposal is created.
The arena hand-off is now durable: the matcher deterministically selects an eligible floor-goal arena from the proposal ID, migration 0008 stores that path on proposals and matches and enforces it for new direct SQL writes, migration 0009 retains it on provider allocations, domain/store/provider boundaries and recovery lookups recheck the same allowlist, allocation claims and idempotency digests retain it, Agones applies it as a match-scoped annotation, and the supervisor overlays the allocated child’s --arena-path. Godot accepts only the same floor-goal ArenaRegistry paths and requires one for allocated ranked matches, so a stale Fleet default, an elevated variant, or an altered retry cannot substitute a ranked arena. The recovery worker also rejects a provider-recovered allocation whose arena differs from the durable request before recording or binding it.
The Godot control-plane client now retains the exact last idempotent mutation and exposes retry_last_mutation() for transport, timeout, rate-limit, and 5xx failures. Retries reuse the original idempotency key and expected revision, while 401 and 409 responses remain non-retryable; the harness covers the policy boundary. This closes the local duplicate-action recovery mechanism for heartbeat/cancel/proposal calls, with broader live UI retry verification still remaining.
The matchmaking UI now exposes that retained replay through its existing action button as Retry Request while a heartbeat, cancellation, or proposal action has a retryable failure. Terminal, authentication, and revision-conflict paths remain ineligible, so the button cannot issue a stale blind command.
The control plane now exports bounded Prometheus-compatible API request counters and latency summaries at GET /metrics, with fixed operation/status labels and no event-stream wrapping. Production and testkit services wire the collector; adversarial tests verify unknown paths cannot inject label cardinality or leak URL secrets, and full Go/race/vet checks pass. Durable SLO dashboards and alert routing remain operational work.
The authenticated event stream now rejects client data/reserved opcodes and oversized control frames at the parser boundary; only RFC 6455 close, ping, and pong frames are accepted from clients, preserving the bounded v1 stream contract.
Event delivery also applies a bounded write deadline, so a client that stops reading cannot strand the event handler after the bounded subscriber queue evicts it.
make verify-multiplayer-local now provides one cloud-free regression gate for the current implementation: the complete Go suite, the Godot harness, OpenAPI parsing, and the migration/Fleet/Kubernetes/supply-chain checks. It fails clearly when the configured Godot executable is unavailable and does not weaken or replace the existing Phase 6/ENet gates; PostgreSQL, Redis, Steam, Agones, and multi-process Internet gates remain separate.
That local gate now also runs go test -race ./..., go vet ./..., and each declared domain fuzz target for a bounded 2-second interval, aligning the one-command gate with the separately recorded 8.46 verification requirements.
Observability redaction now adds content-aware protection on top of denylisted field names: bearer values, compact JWT-like strings, PEM material, and long opaque mixed alphanumeric values are redacted recursively through arbitrary nested maps and string slices. Unknown-key credential canaries pass without leaking; false-positive risk is limited to custom long opaque fields, while canonical correlation IDs remain outside the free-form field map.
The authenticated control-plane WebSocket now caps each player at two
simultaneous connections, releasing capacity on disconnect; this complements
the bounded per-player event queue and prevents connection fan-out from
becoming an unbounded account-level resource cost. Over-limit attempts fail
before upgrade with 429 websocket_connection_limited, rather than becoming
ambiguous post-upgrade disconnects.
Proposal explicit-decline cooldowns are now durable: the declining player is
requeued for recovery, but a subsequent queue create is rejected until the
playlist-specific cooldown computed by domain.CooldownUntil expires. The
operation is idempotent and does not affect the other participants' requeue;
timeout-derived cooldown recording now uses the same durable penalty path,
with deterministic per-proposal/player IDs for replay safety.
Current local completion index (2026-09-01)
The following Phase 8 slices have local implementation and verification evidence in this document: 8.29 dynamic allocated launch flags and endpoint handling; 8.30 allocator claim/reconciliation including provider-outcome recovery fencing and durable arena identity (migrations 0008–0009); 8.31 signed assignment/roster validation; 8.35 initial-connect no-show and casual bot policy; 8.36 controlled drain and shutdown acknowledgment; 8.39–8.43 client state, assignment, profile, recovery, and idempotent action retry; 8.44 structured observability and content-aware redaction; 8.45 bounded API metrics export plus optional Prometheus scrape/alert rules; 8.46 normal/race/vet/fuzz coverage; 8.47–8.48 offline/testkit/Compose coverage; 8.50 atomic stalled-allocation recovery notifications; 8.51 the 10,000-client API load boundary; 8.52 the opt-in per-replica plus shared PostgreSQL regional allocator quota; and 8.53 the fail-closed promotion validator. Their remaining acceptance text is infrastructure or production dependent where explicitly noted below the corresponding row.
The following are not locally certifiable from this workspace and remain open prerequisites rather than silently “done”: Valve/GodotSteam credentials and hosted SDR (7.1–7.8), live PostgreSQL/Redis execution where Docker is unavailable, live Agones/kind lifecycle (8.30–8.38, 8.49), public-network chaos/load/cost/release gates (8.50–8.53), and real-hardware graphics profiling (0.15b onward). make verify-kind-agones is the committed runner for 8.49; its response validator is unit-tested against malformed/ambiguous allocation payloads, but the runner still requires a running Docker daemon plus kind, kubectl, and Helm. TODO.md’s AI-training and presentation tasks remain separate from multiplayer and are not marked by this index.
The deferred teamplay TODO prerequisite is now implemented locally but not
enabled: team-touch credit is opt-in and the evaluator can run paired 2v2
matches with --team-size=2. No Stage 7 training run or promotion is claimed;
teamplay still needs recorded behaviour thresholds and a working Godot runtime
for its end-to-end evaluation.
The generation-5 environment now also has opt-in wall-play and pre-rebound episode starts. Stage 6's next command enables each at 10% after the Stage 5 aerial baseline; Stages 4–5 retain their prior distributions. The state generator and configuration are covered statically, but no training pass or promotion is claimed until the Godot runtime and telemetry gates are available.
The three declared domain fuzz targets have now each completed a bounded 4-second run (FuzzQueueCreateDoesNotPanic, FuzzResultDigestIsDeterministic, and FuzzSyncEventApplicationDoesNotPanic) with no failures; this closes the locally runnable fuzz portion of 8.46. Live Redis failover, further transaction races, and cloud/runtime gates remain explicitly unverified.
The real ENet integration gate now passes with GODOT_BIN=/Applications/Godot.app/Contents/MacOS/Godot bash scripts/verify_enet_integration.sh, covering the net, match-net, clock, lobby, and networked match process scenarios. The default GODOT_BIN remains the portable godot PATH lookup for CI; this machine requires the explicit app-bundle path.
Stalled-allocation recovery now emits a participant-targeted state_changed
outbox event in the same serializable transaction that fails the abandoned
match, releases its participants, and requeues their tickets. The maintenance
adapter verifies that every reclaimed match produced its durable event, so an
API/WebSocket restart cannot turn a successful infrastructure recovery into a
silent client-side stale state. Normal, race, vet, and SQL-shape checks pass;
the live PostgreSQL chaos/restart gate remains part of 8.50.
The ENet gate now auto-detects /Applications/Godot.app/Contents/MacOS/Godot when no PATH executable or GODOT_BIN override exists, while retaining explicit override precedence. The same gate passes without an environment override on this macOS host.
The client matchmaking projection now preserves the server's enqueued_at timestamp through normalization, snapshots, and recovery, and uses it for the displayed queue wait when available. This prevents a client restart or delayed response from resetting the user's perceived wait to local process uptime; a local timer remains the fallback when older responses omit the timestamp. Godot state and normalization tests cover the projection and restore path.
The ranked profile projection now also carries the active season's authoritative end timestamp from PostgreSQL through the API and Godot client. Ranked matchmaking displays a bounded days-remaining countdown, while providers without an active season remain compatible and omit the countdown.
The versioned OpenAPI contract now declares the implemented /profile/ranked surface and its server-authoritative ranked profile schema, including optional active-season metadata. Contract tests reject omission of this operation, extra response fields, and credential leakage.
The Godot client now defers reconnect-triggered authoritative recovery when an HTTP mutation is still in flight, closing the ERR_BUSY recovery-drop race. An adversarial client test verifies that the active ticket remains queued for recovery rather than silently staying stale.
The Godot queue projection now includes the contract's ACCEPTED ticket phase. Accepted events are no longer rejected as an unknown state; the UI keeps the accepted status visible and proceeds through allocation recovery. State and WebSocket vocabulary tests cover the transition.
The queue projection now also accepts the contract's post-allocation/result states (ASSIGNED, RESULT_PENDING, and COMPLETED). These states remain visible, cannot issue queue cancellation, and completed matches return the search action to a valid new-search state; adversarial lifecycle and WebSocket vocabulary tests cover them.
Client ticket updates now enforce the versioned legal transition graph as well as revision ordering. Same-state heartbeat revisions remain valid, while higher-revision jumps and rewinds request authoritative recovery without mutating the visible phase; adversarial tests cover both boundaries.
Reconnect recovery now treats COMPLETED as terminal, avoiding a needless queue read after a finished match. UI policy tests cover the complete expanded lifecycle, including the completed-to-new-search boundary.
Ticket, proposal, and WebSocket revisions now fail closed unless they are finite, non-negative integers; fractional values are no longer silently truncated into valid revisions. Adversarial client tests cover fractional and negative inputs.
Proposal updates now enforce the documented OPEN → ACCEPTED/DECLINED/EXPIRED/CANCELLED graph, including rejecting higher-revision reopen/accept attempts after terminal decisions while preserving same-state duplicates. Adversarial proposal-transition tests cover accepted and declined terminal paths.
Proposal decline/expiry/cancellation now leaves a still-proposed ticket in QUEUED, matching the durable server requeue transaction; the proposal’s terminal message remains visible without making the ticket terminal. A cancelled ticket is never resurrected by a later proposal event, covered by adversarial cross-aggregate tests.
Recovery targeting now follows the same boundary: only an OPEN proposal is polled as a proposal; terminal proposal outcomes fall back to the ticket recovery endpoint. This prevents repeated reads of a finished proposal from starving recovery of the requeued ticket.
Client queue/proposal expiry and enqueue epoch metadata now fail closed on malformed, negative, or fractional values instead of being silently coerced to zero. Adversarial metadata tests cover string, negative, and fractional timestamps.
All client resync entry points now apply the open-proposal boundary: a terminal proposal always recovers the durable ticket instead of polling the finished proposal. A direct-resync regression test covers this path.
Ticket projections now validate playlist metadata on every update, rejecting unknown values before either phase or playlist state can mutate. An adversarial higher-revision update test covers this boundary.
Client sessions now fail closed at the expiry boundary and proactively clear credentials before reconnects or authenticated requests. Boundary and malformed-expiry tests cover the lifecycle guard.
Queue heartbeat and cancellation revision conflicts now schedule the same authoritative ticket recovery as proposal conflicts, preventing stale client actions from leaving the visible queue state unresolved. Adversarial operation/status/identity coverage is included.
WebSocket event envelopes now require RFC3339 timestamps rather than merely non-empty text, matching the versioned contract; session-expiry format checks use the same boundary validator. Malformed-format adversarial coverage is included.
WebSocket event resource identifiers now enforce the contract’s opaque 16–128 character allowlist, preventing path/separator text or undersized identifiers from entering the client projection.
The Go event hub now enforces the same resource-ID allowlist before publication, so malformed identifiers are rejected at the server boundary rather than only discarded by clients.
The matchmaking UI now displays the authoritative proposal countdown from the server expiry epoch, clamped at zero and retaining compatible copy when older responses omit expiry metadata. Adversarial countdown tests cover delayed and missing-expiry responses.
The UI now provides explicit detail copy for every non-terminal allocation and connection phase (ACCEPTED through LIVE), so server progress remains understandable throughout assignment and transport startup.