multiplayer-next.md carried the Phase 8 checklist but not the numbered
tasks with acceptance criteria that work actually gets picked up from.
That format lives in multiplayer-todo.md section 7, which already hosts
Phase 7 as in-progress, so Phase 8 goes there too.
Tasks 8.1-8.20 across four groups: backend service (identity, rating
store, Glicko-2, queue), server orchestration and autoscaling, playlists
and client UI, and keeping Docker/CI green. Section 0's short list gains
an index entry, and the status header now says Phase 8 is a 1.0 launch
blocker and the first phase to add a component outside the Godot
project.
Three entries are measured findings rather than plans, each of which
would break a naive implementation: stdout block-buffering making a
log-grep readiness probe hang forever, the hardcoded 7777/udp port
preventing more than one match per host, and compose.phase6-smoke.yml's
dependence on the exact behaviour allocation work would change.
Also fixes a now-false cross-reference: a Phase 4 note read 'not
Phase 8' meaning 'not a later phase', written when no Phase 8 existed.
CLAUDE.md's 'never add new work to multiplayer-todo.md' rule gains the
new-phase exception it always had in practice - Phase 7 was already
there.
The document had no one place that answered "what is left". Outstanding
items were spread across two phase-gate lines, two phase tables, §11, and
prose buried in the phase notes - and the name-keyed slot-reservation
hijack, which an adversarial review demonstrated with a real three-process
run, was not written down anywhere at all. It existed only in a
conversation.
New §0 indexes everything not done, in four groups: verification a
machine cannot do (the Phase 4 playtest, the Phase 5 3v3 gate), known
defects left unfixed with their severity, the one open architectural
question, and the two unstarted phases. Each row points at the detailed
write-up rather than duplicating it, and the phase gates now point back.
§11 gains the identity defect in full: reservations match on
slot.player_name and nothing else, with no uniqueness constraint on names
anywhere, so a peer joining during the 30s window with a departed
player's name is handed their slot, ship and team. A bespoke token would
be half of task 7.4 thrown away, so it stays deferred - with the
consequence stated plainly rather than implied, and listed as a
precondition of Phase 6's internet-facing gate.
Also refreshes the stale status paragraph and task 5.10 for the replay
log's reject recording, write-failure handling, close(), and dump tool.
"Spectate now, take the slot at the next kickoff" was a print statement.
The server logged it and never acted; on the client, _is_spectator was
assigned once in _on_match_config_received and never revisited - and that
handler returns early whenever _slots is non-empty, so no rebroadcast
could promote an in-match spectator. The reconnect path only worked
because a returning player is a fresh process.
Server: late joiners are queued in arrival order and the queue is drained
from _begin_kickoff, before the reset transforms are read, so a promoted
player's ship is placed by that same kickoff and the controller swap
lands on an already-frozen body. A slot is available only once its player
has gone AND their 30s reservation has lapsed - §6.4 outranks §6.3, since
taking a reserved slot would quietly break the reconnect promise.
_abort_if_abandoned now counts a waiting spectator as somebody present,
or the one person queued for the slot that just opened is dumped to the
lobby at the moment they were about to get it.
Client: new broadcast slot_assigned (reliable, channel 0). Broadcast
because every client holds its own slot list and one naming the wrong
peer keeps flying somebody else's ship as a remote body; reliable because
no per-snapshot field would re-converge a client that missed it. The
promoted client undoes what made the body remote - fresh interpolator,
physics interpolation back on, offsets cleared - and deliberately does
not unfreeze, clearing _local_prediction_ready so the next snapshot
teleports it to a real authoritative pose first. The controller-attach
block moved to _take_local_ownership rather than being copied.
New --role=host-latejoin/--role=client-latejoin and
--slot-reservation-seconds=. Verified 4/4 both sides: queued, NOT
promoted merely because the reservation lapsed, takes the slot at the
kickoff, same ship instance, and both peers independently measure ~45.7m
under its input. Control with a 90s reservation: kickoff fires, nothing
is promoted, the slot still reads the departed player's name.
The disconnect scenario only ever asserted the server's bookkeeping, and
the client's half was failing every run. run_disconnect_host_check ticked
60 physics frames past the reclaim and then shut the server down, so the
reconnecting client - whose wiring check waits a 2.0s settle before it
looks at anything - had its peer torn out from under it and reported
"current_scene is not NetworkedMatch after 2.0s". The host printed PASS
throughout, and the host was the side anyone read.
The hold is now a real window (8s), and the host also asserts that the
reconnected player's input reaches the server and moves the ship the
server owns - every other assertion there is slot bookkeeping that would
hold identically for a client whose input pipeline came back dead. Both
position and connection state are sampled while the peer is still
connected: the client leaves on its own schedule, and an end-of-hold
sample reported still_connected=false for a good run.
New --role=client-reconnect asserts the returning player is not a
spectator, owns a slot with its own peer_id, has a real ship, rejoined a
live match with the clock already known (§6.2 step 2's bootstrap), and
can still drive. That set is chosen because a stale _last_match_config
once made a reconnecting player a spectator, and that bug was visible in
this scenario's own logs while it reported PASS.
Verified 3/3 both sides. Control: rejoining while the slot is still
occupied fails on is_player=false - and since the first control run
reported it as the generic "lost its ship mid-drive", the spectator case
is now diagnosed before the drive rather than after.
Closes task 5.10's three recording gaps, and the gap-closing found a real
input-loss bug.
Replay log: a failed write now ends the log permanently instead of
desyncing every later record's framing; close() is called from _exit_tree
with a summary, since the RefCounted destructor closes it implicitly but
never says whether the log is complete; rejected packets are recorded
with their reason in the kind byte (framing unchanged, FORMAT_VERSION 2
so "no rejects" differs from "this build never recorded them"). Recording
is capped at 8 per peer per window - uncapped, the diagnostic is a remote
disk-fill amplifier, since the attacker picks the packet rate. Uncapped
totals live on MatchSim and survive the peer's disconnect.
The bug: a 2s host stall has the client sending at 60Hz throughout, and
ENet delivers that whole backlog in the first window after resume - 70 of
an honest client's packets rejected as "rate limit exceeded". Redundancy
does not cover it, because the dropped packets are contiguous: 0 of 70
rescued, and 82 of 923 sequences (8.88%, ~1.4s of input) never reached
the server, against 0.00% with no stall. Every prediction gate passed.
Fixed by granting each already-tracked peer a capped, two-window packet
grace when the server detects its own wall-clock stall. Rate-limit
rejects 70 -> 0, sequences missing 8.88% -> 0.00%, seq-guard rejects
9 -> 0. Controls on the unfixed build lost 4.34/7.52/7.86%. All three
abuse roles still disconnect and no flood induced a stall, so the grace
cannot be farmed.
Also corrects an earlier wrong conclusion: the reviewer's free-flight
p95 0.688 is real and reproduces on two processes with 0.0% snapshot
loss. The plain --role=client drive fails the 0.5 free-flight bound in
3 of 8 runs because that drive is mostly a contact test - the harness
comment already said so - leaving a cohort as small as 12 samples.
Near-surface error is genuinely several times open-air error, so the
calibrated bound now belongs to --exercise-free-flight alone and the
plain role asserts the always-well-sampled all-cohort percentiles at
1.2/2.0, printing the free-flight numbers as reported-not-asserted.
6/6 plain runs pass where 3/7 failed; tightening to 0.3 still fails.
tools/replay_dump.gd reads a log back: counts by kind, plus how much of
the input sequence stream reached the server once redundancy is counted.
A percentile alone cannot tell "the predictor regressed" from "the client
never received the data". The client gate now prints snapshot_loss /
snapshot_age / rtt on every run, and on a quality failure with >20% loss
says explicitly that the run was transport-starved. It deliberately does
not convert the failure into a pass: a client that cannot receive
snapshots is still a failed run, just a differently-diagnosed one.
Both directions of the new branch verified non-vacuously (forced true so
it fires and formats; restored so it stays quiet on a healthy run while
the INFO line still prints).
Records the investigation behind it in multiplayer-todo.md: the reviewer's
3-process p95 0.688 did not reproduce. An idle third process costs nothing
(p99 0.094), a spectator costs a small but real amount (p99 0.094-0.146),
and snapshot loss held at 0.0% even under 2x CPU oversubscription - all an
order of magnitude inside the 0.5/2.0 gates. Also notes that a previously
working class_name can silently drop out of the .godot class cache, which
surfaces as a bogus parse error with nothing in git status to explain it.
Completes Phase 5's implementation. Every task is verified at 1v1; the
3v3 phase gate itself has not been run and remains outstanding.
5.6/5.7 disconnects: a ship is never despawned. The slot keeps it and
swaps the controller (--fill-bots gives it a bot, the default leaves it
inert per §1.4), sets `stalled` immediately so the nameplate greys out
rather than waiting ~500ms for the abandoned jitter buffer to starve,
and reserves the slot for 30s keyed by player name so a reconnect gets
the same ship back.
5.7 was a real bug, found by the test rather than by review:
SlotInfo.controller was declared RLShipController, but the takeover
swaps in an AIShipController or the base controller - the narrower type
makes that assignment fail its type check, leaving the field pointing at
the controller set_controller() just queue_free()d. It surfaced as
controller_valid=false on the first run. The per-tick action write is
now also gated on `is RLShipController`, since a disconnected slot's bot
drives itself and overwriting it from a starving buffer would pin it to
the departed player's last input.
§6.4's two rules conflict: reserve for 30s, but abort when the last
human leaves. Applied naively the abort wins instantly in a 1v1 and the
reservation can never be redeemed, making reconnect unreachable exactly
when it matters. Abort now waits for no connections AND no outstanding
reservations.
5.8 spectators: a slotless peer spawns no ship and receives the same
snapshot broadcast. HUDController.spectator_mode keeps the clock, score
and goal celebration and hides only the ship instrument cluster - it
previously push_error'd and bailed, leaving a spectator with a dead HUD.
Camera cycles ships in slot order then the ball. --max-spectators caps
it, counted from the live peer list so a dropped spectator cannot leak a
unit of the cap.
5.9 escape respawn: new GameMode._on_bodies_respawned() virtual;
NetworkedMatch bumps reset_gen through Phase 2's deferred path so the
bump and the respawned pose land in the same broadcast. Single-player
modes are unaffected - the base is a no-op.
5.10 replay log: scripts/replay_log.gd, --replay-log=<path>, storing the
wire bytes verbatim in both directions rather than re-serialising - a
re-encode would launder away precisely the malformed payload being
chased. A live 6s match recorded 1115 records (557 inputs / 558
snapshots) and a stored snapshot decodes back to server_tick=100
match_state=WARMUP bodies=2.
Note for future work: --check-only --script is the only thing that
catches a parse error in networked_match.gd, because the unit runner
never loads it. Two separate breakages passed the full unit suite while
breaking every two-process run. A new class_name also needs --import
before it resolves.
Test surface: --role=host-disconnect (three-process 5.6/5.7 scenario),
--match-length=<s>, --replay-log, --fill-bots/--no-fill-bots,
--max-spectators. The ball-contact scenario now steers at the ball with
closed-loop real input instead of a hand-tuned fixed heading, which 5.3
broke by adding KICKOFF_YAW_JITTER; thrusting while turning took it from
2/3 to 5/5.
Regression: 87 unit tests; free-flight LAN p99 0.094m with 0 hard snaps;
transition gate 0.00%; ball contact 5/5; lifecycle goal cycle and full
match to RESULTS/LOBBY; disconnect+reconnect; two-bot CI.
Implements the rest of the §6.2 lifecycle on top of 5.1's state machine.
5.3 kickoff: the server resets every body and broadcasts the RESULTING
transforms, never a seed - §1's locked decision, because shared-seed
determinism needs both sides to consume the RNG stream in identical
order forever and the first randf() added to the reset path desyncs
silently. Countdown is derived from server_tick on both peers, and a
kickoff that lands after its own resume tick applies immediately and
skips the countdown rather than scheduling into the past.
5.4 goals: goal_scored(scoring_team, score, goal_tick, resume_tick).
Score is authoritative at sensor time, before any presentation. The
reset moved OUT of the sensor path and into the kickoff at resume_tick,
which is what stops the server resetting while clients are still
mid-celebration. Engine.time_scale is never touched.
5.2 clock: tick-derived, no Timer and no _process polling. The goal
pause shifts the absolute end_tick by (resume_tick - goal_tick) rather
than pausing anything, so no float drift accumulates across goals.
5.5 full time: clock expiry -> FULL_TIME -> sudden death on a draw or
RESULTS, golden goal in overtime, then LOBBY on both peers - clients
return to the lobby, not the main menu. get_tree().paused is never used.
Four bugs found and fixed while building this, each by a failing run
rather than by inspection:
- Tick order was load-bearing: _update_kickoff_countdown() clears the
same _kickoff_resume_tick that _update_match_state() reads to leave
WARMUP, so running the countdown first wiped the transition condition
and the match sat frozen in WARMUP forever.
- _apply_match_state resets _state_deadline_tick on every transition, so
a GOAL_PAUSE deadline assigned before _set_match_state was wiped and
the match never resumed. Deadlines are now owned by _apply_match_state.
- Freezing "all bodies" is wrong on a client. Remote ships and the ball
are permanently FREEZE_MODE_KINEMATIC and transform-driven; freezing
them all unfroze the remote ones on the way back out, so they fell
under gravity while the interpolator fought them - 210 hard snaps and
an infinite p99. A client now freezes only the one body it simulates.
- A frozen body never runs _integrate_forces, so the queued kickoff
teleport was stranded by an immediate set_deferred("freeze", true).
Freeze now happens on a strictly later tick, the same pattern Phase 2
used for _pending_reset_gen_bump_tick.
Prediction and reconciliation are suspended while the match is not live:
during a countdown or goal pause the local ship is frozen on both peers,
and running delta transport over those frozen states produced a p95
position error of 2.4e10 m. Input keeps flowing so the server's jitter
buffer does not starve into `stalled`.
Also fixed: a kickoff can arrive before match_config, and body order is
slot order - applying it early placed the BALL at positions[0], on top
of the first ship, which the ball-cam reported as "target vector can't
be zero" 95 times. It is now held until the roster exists.
Test changes: the ball-contact scenario steered by a hand-tuned fixed
heading, which 5.3 broke because kickoff applies KICKOFF_YAW_JITTER - it
flew past the ball in 3/3 runs. It now closes the loop on the actual
bearing using real input actions. Assertions that read a frozen ship
(freeze, thrust) are gated on the match being live, and the hooks now
survive the scene teardown at RESULTS instead of hanging on freed
objects for the full timeout.
Regression: 81 unit tests; free-flight LAN p99 0.143m and 80±20ms, both
0 hard snaps; transition gate 0.00%; ball contact 3/3; two-bot CI.
Adds the §6.1 state machine, its broadcast, and the client side that
follows it. Physics, freezing and input are deliberately NOT gated on
state yet - 5.3 and 5.4 own freeze/unfreeze at kickoff and goal, and
doing it here would change the conditions every Phase 4 prediction gate
was measured under.
scripts/match_state.gd holds the enum and transition table as pure data
with no scene or RPC dependency, so the table is checked exhaustively
rather than by example: every state reachable, every state has an exit,
no self-transitions, abort-to-LOBBY from anywhere per §6.4, illegal
shortcuts rejected, unknown values refused rather than coerced. The enum
values are the wire format - match_state has been a u8 in the snapshot
header since §2.4 - so a test pins them; only append, never renumber.
The server validates every transition and push_errors an illegal one
rather than following it. Clients deliberately do NOT enforce the table:
authoritative state must be accepted, and a late joiner legitimately
jumps straight to PLAYING.
Two channels carry the state. state_change (reliable, channel 0) is
prompt and carries an absolute at_tick, never a duration. The snapshot's
match_state byte is the catch-up path for a client not yet sent a
transition - a late joiner, or the window between scene load and the
first RPC.
The byte needs a tick guard, and this was found the hard way. Snapshots
are unreliable_ordered on channel 2 and ordering holds only within a
channel, so a state_change for tick N routinely arrives before an
in-flight snapshot from tick N-2. Without the guard the client applies
the new state then gets dragged back by the older byte, oscillating on
every transition - observed directly as LOADING -> WARMUP -> LOBBY ->
PLAYING -> LOBBY while running a deliberately-broken-byte control. Only
a byte at least as new as match_state_since_tick is accepted.
WARMUP_TICKS/GOAL_PAUSE_TICKS are honest placeholders so 5.1 drives real
transitions to verify against; 5.3 and 5.4 replace them. The server also
leaves LOADING immediately rather than waiting for scene_ready, which
does not exist yet.
New smoke flag --exercise-match-state, passed to both roles: the host
forces a goal to drive a GOAL_PAUSE cycle, the client records the
sequence and asserts every consecutive pair is legal, that ticks are
monotonic, and that the wire byte agrees with its own state. Observed
LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP with tick deltas
matching the configured durations exactly.
Verified against a control: hardcoding the snapshot byte back to 0 fails
both the byte assertion and the transition-legality assertion. The byte
is asserted separately from the RPC precisely because everything else in
the check is RPC-driven and would pass with a dead byte - the same gap
that hid the Phase 4 label bug (gotcha 47).
Regression: 81 unit tests; 60s free-flight LAN (p99 0.148m, 0 hard
snaps, marker 0/3364); transition gate 0.00%; ball contact; two-bot CI.
Josh Creek