Adds fetch_ranked_profile() right after login, asserting the expected
404 for a brand-new identity round-trips correctly through the real
RankedProfileProvider (previous commit) before proceeding to the
existing queue_create -> heartbeat -> cancel sequence. Verified stable
across 3 consecutive full runs.
Adds a fourth real round trip to control_plane_smoke.gd: heartbeat ->
cancel_queue -> CANCELLED, using the same idle-wait pattern the
heartbeat step already needed. Verified stable across 3 consecutive
full runs (real Postgres, real testkit-api, real headless Godot
client), plus the full Go and Godot unit suites clean.
Every existing test of the client/control-plane boundary is either a
Go unit test with a mocked HTTP layer or a GDScript unit test with no
network at all (multiplayer-next.md 8.40's own evidence names "live
multi-process control-plane/game verification" as remaining). Nothing
before this actually ran the real compiled Go binary, a real
PostgreSQL instance, and a real headless Godot process talking real
HTTP to each other -- and it immediately found a real bug (previous
commit).
server/cmd/testkit-api is a new, deliberately separate, clearly-marked
test-only binary wired identically to cmd/control-plane except for
SteamLogin: cmd/control-plane has no way to authenticate against a
real Steam Web API from this sandbox (task 8.7's own documented
blocker), so testkit-api accepts any non-empty ticket string and
derives a deterministic identity instead. This bypass is confined to
its own binary -- never a flag on cmd/control-plane, never referenced
by any Dockerfile stage or Kubernetes manifest -- specifically so it
can't become a footgun on the real one.
Game/tests/control_plane_smoke.gd drives the real ControlPlaneClient
autoload through login -> queue_create -> heartbeat against a real
server and prints SMOKE PASS/FAIL, matching the existing net_smoke.gd
convention. scripts/verify_control_plane_integration.sh orchestrates
both sides (real postgres:17-alpine, the built testkit-api binary, the
Godot client) end to end.
Two real bugs surfaced building this, both fixed and re-verified, not
just the target bug: the smoke script's own use of `go run` left a
zombie process that survived cleanup and squatting on its port
corrupted the NEXT run with a misleading "http=401 unauthorized" (now
builds and runs a real binary directly, plus a belt-and-suspenders
port-kill in cleanup); and calling heartbeat() synchronously from
within a request_succeeded handler produced a spurious "Busy" because
ControlPlaneClient's own internal resync (see previous commit) was
still in flight -- the test now waits for ControlPlaneClient to go
idle via a real Timer (call_deferred alone floods the message queue
without ever yielding a frame for the in-flight request to complete).
Verified stable across 3 consecutive full runs: real PostgreSQL
container up, migrations applied, testkit-api built and started, real
headless Godot client round-tripping login/queue/heartbeat, clean
teardown with no leftover processes, containers, or bound ports each
time.
multiplayer-todo.md and multiplayer-next.md tracked overlapping
information in two places. Fold everything into multiplayer-next.md
(architecture decisions, wire format, task breakdown with checkboxes,
gotchas list, testing notes) and delete multiplayer-todo.md. Section
numbers are unchanged, so existing code comments citing them by
section/task number still resolve; update every such reference to
point at the new filename.
CLAUDE.md and README.md described the pre-multiplayer state (local-only
MVP, planned C# backend) even though server-authoritative multiplayer,
the dedicated server, Docker/CI verification, and Steam transport have
since shipped (Phases 1-6). Update both to reflect reality and add a
docs index in CLAUDE.md pointing at multiplayer-next.md as the current
checklist.
- Add docs/TECH_STACK.md, linked from README, explaining the stack and
why it's a single GDScript project with no separate backend.
- Add one TODO item for the video settings menu (missing presets/vsync/
resolution scaling), blocked on the same profiling gate as the
multiplayer 0.17 tasks.
- Pick up editor-generated .gd.uid sidecars and minor project.godot
formatting noise from opening the project in Godot 4.7.
Godot's ConfigFile writer does not round-trip comments in project.godot. An
observed rewrite deleted both `;` blocks outright and spliced the three-line
`#` block above run/main_scene.dedicated_server onto the setting's own line,
leaving it commented out — which would send dedicated builds to the
interactive main menu instead of server_boot.tscn, with nothing failing until
someone noticed a server process rendering a menu.
Move the explanations into the code that owns the settings (server_boot.gd for
the dedicated-server override, video_settings.gd for stretch mode and vsync)
so they cannot be destroyed by a rewrite, and leave project.godot holding only
assignments plus Godot's own regenerated header.
Add tests/cases/test_project_settings.gd as the backstop: the feature-override
assertions read project.godot as text and reject a line that has been folded
into a comment, since ProjectSettings resolves `key.<feature>` overrides at
load time and never exposes the suffixed key. Verified by reproducing the
exact corruption, which fails the test, and it also covers the Jolt physics
engine, the required autoloads, and that no test-hook autoload is ever shipped
registered.
run_ci_host_check asserted input reached the server by comparing each bot
ship's position against one recorded before the check forced a goal. But a
goal's kickoff teleports every ship back to spawn (_begin_kickoff ->
reset_ships), so that comparison measured only the distance covered since the
last reset — a window whose length depends on when the sample lands relative
to the kickoff rather than on whether input was flowing at all.
It failed on master with peers at 0.51m and 0.23m against a 0.5m threshold:
one passed by a centimetre, the other failed, with both connected, neither
stalled, and every other assertion in the run green. The commit it failed on
touches only training JSON, and the push two minutes earlier passed on
identical game code.
Accumulate per-tick path length in _await_recording_score instead, discarding
any single-frame step over 2.0m as a teleport — Ship.max_speed (35 m/s) is
hard-clamped each tick in _integrate_forces, so 60Hz caps legitimate travel at
~0.58m. Same 0.5m threshold now reads 28-75m across runs, and it is strictly
stronger than before: it asserts input kept arriving for the whole wait rather
than that the ship merely ended up somewhere else.
server_boot.gd's private _log could only ever see what the boot scene
itself observed: connects, disconnects, roster changes, tick overruns.
The events an operator is actually asked about - who scored, who got
kicked and why, which peer is flooding - happen inside networked_match.gd
and match_sim.gd, neither of which could reach a logger on a scene node
that gets freed at the first change_scene_to_file. scripts/server_log.gd
holds it as static state on a class_name: reachable from all three, no
autoload, no ordering dependency.
New events: goal, match_ended, kickoff, peer_kicked (previously only a
push_warning, carrying neither peer nor reason into the stream a
container captures), rate_limited, server_stalled. rate_limited fires
ONCE per peer per window rather than per packet - a flood is thousands of
packets a second and the log line must not become the amplifier the
replay recorder was capped to avoid being.
Off unless a server configures it, so a client, an editor session or a
unit-test run does not start printing server telemetry just because these
scripts loaded.
Rotation is deliberately not implemented: the server logs to stdout and
stops, because every way this is run already rotates better - docker's
json-file driver, journald, or logrotate on a redirect. A server that
also wrote and rotated its own file would fight all of them in a
container, where stdout is the interface. SERVER.md (6.6) documents the
three configurations.
Five tests on the one piece with real logic - the one-line contract.
Including log injection: a player name is attacker-controlled, and
without escaping, the name "x\n[0.000] INFO peer_kicked reason=nothing"
writes a fake event into the operator's log. Newlines are escaped rather
than dropped so the attempt stays visible.
End-to-end verification of the new events comes with 6.5, which is what
first makes a server run a match at all.
6.1: "Linux Dedicated Server" preset (dedicated_server=true,
custom_features="dedicated_server") mirroring the existing training
preset, plus run/main_scene.dedicated_server so the server binary reaches
its own entry point with no flag. Builds: an 85MB Linux x86_64 binary,
gitignored like the training one.
6.3: scripts/server_config.gd declares every server flag once - name,
type, default, section, help - and one parser turns that into parsing,
type checking, range validation, config-file backing and --help. The
flags had grown to ~30 across server_boot.gd and networked_match.gd, each
parsed inline with begins_with, none documented, and an unrecognised flag
was SILENTLY IGNORED: --max-clientss=8 ran a server on the default cap
and said nothing. Unknown flags, missing values, wrong types, duplicates
and out-of-range values are now hard errors, reported all at once.
Precedence is command line > config file > default. server_boot.gd parses
strictly because it owns the whole command line; networked_match.gd reads
the same declaration leniently because it is one consumer of an argv the
smoke harnesses also fill with --role= and --drive-seconds=. Nothing is
lost - every server flag is declared, so the strict pass already caught
any typo before the match scene re-reads its own.
13 unit tests covering the precedence order, the typo rejection that
motivated this, --no-<bool> not double-listing in --help, and --help
documenting every flag asserted against the declaration rather than a
hand-kept list. Verified end to end: --help prints, a typo'd flag refuses
to start, and the plain/replay-log/late-joiner smoke scenarios still pass.
"Spectate now, take the slot at the next kickoff" was a print statement.
The server logged it and never acted; on the client, _is_spectator was
assigned once in _on_match_config_received and never revisited - and that
handler returns early whenever _slots is non-empty, so no rebroadcast
could promote an in-match spectator. The reconnect path only worked
because a returning player is a fresh process.
Server: late joiners are queued in arrival order and the queue is drained
from _begin_kickoff, before the reset transforms are read, so a promoted
player's ship is placed by that same kickoff and the controller swap
lands on an already-frozen body. A slot is available only once its player
has gone AND their 30s reservation has lapsed - §6.4 outranks §6.3, since
taking a reserved slot would quietly break the reconnect promise.
_abort_if_abandoned now counts a waiting spectator as somebody present,
or the one person queued for the slot that just opened is dumped to the
lobby at the moment they were about to get it.
Client: new broadcast slot_assigned (reliable, channel 0). Broadcast
because every client holds its own slot list and one naming the wrong
peer keeps flying somebody else's ship as a remote body; reliable because
no per-snapshot field would re-converge a client that missed it. The
promoted client undoes what made the body remote - fresh interpolator,
physics interpolation back on, offsets cleared - and deliberately does
not unfreeze, clearing _local_prediction_ready so the next snapshot
teleports it to a real authoritative pose first. The controller-attach
block moved to _take_local_ownership rather than being copied.
New --role=host-latejoin/--role=client-latejoin and
--slot-reservation-seconds=. Verified 4/4 both sides: queued, NOT
promoted merely because the reservation lapsed, takes the slot at the
kickoff, same ship instance, and both peers independently measure ~45.7m
under its input. Control with a 90s reservation: kickoff fires, nothing
is promoted, the slot still reads the departed player's name.
The disconnect scenario only ever asserted the server's bookkeeping, and
the client's half was failing every run. run_disconnect_host_check ticked
60 physics frames past the reclaim and then shut the server down, so the
reconnecting client - whose wiring check waits a 2.0s settle before it
looks at anything - had its peer torn out from under it and reported
"current_scene is not NetworkedMatch after 2.0s". The host printed PASS
throughout, and the host was the side anyone read.
The hold is now a real window (8s), and the host also asserts that the
reconnected player's input reaches the server and moves the ship the
server owns - every other assertion there is slot bookkeeping that would
hold identically for a client whose input pipeline came back dead. Both
position and connection state are sampled while the peer is still
connected: the client leaves on its own schedule, and an end-of-hold
sample reported still_connected=false for a good run.
New --role=client-reconnect asserts the returning player is not a
spectator, owns a slot with its own peer_id, has a real ship, rejoined a
live match with the clock already known (§6.2 step 2's bootstrap), and
can still drive. That set is chosen because a stale _last_match_config
once made a reconnecting player a spectator, and that bug was visible in
this scenario's own logs while it reported PASS.
Verified 3/3 both sides. Control: rejoining while the slot is still
occupied fails on is_player=false - and since the first control run
reported it as the generic "lost its ship mid-drive", the spectator case
is now diagnosed before the drive rather than after.
Closes task 5.10's three recording gaps, and the gap-closing found a real
input-loss bug.
Replay log: a failed write now ends the log permanently instead of
desyncing every later record's framing; close() is called from _exit_tree
with a summary, since the RefCounted destructor closes it implicitly but
never says whether the log is complete; rejected packets are recorded
with their reason in the kind byte (framing unchanged, FORMAT_VERSION 2
so "no rejects" differs from "this build never recorded them"). Recording
is capped at 8 per peer per window - uncapped, the diagnostic is a remote
disk-fill amplifier, since the attacker picks the packet rate. Uncapped
totals live on MatchSim and survive the peer's disconnect.
The bug: a 2s host stall has the client sending at 60Hz throughout, and
ENet delivers that whole backlog in the first window after resume - 70 of
an honest client's packets rejected as "rate limit exceeded". Redundancy
does not cover it, because the dropped packets are contiguous: 0 of 70
rescued, and 82 of 923 sequences (8.88%, ~1.4s of input) never reached
the server, against 0.00% with no stall. Every prediction gate passed.
Fixed by granting each already-tracked peer a capped, two-window packet
grace when the server detects its own wall-clock stall. Rate-limit
rejects 70 -> 0, sequences missing 8.88% -> 0.00%, seq-guard rejects
9 -> 0. Controls on the unfixed build lost 4.34/7.52/7.86%. All three
abuse roles still disconnect and no flood induced a stall, so the grace
cannot be farmed.
Also corrects an earlier wrong conclusion: the reviewer's free-flight
p95 0.688 is real and reproduces on two processes with 0.0% snapshot
loss. The plain --role=client drive fails the 0.5 free-flight bound in
3 of 8 runs because that drive is mostly a contact test - the harness
comment already said so - leaving a cohort as small as 12 samples.
Near-surface error is genuinely several times open-air error, so the
calibrated bound now belongs to --exercise-free-flight alone and the
plain role asserts the always-well-sampled all-cohort percentiles at
1.2/2.0, printing the free-flight numbers as reported-not-asserted.
6/6 plain runs pass where 3/7 failed; tightening to 0.3 still fails.
tools/replay_dump.gd reads a log back: counts by kind, plus how much of
the input sequence stream reached the server once redundancy is counted.
A percentile alone cannot tell "the predictor regressed" from "the client
never received the data". The client gate now prints snapshot_loss /
snapshot_age / rtt on every run, and on a quality failure with >20% loss
says explicitly that the run was transport-starved. It deliberately does
not convert the failure into a pass: a client that cannot receive
snapshots is still a failed run, just a differently-diagnosed one.
Both directions of the new branch verified non-vacuously (forced true so
it fires and formats; restored so it stays quiet on a healthy run while
the INFO line still prints).
Records the investigation behind it in multiplayer-todo.md: the reviewer's
3-process p95 0.688 did not reproduce. An idle third process costs nothing
(p99 0.094), a spectator costs a small but real amount (p99 0.094-0.146),
and snapshot loss held at 0.0% even under 2x CPU oversubscription - all an
order of magnitude inside the 0.5/2.0 gates. Also notes that a previously
working class_name can silently drop out of the .godot class cache, which
surfaces as a bogus parse error with nothing in git status to explain it.
A second adversarial review (this one able to RUN things, unlike the
first) reproduced five defects. Fixing the critical and high ones.
CRITICAL - Esc no longer left a networked match, and a client whose
server vanished was stranded forever. Two independent bugs composing:
_unhandled_input (added for spectator target cycling) overrode
GameMode._unhandled_input and returned early for every non-spectator
without ever calling super(), silently killing ui_cancel -> main menu;
and NetworkedMatch never connected NetworkManager.disconnected_from_
server the way lobby.gd does. Measured: a client whose host exited
emitted 7,235 engine errors in ~18s and only left because a test timer
fired. Now 1 benign teardown error, and it returns to the main menu.
HIGH - the match clock lost up to 3 seconds of regulation per goal.
_on_goal_registered extended end_tick by the celebration only
(resume_tick - goal_tick) and never by the 180-tick kickoff countdown
that follows it, while _update_clock derived remaining time from the
current tick regardless of _clock_running - so regulation drained during
every stoppage. Measured 660 PLAYING ticks for a 14s match against 840
expected: exactly one WARMUP lost. The HUD also opened at 0:17 for a 14s
match because the initial arm folded WARMUP into end_tick.
Replaced the per-goal arithmetic with bank-and-rebase: entering any
non-live state banks the remaining ticks, leaving it rebases end_tick
off the banked value. That covers celebration and countdown together and
cannot drift, since nothing has to predict how long a stoppage will be.
clock_state and match_bootstrap now carry remaining_ticks, which is
authoritative whenever the clock is stopped. Verified with the
reviewer's own metric: 840 PLAYING ticks for a 14s match, exactly.
MEDIUM - clients never froze at FULL_TIME/RESULTS. The freeze handling
sat inside `if multiplayer.is_server()`, so a local player flew around
for the whole 8s results screen while every other peer saw their ship
parked.
Not fixed, and now demonstrated rather than merely suspected:
- The 30s slot reservation is keyed on display NAME, so a stranger can
take a departed player's ship and the real player is then locked out
(reproduced). Worse than first thought: MatchNet.local_player_name
defaults to "Player" and uniqueness is never enforced, so collisions
are the common case, not an attack setup. Needs a real identity token;
§6.2 step 1 reserves auth_ticket for Phase 7.
- §6.3's "late joiner takes the slot at the next kickoff" is
unimplemented - _is_spectator is assigned once and never revisited -
while the server logs that it happened.
- Replay log still ignores store_* return values, never records
malformed/rejected inputs, and close() has no caller.
- --role=host-disconnect grades the reconnecting client on ~1s of life
before the host quits, and never asserts the client owns _my_slot.
Regression: 87 unit tests; free-flight LAN; transition gate 0.00%; goal
cycle; spectator; disconnect and reconnect; full match to RESULTS.
Josh Creek