Stage 5 blocked after nine attempts and ~540M steps, every one on
productive_air_touch_fraction. Instrumenting the environment rather than
retuning the reward again found three separate causes, none of which was the
policy's competence.
The gate could not register the behaviour. productive_air_touch_fraction
divides by TOTAL touches in the episode, so a strong ground game dilutes it for
identical aerial play. Stage 4's entire purpose is improving that ground game
(it took forward_motion_fraction 0.24 -> 0.48), so Stage 4's success drove
Stage 5's gate toward zero and the two stages were working against each other.
It also explains why every non-zero reading in the whole lineage came from
degenerate episodes whose single touch happened to be aerial: per-episode 1.0,
which is exactly 0.0100 once meaned over SB3's 100-episode buffer, and 0.0100
was every run's observed maximum. Replaced with
productive_air_touch_episode_fraction, which asks whether the episode contained
a productive aerial at all and cannot be diluted by ground play.
The bar was never derived from anything. AIR_TOUCH_HEIGHT was 5.0 and four
rounds of aerial mechanisms were built on top of it without anyone measuring
where the ball goes. New ball-altitude telemetry over normal match play: the
ball averages ~1.6m, the average episode's peak is ~2.4m, and it clears 5m for
~5% of ticks. Lowered to 3.0, this project's existing airborne threshold, with
_place_air_intercept's band retuned 8-14m -> 6-10m. Simulated against real
physics the pair strictly dominates the old one: 67.8% reach (was 53.2%), 57.3%
above-bar touches (was 41.2%), 5.2m of climb instead of 8.2m. The band could
not be lowered alone -- at a 5m bar, 8-14m was optimal and 5-8m collapses
above-bar touches to 4.3%. This reverses Round 9's explicit "AIR_TOUCH_HEIGHT
stays 5.0"; that objection was about comparability, and a metric that read 0.0
for nine attempts has no history to protect. Pre-2026-08-24 air-touch figures
are not comparable with later ones.
Note AIR_TOUCH_HEIGHT also gates air_touch_bonus_weight's payout, so unlike
Round 9 this DOES change the reward function and the usual "don't resume a
policy shaped by a different reward balance" rule is engaged rather than exempt.
Resuming retry2 anyway is justified on narrower grounds: the changed term has
never once fired (productive_air_touch_fraction exactly 0.0 across nine
attempts, air_touch_fraction at ~0.0003 noise), so no learned value estimate is
attached to it, while the ground handling and scoring retry2 does know are
untouched. The flip side is that at a 3m bar a fully-aligned aerial touch now
pays 0.7 + 0.5 = 1.2 against a ground touch's 0.7, which is the intended
incentive but is a live reward change -- if attempts show touch farming near 3m
rather than genuine intercepts, cut air_touch_bonus_weight rather than raising
the threshold back.
The policy could not climb, and the entropy controller could not see it. Its
target is a sum over heads, which read 21% of h_max -- on target -- while
thrust_y alone sat at 14% of its own ceiling. The measured consequence was a
policy commanding ~0.03 mean vertical thrust when hovering needs 0.408
(120/5 = 24 m/s^2 against 9.8 gravity), leaving it in free fall ~84% of every
episode. Added --min-head-entropy-frac so one starved head raises ent_coef
regardless of the aggregate, and --ent-coef-max because a probe pinned the old
0.05 ceiling for its entire duration with the head still starved.
A 200k-step probe from retry2 with all three in place moved air_touch_fraction
from 0/74 rollouts non-zero to 5/98, ent_coef 0.0102 -> 0.0416 and
vertical_thrust_mean 0.031 -> 0.089, with goal_rate, upright_fraction and
forward_motion_fraction all holding. The gate metric was still 0.0 at that
scale, so its 0.02 floor is marked provisional in generation5.py and should be
re-derived from attempt 1's tail rather than trusted.
Stage 5 expands to 90M timesteps and MAX_RETRIES 4, its goal_rate floor drops
0.75 -> 0.72 (every attempt landed 0.7217-0.7369 and was failed by ~2-4% while
winning its paired evaluations 54-25, 63-23 and 47-32), and state resumes from
20260823-1734-gen5-s5-intercepts-retry2 via resume_override.
Verified: generation5.py --dry-run resolves the resume to retry2 with the new
flags, 123 unit tests pass, probe artifacts removed.
The Hard-tier promotion noted a 17% physical side imbalance (physical teams
0-1 = 29-46) and flagged it as worth investigating, possibly in the arena or in
ship_observations.gd's team-1 mirroring. Testing it directly shows that was
wrong.
Ran hard.json against itself — self-play, so any team_0/team_1 split is purely
positional and cannot be a strength difference — over 10 independent seeds at
30 episodes each. Pooled: 113-125 across 300 episodes, 4.0% imbalance, sign
test p = 0.48, team 1 ahead in only 3 of 10 seeds. Per-seed imbalance ranged
0.0% to 43.3%, so swings larger than the original observation happen by chance
at this episode count.
The underlying mistake is worth recording, and is now in TRAINING.md:
evaluate.py --seed defaults to 1, so the two measurements that appeared to
agree were the same paired starting-state sequence rather than independent
samples, and seed 1 happens to favour team 1. Same reason the
physical_side_imbalance_ceiling gate in generation5.py is a single-seed
catastrophe check, not evidence about side balance.
Hard has been a label-only duplicate of medium.json since medium was promoted
on 2026-08-17. Promote 20260823-1734-gen5-s5-intercepts-retry2 into
hard.json so the tier is a genuinely distinct policy, and so the strongest bot
the curriculum has produced survives the next round's checkpoint pruning —
promoted files are never touched by training scripts.
Stage 5 blocked after three attempts, so like medium.json this comes from a run
recorded as decision: "fail". Both failing floors are covered in TRAINING.md:
goal_rate 0.7369 vs 0.75 is marginal, and productive_air_touch_fraction 0.0001
vs 0.005 is a bar no policy in the lineage has approached, against a metric
quantised at 0.01 per ~100-episode window. On every other axis it is the best
yet: upright_fraction 0.757 against a 0.40 floor that the pre-Round-6 lineage
never pushed past 0.331, and forward_motion_fraction 0.479 against 0.20.
Chosen over attempt 2 (retry1) on a tiebreak, not a margin. retry1 posts a much
wider indirect result against medium.json (63-23-14 vs 47-32-21), but a direct
100-episode head-to-head between the two finished 36-39 with 25 draws, so that
gap does not reflect a real strength difference. Attempt 3 is the later
checkpoint (it resumed from attempt 2) and edges every telemetry metric.
Verified: hard.json is byte-identical to its source export, matches easy/medium
on input_size 83, 3 layers and action space, and beats medium.json 19-7-4 in a
fresh 30-episode paired run. Tiers stay monotonic: hard > medium > easy.
That head-to-head also showed a 17% physical side imbalance (physical teams
0-1 = 29-46), reproduced at 13% in the 30-episode check. Inside the 20% bar
used elsewhere and equal across both models, but noted in TRAINING.md as worth
investigating rather than assuming variance.
Godot's ConfigFile writer does not round-trip comments in project.godot. An
observed rewrite deleted both `;` blocks outright and spliced the three-line
`#` block above run/main_scene.dedicated_server onto the setting's own line,
leaving it commented out — which would send dedicated builds to the
interactive main menu instead of server_boot.tscn, with nothing failing until
someone noticed a server process rendering a menu.
Move the explanations into the code that owns the settings (server_boot.gd for
the dedicated-server override, video_settings.gd for stretch mode and vsync)
so they cannot be destroyed by a rewrite, and leave project.godot holding only
assignments plus Godot's own regenerated header.
Add tests/cases/test_project_settings.gd as the backstop: the feature-override
assertions read project.godot as text and reject a line that has been folded
into a comment, since ProjectSettings resolves `key.<feature>` overrides at
load time and never exposes the suffixed key. Verified by reproducing the
exact corruption, which fails the test, and it also covers the Jolt physics
engine, the required autoloads, and that no test-hook autoload is ever shipped
registered.
run_ci_host_check asserted input reached the server by comparing each bot
ship's position against one recorded before the check forced a goal. But a
goal's kickoff teleports every ship back to spawn (_begin_kickoff ->
reset_ships), so that comparison measured only the distance covered since the
last reset — a window whose length depends on when the sample lands relative
to the kickoff rather than on whether input was flowing at all.
It failed on master with peers at 0.51m and 0.23m against a 0.5m threshold:
one passed by a centimetre, the other failed, with both connected, neither
stalled, and every other assertion in the run green. The commit it failed on
touches only training JSON, and the push two minutes earlier passed on
identical game code.
Accumulate per-tick path length in _await_recording_score instead, discarding
any single-frame step over 2.0m as a teleport — Ship.max_speed (35 m/s) is
hard-clamped each tick in _integrate_forces, so 60Hz caps legitimate travel at
~0.58m. Same 0.5m threshold now reads 28-75m across runs, and it is strictly
stronger than before: it asserts input kept arriving for the whole wait rather
than that the ship merely ended up somewhere else.
server_boot.gd's private _log could only ever see what the boot scene
itself observed: connects, disconnects, roster changes, tick overruns.
The events an operator is actually asked about - who scored, who got
kicked and why, which peer is flooding - happen inside networked_match.gd
and match_sim.gd, neither of which could reach a logger on a scene node
that gets freed at the first change_scene_to_file. scripts/server_log.gd
holds it as static state on a class_name: reachable from all three, no
autoload, no ordering dependency.
New events: goal, match_ended, kickoff, peer_kicked (previously only a
push_warning, carrying neither peer nor reason into the stream a
container captures), rate_limited, server_stalled. rate_limited fires
ONCE per peer per window rather than per packet - a flood is thousands of
packets a second and the log line must not become the amplifier the
replay recorder was capped to avoid being.
Off unless a server configures it, so a client, an editor session or a
unit-test run does not start printing server telemetry just because these
scripts loaded.
Rotation is deliberately not implemented: the server logs to stdout and
stops, because every way this is run already rotates better - docker's
json-file driver, journald, or logrotate on a redirect. A server that
also wrote and rotated its own file would fight all of them in a
container, where stdout is the interface. SERVER.md (6.6) documents the
three configurations.
Five tests on the one piece with real logic - the one-line contract.
Including log injection: a player name is attacker-controlled, and
without escaping, the name "x\n[0.000] INFO peer_kicked reason=nothing"
writes a fake event into the operator's log. Newlines are escaped rather
than dropped so the attempt stays visible.
End-to-end verification of the new events comes with 6.5, which is what
first makes a server run a match at all.
6.1: "Linux Dedicated Server" preset (dedicated_server=true,
custom_features="dedicated_server") mirroring the existing training
preset, plus run/main_scene.dedicated_server so the server binary reaches
its own entry point with no flag. Builds: an 85MB Linux x86_64 binary,
gitignored like the training one.
6.3: scripts/server_config.gd declares every server flag once - name,
type, default, section, help - and one parser turns that into parsing,
type checking, range validation, config-file backing and --help. The
flags had grown to ~30 across server_boot.gd and networked_match.gd, each
parsed inline with begins_with, none documented, and an unrecognised flag
was SILENTLY IGNORED: --max-clientss=8 ran a server on the default cap
and said nothing. Unknown flags, missing values, wrong types, duplicates
and out-of-range values are now hard errors, reported all at once.
Precedence is command line > config file > default. server_boot.gd parses
strictly because it owns the whole command line; networked_match.gd reads
the same declaration leniently because it is one consumer of an argv the
smoke harnesses also fill with --role= and --drive-seconds=. Nothing is
lost - every server flag is declared, so the strict pass already caught
any typo before the match scene re-reads its own.
13 unit tests covering the precedence order, the typo rejection that
motivated this, --no-<bool> not double-listing in --help, and --help
documenting every flag asserted against the declaration rather than a
hand-kept list. Verified end to end: --help prints, a typo'd flag refuses
to start, and the plain/replay-log/late-joiner smoke scenarios still pass.
The document had no one place that answered "what is left". Outstanding
items were spread across two phase-gate lines, two phase tables, §11, and
prose buried in the phase notes - and the name-keyed slot-reservation
hijack, which an adversarial review demonstrated with a real three-process
run, was not written down anywhere at all. It existed only in a
conversation.
New §0 indexes everything not done, in four groups: verification a
machine cannot do (the Phase 4 playtest, the Phase 5 3v3 gate), known
defects left unfixed with their severity, the one open architectural
question, and the two unstarted phases. Each row points at the detailed
write-up rather than duplicating it, and the phase gates now point back.
§11 gains the identity defect in full: reservations match on
slot.player_name and nothing else, with no uniqueness constraint on names
anywhere, so a peer joining during the 30s window with a departed
player's name is handed their slot, ship and team. A bespoke token would
be half of task 7.4 thrown away, so it stays deferred - with the
consequence stated plainly rather than implied, and listed as a
precondition of Phase 6's internet-facing gate.
Also refreshes the stale status paragraph and task 5.10 for the replay
log's reject recording, write-failure handling, close(), and dump tool.
"Spectate now, take the slot at the next kickoff" was a print statement.
The server logged it and never acted; on the client, _is_spectator was
assigned once in _on_match_config_received and never revisited - and that
handler returns early whenever _slots is non-empty, so no rebroadcast
could promote an in-match spectator. The reconnect path only worked
because a returning player is a fresh process.
Server: late joiners are queued in arrival order and the queue is drained
from _begin_kickoff, before the reset transforms are read, so a promoted
player's ship is placed by that same kickoff and the controller swap
lands on an already-frozen body. A slot is available only once its player
has gone AND their 30s reservation has lapsed - §6.4 outranks §6.3, since
taking a reserved slot would quietly break the reconnect promise.
_abort_if_abandoned now counts a waiting spectator as somebody present,
or the one person queued for the slot that just opened is dumped to the
lobby at the moment they were about to get it.
Client: new broadcast slot_assigned (reliable, channel 0). Broadcast
because every client holds its own slot list and one naming the wrong
peer keeps flying somebody else's ship as a remote body; reliable because
no per-snapshot field would re-converge a client that missed it. The
promoted client undoes what made the body remote - fresh interpolator,
physics interpolation back on, offsets cleared - and deliberately does
not unfreeze, clearing _local_prediction_ready so the next snapshot
teleports it to a real authoritative pose first. The controller-attach
block moved to _take_local_ownership rather than being copied.
New --role=host-latejoin/--role=client-latejoin and
--slot-reservation-seconds=. Verified 4/4 both sides: queued, NOT
promoted merely because the reservation lapsed, takes the slot at the
kickoff, same ship instance, and both peers independently measure ~45.7m
under its input. Control with a 90s reservation: kickoff fires, nothing
is promoted, the slot still reads the departed player's name.
The disconnect scenario only ever asserted the server's bookkeeping, and
the client's half was failing every run. run_disconnect_host_check ticked
60 physics frames past the reclaim and then shut the server down, so the
reconnecting client - whose wiring check waits a 2.0s settle before it
looks at anything - had its peer torn out from under it and reported
"current_scene is not NetworkedMatch after 2.0s". The host printed PASS
throughout, and the host was the side anyone read.
The hold is now a real window (8s), and the host also asserts that the
reconnected player's input reaches the server and moves the ship the
server owns - every other assertion there is slot bookkeeping that would
hold identically for a client whose input pipeline came back dead. Both
position and connection state are sampled while the peer is still
connected: the client leaves on its own schedule, and an end-of-hold
sample reported still_connected=false for a good run.
New --role=client-reconnect asserts the returning player is not a
spectator, owns a slot with its own peer_id, has a real ship, rejoined a
live match with the clock already known (§6.2 step 2's bootstrap), and
can still drive. That set is chosen because a stale _last_match_config
once made a reconnecting player a spectator, and that bug was visible in
this scenario's own logs while it reported PASS.
Verified 3/3 both sides. Control: rejoining while the slot is still
occupied fails on is_player=false - and since the first control run
reported it as the generic "lost its ship mid-drive", the spectator case
is now diagnosed before the drive rather than after.
Closes task 5.10's three recording gaps, and the gap-closing found a real
input-loss bug.
Replay log: a failed write now ends the log permanently instead of
desyncing every later record's framing; close() is called from _exit_tree
with a summary, since the RefCounted destructor closes it implicitly but
never says whether the log is complete; rejected packets are recorded
with their reason in the kind byte (framing unchanged, FORMAT_VERSION 2
so "no rejects" differs from "this build never recorded them"). Recording
is capped at 8 per peer per window - uncapped, the diagnostic is a remote
disk-fill amplifier, since the attacker picks the packet rate. Uncapped
totals live on MatchSim and survive the peer's disconnect.
The bug: a 2s host stall has the client sending at 60Hz throughout, and
ENet delivers that whole backlog in the first window after resume - 70 of
an honest client's packets rejected as "rate limit exceeded". Redundancy
does not cover it, because the dropped packets are contiguous: 0 of 70
rescued, and 82 of 923 sequences (8.88%, ~1.4s of input) never reached
the server, against 0.00% with no stall. Every prediction gate passed.
Fixed by granting each already-tracked peer a capped, two-window packet
grace when the server detects its own wall-clock stall. Rate-limit
rejects 70 -> 0, sequences missing 8.88% -> 0.00%, seq-guard rejects
9 -> 0. Controls on the unfixed build lost 4.34/7.52/7.86%. All three
abuse roles still disconnect and no flood induced a stall, so the grace
cannot be farmed.
Also corrects an earlier wrong conclusion: the reviewer's free-flight
p95 0.688 is real and reproduces on two processes with 0.0% snapshot
loss. The plain --role=client drive fails the 0.5 free-flight bound in
3 of 8 runs because that drive is mostly a contact test - the harness
comment already said so - leaving a cohort as small as 12 samples.
Near-surface error is genuinely several times open-air error, so the
calibrated bound now belongs to --exercise-free-flight alone and the
plain role asserts the always-well-sampled all-cohort percentiles at
1.2/2.0, printing the free-flight numbers as reported-not-asserted.
6/6 plain runs pass where 3/7 failed; tightening to 0.3 still fails.
tools/replay_dump.gd reads a log back: counts by kind, plus how much of
the input sequence stream reached the server once redundancy is counted.
A percentile alone cannot tell "the predictor regressed" from "the client
never received the data". The client gate now prints snapshot_loss /
snapshot_age / rtt on every run, and on a quality failure with >20% loss
says explicitly that the run was transport-starved. It deliberately does
not convert the failure into a pass: a client that cannot receive
snapshots is still a failed run, just a differently-diagnosed one.
Both directions of the new branch verified non-vacuously (forced true so
it fires and formats; restored so it stays quiet on a healthy run while
the INFO line still prints).
Records the investigation behind it in multiplayer-todo.md: the reviewer's
3-process p95 0.688 did not reproduce. An idle third process costs nothing
(p99 0.094), a spectator costs a small but real amount (p99 0.094-0.146),
and snapshot loss held at 0.0% even under 2x CPU oversubscription - all an
order of magnitude inside the 0.5/2.0 gates. Also notes that a previously
working class_name can silently drop out of the .godot class cache, which
surfaces as a bogus parse error with nothing in git status to explain it.
productive_air_touch_fraction sat at exactly 0.0 across nine Stage-5
attempts and 540M timesteps. Two rounds of reward shaping were aimed at
it (air_approach_weight, then air_touch_bonus_weight); both worked --
airborne_fraction 0.223->0.258, mean_altitude 2.59->3.25,
vertical_thrust_mean 0.004->0.063 -- and the ship now visibly plays the
ball in the air. The metric could not see it because it counts only
touches with the ball above AIR_TOUCH_HEIGHT (5m), and
_place_air_intercept never produced a reachable one.
Simulating the spawn distribution against the ship's flight envelope
(vertical_thrust 120 / mass 5 = 24 m/s^2 less gravity, drag capping
climb near 12 m/s): a ball spawned 6-12m up at 6-11 m/s is above 5m for
a median of 0.80s, while the ship spawned 7-13m behind, 3-10m below, and
at a dead stop. An ideal interceptor -- point mass, instant attitude, no
righting torque, zero reaction delay -- makes that touch in 0.00% of
episodes and reaches the ball at all in 0.5%.
Retune the drill instead of the reward: ball higher (8-14m) and slower
(4-8 m/s), ship closer (4-9m behind), narrower lateral spread, and a
6-14 m/s planar run-up rather than a standing start -- the dead stop was
the largest single factor. Ideal interceptor now reaches the ball in
~98% of episodes and above 5m in ~37%, so the 0.005 floor has headroom.
AIR_TOUCH_HEIGHT stays 5.0 so the metric remains comparable with earlier
generations.
Resume from retry2 rather than restarting from Stage 4: that rule guards
against a changed reward function invalidating the value function, and
the reward function is untouched here -- only the state distribution
moved, so the policy that already learned to fly is what should be
pointed at a reachable target. Adds a one-shot resume_override to
generation5_state.json, consumed on first use.
A second adversarial review (this one able to RUN things, unlike the
first) reproduced five defects. Fixing the critical and high ones.
CRITICAL - Esc no longer left a networked match, and a client whose
server vanished was stranded forever. Two independent bugs composing:
_unhandled_input (added for spectator target cycling) overrode
GameMode._unhandled_input and returned early for every non-spectator
without ever calling super(), silently killing ui_cancel -> main menu;
and NetworkedMatch never connected NetworkManager.disconnected_from_
server the way lobby.gd does. Measured: a client whose host exited
emitted 7,235 engine errors in ~18s and only left because a test timer
fired. Now 1 benign teardown error, and it returns to the main menu.
HIGH - the match clock lost up to 3 seconds of regulation per goal.
_on_goal_registered extended end_tick by the celebration only
(resume_tick - goal_tick) and never by the 180-tick kickoff countdown
that follows it, while _update_clock derived remaining time from the
current tick regardless of _clock_running - so regulation drained during
every stoppage. Measured 660 PLAYING ticks for a 14s match against 840
expected: exactly one WARMUP lost. The HUD also opened at 0:17 for a 14s
match because the initial arm folded WARMUP into end_tick.
Replaced the per-goal arithmetic with bank-and-rebase: entering any
non-live state banks the remaining ticks, leaving it rebases end_tick
off the banked value. That covers celebration and countdown together and
cannot drift, since nothing has to predict how long a stoppage will be.
clock_state and match_bootstrap now carry remaining_ticks, which is
authoritative whenever the clock is stopped. Verified with the
reviewer's own metric: 840 PLAYING ticks for a 14s match, exactly.
MEDIUM - clients never froze at FULL_TIME/RESULTS. The freeze handling
sat inside `if multiplayer.is_server()`, so a local player flew around
for the whole 8s results screen while every other peer saw their ship
parked.
Not fixed, and now demonstrated rather than merely suspected:
- The 30s slot reservation is keyed on display NAME, so a stranger can
take a departed player's ship and the real player is then locked out
(reproduced). Worse than first thought: MatchNet.local_player_name
defaults to "Player" and uniqueness is never enforced, so collisions
are the common case, not an attack setup. Needs a real identity token;
§6.2 step 1 reserves auth_ticket for Phase 7.
- §6.3's "late joiner takes the slot at the next kickoff" is
unimplemented - _is_spectator is assigned once and never revisited -
while the server logs that it happened.
- Replay log still ignores store_* return values, never records
malformed/rejected inputs, and close() has no caller.
- --role=host-disconnect grades the reconnecting client on ~1s of life
before the host quits, and never asserts the client owns _my_slot.
Regression: 87 unit tests; free-flight LAN; transition gate 0.00%; goal
cycle; spectator; disconnect and reconnect; full match to RESULTS.
An adversarial review found five real defects in the Phase 5 lifecycle
work. Two were critical and both were verified against controls.
CRITICAL - a reconnecting client silently became a spectator.
_try_reclaim_slot() swapped slot.peer_id, but MatchSim caches the last
match_config and replays THAT to whoever asks. A reconnecting client in
a fresh process requested config, received the pre-disconnect peer-id
array, could not find itself, left _my_slot null and fell through to the
spectator path - no ship, no input, for the rest of the match. The
evidence was already in my own disconnect-test logs ("no slot for this
peer - spectating", my_slot_ok=false) and I dismissed it: the host-side
check only asserted the SERVER reclaimed the slot, never that the
returning client owned it. Config is now rebroadcast on reclaim.
Verified: my_slot_ok=false -> true.
CRITICAL - spectators received no snapshots at all. §6.3 says a
spectator "receives identical snapshots (the snapshot is already a
broadcast - zero extra server work)". That was only ever true of the
body SEGMENT: _broadcast_snapshot unicasts one packet per SLOT, so a
peer without a slot got nothing - no poses, no reset_gen, no
match_state byte. Spectating was entirely non-functional. The segment is
still shared, so this is one extra send per spectator. Verified against
a control: 0 snapshots and state stuck at LOADING before, 361 snapshots
and PLAYING after.
HIGH - cycling the spectator camera to the ball was a type error.
ShipCameraRig.target is declared `var target: Ship` and the rig reaches
into ship-only API, so it would have fired the moment anyone cycled past
the last ship. Cycling is ships-only; the rig already has its own
ball-cam mode for watching the ball.
MEDIUM - clients never received match_ended or overtime_started. Both
emitted only inside server-side logic, so a client froze and returned to
the lobby without a result and its timer never switched to overtime.
Derived from replicated state instead of adding two more RPCs: the
client already has the authoritative score, and the transition is the
event.
MEDIUM - the goal cinematic ignored its authoritative window. goal_tick
and resume_tick arrived and were unused; the client started a fresh
fixed-length timer on RPC receipt, so a reliable retransmit could run
the celebration past the server's window and into the next kickoff.
_goal_pause_seconds() now returns the time actually remaining, clamped
so an elapsed window cannot produce a non-positive timer.
Also added: a match_bootstrap RPC carrying state, score, clock and
reset_gen to one peer. match_config alone carries arena and roster only,
so a late joiner or reconnecting player had no score or clock until the
next goal happened to fire. It is sent on join AND on every
request_match_config retry - the join-time send has exactly the same
race match_config already had (the server sends it before the peer has
loaded the match scene and connected its listeners), which the control
run exposed: state was reaching PLAYING via the snapshot byte, not the
bootstrap.
New test: --role=client-spectator asserts a slotless peer receives the
snapshot stream, follows the lifecycle, agrees with the wire byte, and
can cycle targets without ever handing the camera a non-Ship. Verified
non-vacuous. The ball-contact steering now closes all the way to 1.2m
instead of coasting from 3m, which was missing the ball outright in
roughly 1 run in 4.
Not fixed, and still open: the 30s slot reservation is keyed on the
player's display name, so any peer can claim a departed player's ship by
choosing their name. §6.2 step 1 reserves auth_ticket for Phase 7; this
needs a real identity token, not a name.
Regression: 87 unit tests; free-flight LAN; transition gate 0.00%; ball
contact 4/4; goal cycle; full match to RESULTS/LOBBY; disconnect and
reconnect; spectator; two-bot CI.
Completes Phase 5's implementation. Every task is verified at 1v1; the
3v3 phase gate itself has not been run and remains outstanding.
5.6/5.7 disconnects: a ship is never despawned. The slot keeps it and
swaps the controller (--fill-bots gives it a bot, the default leaves it
inert per §1.4), sets `stalled` immediately so the nameplate greys out
rather than waiting ~500ms for the abandoned jitter buffer to starve,
and reserves the slot for 30s keyed by player name so a reconnect gets
the same ship back.
5.7 was a real bug, found by the test rather than by review:
SlotInfo.controller was declared RLShipController, but the takeover
swaps in an AIShipController or the base controller - the narrower type
makes that assignment fail its type check, leaving the field pointing at
the controller set_controller() just queue_free()d. It surfaced as
controller_valid=false on the first run. The per-tick action write is
now also gated on `is RLShipController`, since a disconnected slot's bot
drives itself and overwriting it from a starving buffer would pin it to
the departed player's last input.
§6.4's two rules conflict: reserve for 30s, but abort when the last
human leaves. Applied naively the abort wins instantly in a 1v1 and the
reservation can never be redeemed, making reconnect unreachable exactly
when it matters. Abort now waits for no connections AND no outstanding
reservations.
5.8 spectators: a slotless peer spawns no ship and receives the same
snapshot broadcast. HUDController.spectator_mode keeps the clock, score
and goal celebration and hides only the ship instrument cluster - it
previously push_error'd and bailed, leaving a spectator with a dead HUD.
Camera cycles ships in slot order then the ball. --max-spectators caps
it, counted from the live peer list so a dropped spectator cannot leak a
unit of the cap.
5.9 escape respawn: new GameMode._on_bodies_respawned() virtual;
NetworkedMatch bumps reset_gen through Phase 2's deferred path so the
bump and the respawned pose land in the same broadcast. Single-player
modes are unaffected - the base is a no-op.
5.10 replay log: scripts/replay_log.gd, --replay-log=<path>, storing the
wire bytes verbatim in both directions rather than re-serialising - a
re-encode would launder away precisely the malformed payload being
chased. A live 6s match recorded 1115 records (557 inputs / 558
snapshots) and a stored snapshot decodes back to server_tick=100
match_state=WARMUP bodies=2.
Note for future work: --check-only --script is the only thing that
catches a parse error in networked_match.gd, because the unit runner
never loads it. Two separate breakages passed the full unit suite while
breaking every two-process run. A new class_name also needs --import
before it resolves.
Test surface: --role=host-disconnect (three-process 5.6/5.7 scenario),
--match-length=<s>, --replay-log, --fill-bots/--no-fill-bots,
--max-spectators. The ball-contact scenario now steers at the ball with
closed-loop real input instead of a hand-tuned fixed heading, which 5.3
broke by adding KICKOFF_YAW_JITTER; thrusting while turning took it from
2/3 to 5/5.
Regression: 87 unit tests; free-flight LAN p99 0.094m with 0 hard snaps;
transition gate 0.00%; ball contact 5/5; lifecycle goal cycle and full
match to RESULTS/LOBBY; disconnect+reconnect; two-bot CI.
Implements the rest of the §6.2 lifecycle on top of 5.1's state machine.
5.3 kickoff: the server resets every body and broadcasts the RESULTING
transforms, never a seed - §1's locked decision, because shared-seed
determinism needs both sides to consume the RNG stream in identical
order forever and the first randf() added to the reset path desyncs
silently. Countdown is derived from server_tick on both peers, and a
kickoff that lands after its own resume tick applies immediately and
skips the countdown rather than scheduling into the past.
5.4 goals: goal_scored(scoring_team, score, goal_tick, resume_tick).
Score is authoritative at sensor time, before any presentation. The
reset moved OUT of the sensor path and into the kickoff at resume_tick,
which is what stops the server resetting while clients are still
mid-celebration. Engine.time_scale is never touched.
5.2 clock: tick-derived, no Timer and no _process polling. The goal
pause shifts the absolute end_tick by (resume_tick - goal_tick) rather
than pausing anything, so no float drift accumulates across goals.
5.5 full time: clock expiry -> FULL_TIME -> sudden death on a draw or
RESULTS, golden goal in overtime, then LOBBY on both peers - clients
return to the lobby, not the main menu. get_tree().paused is never used.
Four bugs found and fixed while building this, each by a failing run
rather than by inspection:
- Tick order was load-bearing: _update_kickoff_countdown() clears the
same _kickoff_resume_tick that _update_match_state() reads to leave
WARMUP, so running the countdown first wiped the transition condition
and the match sat frozen in WARMUP forever.
- _apply_match_state resets _state_deadline_tick on every transition, so
a GOAL_PAUSE deadline assigned before _set_match_state was wiped and
the match never resumed. Deadlines are now owned by _apply_match_state.
- Freezing "all bodies" is wrong on a client. Remote ships and the ball
are permanently FREEZE_MODE_KINEMATIC and transform-driven; freezing
them all unfroze the remote ones on the way back out, so they fell
under gravity while the interpolator fought them - 210 hard snaps and
an infinite p99. A client now freezes only the one body it simulates.
- A frozen body never runs _integrate_forces, so the queued kickoff
teleport was stranded by an immediate set_deferred("freeze", true).
Freeze now happens on a strictly later tick, the same pattern Phase 2
used for _pending_reset_gen_bump_tick.
Prediction and reconciliation are suspended while the match is not live:
during a countdown or goal pause the local ship is frozen on both peers,
and running delta transport over those frozen states produced a p95
position error of 2.4e10 m. Input keeps flowing so the server's jitter
buffer does not starve into `stalled`.
Also fixed: a kickoff can arrive before match_config, and body order is
slot order - applying it early placed the BALL at positions[0], on top
of the first ship, which the ball-cam reported as "target vector can't
be zero" 95 times. It is now held until the roster exists.
Test changes: the ball-contact scenario steered by a hand-tuned fixed
heading, which 5.3 broke because kickoff applies KICKOFF_YAW_JITTER - it
flew past the ball in 3/3 runs. It now closes the loop on the actual
bearing using real input actions. Assertions that read a frozen ship
(freeze, thrust) are gated on the match being live, and the hooks now
survive the scene teardown at RESULTS instead of hanging on freed
objects for the full timeout.
Regression: 81 unit tests; free-flight LAN p99 0.143m and 80±20ms, both
0 hard snaps; transition gate 0.00%; ball contact 3/3; two-bot CI.
Adds the §6.1 state machine, its broadcast, and the client side that
follows it. Physics, freezing and input are deliberately NOT gated on
state yet - 5.3 and 5.4 own freeze/unfreeze at kickoff and goal, and
doing it here would change the conditions every Phase 4 prediction gate
was measured under.
scripts/match_state.gd holds the enum and transition table as pure data
with no scene or RPC dependency, so the table is checked exhaustively
rather than by example: every state reachable, every state has an exit,
no self-transitions, abort-to-LOBBY from anywhere per §6.4, illegal
shortcuts rejected, unknown values refused rather than coerced. The enum
values are the wire format - match_state has been a u8 in the snapshot
header since §2.4 - so a test pins them; only append, never renumber.
The server validates every transition and push_errors an illegal one
rather than following it. Clients deliberately do NOT enforce the table:
authoritative state must be accepted, and a late joiner legitimately
jumps straight to PLAYING.
Two channels carry the state. state_change (reliable, channel 0) is
prompt and carries an absolute at_tick, never a duration. The snapshot's
match_state byte is the catch-up path for a client not yet sent a
transition - a late joiner, or the window between scene load and the
first RPC.
The byte needs a tick guard, and this was found the hard way. Snapshots
are unreliable_ordered on channel 2 and ordering holds only within a
channel, so a state_change for tick N routinely arrives before an
in-flight snapshot from tick N-2. Without the guard the client applies
the new state then gets dragged back by the older byte, oscillating on
every transition - observed directly as LOADING -> WARMUP -> LOBBY ->
PLAYING -> LOBBY while running a deliberately-broken-byte control. Only
a byte at least as new as match_state_since_tick is accepted.
WARMUP_TICKS/GOAL_PAUSE_TICKS are honest placeholders so 5.1 drives real
transitions to verify against; 5.3 and 5.4 replace them. The server also
leaves LOADING immediately rather than waiting for scene_ready, which
does not exist yet.
New smoke flag --exercise-match-state, passed to both roles: the host
forces a goal to drive a GOAL_PAUSE cycle, the client records the
sequence and asserts every consecutive pair is legal, that ticks are
monotonic, and that the wire byte agrees with its own state. Observed
LOADING -> WARMUP -> PLAYING -> GOAL_PAUSE -> WARMUP with tick deltas
matching the configured durations exactly.
Verified against a control: hardcoding the snapshot byte back to 0 fails
both the byte assertion and the transition-legality assertion. The byte
is asserted separately from the RPC precisely because everything else in
the check is RPC-driven and would pass with a dead byte - the same gap
that hid the Phase 4 label bug (gotcha 47).
Regression: 81 unit tests; 60s free-flight LAN (p99 0.148m, 0 hard
snaps, marker 0/3364); transition gate 0.00%; ball contact; two-bot CI.
Closes Phase 4's outstanding action-sequence-correctness invariant, then
fixes two server-side bugs an adversarial review of that work uncovered.
Server simulation, bot observations, collision resources and tick rate are
unchanged: the server_physics_parity trace is byte-for-byte identical to
HEAD across 360 ticks including both ships' full observation vectors.
4.11 - prediction history filed under the ISSUING sequence
_send_local_input filed each post-step predicted state under the timeline's
estimate of the sequence the server would consume this tick, trailing
issuance by input_lead. The body had integrated the intent issued under
_input_seq, so predicted[S] held "state after the intent from now" while
the server's authority for S is "state after action(S)". They agree only
while the stick is still. Filing under _input_seq costs nothing: which
action the ship uses is decided in LocalNetShipController.get_action() and
is untouched.
Every prior Phase 4 gate held its input steady, and a steady input cannot
falsify a sequence label - the 60s runs honestly reported marker=0/3784.
New --exercise-input-transitions role toggles thrust every 6 ticks; it is
the only gate that can catch a label regression. Verified non-vacuous: the
old label fails it at 50%.
4.12 - issued-but-unsimulated sequences, and the release path
An attack (delta > 1) issues and sends several sequences for one local
physics step. Those gap sequences had no recorded prediction, so a server
ack of one reported missing_not_recorded - indistinguishable from ring
loss, costing a teleport and resync suppression several times a minute.
They are now recorded stateless via record_unsimulated() and answered with
a new "skip" decision mode. Free-flight hard snaps: 25/8/4 -> 0/0/0.
A release (delta == 0) re-recorded at the unchanged _input_seq, filing the
current intent under a sequence that went out carrying a different action;
LocalInputTimeline deliberately refuses to mutate an issued sequence, so
the ring contradicted the wire. Recording is now skipped on release ticks.
4.13 - two Phase 3 bugs silently killing player input
(a) InputJitterBuffer.consume() advanced last_applied_seq on every tick
including a starve. Since ingest() discards seq <= last_applied_seq, one
starve on a sequence the client had not sent yet stranded the stream one
ahead of arrivals permanently - both sides advancing in lockstep, every
honest packet discarded on arrival. The client's own input_lead release is
enough to trigger it, so input died for ~30 ticks roughly every 6.5s on a
clean LAN. Now only gives up on a sequence once strictly newer data proves
it lost. Silent-client stall and ring-overflow resync are unchanged.
(b) The seq-range guard bounded incoming seq against highest_ingested_seq,
which only advances inside ingest(), which that guard gates. After a ~2s
host hitch every packet was rejected forever with no diagnostic (600+
consecutive rejections reproduced via SIGSTOP). Third iteration of this
guard; each previous version bounded against a value only the accepted
path could advance. Adds an escape after 10 consecutive rejections, which
grants an attacker nothing the rate limiter does not already bound.
(c) The transitions gate reported PASS at 3.76% while input was completely
dead, because suppression stops _record_metrics - a worse outage yields
fewer samples and a LOWER rate. Now scales the required sample count with
run length and asserts the wire's server_stalled bit. Reverting both fixes
makes it fail at samples 292/600, server_stalled=true, input_lead=12.
Fixing (a) also explained a residual the review had already traced: 151 of
151 action-marker mismatches were the server repeating a stale action on a
starve, not a prediction defect. Marker is now 0.00% in all three
conditions (was 1.7-2.5%), and free-flight p99 improved to
0.141/0.168/0.154m from 0.170/0.176/0.184m.
Two pre-existing test defects fixed alongside: the ball gate asserted
RTT-masking on a link with no RTT (flaked 2 in 5; now asserted only at
rtt >= 20ms, 5/5 under latency), and the two-bot CI compared scores across
a 3-5s window (now polls the scores the server actually held; note
score_changed is emitted only on the client path).
QA: 72 unit tests; 60s free-flight at LAN/80+-20ms/5% loss; transition
gate in all three; 2.0s and 3.5s host-freeze recovery; ball contact x5;
two-bot CI x3; all three abuse roles; net/match_net/clock/lobby smokes.
Phase 4 sign-off still pending a human playtest at ~100ms RTT - the
milestone asks how it feels, which no gate here answers.
Adds LocalPredictionHistory, a client-owned seq-tagged ring recording
predicted ship state per input sequence, plus wiring in
NetworkedMatch to record predictions on send and compare them against
authoritative snapshots on arrival. Ships stay frozen/interpolated
until 4.3 lands actual correction logic; this round only builds the
comparison machinery and its data.
Includes fixes from two review rounds: resync_required now
self-clears once acknowledgements catch back up (mirrors
InputJitterBuffer's stalled flag), NetBodyState gained a copy()
method to stop diagnostic accessors aliasing ring-owned state, and
corrected comments that had described the local ship as being
force-simulated pre-4.3 when it is still driven by interpolated
transform writes.
A second adversarial review of the previous fix commit found two of its
nine fixes silently defeated each other: the seq-range guard (fix for a
MEDIUM epoch-mismatch finding) capped the exact variable the ring-overflow
resync (fix for the original CRITICAL finding) depends on, making the
resync unreachable in production and recreating permanent input death at
a lower failure threshold, reachable via ordinary server tick loss alone.
- CRITICAL: rebind the seq-range guard to InputJitterBuffer's own
highest_ingested_seq (now public) instead of the consumer-side
last_applied_seq, so it tracks the client's send epoch rather than a
value that can lag arbitrarily far behind during a stall.
- HIGH: InputLeadController's release logic still ANDed the old
`lead > LEAD_MIN` gate onto the new depth-driven condition, so a
backlog the controller never caused still couldn't drain. Split into
two independent decisions: the seq-duplicate action follows real
depth alone; lead's own bookkeeping separately never drops below its
floor.
- MEDIUM: widen the CI driver's movement/stalled sampling margin
(run_seconds - 2.0, was - 0.5) and assert the peer is still in
multiplayer.get_peers() at sample time, since the old margin let the
check pass on residual starvation grace after a bot had already
disconnected.
- LOW: measure horizontal-only displacement in the human smoke test's
movement check — the old 3D-distance bar was beatable by pure
gravity settling with fully dead input.
- LOW: fix a real "clean stderr" violation (match_net.gd broadcasting
a departure notice to a peer whose ENet channels are already torn
down, including a second peer disconnecting in the same poll batch)
by deferring the notification to the next idle frame.
- Wire the server's per-slot stalled bit into the client debug overlay
for real — a prior commit message claimed this already reached the
overlay when only the CI gate actually read it.
Re-verified end-to-end against the real production RPC path (not just
unit tests in isolation, which is how the composition bug got past the
first round): a 2-bot CI match with a 1.5s host SIGSTOP freeze injected
mid-run, well past the 0.6s threshold the review reproduced the bug at,
now recovers cleanly on repeated runs with zero stderr noise.
An Opus subagent's adversarial review of Phase 3 found a critical, silent,
permanent bug plus eight smaller real issues, all empirically verified
with real two- and three-process runs:
CRITICAL: InputJitterBuffer's 32-entry ring permanently bricked a
player's input once the un-consumed backlog exceeded the ring's
capacity - a fresh arrival would land in the exact slot consume() was
still waiting on, and since both counters only ever advance, the gap
never closed. Reproduced with a real SIGSTOP/SIGCONT host freeze:
client movement dropped from ~26m to 0.00m at ~0.7s, worse under real
loss (a lossy link lowered the fatal threshold to ~400ms), and
reachable via ordinary clock drift with no external trigger at all.
Fixed by tracking the highest seq ever ingested and having consume()
jump directly to what the ring can still provide once the gap exceeds
capacity, instead of starving through an unrecoverable span. Re-verified
with a 3s freeze (well past the original threshold): full recovery.
HIGH: InputLeadController's release logic was gated on its own past
attacks (lead > LEAD_MIN) rather than the real server-reported depth, so
a backlog it didn't itself cause was never drained. Fixed to gate on
actual depth vs target.
MEDIUM-HIGH: the rate limiter's "N consecutive over-budget seconds"
streak hard-reset to 0 on any clean window, letting a duty-cycled flood
(burst, one clean window, repeat) sustain ~33x budget indefinitely with
zero warnings. Replaced with a leaky-bucket accumulator immune to the
same evasion by construction.
MEDIUM: the seq > server_tick + 20 guard compared two unrelated clock
epochs (server process uptime vs. client's own from-zero seq numbering),
so it never actually protected anything on a long-running server and
could silently drop an honest client's input forever. Bound against the
buffer's own last_applied_seq instead.
MEDIUM: InputJitterBuffer.stalled was computed but never reached the
wire - the one signal that would have made the ring-overflow bug visible
anywhere. Now wired through _ship_to_net_body_state.
MEDIUM: task 3.6's CI driver's assertions didn't depend on client input
reaching the server at all, so it kept passing with the ring-overflow
bug actively triggered. Added real ship-movement and non-stalled checks,
sampled while bots are still connected (an initial attempt sampled after
their own legitimate disconnect, which starves identically to the bug).
LOW-MEDIUM: a lead change silently mislabelled _input_history's older
entries, since the wire format has no per-entry seq field. Fixed by
handling each delta case (ordinary/release/attack) on its own terms.
LOW: bandwidth and snapshot-loss overlay metrics froze at their last
value during a total outage instead of decaying - exactly when they
matter most. Both now report honest post-outage values.
LOW: a guard comment on NetworkManager._ping misdescribed the actual
disconnect_peer() arguments in use. Corrected.
New permanent regression tests: test_ring_overflow_resyncs_to_fresh_data
_instead_of_starving_forever, test_release_drains_a_backlog_it_never_
caused_itself, and client-abuse-flood-dutycycle (reproduces the exact
duty-cycle evasion). Full regression suite, including the net-sim-latency
milestone gate, all abuse roles, and the CI driver, re-run clean after
every fix.
Documents all seven Phase 3 tasks (3.1-3.7) with DONE status and
verification evidence, updates the top-level status summary, and records
the phase gate as met - re-verified today under the gate's own exact
condition (--net-sim-latency 80 --net-sim-loss 0.05) on both the human
smoke test and the two-bot CI driver, not just the looser conditions
used during individual task development.
Adds one new gotcha (#38): GDScript lambdas capture enclosing locals by
value, not by reference, which silently broke two separate Phase 3 test
scripts' own disconnect-detection assertions this session (the
production disconnect logic was correct both times; only the test's own
flag-capture pattern was wrong).
Also records a deliberate scope decision for task 3.4: server-side
input_lead enforcement from arrival times was scoped down to
observability rather than built as active enforcement, since the
concrete security requirements (rate limiting, malformed-packet
counting, seq-range rejection, disconnect policy) already close the
load-bearing gaps and the doc's own text calls the remaining edge
"small" - flagged to revisit once Phase 4's prediction work exists to
judge against.
networked_match.gd's client can now swap its input sampler for a real
AIShipController (--test-bot, optionally --test-bot-model=<path>,
defaulting to bots/promoted/medium.json) instead of PlayerShipController.
Unlike the human sampler, AIShipController needs real scene context
(get_parent() as Ship, plus ball/teammate/opponent discovery via groups),
so it's parented onto the client's own ship via Ship.set_controller()
rather than left floating - and the field's static type widened from
PlayerShipController to the shared ShipController base to allow either.
Known, documented limitation: this client's ships are all
FREEZE_MODE_KINEMATIC and driven purely by transform writes, so nothing
ever writes linear_velocity/angular_velocity onto them - the bot's
observations always see every ship as stationary. It still produces
well-formed, bounded actions from that degraded input (the policy
network's output layer is bounded regardless of input quality), which is
sufficient for this task's actual job: generating realistic sustained
network traffic for CI, not winning matches.
New CI driver (tests/networked_match_ci.gd/.tscn): a headless server plus
two headless --test-bot clients playing a real match. task 3.6's original
acceptance text also named "p95/p99 prediction error" and "snap count" -
both Phase 4 concepts that don't exist until client-side prediction and
its hard-snap threshold are built, so asserting on them now would be
fabricated. What's checked instead: snapshot throughput (500+ received
over an 8s run, comfortably above a 60Hz-scaled floor), and genuine
cross-peer score agreement - forced via a deterministic server-side goal
(bot-vs-bot scoring isn't reliable enough within a short run to gate on),
with each client independently writing its own final score to a peer-id-
keyed file for the host to compare against the other bot's, not just
trusting the server's own view. "Clean stderr" is left as the external
invocation's job, same as every other smoke test in this project.
Verified with real 3-process runs (host + two bots): both clients
independently confirmed identical scores after a forced goal, both saw
500+ snapshots, and all three processes exited 0 with clean stderr on a
representative run (one run separately hit the same known, already-
documented single-benign-error disconnect-timing race task 3.4's own
abuse tests hit - not a new issue). Full regression suite, including the
net-sim-latency milestone gate and the abuse-detection tests, re-run
clean.
Extends net_debug_overlay.gd (Phase 1's RTT/offset display) with the
rest of task 3.7's list: jitter (new RFC3550-style EWMA in
NetworkManager, computed from raw per-sample RTT before Phase 1's own
min-filtering, since that filter is deliberately jitter-insensitive by
design), input buffer depth and input_lead (both already tracked
client-side for task 3.3), snapshot loss (a new EWMA in networked_match.gd
over each received snapshot's own server_tick gap - snapshots go out at a
steady one-tick cadence, so a gap is direct evidence of a drop or
reorder), snapshot age (computed on demand from the same bias-corrected
tick estimate the interpolator itself uses), and bandwidth (new rolling
per-second byte counters in MatchSim, on the two 60Hz hot-path channels
only). Prediction error is deliberately omitted with a comment explaining
why: there's no client-side prediction to measure until Phase 4.
Verified values are live and plausible, not just present, by calling
get_net_debug_stats() directly in a real two-process test and checking
the numbers make sense: bandwidth matched the wire format's own byte
math almost exactly (measured ~2400 B/s sent against a computed 40B x
60Hz, ~3540 B/s received against 59B x 60Hz), and buffer depth/lead/loss
all moved in the correct direction between a clean LAN run and one under
simulated 60ms latency + 10% loss. Full regression suite re-run clean.
MatchSim._recv_input now validates before decoding (§3.1 steps 2-3):
per-peer rolling-1s rate limiting (packet count AND byte budget, dropping
over-budget packets and disconnecting after 3 consecutive over-budget
seconds), and framing validation (redundancy count and payload size
checked against NetCodec's own layout before unpack_input ever runs,
disconnecting after 20 malformed packets). Framing has to be validated
explicitly rather than relying on decode failure: StreamPeerBuffer
silently zero-fills past EOF instead of erroring, a finding from Phase
2's adversarial review.
networked_match.gd's _on_input_received now rejects any seq claiming to
be more than 20 ticks ahead of the current server tick (§3.1 step 4) and
counts (rather than silently ignoring) input from a peer with no slot,
for observability.
Verified with two new permanent regression tests (networked_match_smoke.gd
--role=client-abuse-malformed / client-abuse-flood) that call
MatchSim._recv_input directly with garbage bytes and a legitimate-but-
too-frequent flood, respectively, bypassing the honest client encoder
entirely - the same thing a hostile custom client sending raw ENet
packets would look like. Both confirm real disconnection, not just that
the server tolerates the abuse.
Two bugs surfaced by getting these tests to actually pass cleanly: a
GDScript lambda-capture-by-value mistake in the tests themselves (a
plain `var disconnected := false` mutated inside a signal-handler lambda
never became visible to the enclosing function - fixed by capturing a
single-element Array instead, which is captured by reference); and a
narrow real race where NetworkManager's own ping/pong reply could target
a peer that a concurrent abuse-triggered disconnect had just removed
from the same poll() batch, now guarded. (Passing disconnect_peer's
`force` parameter as an attempted fix for a related one-off benign error
was tried and reverted - it made Godot's own peer-list bookkeeping
inconsistent, producing hundreds of errors instead of one; verified
empirically rather than assumed.)
Full regression suite, including the net-sim-latency milestone gate,
re-run clean.
Josh Creek