fix(tests): stop the ENet residual gate failing on host scheduling noise

The client-bot smoke asserted remote_residual_position_p99 < 0.3. Commit
52ee1810 both passed and failed on that assertion, in runs three seconds
apart on identical code: two clients in one run reported 0.324 and 0.187
with matching snapshot counts, scores and slot checks.

A p99 over a few hundred samples is its worst handful, so on a shared CI
host it measures how often the process was descheduled as much as how
well the interpolator tracks. Locally the same test reports p95 0.025m
and p99 0.081-0.149m; CI's p99 runs 2-4x higher on a healthy build,
which is the entire margin the 0.3 bar had.

Assert two bars instead of one. The tight numeric bar moves to p95,
which is stable run to run, and p99 is bounded by the product's own
REMOTE_VISUAL_MAX_OFFSET/REMOTE_VISUAL_MAX_ROTATION_DEGREES: past those
the visual smoother stops absorbing a correction in a single step, so
exceeding them is a real defect rather than a slow runner. Referencing
the constants also means the test follows the product if that tolerance
is ever retuned.

This is a genuine trade, not a free win: a regression that pushed p99
from 0.2 to 0.35 while leaving p95 healthy now passes where it once
failed. That band is exactly where the noise lives -- 0.324 was observed
on a healthy build -- so the old bar could not tell that regression from
a busy runner either, and paid for the ambiguity with false reds.

Both percentiles are printed with their bars so a future failure shows
which one moved.

Checked first whether goal-driven kickoff teleports were polluting the
metric; they are not. Both the ship and ball paths already skip
accumulation across a reset_gen change.
This commit is contained in:
Josh Creek
2026-09-05 22:22:30 +01:00
parent 52ee181042
commit aac00f148e
3 changed files with 28 additions and 3 deletions
+6
View File
@@ -2216,6 +2216,12 @@ func get_net_debug_stats() -> Dictionary:
"ball_proxy_moved_before_authority": _ball_proxy_moved_before_authority_count > 0,
"ball_proxy_moved_before_authority_count": _ball_proxy_moved_before_authority_count,
"ball_authority_changed_since_contact": _ball_authority_changed_since_contact,
# p95 alongside p99. A p99 over a few hundred samples is only its worst
# handful, so on a loaded host it reports scheduling jitter as much as
# interpolation quality. p95 is stable enough to carry a tight bar,
# leaving p99 to catch genuine tail blow-ups.
"remote_residual_position_p95": _remote_percentile(_remote_position_residuals, 0.95),
"remote_residual_rotation_p95": _remote_percentile(_remote_rotation_residuals, 0.95),
"remote_residual_position_p99": _remote_percentile(_remote_position_residuals, 0.99),
"remote_residual_rotation_p99": _remote_percentile(_remote_rotation_residuals, 0.99),
"latest_prediction_error": _last_local_prediction_comparison.get("position_error", Vector3.ZERO),
+21 -3
View File
@@ -1378,9 +1378,25 @@ func run_ci_client_check(run_seconds: float) -> void:
var snapshot_count_ok: bool = snapshot_count[0] >= min_expected
var net_stats: Dictionary = match_scene.get_net_debug_stats()
var present_time := bool(match_scene.remote_visual_present_time_enabled)
var remote_position_p95 := float(net_stats.get("remote_residual_position_p95", INF))
var remote_rotation_p95 := float(net_stats.get("remote_residual_rotation_p95", INF))
var remote_position_p99 := float(net_stats.get("remote_residual_position_p99", INF))
var remote_rotation_p99 := float(net_stats.get("remote_residual_rotation_p99", INF))
var remote_quality_ok := not present_time or (remote_position_p99 < 0.3 and remote_rotation_p99 < 5.0)
# Two bars rather than one loose one. The tight bar moved to p95, which is
# stable across runs; p99 over a few hundred samples is its worst handful,
# so on a shared CI host it measures scheduling jitter as much as
# interpolation. The p99 bar is the product's own tolerance: beyond
# REMOTE_VISUAL_MAX_OFFSET the visual smoother stops absorbing a correction
# in one step, so exceeding it is a real defect rather than a slow runner.
#
# The single p99 < 0.3 bar produced false failures: two clients in one run
# reported 0.324 and 0.187 with everything else identical, and the same
# commit passed and failed in the same minute.
var remote_quality_ok := not present_time or (
remote_position_p95 < 0.3 and remote_rotation_p95 < 5.0
and remote_position_p99 < NetworkedMatch.REMOTE_VISUAL_MAX_OFFSET
and remote_rotation_p99 < NetworkedMatch.REMOTE_VISUAL_MAX_ROTATION_DEGREES
)
var my_id := multiplayer.get_unique_id()
var score_path := "/tmp/cosmicclash_ci_score_%d.txt" % my_id
@@ -1388,8 +1404,10 @@ func run_ci_client_check(run_seconds: float) -> void:
f.store_string(JSON.stringify(match_scene.score))
f.close()
print("SMOKE INFO: client-bot snapshot_count=%d (want >= %d) slots_ok=%s final_score=%s remote_present_time=%s residual_p99=%.3fm/%.3fdeg" % [
snapshot_count[0], min_expected, str(slots_ok), str(match_scene.score), str(present_time), remote_position_p99, remote_rotation_p99,
print("SMOKE INFO: client-bot snapshot_count=%d (want >= %d) slots_ok=%s final_score=%s remote_present_time=%s residual_p95=%.3fm/%.3fdeg residual_p99=%.3fm/%.3fdeg (p95 bar %.2fm/%.1fdeg, p99 bar %.2fm/%.1fdeg)" % [
snapshot_count[0], min_expected, str(slots_ok), str(match_scene.score), str(present_time),
remote_position_p95, remote_rotation_p95, remote_position_p99, remote_rotation_p99,
0.3, 5.0, NetworkedMatch.REMOTE_VISUAL_MAX_OFFSET, NetworkedMatch.REMOTE_VISUAL_MAX_ROTATION_DEGREES,
])
var success: bool = slots_ok and snapshot_count_ok and remote_quality_ok
print("SMOKE %s: CI client-bot run" % ("PASS" if success else "FAIL"))
+1
View File
@@ -52,6 +52,7 @@ Each item is also a GitHub issue (linked inline), labelled `needs:human` plus a
- [ ] ([#21](https://github.com/jcreek/CosmicClash/issues/21)) **Reference-hardware profiling (task 0.15b)** in the live editor on real low/mid-tier hardware — blocks 0.16, 0.17/0.17b/0.17c/0.17d, 0.26 (arena GI bake), and 0.28 (physics separate-thread prototype). Covered above; listed again here because it also gates Phase 5.5's graphics QA gate for multiplayer sign-off.
- [ ] ([#17](https://github.com/jcreek/CosmicClash/issues/17)) **Stand up the live Kubernetes cluster and Agones deployment** for Phase 8 — provider-portable manifests exist, but nothing has run against a real cluster; needs the provider-specific deployment overlay (network, DNS, secrets) per `docs/MATCHMAKING.md`.
- [ ] ([#31](https://github.com/jcreek/CosmicClash/issues/31)) **Build, push and pin the container images the Kubernetes manifests reference.** Every image target builds, but no workflow publishes any of them and all manifest digests are still all-zero placeholders, so `deploy/k8s/base` cannot pull running images. Needs a registry namespace, publish credentials, and a signing/provenance decision; once digests are real, turn on `--require-concrete` in `make verify-supply-chain` so a placeholder can no longer pass. Blocks [#17](https://github.com/jcreek/CosmicClash/issues/17).
- [ ] ([#33](https://github.com/jcreek/CosmicClash/issues/33)) **Move game servers to their own namespace** so `cosmic-clash` can enforce `restricted` again. Agones' Dynamic port policy needs a `hostPort`, which `baseline`/`restricted` forbid, so the whole namespace dropped to `privileged` — including the control plane, which mounts the database DSN, workload secret and Steam publisher key. Deferred until the Agones gate was green so a new failure could not be ambiguous.
- [x] (no issue — agent-actionable) **Phase 8.48 has its own Compose smoke fixture.** `compose.allocated-smoke.yml` and `scripts/verify_allocated_compose.sh` are independent of `compose.phase6-smoke.yml` — the script states so explicitly and reuses none of its ports — so the allocated-mode flow no longer inherits that fixture's hardcoded port, first-come slots or `--max-matches=2`. Exercised by `make verify-allocated-compose`.
- [ ] ([#22](https://github.com/jcreek/CosmicClash/issues/22)) **Release-evidence and human sign-off gates for Phase 8 production launch** — once the above are done, someone needs to actually run and sign off the production-shaped checks `multiplayer-next.md` §7 lists as infrastructure/production-dependent.