mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
fix(multiplayer): stop a doomed formation from wedging the matcher
Closes the 'innocent-ticket restoration' gap noted in §8.20 and found by re-examining §8.16's matcher worker. domain.FormFromQueue's anchor is always the single oldest candidate, deterministically. If domain.PrepareProposal then rejected that exact formation for a reason specific to those particular players — mismatched protocol, incomplete ranked identity metadata, a duplicate-SteamID pair, ranked admission generally — RunOnce returned immediately and the next matcher interval reproduced the identical formation and failed again. Forever: nothing in the queue ever changes, so the same doomed anchor group would be retried every single pass, permanently head-of-line- blocking every other waiting player behind it too, not just the players actually at fault. This is worse than the already-fixed no-common-region crash-loop (§8.16) — that one killed the process; this one fails silently and just never matches anyone again. Two changes, both required together: 1. RunOnce now excludes a failed formation's players and retries with the remaining candidate pool, bounded to 8 attempts per pass. A batch with no viable formation at all (the pre-existing no-common-region case) still returns immediately, since retrying that can't help. 2. That fix was inert without a second one: RunOnce was asking Source for exactly w.Size candidates, so after excluding one failed formation's players there was nothing left to retry against. domain.SelectCandidates was always designed to search a larger pool (anchor plus an arbitrary remainder, widening through it) — the call site just never gave it one. RunOnce now requests up to 10x w.Size, capped at 200. Verified: go build/vet/test -race clean across every server package. Three new matcher tests cover the exclusion retry (an 8-candidate batch whose permanently-doomed oldest 4 still lets the remaining 4 form and claim, correctly excluding the doomed players from the claimed ticket set), that exhausting every attempt surfaces the last real error rather than a silent false/nil, and that Source is actually asked for more than w.Size candidates — a regression guard for exactly the companion bug above. All six pre-existing worker tests still pass unmodified, confirming the fix preserves every prior guarantee (mixed-playlist/duplicate-identity rejection, incomplete batch handling, durable claim failure propagation, Run's existing per-pass-error survival).
This commit is contained in:
Josh Creek
+2
-2
@@ -1204,11 +1204,11 @@ production fallback.
|
||||
|---|---|---|
|
||||
| 8.14 `[D:8.4,8.5,8.8]` | **IN PROGRESS.** Pure Go queue policy and PostgreSQL enforce one active ticket per verified player, 10 s heartbeat/30 s expiry, retry-safe owner/revision-scoped create/heartbeat/cancel, and deterministic candidate projection. Client cancellation is limited to `QUEUED`/`PROPOSED`; it cannot overwrite match-owned `ACCEPTED` through `LIVE` lifecycle states. A locked rejection classifier maps missing ticket, wrong owner, expiry, stale revision, and invalid state to distinct domain/API outcomes without weakening the atomic mutation predicate. Queue admission also honors both pre-live and live ranked abandonment penalties, so an expired reconnect cannot immediately requeue after result completion. Redis is an optional rebuildable projection over authoritative PostgreSQL | Domain/store/API tests cover ownership, expiry, idempotency, candidate binding, exact mutation-state fences, live-ticket cancellation rejection, stale revision classification, abandonment cooldown selection, concurrent create/heartbeat races, durable-source cache repair, Redis TTL/lost-keyspace behavior, and playlist/build/protocol compatibility. PostgreSQL-tagged lifecycle regressions compile and prior live runs cover the queue races; live database reruns remain blocked by Docker storage. Live Redis failover-under-load and worker integration remain |
|
||||
| 8.15 `[D:7.8,8.3]` | **IN PROGRESS.** Pure Go probe validation treats Steam location as opaque, requires nonce/freshness/region and server-computed RTT, and implements discrepancy quarantine/release; authenticated HTTP now accepts only opaque location/nonce input through a server-owned probe provider and records validated RTT into the active player's durable queue ticket; durable queue projections have a server-derived RTT JSON field for matcher reads | `server/domain/probes.go`, `server/migrations/0003_queue_probe_metadata.sql`, adversarial fixtures and `server/api/service.go`/`store/queue_sql.go` cover stale/wrong/forged evidence, the 25 ms/30% threshold, three-sample quarantine, five-clean release, authenticated provider arguments, rejection of client RTT fields, player/ticket/expiry binding, persistence failure, playlist-scoped candidate reads and bounded metadata decoding; Steam coordinator, regional probe adapters and multi-region probe population remain |
|
||||
| 8.16 `[D:8.14,8.15]` | **IN PROGRESS.** Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction; the runnable matcher now supports explicitly enabled ranked six-player polling with durable verified-Steam identity lookup. **Fixed a real crash-loop**: `Worker.Run` treated every `RunOnce` error as fatal to the whole loop, including "no compatible candidates" (`FormFromQueue`'s completely routine answer when currently-queued players share no verified region) — found building a live two-player integration attempt (see below): two real players with no common region crashed the entire matcher process, taking matchmaking down for every other player in the playlist, and would crash-loop again on restart since the same incompatible candidates stay queued. Now only genuine static misconfiguration (`ErrWorkerNotConfigured`/`ErrUnsupportedPlaylist`/`ErrInvalidMatcherSize`) stops the loop; everything else retries next interval | `server/domain/matcher.go`, `teams.go`, `server/matcher/worker.go`, `server/store/queue_sql.go` and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure, queue-backed oldest-anchor formation and incomplete ranked identity metadata; two new tests cover `Run` (not just `RunOnce`) surviving a per-pass error via a real concurrent goroutine, and still stopping immediately on a real configuration error, both clean across repeated `-race` runs. **A live two-player Godot proposal integration attempt is on disk but not committed**: `Game/tests/control_plane_proposal_smoke.gd`/`.tscn` and `scripts/verify_control_plane_proposal_integration.sh` exist and found the bug above, but the session paused running further concurrent headless Godot processes after discovering they'd been causing native engine crashes (macOS crash reporter, `EXC_BAD_ACCESS`/`SIGBUS`) intermittently all session, confirmed by the user; the two-player script was never itself verified to a clean pass. Arena selection and long-running worker integration remain |
|
||||
| 8.16 `[D:8.14,8.15]` | **IN PROGRESS.** Pure Go candidate/team selection implements the <=100 ms ceiling, pairwise widening tolerance, anchor inclusion, deterministic set/region scoring and balanced team partitioning; queue-backed formation now consumes the server-owned projection, fences duplicate player identities and rejects playlist/build/protocol mixing; the matcher worker performs bounded formation and delegates the final claim to the durable proposal transaction; the runnable matcher now supports explicitly enabled ranked six-player polling with durable verified-Steam identity lookup. **Fixed a real crash-loop**: `Worker.Run` treated every `RunOnce` error as fatal to the whole loop, including "no compatible candidates" (`FormFromQueue`'s completely routine answer when currently-queued players share no verified region) — found building a live two-player integration attempt (see below): two real players with no common region crashed the entire matcher process, taking matchmaking down for every other player in the playlist, and would crash-loop again on restart since the same incompatible candidates stay queued. Now only genuine static misconfiguration (`ErrWorkerNotConfigured`/`ErrUnsupportedPlaylist`/`ErrInvalidMatcherSize`) stops the loop; everything else retries next interval. **Fixed a second, quieter wedge in the same area**: `FormFromQueue`'s anchor is always the single oldest candidate, deterministically, so when `domain.PrepareProposal` rejected that exact formation for a reason specific to those particular players (mismatched protocol, incomplete ranked identity metadata, a duplicate-SteamID pair) rather than "no compatible batch exists", `RunOnce` returned immediately and the next interval reproduced the identical formation and failed again — forever, permanently head-of-line-blocking every other waiting player behind that anchor too, not just the players actually at fault (this is the "innocent-ticket restoration" gap task 8.20 named: the innocents were never stuck in the database, since no claim had happened yet, but they were durably starved of ever being tried). `RunOnce` now excludes a failed formation's players and retries with the remaining pool, bounded to 8 attempts per pass; a batch that has no viable formation at all (the pre-existing no-common-region case) still returns immediately rather than looping pointlessly. **This fix was inert without a companion one**: `RunOnce` was asking `Source` for exactly `w.Size` candidates -- `SelectCandidates` was always designed to search a larger pool (it takes an anchor plus an arbitrary remainder and widens through it), but the call site never gave it one, so there was never a "remainder" for the exclusion retry to fall back to in production. `RunOnce` now requests up to 10x `w.Size` (capped at 200) instead | `server/domain/matcher.go`, `teams.go`, `server/matcher/worker.go`, `server/store/queue_sql.go` and adversarial fixtures cover no-common-region, tolerance boundaries, lexical ties, mean-rating balance, malformed candidates, duplicate identities, compatibility mismatches, incomplete batches, source failure, durable claim failure, queue-backed oldest-anchor formation and incomplete ranked identity metadata; two new tests cover `Run` (not just `RunOnce`) surviving a per-pass error via a real concurrent goroutine, and still stopping immediately on a real configuration error, both clean across repeated `-race` runs; three further tests cover the formation-exclusion retry (an 8-candidate batch whose oldest 4 are permanently doomed still forms and claims the remaining 4, excluding the doomed players from the claimed ticket set), that exhausting every attempt still surfaces the last real error rather than a silent `false,nil`, and that `Source` is actually asked for more than `w.Size` candidates. **A live two-player Godot proposal integration attempt is on disk but not committed**: `Game/tests/control_plane_proposal_smoke.gd`/`.tscn` and `scripts/verify_control_plane_proposal_integration.sh` exist and found the original crash-loop bug above, but the session paused running further concurrent headless Godot processes after discovering they'd been causing native engine crashes (macOS crash reporter, `EXC_BAD_ACCESS`/`SIGBUS`) intermittently all session, confirmed by the user; the two-player script was never itself verified to a clean pass. Arena selection and long-running worker integration remain |
|
||||
| 8.17 `[D:8.14,8.16]` | **IN PROGRESS.** Pure Go proposal policy sends a 10-second response window to every selected human, requires unanimous acceptance, applies exact casual/ranked decline and timeout cooldowns with ranked escalation, and exposes revisioned idempotent responses through the authenticated API. Proposal closure now atomically separates offenders from innocents: a decliner's ticket is `CANCELLED`; a timed-out player's ticket is `EXPIRED`; accepted or otherwise innocent participants return to `QUEUED` with their original `enqueued_at` and refreshed expiry. Direct queue cancellation closes the open proposal and requeues remaining participants immediately. Late API responses commit expiry, timeout penalties, and ticket release before returning `ErrProposalClosed`; recovery of an old declined proposal cannot misclassify its pending innocents as timeouts. Cooldown history rejects future, foreign-playlist, and invalid-kind events, and database rows are closed before penalty writes | Domain/store/API fixtures cover partial/unanimous response, expiry, replay/conflict, stale revision, exact cooldown windows/escalation, corrupt history filtering, offender ticket termination, innocent precedence preservation, direct-cancel cascade, and the former late-response rollback. PostgreSQL-tagged regressions compile and assert the durable split and penalty rows; the full local Go suite passes. Live PostgreSQL execution and allocation integration remain |
|
||||
| 8.18 `[D:8.5,8.14,8.17]` | **IN PROGRESS.** Go store layer defines PostgreSQL SERIALIZABLE whole-transaction retries and queue candidate/proposal claim SQL using `FOR UPDATE SKIP LOCKED` plus durable uniqueness/revision fences; proposal creation persists matcher-selected region/protocol/team/slot topology before acceptance, and the same serializable final-acceptance transaction now promotes the exact roster into one `ALLOCATING` match, closing the process-crash gap that could otherwise strand an accepted proposal before the former second promotion transaction. The API promoter remains a replay check. Promotion replay validates immutable playlist/region/protocol/arena, participant, ticket, team, and slot identity but deliberately ignores mutable match state/server ownership, so a retry after a lost response still succeeds after allocation has advanced. Result sets are closed before crossing into promotion writes, avoiding one-connection pool stalls. Redis remains a rebuildable candidate projection over PostgreSQL authority | Store/API tests cover retries, claims, owner/revision fencing, expiry, exact promotion replay/conflict, progressed-match replay, rollback of partial claims, concurrent contested-ticket formation, and lost-cache repair. PostgreSQL-tagged regressions compile and assert acceptance, ticket transitions, match creation, and roster insertion are one durable outcome; prior live runs covered queue/proposal promotion and races, while this atomic-promotion change awaits a live database rerun. Allocation runtime integration remains |
|
||||
| 8.19 `[D:8.18]` | **IN PROGRESS.** Pure Go casual lineup requires 2–6 humans with at least one per team, fills missing slots with explicit bots, permits kickoff-only bot-slot backfill and assigns no backfill penalty/rating update; proposal preparation now derives the lineup from formed teams | `server/domain/casual.go`, `formation.go` cover both-team minimum, bot shape, live-play rejection, zero-penalty backfill and casual proposal composition; queue candidate selection, opt-in 10 s backfill proposals, reconnect/leave penalties and live integration remain |
|
||||
| 8.20 `[D:8.18]` | **IN PROGRESS.** Pure Go ranked admission requires six unique verified solo humans, rejects bots/backfill/parties, and allows only random-enabled non-elevated arenas; proposal preparation requires matching metadata for every formed player | `server/domain/ranked.go`, `formation.go` cover count, identity, party, bot/backfill, arena eligibility and formed-player metadata rejection; `ArenaRegistry` integration, allocation wiring and innocent-ticket restoration remain |
|
||||
| 8.20 `[D:8.18]` | **IN PROGRESS.** Pure Go ranked admission requires six unique verified solo humans, rejects bots/backfill/parties, and allows only random-enabled non-elevated arenas; proposal preparation requires matching metadata for every formed player | `server/domain/ranked.go`, `formation.go` cover count, identity, party, bot/backfill, arena eligibility and formed-player metadata rejection; **innocent-ticket restoration is fixed, see §8.16**: a formation rejected by ranked admission (or any other formation-specific `PrepareProposal` failure) no longer permanently wedges the matcher on the same doomed anchor group, starving every other waiting player behind it. `ArenaRegistry` integration and allocation wiring remain |
|
||||
| 8.21 `[D:8.5,8.20]` | **IN PROGRESS.** Pure Go rating core implements canonical Glicko-2, daily inactivity, ranked 1/3 and casual 1/N human-opponent weights, deterministic opponent ordering, and authoritative draw/overtime/abandon scoring; certified result completion now applies per-player updates inside the durable transaction with lexical row locks and revision increments | `server/domain/rating.go`, `server/store/result_sql.go` and tests cover canonical/inactivity/weight/invalid-input, draw/OT/abandon, ordered participant snapshots, lock/value re-read and rating update SQL; live PostgreSQL rating and seasons execution now covered (§8.23), and concurrent result transaction cases are covered: `TestPostgreSQLConcurrentIdenticalResultSubmissionAppliesRatingsExactlyOnce` races 5 identical submissions and confirms one rating application, while `TestPostgreSQLConcurrentConflictingResultSubmissionsKeepOneReceipt` races different payloads and confirms exactly one winner, one conflict, one receipt and one completion event; live maintenance/DB execution remains |
|
||||
| 8.22 `[D:8.21]` | **IN PROGRESS.** Pure Go ranked profile exposes the first ten games as provisional, derives tiers only through validated backend-owned rating bands, and keeps casual ratings outside the API; authenticated HTTP now returns the authoritative ranked view. **"Authoritative ranked view" was durable-adapter-shaped but had no durable adapter**: `rankedProfile`/`profile` only ever read an in-memory map, so every real `GET /v1/profile/ranked` 404'd regardless of a player's actual rating. `RankedProfileProvider` (interface) + `store.PostgresRankedProfiles` close it, preferred over the map when set so existing tests/literals are unaffected; `LastSeasonID`/`SeasonHistory` deliberately left unset (no season pointer on `ratings`, needs its own query/semantics) | `server/domain/rating.go`, `tier_test.go` and `server/api/service.go` cover provisional override, exact band boundaries, malformed policy rejection, session authentication and ranked-only response fields; `server/store/ranked_profile_sql.go`, verified against real PostgreSQL via curl (a fresh identity correctly 404s through the real adapter) and via §8.40's integration test (`control_plane_smoke.gd` now asserts this exact 404 round-trips before queueing); persisted tier policy, client UI and reconnect transport remain |
|
||||
| 8.23 `[D:8.21]` | **IN PROGRESS.** Pure Go ranked-only season policy compresses 25% toward 1500, clamps RD to 200–350, preserves volatility/history, is idempotent by season ID, and defines exact 12-week windows/due detection; migration and Go store now persist a per-player/per-season marker and rating update atomically; `cmd/maintenance` runs bounded due-season batches with signal-bound shutdown | `server/domain/rating.go`, `season_test.go`, `server/migrations/0001_initial.sql` and `server/store/maintenance_sql.go` cover compression, floor/cap, duplicate replay, window boundary, completed-season idempotence, bounded enumeration, row locking and conflict-safe rollover markers; opt-in PostgreSQL execution now covers the durable rating update, marker creation and duplicate replay without a second compression — re-run live for the first time as part of the wider integration-suite verification below, after fixing a test setup gap (a missing `seasons` row tripped the `ranked_season_rollovers` foreign key before the rollover logic itself ran); live maintenance/DB execution remains |
|
||||
|
||||
+80
-10
@@ -90,30 +90,100 @@ func (w Worker) RunOnce(ctx context.Context) (bool, error) {
|
||||
return false, ErrInvalidMatcherSize
|
||||
}
|
||||
now := w.Now()
|
||||
candidates, err := w.Source(ctx, now, w.Playlist, w.Size)
|
||||
// Request headroom beyond exactly w.Size: the exclusion-retry loop below
|
||||
// needs other candidates to fall back to when the oldest-anchor formation
|
||||
// fails, and a pool capped at exactly w.Size leaves nothing to retry with
|
||||
// -- silently reintroducing the same head-of-line wedge the loop exists
|
||||
// to fix. Bounded (not unlimited) so a large playlist backlog doesn't turn
|
||||
// every pass into an expensive scan.
|
||||
candidates, err := w.Source(ctx, now, w.Playlist, w.candidatePoolSize())
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if len(candidates) < w.Size {
|
||||
return false, nil
|
||||
}
|
||||
queue := domain.NewQueue()
|
||||
// A first full pass over every candidate validates playlist and identity
|
||||
// shape once, exactly as before -- these are hard input-format errors, not
|
||||
// "this particular formation didn't work out", so they still fail the pass
|
||||
// immediately rather than being retried below.
|
||||
for _, candidate := range candidates {
|
||||
if candidate.Playlist != w.Playlist {
|
||||
return false, fmt.Errorf("candidate playlist does not match worker")
|
||||
}
|
||||
if _, err := queue.Create(candidate.PlayerID, candidate.TicketID, "matcher-"+candidate.TicketID, candidate, now); err != nil {
|
||||
}
|
||||
|
||||
// FormFromQueue's anchor is always the oldest candidate, deterministically.
|
||||
// If domain.PrepareProposal then rejects that exact formation (mismatched
|
||||
// protocol, incomplete ranked identity metadata, a duplicate-SteamID pair,
|
||||
// etc.), retrying next interval reproduces the identical formation and
|
||||
// fails again -- forever, permanently head-of-line-blocking every other
|
||||
// waiting player behind that anchor, not just the players actually at
|
||||
// fault. Excluding the failed formation's players and retrying with the
|
||||
// remainder, bounded within this one pass, means one bad combination can
|
||||
// no longer wedge the whole playlist; Worker.Run's existing non-fatal
|
||||
// per-pass-error handling still applies if every attempt is exhausted.
|
||||
remaining := candidates
|
||||
var lastErr error
|
||||
for attempt := 0; attempt < maxFormationAttemptsPerPass && len(remaining) >= w.Size; attempt++ {
|
||||
queue := domain.NewQueue()
|
||||
for _, candidate := range remaining {
|
||||
if _, err := queue.Create(candidate.PlayerID, candidate.TicketID, "matcher-"+candidate.TicketID, candidate, now); err != nil {
|
||||
return false, err
|
||||
}
|
||||
}
|
||||
formation, err := domain.FormFromQueue(queue, w.Size, now)
|
||||
if err != nil {
|
||||
// No compatible batch exists at all within what's left of the pool
|
||||
// (e.g. no shared region) -- not specific to one formation, so
|
||||
// retrying within this pass cannot help either.
|
||||
return false, err
|
||||
}
|
||||
prepared, err := w.Prepare(w.NextID(), w.Playlist, formation, now)
|
||||
if err != nil {
|
||||
lastErr = err
|
||||
excluded := make(map[string]bool, len(formation.Selection.Players))
|
||||
for _, player := range formation.Selection.Players {
|
||||
excluded[player.PlayerID] = true
|
||||
}
|
||||
next := make([]domain.Candidate, 0, len(remaining))
|
||||
for _, candidate := range remaining {
|
||||
if !excluded[candidate.PlayerID] {
|
||||
next = append(next, candidate)
|
||||
}
|
||||
}
|
||||
remaining = next
|
||||
continue
|
||||
}
|
||||
return w.claim(ctx, formation, prepared, now)
|
||||
}
|
||||
formation, err := domain.FormFromQueue(queue, w.Size, now)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
prepared, err := w.Prepare(w.NextID(), w.Playlist, formation, now)
|
||||
if err != nil {
|
||||
return false, err
|
||||
return false, lastErr
|
||||
}
|
||||
|
||||
// maxFormationAttemptsPerPass bounds how many distinct formations RunOnce
|
||||
// will try excluding prior failures before deferring to the next interval.
|
||||
// Each attempt is pure in-memory work (no durable claim happens until
|
||||
// Prepare succeeds), so this is cheap; it exists to keep one pass bounded
|
||||
// rather than to conserve resources.
|
||||
const maxFormationAttemptsPerPass = 8
|
||||
|
||||
const (
|
||||
candidatePoolMultiplier = 10
|
||||
maxCandidatePoolSize = 200
|
||||
)
|
||||
|
||||
// candidatePoolSize is how many candidates RunOnce asks Source for. It is
|
||||
// deliberately larger than w.Size (see RunOnce) and bounded independently of
|
||||
// the playlist's actual backlog size.
|
||||
func (w Worker) candidatePoolSize() int {
|
||||
poolSize := w.Size * candidatePoolMultiplier
|
||||
if poolSize > maxCandidatePoolSize {
|
||||
return maxCandidatePoolSize
|
||||
}
|
||||
return poolSize
|
||||
}
|
||||
|
||||
func (w Worker) claim(ctx context.Context, formation domain.MatchFormation, prepared domain.PreparedProposal, now time.Time) (bool, error) {
|
||||
ticketIDs := make(map[string]string, len(prepared.Proposal.Participants))
|
||||
for _, participant := range prepared.Proposal.Participants {
|
||||
for _, candidate := range formation.Selection.Players {
|
||||
|
||||
@@ -3,6 +3,7 @@ package matcher
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -163,6 +164,109 @@ func TestRunSurvivesPerPassErrorsAndKeepsRetrying(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunOnceRequestsMoreCandidatesThanASingleFormationNeeds guards the
|
||||
// companion half of the wedge fix below: excluding a failed formation and
|
||||
// retrying is a no-op if Source was only ever asked for exactly w.Size
|
||||
// candidates in the first place, since nothing is left afterward. RunOnce
|
||||
// must ask Source for headroom beyond one formation's worth.
|
||||
func TestRunOnceRequestsMoreCandidatesThanASingleFormationNeeds(t *testing.T) {
|
||||
var requestedLimit int
|
||||
worker := workerFor(func(_ context.Context, _ time.Time, _ domain.Playlist, limit int) ([]domain.Candidate, error) {
|
||||
requestedLimit = limit
|
||||
return candidates(), nil
|
||||
}, &creatorSpy{})
|
||||
if _, err := worker.RunOnce(context.Background()); err != nil {
|
||||
t.Fatalf("RunOnce: %v", err)
|
||||
}
|
||||
if requestedLimit <= worker.Size {
|
||||
t.Fatalf("Source was asked for limit=%d, want more than worker.Size=%d so a failed formation has a remainder to retry against", requestedLimit, worker.Size)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunOnceExcludesAFailingFormationAndTriesTheRemainingCandidates covers
|
||||
// a wedge distinct from the no-common-region crash-loop above:
|
||||
// domain.FormFromQueue's anchor is always the oldest candidate, so if
|
||||
// domain.PrepareProposal rejects that exact formation (ranked admission,
|
||||
// mismatched protocol, incomplete identity metadata -- anything formation-
|
||||
// specific rather than "no batch exists at all"), retrying next interval
|
||||
// reproduces the identical formation and fails again forever, permanently
|
||||
// head-of-line-blocking every other waiting player behind that anchor too,
|
||||
// not just the players actually at fault. RunOnce must exclude the failed
|
||||
// formation's players and try the remaining pool within the same pass.
|
||||
func TestRunOnceExcludesAFailingFormationAndTriesTheRemainingCandidates(t *testing.T) {
|
||||
now := time.Unix(1000, 0).UTC()
|
||||
batch := make([]domain.Candidate, 8)
|
||||
for i := range batch {
|
||||
batch[i] = domain.Candidate{TicketID: fmt.Sprintf("ticket-%d", i), PlayerID: fmt.Sprintf("player-%d", i), Playlist: domain.Casual, ProtocolVersion: 1, EnqueuedAt: now.Add(time.Duration(i) * time.Second), PredictedRTT: map[string]float64{"EU": 20}}
|
||||
}
|
||||
creator := &creatorSpy{}
|
||||
prepareCalls := 0
|
||||
worker := Worker{
|
||||
Source: func(context.Context, time.Time, domain.Playlist, int) ([]domain.Candidate, error) { return batch, nil },
|
||||
Creator: creator,
|
||||
Playlist: domain.Casual,
|
||||
Size: 4,
|
||||
Now: func() time.Time { return now },
|
||||
NextID: func() string { return "proposal-1234567890123456" },
|
||||
Prepare: func(id string, playlist domain.Playlist, formation domain.MatchFormation, at time.Time) (domain.PreparedProposal, error) {
|
||||
prepareCalls++
|
||||
for _, player := range formation.Selection.Players {
|
||||
// The oldest four players (the deterministic anchor group) are
|
||||
// the "doomed" combination -- always reject them, every time.
|
||||
if player.PlayerID == "player-0" {
|
||||
return domain.PreparedProposal{}, errors.New("simulated formation-specific rejection")
|
||||
}
|
||||
}
|
||||
return domain.PrepareProposal(id, playlist, formation, nil, domain.RankedArena{}, at)
|
||||
},
|
||||
}
|
||||
formed, err := worker.RunOnce(context.Background())
|
||||
if err != nil || !formed {
|
||||
t.Fatalf("formed=%v err=%v, want the second (players 4-7) formation to succeed", formed, err)
|
||||
}
|
||||
if prepareCalls != 2 {
|
||||
t.Fatalf("Prepare calls=%d, want exactly 2 (the doomed anchor group, then the remainder)", prepareCalls)
|
||||
}
|
||||
if creator.calls != 1 {
|
||||
t.Fatalf("creator calls=%d, want exactly 1", creator.calls)
|
||||
}
|
||||
for _, doomed := range []string{"player-0", "player-1", "player-2", "player-3"} {
|
||||
if _, claimed := creator.ids[doomed]; claimed {
|
||||
t.Fatalf("doomed player %s must not have been claimed by the surviving proposal: %+v", doomed, creator.ids)
|
||||
}
|
||||
}
|
||||
if len(creator.ids) != 4 {
|
||||
t.Fatalf("claimed ticket count=%d, want 4", len(creator.ids))
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunOnceReturnsTheLastFormationErrorWhenEveryAttemptFails proves the
|
||||
// exclusion loop is bounded and still surfaces a real error to Run's
|
||||
// existing non-fatal per-pass handling, rather than silently reporting
|
||||
// formed=false,err=nil when nothing could ever have worked this pass.
|
||||
func TestRunOnceReturnsTheLastFormationErrorWhenEveryAttemptFails(t *testing.T) {
|
||||
now := time.Unix(1000, 0).UTC()
|
||||
batch := make([]domain.Candidate, 8)
|
||||
for i := range batch {
|
||||
batch[i] = domain.Candidate{TicketID: fmt.Sprintf("ticket-%d", i), PlayerID: fmt.Sprintf("player-%d", i), Playlist: domain.Casual, ProtocolVersion: 1, EnqueuedAt: now.Add(time.Duration(i) * time.Second), PredictedRTT: map[string]float64{"EU": 20}}
|
||||
}
|
||||
worker := Worker{
|
||||
Source: func(context.Context, time.Time, domain.Playlist, int) ([]domain.Candidate, error) { return batch, nil },
|
||||
Creator: &creatorSpy{},
|
||||
Playlist: domain.Casual,
|
||||
Size: 4,
|
||||
Now: func() time.Time { return now },
|
||||
NextID: func() string { return "proposal-1234567890123456" },
|
||||
Prepare: func(string, domain.Playlist, domain.MatchFormation, time.Time) (domain.PreparedProposal, error) {
|
||||
return domain.PreparedProposal{}, errors.New("every formation is doomed")
|
||||
},
|
||||
}
|
||||
formed, err := worker.RunOnce(context.Background())
|
||||
if formed || err == nil || err.Error() != "every formation is doomed" {
|
||||
t.Fatalf("formed=%v err=%v, want the last formation-specific error surfaced", formed, err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunStopsImmediatelyOnConfigurationErrors is the other half of the
|
||||
// fix: a genuinely static misconfiguration (true on every future pass, not
|
||||
// just this one) must still stop the worker rather than spin forever.
|
||||
|
||||
Reference in New Issue
Block a user