fix(store): widen the serializable retry budget, stop leaking test volumes

Two things that made the integration gate untrustworthy.

The retry budget was too small for expected contention.
TestPostgreSQLConcurrentIdenticalResultSubmission fires five identical
concurrent submissions and requires all five to succeed; it failed 4 runs
in 20. The error was retryable and retries did fire -- three attempts
simply was not enough. Contention here is normal rather than
exceptional: several game servers can submit results, and several
matchers can claim candidates, against the same rows at once. Raised to
five attempts, which is 0 failures in 40 runs.

Also jittered the backoff, but measured rather than assumed: my first
theory was a thundering herd, since the delay was exactly
RetryBackoff*(attempt+1) and every loser of a race woke at the same
instant. Isolating the two changes showed jitter alone moved 4/20 to
3/20, while the budget alone reached 0/20. The budget was the real
constraint. Jitter is kept because it costs nothing and its benefit
grows with the number of contending writers -- production is not capped
at five -- but the comment now says plainly that it is the smaller half,
so nobody inherits my wrong explanation.

Second, the integration scripts leaked one throwaway database volume per
run. --rm does reclaim anonymous volumes on a normal exit, but these
scripts force-remove the container from a trap, and `docker rm -f`
without -v keeps the volume. Sixty-four accumulated during this branch
until PostgreSQL stopped starting, surfacing only as the scripts' own
readiness timeout rather than as a disk error -- which is what the
"Docker storage exhausted locally" notes were really describing.
Measured at one volume per run before, zero after, across all five
scripts.
This commit is contained in:
Josh Creek
2026-09-05 17:14:43 +01:00
parent a1f30f6af9
commit 61a073099d
7 changed files with 70 additions and 9 deletions
+33 -3
View File
@@ -6,13 +6,22 @@ import (
"context"
"database/sql"
"fmt"
"math/rand/v2"
"strings"
"time"
)
const (
DefaultSerializableAttempts = 3
RetryBackoff = 10 * time.Millisecond
// DefaultSerializableAttempts is the retry budget for one logical
// mutation. Contention here is expected rather than exceptional: several
// game servers can submit results, and several matchers can claim
// candidates, against the same rows at once. Three attempts was too tight
// for even five-way contention on identical rows.
DefaultSerializableAttempts = 5
// RetryBackoff is the base delay. The actual wait is jittered -- see
// retryDelay -- because an unjittered backoff makes every contending
// transaction wake at the same instants and collide again.
RetryBackoff = 10 * time.Millisecond
)
// RunSerializable executes one logical mutation with PostgreSQL SERIALIZABLE
@@ -44,12 +53,33 @@ func RunSerializable(ctx context.Context, db *sql.DB, attempts int, fn func(cont
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(RetryBackoff * time.Duration(attempt+1)):
case <-time.After(retryDelay(attempt)):
}
}
return last
}
// retryDelay applies full jitter to a linearly growing ceiling, so contending
// transactions do not all wake at the same instant and collide again.
//
// Measured honestly: jitter is the smaller half of this fix. Against the
// five-way contention in TestPostgreSQLConcurrentIdenticalResultSubmission,
// jitter alone moved the failure rate from 4/20 to 3/20, while raising the
// attempt budget from 3 to 5 took it to 0/20 on its own. The budget was the
// real constraint. Jitter is kept because it costs nothing and its benefit
// grows with the number of contending writers, which in production is not
// capped at five -- but it should not be mistaken for the reason this got
// better.
func retryDelay(attempt int) time.Duration {
ceiling := RetryBackoff * time.Duration(attempt+1)
if ceiling <= 0 {
return 0
}
// math/rand/v2's top-level functions are safe for concurrent use, which
// matters because every contending goroutine calls this.
return time.Duration(rand.Int64N(int64(ceiling)))
}
func retryable(err error) bool {
if err == nil {
return false