fix(server): fan outbox events out to every control-plane replica

The Deployment runs two replicas, but WebSocket subscribers live only in
each process's in-memory hub. Every replica races to read the same
global unpublished outbox rows, and publishing succeeded even when the
winning replica held no matching local subscriber -- that replica then
set the single global published_at. A client connected to the other
replica never received the event, and delivery degraded further with
each replica added. REST recovery eventually converged, but short-lived
proposal transitions could be observed late or not at all.

Publish committed events through PostgreSQL LISTEN/NOTIFY so the replica
that owns the subscriber's connection delivers it, regardless of which
replica drained the row. The listener holds its own pgx connection --
LISTEN is session state, so a pooled database/sql connection cannot
carry it -- and reconnects with backoff, since losing it would silently
downgrade that replica's subscribers to REST-only recovery.

The fan-out is optional: without EventFanout configured, behaviour is
unchanged local-hub publication, which stays correct for a single
replica and for tests. Only outbox-sourced events are routed through it;
the in-request-path publishes remain local, as those are a latency
optimisation for the caller's own connection.

Fan-out needs a wire shape of its own because ControlPlaneEvent hides
PlayerID from clients, and the recipient is exactly what a peer replica
needs to route on.
This commit is contained in:
Josh Creek
2026-09-05 10:29:23 +01:00
parent 4248e51c60
commit 129b0c7ef0
6 changed files with 233 additions and 5 deletions
+5
View File
@@ -118,6 +118,11 @@ type Service struct {
CandidateV2 CandidateProviderV2
QueueBackend QueueBackend
CandidateIndex CandidateIndex
// EventFanout, when set, publishes outbox-sourced events through a shared
// transport instead of only this replica's in-memory hub. Without it a
// client connected to a replica other than the one that drained the outbox
// row never receives the event.
EventFanout func(ControlPlaneEvent) error
Probe ProbeProvider
ProbeRecorder ProbeRecorder
WorkloadVerify WorkloadVerifier