jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-11 08:23:45 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(server): fan outbox events out to every control-plane replica

Browse Source 
The Deployment runs two replicas, but WebSocket subscribers live only in
each process's in-memory hub. Every replica races to read the same
global unpublished outbox rows, and publishing succeeded even when the
winning replica held no matching local subscriber -- that replica then
set the single global published_at. A client connected to the other
replica never received the event, and delivery degraded further with
each replica added. REST recovery eventually converged, but short-lived
proposal transitions could be observed late or not at all.

Publish committed events through PostgreSQL LISTEN/NOTIFY so the replica
that owns the subscriber's connection delivers it, regardless of which
replica drained the row. The listener holds its own pgx connection --
LISTEN is session state, so a pooled database/sql connection cannot
carry it -- and reconnects with backoff, since losing it would silently
downgrade that replica's subscribers to REST-only recovery.

The fan-out is optional: without EventFanout configured, behaviour is
unchanged local-hub publication, which stays correct for a single
replica and for tests. Only outbox-sourced events are routed through it;
the in-request-path publishes remain local, as those are a latency
optimisation for the caller's own connection.

Fan-out needs a wire shape of its own because ControlPlaneEvent hides
PlayerID from clients, and the recipient is exactly what a peer replica
needs to route on.
This commit is contained in:
Josh Creek
2026-09-05 10:29:23 +01:00
parent 4248e51c60
commit 129b0c7ef0
6 changed files with 233 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
server/api/outbox.go
+3 -3
View File
@@ -155,7 +155,7 @@ func deliverProposalOutboxEvent(_ context.Context, event store.OutboxEvent, serv
return fmt.Errorf("invalid proposal outbox event")
}
for _, playerID := range envelope.PlayerIDs {
if err := service.PublishControlPlaneEvent(ControlPlaneEvent{
if err := service.publishOutboxEvent(ControlPlaneEvent{
Event: envelope.Event, Revision: envelope.Revision, ResourceID: envelope.ResourceID,
OccurredAt: envelope.OccurredAt, State: envelope.State, PlayerID: playerID,
}); err != nil {
@@ -181,7 +181,7 @@ func deliverResultOutboxEvent(ctx context.Context, db *sql.DB, event store.Outbo
return fmt.Errorf("result outbox event has no participants")
}
for _, playerID := range players {
if err := service.PublishControlPlaneEvent(ControlPlaneEvent{
if err := service.publishOutboxEvent(ControlPlaneEvent{
Event: "state_changed", Revision: event.Revision, ResourceID: event.AggregateID,
OccurredAt: event.CreatedAt, State: "COMPLETED", MatchID: event.AggregateID,
PlayerID: playerID,
@@ -215,7 +215,7 @@ func deliverStateOutboxEvent(_ context.Context, event store.OutboxEvent, service
if playerID == "" {
return fmt.Errorf("state outbox event has empty participant")
}
if err := service.PublishControlPlaneEvent(ControlPlaneEvent{Event: "state_changed", Revision: envelope.Revision, ResourceID: envelope.ResourceID, OccurredAt: envelope.OccurredAt, State: envelope.State, MatchID: envelope.MatchID, PlayerID: playerID}); err != nil {
if err := service.publishOutboxEvent(ControlPlaneEvent{Event: "state_changed", Revision: envelope.Revision, ResourceID: envelope.ResourceID, OccurredAt: envelope.OccurredAt, State: envelope.State, MatchID: envelope.MatchID, PlayerID: playerID}); err != nil {
return err
}
}