mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-11 08:23:45 +00:00
feat(multiplayer): alert on workload server-mutation conflicts
Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.
observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.
Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.
Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
This commit is contained in:
@@ -52,6 +52,27 @@ spec:
|
||||
The 5-minute 5xx ratio for operation {{ $labels.operation }}
|
||||
has exceeded 1 percent for 5 minutes.
|
||||
runbook_url: https://example.invalid/cosmic-clash/runbooks/control-plane-api
|
||||
- alert: CosmicClashControlPlaneServerConflicts
|
||||
expr: |
|
||||
sum by (kind) (
|
||||
increase(cosmic_clash_api_server_conflicts_total[15m])
|
||||
) > 3
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
owner: api
|
||||
annotations:
|
||||
summary: Cosmic Clash workload-authenticated server mutations are conflicting
|
||||
description: >-
|
||||
More than 3 workload-authenticated {{ $labels.kind }} requests
|
||||
(register/connect/disconnect/shutdown/result) have been rejected
|
||||
as durable conflicts in the last 15 minutes; this is a distinct,
|
||||
tighter-scoped signal than the generic 4xx ratio above and can
|
||||
indicate a raced/duplicate GameServer registration, a replayed
|
||||
result, or a reconnect fencing bug rather than ordinary client
|
||||
noise. Correlate with server_{{ $labels.kind }} "conflict"-stage
|
||||
log events for the affected match/server IDs.
|
||||
runbook_url: https://example.invalid/cosmic-clash/runbooks/control-plane-api
|
||||
- name: cosmic-clash.allocator
|
||||
rules:
|
||||
- alert: CosmicClashAllocatorQuotaDenials
|
||||
|
||||
Reference in New Issue
Block a user