mirror of
https://github.com/jcreek/CosmicClash.git
synced 2026-09-10 16:04:04 +00:00
Josh Creek
ce17a45afb
Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.
observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.
Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.
Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
94 lines
3.6 KiB
YAML
94 lines
3.6 KiB
YAML
apiVersion: monitoring.coreos.com/v1
|
|
kind: PrometheusRule
|
|
metadata:
|
|
name: cosmic-clash-control-plane
|
|
namespace: cosmic-clash
|
|
labels:
|
|
app.kubernetes.io/name: cosmic-clash
|
|
app.kubernetes.io/component: observability
|
|
spec:
|
|
groups:
|
|
- name: cosmic-clash.control-plane
|
|
rules:
|
|
- alert: CosmicClashControlPlaneAPIP95High
|
|
expr: |
|
|
histogram_quantile(
|
|
0.95,
|
|
sum by (le, operation) (
|
|
rate(cosmic_clash_api_latency_seconds_bucket[5m])
|
|
)
|
|
) > 0.25
|
|
for: 5m
|
|
labels:
|
|
severity: page
|
|
owner: api
|
|
annotations:
|
|
summary: Cosmic Clash control-plane API p95 latency is high
|
|
description: >-
|
|
The 5-minute p95 latency for operation {{ $labels.operation }}
|
|
has exceeded the 250 ms API SLO for 5 minutes.
|
|
runbook_url: https://example.invalid/cosmic-clash/runbooks/control-plane-api
|
|
- alert: CosmicClashControlPlaneAPI5xxHigh
|
|
expr: |
|
|
(
|
|
sum by (operation) (
|
|
rate(cosmic_clash_api_requests_total{status="5xx"}[5m])
|
|
)
|
|
/
|
|
clamp_min(
|
|
sum by (operation) (
|
|
rate(cosmic_clash_api_requests_total[5m])
|
|
),
|
|
0.001
|
|
)
|
|
) > 0.01
|
|
for: 5m
|
|
labels:
|
|
severity: page
|
|
owner: api
|
|
annotations:
|
|
summary: Cosmic Clash control-plane API 5xx rate is high
|
|
description: >-
|
|
The 5-minute 5xx ratio for operation {{ $labels.operation }}
|
|
has exceeded 1 percent for 5 minutes.
|
|
runbook_url: https://example.invalid/cosmic-clash/runbooks/control-plane-api
|
|
- alert: CosmicClashControlPlaneServerConflicts
|
|
expr: |
|
|
sum by (kind) (
|
|
increase(cosmic_clash_api_server_conflicts_total[15m])
|
|
) > 3
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
owner: api
|
|
annotations:
|
|
summary: Cosmic Clash workload-authenticated server mutations are conflicting
|
|
description: >-
|
|
More than 3 workload-authenticated {{ $labels.kind }} requests
|
|
(register/connect/disconnect/shutdown/result) have been rejected
|
|
as durable conflicts in the last 15 minutes; this is a distinct,
|
|
tighter-scoped signal than the generic 4xx ratio above and can
|
|
indicate a raced/duplicate GameServer registration, a replayed
|
|
result, or a reconnect fencing bug rather than ordinary client
|
|
noise. Correlate with server_{{ $labels.kind }} "conflict"-stage
|
|
log events for the affected match/server IDs.
|
|
runbook_url: https://example.invalid/cosmic-clash/runbooks/control-plane-api
|
|
- name: cosmic-clash.allocator
|
|
rules:
|
|
- alert: CosmicClashAllocatorQuotaDenials
|
|
expr: |
|
|
sum by (region) (
|
|
increase(cosmic_clash_allocator_quota_denials_total[15m])
|
|
) > 0
|
|
for: 5m
|
|
labels:
|
|
severity: warning
|
|
owner: allocator
|
|
annotations:
|
|
summary: Cosmic Clash allocator quota is denying allocation attempts
|
|
description: >-
|
|
The {{ $labels.region }} allocator has denied at least one
|
|
allocation attempt in the last 15 minutes; verify quota capacity,
|
|
provider health, and denial-of-wallet activity.
|
|
runbook_url: https://example.invalid/cosmic-clash/runbooks/allocator-quota
|