jcreek/CosmicClash
1
0
Fork 0
mirror of https://github.com/jcreek/CosmicClash.git synced 2026-09-11 23:03:42 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(multiplayer): alert on workload server-mutation conflicts

Browse Source 
Closes the 'live duplicate/conflict alerting also remains' gap noted
in §8.10: a durable domain.ErrConflict/ErrResultConflict rejection on
/v1/servers/{id}/{register,connect,disconnect,shutdown,result} was
already logged as a structured 'conflict' stage event, but had no
Prometheus signal distinct from the generic 4xx-class counter, which
also catches ordinary client noise (malformed bodies, expired
tokens). A real duplicate registration, raced reconnect, or replayed
result would have been invisible to alerting until someone went
looking through logs.

observability.Metrics gains ObserveServerConflict(kind), a bounded
counter keyed to serverMutation's own five routes (an unrecognized
kind folds into "other", so a caller mistake can't grow the label
set), exported as cosmic_clash_api_server_conflicts_total. Wired at
each of serverMutation's four conflict branches in server/api/service.go.
deploy/observability/prometheus-rules.yaml adds
CosmicClashControlPlaneServerConflicts, mirroring the existing
allocator quota-denial alert shape, firing on >3 conflicts of one
kind in 15 minutes.

Verified: go build/vet/test -race clean across every server package;
new unit tests cover per-kind counting, the bounded 'other' fallback,
the counter's absence until first observed, and a nil-receiver no-op;
a service-level test proves a real register conflict is exported
through the live /metrics endpoint. scripts/verify_observability_manifests.py
passes against the edited rules file.

Remaining, and explicitly out of scope here: this alert has only been
validated statically, never against a live Prometheus/Alertmanager
firing on real traffic — that requires the same live cluster this
sandbox has never had.
This commit is contained in:
Josh Creek
2026-09-04 17:24:09 +01:00
parent 817572a6ce
commit ce17a45afb
6 changed files with 162 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
server/api/service.go
+4
View File
@@ -652,6 +652,7 @@ func (s *Service) serverMutation(w http.ResponseWriter, r *http.Request) {
stage := "invalid"
if errors.Is(err, domain.ErrConflict) {
stage = "conflict"
s.Metrics.ObserveServerConflict("register")
writeError(w, http.StatusConflict, "conflict")
} else {
writeError(w, http.StatusUnprocessableEntity, "invalid_request")
@@ -702,6 +703,7 @@ func (s *Service) serverMutation(w http.ResponseWriter, r *http.Request) {
}
if err != nil {
if errors.Is(err, domain.ErrConflict) {
s.Metrics.ObserveServerConflict(parts[1])
writeError(w, http.StatusConflict, "conflict")
} else {
// The request has already passed schema and workload checks. An
@@ -747,6 +749,7 @@ func (s *Service) serverMutation(w http.ResponseWriter, r *http.Request) {
stage := "invalid"
if errors.Is(err, domain.ErrConflict) {
stage = "conflict"
s.Metrics.ObserveServerConflict("shutdown")
writeError(w, http.StatusConflict, "conflict")
} else {
writeError(w, http.StatusUnprocessableEntity, "invalid_request")
@@ -777,6 +780,7 @@ func (s *Service) serverMutation(w http.ResponseWriter, r *http.Request) {
stage := "invalid"
if errors.Is(err, domain.ErrResultConflict) || strings.Contains(err.Error(), "conflict") {
stage = "conflict"
s.Metrics.ObserveServerConflict("result")
writeError(w, http.StatusConflict, "conflict")
} else {
writeError(w, http.StatusUnprocessableEntity, "invalid_request")