Caught something the service itself was mislabelling.

Caught a service reporting client errors as server failures

Digging into an anomaly on the SLO dashboards, I found one service returning HTTP 500 for problems that were really 4xx-class bad-request issues. That mattered on two fronts: 5xx means we failed, 4xx means the client sent something invalid — so we were burning our error budget for failures that weren't ours, and the mislabelling hid a real client-facing problem behind a generic “server error.” Every self-reported service metric showed a legitimate 5xx spike. I dug into each failing case using the distributed tracing I'd set up — a wrapped OpenTelemetry library with trace-to-log correlation across our Go services — cross-referencing traces, logs and code paths to confirm it was malformed input rather than a real server fault. I re-classified those routes as a pragmatic fix and it drove the work to correct the status codes at source — so the metric measured what it was supposed to.