Do you alert on probe failures or on user-facing SLOs?

papager_dev9 hours ago1 replies

Our on-call gets paged when liveness probes fail more than N times. Half of those pages have zero user impact. Considering moving everything to SLO burn-rate alerts. Anyone done this migration?

Discussion

srsre_lena8 hours ago

Yes. Probe failures became tickets, burn-rate became pages. Page volume dropped about 70%. The hard part was getting product to agree on SLO targets, not the tooling.