Day 9/60: Alerting Strategies -- Production Engineering 60 Day Production Engineering Challenge Alert fatigue is the number one reason on-call rotations burn people out. Today I am covering the strategies that cut noise while keeping signal. Symptom-Based Alerting with PromQL Page on what users feel, not what servers report internally. Here is a burn rate alert that fires when your error budget is burning at 14.4x the allowed rate: # Critical burn rate: will exhaust monthly budget in 1 hour ( sum(rate(http_requests_total{code=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) > (14.4 * 0.001) and ( sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) > (14.4 * 0.001) Enter fullscreen mode Exit fullscreen mode The dual window (1h AND 5m) means you only page when the problem has statistical significance AND is actively happening right now. Alertmanager Inhibition Rules When a node dies, you do not need fifty alerts for every pod that was on it.…