Alert fatigue: making error alerts you'll actually read
The failure mode of monitoring isn't missing data — it's a Slack channel nobody reads because it fires 200 times a day. Alert fatigue is a design problem, and it's fixable. This is the playbook we use: alert only on change, kill noise at the source, route to owners, and make silence trustworthy.
The core principle: alert on change, not existence
An error that fires steadily is a backlog item; an error that just appeared or just spiked is an event worth interrupting someone for. Every rule below is a variation of this principle:
- New issue appears → notify. This is the highest-signal alert in existence — something that never happened before just happened.
- Regression (resolved issue comes back) → notify. A fix that didn't fix is exactly what you want to hear about immediately.
- Spike — an existing issue jumps to N× its baseline → notify. Absolute counts lie; a 10/minute error is an outage for a small endpoint and noise for a big one.
- Steady state → dashboard, not notification. It's already known; interrupting someone adds nothing.
Kill noise at ingestion, not in your inbox
Muting an alert while keeping the events is paying to store noise. Push filtering as early as possible:
- 1Inbound filters drop known junk before it counts against quota: browser-extension errors, bot user agents, the four legacy errors from the IE-era code path you'll never fix.
- 2Ignore with intent: 'ignored' should mean 'we decided this is acceptable', recorded in the tracker — not a Slack mute that hides it from everyone else too.
- 3Rate limits per issue: a runaway loop generating 50k identical events should be capped at ingestion; you need the first hundred, not all fifty thousand.
- 4Environment discipline: nobody should be paged by
dev. Route non-production environments to a low-priority channel or nowhere.
Route by ownership, not to everyone
A channel where every alert concerns 10% of readers trains 90% of readers to ignore it. Ownership rules that map file paths or transaction names to teams mean the payments team sees payments errors and nobody else does. The alert that reaches you should usually be about code you can fix.
Design the message for the 2-second glance
An alert is read in two seconds on a phone. It needs: the error title, the project and environment, event/user counts, and a link to the issue. It does not need the full stack trace — that's what the link is for. (This is how jentry formats its Slack and email alerts by default.)
Audit your alerts quarterly
Alert configuration isn't set-and-forget; it decays as your codebase and team change. A quarterly 30-minute review of which rules fired and which were acted on keeps the signal-to-noise ratio honest.
Frequently asked questions
How many alert rules should a project have?
Fewer than you think: 'new issue', 'regression', and one or two spike/threshold rules cover most projects. Every additional rule needs a human who acts when it fires.
Should alerts go to email or Slack?
Both work if the volume is right. The medium matters less than the rate: any channel above a handful of alerts per day stops being read. Split by severity — page/Slack for actionable, email digest for awareness.
What's a reasonable spike threshold?
Start with 'N× the hourly baseline' style rules (anomaly detection) rather than absolute counts — they self-adjust as your traffic grows. Absolute thresholds need constant retuning.
How do I stop dev/staging noise?
Filter by environment in the alert rule (or at ingestion). Production alerts interrupt; staging errors belong on a dashboard you check when working on staging.
Try jentry free
Hosted error tracking & performance monitoring. Works with your Sentry SDKs — send your first event in minutes.