Alert fatigue is measured in minutes, not in annoyance
There is a moment in every monitoring setup where somebody mutes the channel. Not deletes it, not fixes it. Mutes it, intending to look later, and then a fortnight passes.
The usual framing is that too many alerts are tiring. That undersells it. The measurable consequence is the interval between a failure starting and a human deciding to act on it, and that interval is the entire product an agency sells when it offers monitoring. A client is not buying a dashboard. They are buying the promise that somebody looks quickly, and every false alert extends the delay before anyone does.
Where the noise comes from
Six sources, and they account for nearly all of it on the setups we have taken over.
| Source | What it looks like |
|---|---|
| Single failed probe | One dropped packet becomes a down alert and an up alert two minutes later |
| No dependency suppression | DNS fails, and every site on that nameserver alerts separately |
| Alerting per check rather than per incident | One server down produces one message per URL, per port, per certificate |
| Expected states treated as failures | A staging site that permanently answers 401, alerting forever |
| Maintenance not suppressed | Every deploy pages somebody |
| Thresholds set on the metric, not on the impact | Response time crossed a line for ninety seconds at four in the morning |
Notice that only the last row is about tuning a number. The other five are structural, which is good news: they are fixed once, in configuration, and they stay fixed.
The four changes that remove most of it
Confirm before alerting. Require the failure to be seen twice, ideally from two different vantage points, before anything is sent. This costs one check interval of detection time and it eliminates the single largest category of false positives, because transient network faults are almost never simultaneous in two places. If your tool offers a "confirm from a second location" setting, it is the highest-value box on the page.
Alert on the incident, not on the check. When one server hosting twelve sites goes down, that is one event. Twelve messages describing the same event is not twelve times the information, it is a smaller chance that anyone reads any of them. Grouping needs the tool to know which checks share a machine, which means telling it, which is ten minutes of work per client that nobody does and everybody should.
Suppress what is downstream. If the certificate check and the content check and the response-time check all fail because the host is unreachable, only the host failure is news. The rest are consequences, and reporting a consequence as a separate incident is how a single outage produces a wall of red that takes ten minutes to read.
Declare the expected state. Not every healthy URL answers 200. A staging site behind HTTP authentication answers 401 forever and is working. A retired URL should answer 410. During a maintenance window the correct answer is 503. Configure the expectation and the alert inverts usefully: the staging site that stops asking for a password becomes the incident, which it should be, because a staging copy answering 200 ends up in a search index. That inversion, and which codes deserve which treatment, is set out in the nine codes that matter in production.
Severity is about action, not about how bad it feels
The single question that sorts every alert: is there something a human should do about this right now?
If yes, it is a page, and it may arrive at three in the morning. If no, it belongs in a list somebody reads on Monday. There is no third category, and inventing one is how "warning" alerts come into existence and are then ignored by definition, which makes them worse than not existing because they still cost attention on the way past.
Applied honestly, this reclassifies a lot. A certificate expiring in twenty-one days is not a page; nothing useful happens at three in the morning about a date three weeks out. It is a task. A certificate expiring in eighteen hours is a page, because the window for acting is closing and the failure is total. The arithmetic behind picking that threshold, rather than alerting on the expiry date itself, is in renewal is three operations, and any of them can fail alone.
For a solo operator, the rule needs an extra clause: a page at three in the morning is only justified if you can actually fix it at three in the morning. If the answer to a client site being down is a support ticket with a host who answers at nine, then the alert should wake you only when your intervention changes the outcome, and the rest of the time it should be waiting for you at breakfast. Deciding that in advance, in writing, rather than at the moment the phone rings, is most of what an incident process is for.
Measure it, because the drift is invisible
The failure mode here is gradual. Nobody decides to stop trusting alerts. Acknowledgement time creeps from thirty seconds to two minutes to twenty, and each individual step feels reasonable.
Two numbers, tracked monthly, make it visible. How many alerts fired, and what proportion of them led to any action at all. If more than a small minority produced no action, the system is training you, and the training is working.
Also worth counting: incidents you learned about from a client rather than from the tool. That is the number the client is implicitly tracking, and it is the one that determines whether the monitoring line item survives the next budget conversation.
The fix that makes it worse
When the noise becomes unbearable, the instinct is to raise thresholds until it is quiet. This works immediately and it is the wrong direction, because a quiet monitor and a monitor that has been tuned into uselessness are indistinguishable from the outside. Both produce silence, and only one of them means anything.
The distinction between the two is worth building deliberately rather than assuming. A check that has been silently failing to run for six weeks, or one whose target changed so it now tests a URL that no longer exists, reports exactly the same nothing as a healthy site. Nothing in a dashboard of green tiles distinguishes "checked, fine" from "not checked". Which is the same problem as a job that stopped running: the absence of a signal is not the same as good news.
The honest version of "make it quieter" is to remove checks you would not act on, keep the ones you would, and be able to say, for each remaining alert, what the recipient is expected to do in the first sixty seconds. Any alert that cannot answer that question is noise with a good reputation.
