Heartbeat monitoring: watching for what never arrives

Ordinary monitoring is a question asked from outside. Is the site answering. Is the certificate valid. Is the page the right size. Someone asks, something answers, and a bad answer is the signal.

There is a class of failure that produces no bad answer because it produces no answer at all, and no request was ever made. A nightly backup that stopped running does not return an error. A queue worker that died does not return an error. A cron job whose interpreter path changed after an upgrade does not return an error. Each of them is now not happening, and the observable difference between "ran fine" and "did not run" is zero.

Heartbeat monitoring inverts the question. Instead of asking whether something is broken, you require something to report in, and you alert when the report does not arrive.

How it works

The job, at the end of its run and only if it succeeded, makes one HTTP request to a URL that exists only for it. Something on the other end records the time. If more than an expected interval passes with no request, that something raises an alarm.

0 3 * * *  /usr/local/bin/backup.sh && curl -fsS -m 10 https://example.com/hb/a1b2c3

The && is the entire design. The ping happens only if the script exited zero. A script that fails does not ping, so a failure and a non-start produce the same silence, and both are caught by the same rule. That is the property you want: you no longer have to enumerate the ways it could break.

The flags matter more than they look. -f makes curl fail on an HTTP error rather than cheerfully printing the error page. -sS keeps it quiet but still reports real problems. -m 10 stops a hanging ping from holding the job open forever, which would be an outage caused by the monitoring, and those are the most annoying kind.

The dead man's switch

The idea is older than the web and it comes from trains. A driver holds a lever; if the lever is released, because the driver has collapsed, the brakes apply. Safety is the default. Action is what suppresses it.

Applied to software, this reverses an assumption most alerting is built on. Normal alerting assumes the monitoring system is fine and waits for the monitored thing to complain. A dead man's switch assumes nothing and requires proof of life. If the network between them breaks, if the job's host is powered off, if the whole datacentre is gone, the alarm still fires, because the alarm fires on absence and absence is exactly what all of those produce.

There is a limit worth stating plainly: if the thing watching for absence is itself down, nothing fires. You have not eliminated the problem, you have moved it to a system whose only job is to keep counting, which is easier to keep alive than the thing it watches. That is a real improvement and it is not magic.

Choosing the window, which is where people get it wrong

The window is how long you tolerate silence before alarming. Set it to the interval and you will be woken by every job that ran three minutes late. Set it to a day and a backup can be missing for a day.

The useful framing is not "how often does it run" but "how long can this be broken before it costs something".

That last row is the one to take seriously. Frequent jobs get noticed when they stop, because something downstream dries up within the day. A job that runs on the first of the month has eleven chances a year to fail, and each failure has thirty days to go unremarked.

Three mistakes that make it useless

Pinging at the start. A ping at the top of the script proves the script started. It says nothing about whether it finished, and a backup that starts every night and dies halfway through will report perfect health forever. Ping at the end, on success only.

Pinging unconditionally. Writing the ping on its own line after the command, rather than joined with &&, is the same mistake with extra steps. Every failure now reports success.

One heartbeat for several jobs. Tempting when you have forty client sites, and it collapses the signal: one job still running keeps the alarm quiet for all of them. One identifier per job, always. The bookkeeping is the point.

What it does not replace

A heartbeat tells you a job finished without an error. It does not tell you the job did anything useful. A backup script that succeeds while writing a zero-byte file will ping happily every night for a year.

So the heartbeat covers absence, and something else has to cover correctness: a size check, a row count, a restore test. The two are complementary and neither substitutes for the other. If you only have budget for one, take the heartbeat first, because "it stopped running" is far more common than "it ran and produced garbage", and it is the one nobody sees.

The same asymmetry runs through the schedule itself. A cron expression can be perfectly valid and fire at the wrong time, which is the subject of the three traps, and an invalid one fails loudly enough to be found. The dangerous states are always the quiet ones.

How our own engine does it, if you want to read the code

The asymmetry described above is not a figure of speech, it is a different code path. The polling loop selects monitors with kind <> 'heartbeat', so a heartbeat is never contacted at all: nothing goes out, and the only thing that can happen is that a ping fails to arrive. You can read the exclusion in src/Runner.php and the monitor it creates in src/Heartbeat.php.

Two numbers from that file are worth stealing whatever tool you use. The grace period defaults to 300 seconds, and both the expected interval and the grace are floored, at 60 and 30 seconds, so a job declared as running every 10 seconds is still watched on a one minute window. A grace shorter than the jitter of your own scheduler produces alerts about nothing, and alerts about nothing are how a team learns to ignore alerts.