A scheduled job has four states and one of them shouts

Monitoring a website is easy to reason about because a broken site is loud. Somebody loads a page and gets an error. The error exists whether or not you are watching for it.

A scheduled job has no visitors. Its failure has no audience, and depending on how it failed it may not produce anything at all. That difference is the whole subject, and it is why the tooling for one does not transfer to the other.

Four states, one signal

Rows one and two are indistinguishable from rows four and three respectively if all you have is a log, because a job that never started writes nothing to its log, and an empty log is what a quiet week looks like.

The two lower rows are also the expensive ones. A backup that failed loudly gets fixed on Tuesday. A backup that has been producing a valid, well-formed, forty-kilobyte archive of an empty directory since March gets discovered on the day of the restore.

The exit code lies more often than people think

Before adding any monitoring, make the job tell the truth about itself, because everything downstream reads its exit code.

Pipelines report the last command. This is the big one.

mysqldump -u user db | gzip > backup.sql.gz

If mysqldump fails, gzip still succeeds at compressing the nothing it received, and the pipeline exits zero. You get a small, valid gzip file and a success. Fix it at the top of the script:

set -euo pipefail

pipefail makes the pipeline return the first non-zero status in it. -e stops on the first failure instead of ploughing on. -u turns an unset variable into an error rather than an empty string, which is what turns rm -rf "$DIR/"* into a very bad afternoon.

curl succeeds on an error page. curl exits zero when it successfully retrieves a 500, because from its point of view the transfer worked. Add --fail, or better --fail-with-body where available, so that the HTTP status becomes the exit status.

Interpreted scripts frequently exit zero on fatal errors. Depending on the language, the runtime and the error handler in place, a caught exception logged and swallowed leaves a successful process. If the script is yours, end it with an explicit exit code, and make the failure paths use a different one.

Ten minutes of this is worth more than any monitoring product, because monitoring a job that misreports its own outcome tells you the wrong thing more confidently.

Report the run, not the process

The mechanism that catches the fourth state is the job telling something outside the machine that it finished. There is no way around this: nothing on the server can distinguish "the crontab was deleted" from "the crontab is fine and there was nothing to do", because both produce silence, and the thing that would have noticed is the thing that was deleted.

The shape is three lines around the existing command:

#!/bin/bash
set -euo pipefail
URL="https://example-monitor/ping/<token>"

curl -fsS -m 10 "$URL/start" || true
/usr/local/bin/backup.sh
curl -fsS -m 10 "$URL/$?" || true

Three details in that snippet are load bearing. The timeout, so a monitoring endpoint that is itself unreachable cannot hang the backup. The || true, so that a failure to report never becomes a failure of the job, which would be an absurd way to lose a backup. And the start ping as well as the end ping, which is what turns "the job has not reported in twelve hours" into the two much more specific statements "it never started" and "it started and never came back".

Sending the exit code rather than just a completion means the report distinguishes state one from state two, and you no longer have to also parse a log.

The assertion that catches the third state

This is the part that gets left out, and it is the reason the third row of that table exists.

A job reporting a clean exit proves the process ended without complaining. It does not prove it did anything. The check that closes the gap is not on the process, it is on the artefact, and it is specific to each job:

A backup: the file exists, its modification time is within the expected window, and its size is within a sane band of the last one. A dump that is a tenth of yesterday's is a failure regardless of the exit code. So is one that is ten times larger.

An export or a feed: the record count. Not zero, and not wildly different from the previous run.

A synchronisation: the freshness of the newest row on the receiving side. If the newest record is from Thursday and today is Monday, the job has been reporting success for four days.

A report emailed to a client: whether the message actually left. Delivery is a separate system from the job, and it fails separately, which is the whole reason mail can be gone while the site is perfectly up.

Where possible, fold the assertion into the job so that a bad artefact produces a non-zero exit, and the monitoring you already added does the rest. Where that is not possible, because the artefact lands somewhere the job cannot see, the assertion becomes a second scheduled job, and now that one needs the same treatment.

Where the checker has to live

Not on the machine running the job. This sounds obvious written down and is violated constantly, usually by a second cron entry on the same box that checks whether the first one ran.

That arrangement fails in exactly the cases you built it for. The account is suspended: both jobs are gone. The container was rebuilt without the crontab: both jobs are gone. The disk is full: neither can write. Every shared failure mode takes the watcher out with the watched, and the resulting silence is interpreted as calm.

What you need is something with a different failure surface that expects a message and complains about its absence. Ours does this alongside the site checks, so a client's uptime and their nightly backup are one page rather than two products, and the engine is open source if you want to read what the check actually asserts before trusting it: what it watches, and what it deliberately does not.

The expression itself is the least of it, but it is worth reading once for the two behaviours that make a job run at a time nobody intended, because a job monitored on the wrong schedule reports healthy while doing the wrong thing at the wrong hour: the step operator and the day-of-week OR.