Every five minutes, and the run that never finished

*/5 * * * * /usr/local/bin/sync.sh

There is the answer to the question as it is usually asked. Every fifth minute of the hour, twelve times an hour, 288 times a day.

The question worth asking is the next one. At 10:05 the job starts. At 10:10 cron starts it again. It does not check whether the first one finished, it does not queue, it does not warn. Cron's entire contract is that it launches a command at a time; what happens after that is not its concern. If the run takes six minutes, you now have two copies of your job running against the same database, and at 10:15 you will have three. Whether that can happen to a given job is a comparison between two numbers, and one of them is not in the expression: the duration is the figure no parser has.

Three shapes of overlap, in ascending order of expense

Duplicated work. Two copies of an importer read the same queue and both process the same rows. Two copies of a notification job send the same email twice. This is the mild version, because it is visible: somebody notices the duplicate, and you go looking.

Contention. Two copies hold locks the other one wants. On MySQL this shows up as lock wait timeouts in the application log and a job that fails for a reason that has nothing to do with its actual work. On a file-based cache or a WordPress option row, it shows up as a value that flips back and forth depending on which copy wrote last, which is close to undiagnosable from the outside.

The pile-up. Each copy makes the machine slower, which makes the next copy slower, which means it is still running when the one after that starts. The curve is not linear. A job that normally takes forty seconds and one day takes seven minutes will have accumulated a handful of concurrent copies before anything else on the server notices, and on a shared plan the first external symptom is the account being throttled or the process count limit killing something unrelated. The site goes down and the cause is a backup script.

That third one is why the fix is worth applying to every recurring job, not only the ones you suspect. The cost of the guard is one word on the line.

The one-line guard

*/5 * * * * /usr/bin/flock -n /var/lock/sync.lock /usr/local/bin/sync.sh

flock takes a lock on the named file, runs the command, and releases the lock when the command exits. -n means do not wait: if the lock is already held, exit immediately without running anything.

The property that makes this the right tool, rather than a lock file you create and delete yourself, is what happens when the job dies badly. A hand-rolled lock is a file the script writes at the start and removes at the end. Kill the process, run out of memory, have the machine reboot mid-run, and the file stays. Every subsequent run sees it, concludes another copy is working, and skips. The job is now permanently disabled and still perfectly silent, which is the worst of both outcomes.

flock holds a kernel lock tied to an open file descriptor. When the process ends, for any reason including being killed outright, the descriptor closes and the lock is gone. There is no stale state to clean up, because the state was never in the file's contents.

Two flags worth adding once you have the basic form working:

/usr/bin/flock -n -E 0 /var/lock/sync.lock /usr/local/bin/sync.sh

-E sets the exit code used when the lock is busy. By default that is 1, indistinguishable from the command itself failing, so any wrapper that reports non-zero exits will report a normal skip as an error. Setting it to 0 makes a skipped run a success; setting it to a distinctive number such as 75 lets you count skips separately. Pick one deliberately, because the default will lie to whatever is reading exit codes.

One caveat that costs an afternoon when it applies: the lock file must live on a real local filesystem. On network storage the semantics vary by mount option and by server, and a lock that silently does not lock is worse than no lock at all, because you will have stopped worrying about overlap.

Skip or wait

The -n form skips. That is right for jobs where the work is idempotent and the next run picks up whatever the skipped one would have done: a queue drainer, a synchroniser, a cache warmer. Missing a cycle costs five minutes of freshness.

It is wrong for jobs where each run has its own distinct work. An hourly export that produces one file per hour must not skip, because the skipped hour never comes back. There, you want a wait with a bound:

/usr/bin/flock -w 240 /var/lock/export.lock /usr/local/bin/export.sh

Wait up to four minutes for the lock, then give up. The bound matters: without it, a hung run holds the lock forever and every subsequent invocation queues up behind it, and you have converted an overlap problem into a process-count problem.

Frameworks that run their own scheduler usually expose both behaviours as an option on the task definition rather than on the crontab line, which is the better place for it: the constraint belongs next to the code that knows whether the work is idempotent, not in a file on the server. That is one of the arguments for keeping a single crontab entry and putting every schedule behind an application-level dispatcher, along with the dialect problem the dispatcher makes go away.

The skip is silent too

Add the guard and you have solved the pile-up. You have also created a new state that produces no output: the run that did not happen.

A job skipping once is normal. A job skipping every single cycle for three weeks, because a run hung and is holding the lock, or because the work genuinely no longer fits in the interval, is a job that has stopped doing anything. The log shows nothing unusual, because nothing unusual is being written. The exit code is whatever you configured, which if you followed the advice above is success.

Record the skip. The cheapest version is a line of text and a timestamp:

*/5 * * * * /usr/bin/flock -n -E 0 /var/lock/sync.lock /usr/local/bin/sync.sh \
  || echo "$(date -Is) skipped" >> /var/log/sync-skips.log

Then the useful question becomes answerable: how many of the last 288 runs actually ran. If the answer is under half, the interval is wrong and no amount of locking will fix it.

And the state above that one, the job that is neither running nor skipping because the schedule itself is gone, still produces exactly nothing. Locking does not help there, and neither does any log: the absence of output is the failure mode cron cannot report.