A full disk arrives as a 500, or as a login that fails

No visitor ever sees a message about disk space. They see a 500, or a blank page, or a database error, or a login form that accepts their password and returns them to the login form. The disk is the cause and it is never the symptom.

Which is why this one is worth recognising by its shape rather than by its message.

The order things break in

A site under a full quota does not stop working all at once. It degrades in a specific sequence, because reading still works perfectly and only writing fails.

Sessions first. The session handler writes a file per visitor. When it cannot, the session is not persisted, so the next request has no session, so the login appears not to have worked. The site browses normally for anonymous visitors while nobody can log in, and that combination is close to a signature.

Then anything that uploads. Media, form attachments, imports. The upload appears to succeed and the file is not there.

Then the caches. A page cache that cannot write regenerates every page on every request, so the site gets dramatically slower at the same time, which sends everyone looking at performance.

Then the database. Writes start failing, and depending on the engine the result ranges from individual errors to the storage engine refusing all writes. On a shared plan the database may be under a different quota from the files, so this can arrive first or not at all.

Then, on the next restart, the service does not come back. This is the worst version, because it turns a degraded site into an outage at an arbitrary later moment, when something unrelated restarts a daemon that now cannot write its own files.

Reading it, including the number nobody checks

df -h .
df -i .

Two commands, and the second one is the one that solves the confusing cases. A quota can be on the number of files as well as on the number of bytes, and hitting the file limit produces exactly the same "disk quota exceeded" message while df -h shows plenty of space free.

When that happens the cause is almost always millions of very small files: session files that are never cleaned up, a cache directory with one file per URL per variant, or the mail spool holding one message per cron run. A gigabyte of free space and no free inodes is a full disk in every way that matters.

du -xh --max-depth=1 . 2>/dev/null | sort -h | tail -15
find . -xdev -type f -size +100M -printf '%s\t%p\n' 2>/dev/null | sort -rn | head

And for the inode case, count rather than measure:

for d in */; do printf '%8s %s\n' "$(find "$d" -xdev | wc -l)" "$d"; done | sort -rn | head

What actually fills it

In the order these turn up on client sites.

A log left on. A debug log enabled during an investigation eighteen months ago, growing at the rate of the site's traffic. This is the single most common cause and it is entirely self-inflicted.

Backups written locally and never removed. A plugin keeping every archive it has ever produced, in a directory inside the site. Doubly bad: it fills the disk, and it is not a backup, because it shares the failure of the thing it is backing up.

Mail nobody collects. Every scheduled job that produces output has its output mailed to the account, and on a machine with no working transport those messages accumulate in a spool directory forever. Thousands of tiny files, which is the inode case above.

Session files. Written continuously, cleaned up by a garbage collector whose probability setting means it may effectively never run on a low-traffic site.

Generated images. Every upload multiplied by every registered thumbnail size, and themes and plugins register a lot of sizes.

Database write-ahead or binary logs. On a managed database this is somebody else's problem. Where the client runs their own, an unbounded binary log is a classic.

The recovery that fails for lack of space

Worth planning before you need it, because the obvious moves all require space.

You cannot compress a file in place without room for the compressed copy. You cannot move a large file to another filesystem without reading and writing it. On a completely full disk you may not even be able to open an editor that writes a temporary file, and some panels stop working entirely.

So the first action is to free a small amount immediately, by truncating rather than deleting:

: > /path/to/huge.log

Truncation matters over deletion for a reason that catches people out: a process holding the file open keeps the space allocated even after the file name is removed, so rm on an active log frees nothing until the process restarts. Emptying it in place frees it at once.

Once there is a little room, deal with the cause rather than repeating the cleanup. A log that was truncated will be full again in a fortnight.

What to watch, and at what level

Both numbers, and earlier than feels necessary.

Alert on used space at eighty per cent rather than at ninety-five. The last portion disappears quickly on a site that is growing, and the recovery itself needs headroom, so an alert at ninety-five arrives when your options have already narrowed.

Alert on inode usage separately, at the same level. It is the reading that explains the incidents nobody can account for.

And add one assertion from outside, because everything above requires access to a machine you may not have on a shared plan. A check that logs in, or that fetches a page which only renders correctly when a write succeeds, is what turns this from an invisible degradation into an alert. A read-only check on the homepage will report a perfectly healthy site throughout, since reading never fails: a good status code says the transport worked, and nothing else.

When the failure does eventually reach the visitor, it arrives wearing somebody else's number, and the number in front of you names the messenger rather than the cause: what a gateway error is really reporting.