429 Too Many Requests, and the day it was our own monitoring

Most writing about this code assumes you are the one being limited by an API you consume. That is the common case for a developer. It is not the common case for somebody who runs other people's websites, where the interesting version is the reverse: a limiter on a site you manage, counting requests that turn out to be yours.

What a 429 is counting

A rate limiter needs three things: a key, a window, and a ceiling. The key is almost always the client IP address. The window is a few seconds to a few minutes. The ceiling is whatever number the plugin shipped with.

The consequence that matters: the limiter does not know what you are. It knows an address made N requests in the last minute. A browser, a crawler, a monitoring probe and an attack all look the same from there, and the only thing that separates them is how regular they are.

Monitoring is the most regular traffic a small site will ever receive. Perfectly spaced, identical path, identical user agent, forever.

How it starts looking like an outage

The shape is specific and it is easy to misread.

Your checker reports intermittent failures on a client site. You open the site: fine. You ask the client: fine. You check from your phone: fine. The failures cluster, sometimes several in a row, then nothing for a day. Nothing in the server error log, because a 429 from a WAF or a security plugin is a normal response and not an error.

Then you notice the date. The failures began the week you added several sites hosted by the same provider, or moved your checks to a shorter interval, or added a second check per site for the SSL certificate.

Three things stack up here and each one is invisible on its own. Shared hosting puts many sites behind one frontend, and some limiters count per frontend rather than per site. Your probes leave from a small number of addresses. And "check every minute" plus "check the certificate too" plus "check three URLs per site" multiplies faster than anyone expects.

Finding out whose limit it is

The response usually names its author if you read the headers rather than the body.

curl -sSI https://example.com/ | grep -iE 'HTTP/|retry-after|x-ratelimit|ratelimit|server|cf-'

Standardised limiters send RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset. Older ones send the X-RateLimit- variants. Cloudflare sends its own ray identifier and, when a managed rule fired, a body that names it. WordPress security plugins usually send nothing useful and a themed page, which is its own tell: a 429 with a styled HTML page came from PHP, so the request reached the application, so the limiter is a plugin and not the host.

The other half of the question is which address is being counted. If your monitoring runs from a known set of addresses, ask the host to allow them, and if the answer is that they cannot, that is useful information about the host.

Not getting into this in the first place

Four habits, and none of them cost anything.

Stagger the schedule. Forty sites checked at the top of every minute is forty simultaneous requests from one address. Spread across the minute, it is a request every second and a half, which looks like nothing at all. Any scheduler can offset by a hash of the site identifier.

Do not check three URLs when one will do. A homepage check plus a certificate check plus a robots check is triple the count for very little extra signal, since the certificate is visible on the first connection anyway.

Send a user agent that identifies you and can be searched. When a host's admin sees the traffic, the difference between an unidentified probe and a string with a name in it is whether they block first or ask first.

Treat a 429 as a distinct state, not as down. A limited site is up. Recording it as an outage corrupts the availability figure you report to the client, and once that figure is wrong nobody trusts the next one either. That matters more than it sounds, because the availability figure is often the only part of a monitoring tool a client ever reads.

When the 429 is real

Sometimes the limiter is right and it is not you. Two cases come up often on client sites.

A search or filter page that generates a distinct URL per combination will attract crawlers that walk every combination. The limiter fires, and the correct fix is upstream: those URLs should not be crawlable, which is a robots and canonical question rather than a rate question.

And a login endpoint under credential stuffing will trip a limiter continuously. Here the 429 is the system working exactly as designed, and the thing to watch is not the code but its disappearance: an attack that stops producing 429s either stopped, or found a path that is not limited.

In both cases you want the count over time, not an alert per event. This is the same argument as for the intermittent failures in the nine codes worth knowing: the codes you decide not to alert on are exactly the ones whose weekly count tells you something, and a tool that only stores outages has already thrown them away.