Why Does the Website Say Slow Down? Somebody Has to Protect It From You

A limiter counts requests arriving from a particular source and refuses those beyond an allowed rate, returning a response that says the caller has exceeded their allowance and when to try again. The purpose is protection rather than punishment. A service has finite capacity, and without a cap a single misbehaving client, whether malicious or simply a buggy loop, can consume all of it and make the service unavailable for everybody else. The limit converts a total outage affecting all users into a degraded experience affecting one.

Several approaches trade accuracy against memory and complexity:

• •A fixed window, counting requests in each clock minute
• •Which allows a double burst across a window boundary
• •A sliding window, which fixes that at the cost of more bookkeeping
• •A bucket that fills at a steady rate and is emptied by each request
• •Which permits a burst up to the bucket size, then a steady rate
• •A queue that releases requests at a constant rate, smoothing entirely

Deciding what to count and who to count it against is harder than the algorithm. Counting by network address punishes everybody behind a shared connection, including an entire office or a mobile network. Counting by account is fairer and requires the caller to be identified, which the expensive unauthenticated requests are not. Costs differ enormously between requests, so counting all requests equally lets a client consume everything with a few expensive ones. And in a service running on many machines, the count must be shared between them, which means either a central store on every request or accepting that the limit is approximate.

Capping request rates is one of a family of protections and they are frequently confused with each other. Shedding load drops requests when a service is already struggling, regardless of who sent them, and is a response to overload rather than a preventive cap. Queueing holds requests rather than refusing them, which helps with bursts and hurts if the queue grows faster than it drains. Cutting off a failing downstream service entirely, so callers fail immediately instead of waiting, protects against cascading failure. Backpressure pushes the slowdown up the chain so producers stop generating work. Serious systems use several together.

The difference between a well designed limit and a frustrating one is mostly about communication. A good implementation tells the caller their limit, how much remains and when it resets, on every response rather than only on rejection, so a well behaved client can pace itself and never be rejected at all. It distinguishes between slowing a caller down and cutting them off. It applies different limits to different operations according to cost. And it degrades gracefully under its own failure, allowing traffic through rather than blocking everything, on the grounds that a broken limiter should not itself become the outage.

Capping requests per source converts a total outage caused by one misbehaving client into a degraded experience for that client alone. Fixed windows allow a double burst at the boundary, buckets allow a burst then a steady rate, and queues smooth entirely. The harder problems are what to count against, since addresses are shared, and how to share a count across many machines.