Rate Limiting

Without rate limiting, a single misbehaving client can consume all your server resources. Rate limiting enforces a ceiling on how many requests a client can make in a given time window — protecting the system from abuse, overload, and ensuring fair usage.

Why Rate Limit

The obvious reason is abuse prevention — a scraper hammering your API, a bot brute-forcing logins, or a DDoS attack can all bring a service to its knees without any limit in place. But rate limiting isn't only defensive. It's also how you enforce fair usage: one client shouldn't be able to starve everyone else.

There's also a cost angle. If your API calls a paid third-party service (an LLM, a maps provider, a payment processor), unbounded usage translates directly into unbounded bills. Rate limits let you cap spend per customer tier. They also protect downstream services from being overwhelmed by a burst from a single upstream caller.

Token Bucket Algorithm

The token bucket is the most widely used algorithm — it's what AWS API Gateway and Stripe use under the hood. The idea is simple: each client gets a bucket that holds up to a maximum number of tokens. Tokens refill at a fixed rate (say, 10 per second). Every request costs one token. If the bucket has tokens, the request goes through and a token is consumed. If the bucket is empty, the request is rejected with a 429 Too Many Requests.

The key property is that it allows bursting. A client that hasn't made requests in a while can accumulate tokens up to the bucket capacity and then fire them all at once. This is realistic — legitimate clients often have bursty traffic patterns.

class TokenBucket:
    def __init__(self, capacity, refill_rate):
        self.capacity = capacity       # max tokens
        self.tokens = capacity         # start full
        self.refill_rate = refill_rate # tokens per second
        self.last_refill = time.now()

    def allow_request(self):
        now = time.now()
        elapsed = now - self.last_refill
        # Refill tokens based on elapsed time
        self.tokens = min(
            self.capacity,
            self.tokens + elapsed * self.refill_rate
        )
        self.last_refill = now

        if self.tokens >= 1:
            self.tokens -= 1
            return True  # allow
        return False     # reject with 429

Leaky Bucket

The leaky bucket flips the mental model. Requests enter a queue (the bucket) and are processed at a fixed, constant rate — regardless of how fast they arrive. If the queue fills up, incoming requests overflow and are dropped.

Where the token bucket allows bursts to pass through, the leaky bucket smooths them out completely. A burst of 100 requests becomes a steady drip of 10 per second. This makes it useful when you're rate limiting outbound requests to a downstream service that can't handle spikes — you're shaping your traffic, not just capping it.

Fixed Window Counter

The simplest algorithm to implement: count requests per fixed time window. Give each client a counter keyed to the current minute. Every request increments it. Once the counter hits the limit, reject until the window resets.

key = "rate_limit:{client_id}:{current_minute}"
count = redis.INCR(key)
redis.EXPIRE(key, 60)

if count > limit:
    reject with 429

The problem is the boundary burst. If the limit is 100 requests per minute, a client can send 100 at 11:59 and another 100 at 12:00 — 200 requests in two seconds, both technically within their respective windows. For most use cases this is acceptable, but for strict enforcement it's a real gap.

Sliding Window

The sliding window fixes the boundary problem by tracking requests in a rolling time window rather than fixed buckets. Instead of "requests since the start of this minute", it's "requests in the last 60 seconds from now".

The sliding window log approach stores a timestamp for every request and counts how many fall within the window. Accurate, but memory-intensive for high-volume clients.

The sliding window counter is a hybrid: use two fixed window counters (current and previous) and calculate a weighted estimate of the rolling count. It's 99% as accurate as the log approach with a fraction of the memory cost. Redis is typically used to back both variants.

Where To Implement

The best place is your API Gateway — it sits in front of all your services, so rate limiting is enforced centrally before any application code runs. AWS API Gateway, Kong, and Apigee all have built-in rate limiting.

A reverse proxy like Nginx is the next option:

limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

location /api/ {
    limit_req zone=api burst=20 nodelay;
}

Implementing rate limiting in application code is the last resort. It's not shared across instances — each server has its own counter, so a client can multiply their effective limit by the number of app servers. If you go this route, you need a shared counter store (see below).

Distributed Rate Limiting

The moment you run more than one app server, per-instance counters break down. Five servers each allowing 100 requests per minute means the effective limit is 500. The fix is a shared Redis counter that all instances increment atomically.

Redis's INCR command is atomic, making it a natural fit:

def is_allowed(client_id, limit, window_seconds):
    key = f"rate:{client_id}:{current_window()}"
    pipe = redis.pipeline()
    pipe.incr(key)
    pipe.expire(key, window_seconds)
    count, _ = pipe.execute()
    return count <= limit

The pipeline groups the INCR and EXPIRE into a single round trip, keeping it fast. For even tighter consistency you can use a Lua script to make the check-and-increment atomic in a single operation. Latency to Redis is typically under 1ms on the same network, making this overhead negligible.

Rate Limit Headers

Good API citizenship means telling clients what their limits are and where they stand. Don't just return 429 — give them the information they need to back off gracefully.

HTTP/1.1 200 OK
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 743
X-RateLimit-Reset: 1711929600

# On a 429 response:
HTTP/1.1 429 Too Many Requests
Retry-After: 30
X-RateLimit-Limit: 1000
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1711929600

X-RateLimit-Reset is a Unix timestamp for when the window resets. Retry-After (in seconds) tells the client exactly how long to wait before retrying. Well-behaved clients will use these headers to implement exponential backoff rather than hammering the API repeatedly.

Real-World Limits

Seeing how production systems configure this makes the tradeoffs concrete. GitHub allows 5,000 requests per hour for authenticated users (about 1.4/second), dropping to 60/hour unauthenticated. Stripe enforces 100 requests per second per secret key. OpenAI limits are per-minute token counts that vary by tier — rate limiting compute, not just request count.

Twitter (now X) uses write limits to prevent spam: a few hundred tweets per day for regular accounts. These aren't pure technical limits — they're product decisions encoded as rate limits.

Key Takeaways

• Rate limiting protects against abuse, DoS, unexpected cost spikes, and protects downstream services from overload
• Token bucket is the most practical algorithm — it allows legitimate bursting while enforcing an average rate ceiling
• Fixed window counters are simple but vulnerable to boundary bursts; sliding window solves this at slightly more complexity
• In distributed systems, all rate limiting counters must live in a shared store like Redis — per-instance counters are not effective
• The API Gateway is the right place to enforce limits centrally; application-level rate limiting is a last resort
• Always return X-RateLimit-* headers so clients can make informed decisions rather than blindly retrying

Next up: how to structure your application itself — one big deployable unit or many small ones, and why that decision matters more than most.