Leaky Bucket: Smooth Output Under Load

The failure mode it solves

Some systems fail not because of total volume, but because they cannot handle short spikes. Databases, ML inference endpoints, and payment processors often need a steady flow. Leaky Bucket turns bursty input into a controlled output rate so those backends never see more traffic than they can handle.

Use the bucket analogy once: requests pour in, the bucket leaks at a constant rate. After that, it is just queue math.

How the algorithm works in this service

Leaky Bucket is modeled as a queue with a fixed capacity. Each request tries to enqueue. On every request, the algorithm first leaks items based on elapsed time, then decides whether there is room for the new request.

Step-by-step flow:

Read queue_size and last_leak_time from Redis for rate-limit:{ip}:{endpoint}.
Compute how many slots have leaked: leaked = floor(elapsed_seconds * flowRate).
Reduce queue size: queue_size = max(queue_size - leaked, 0).
If queue_size + 1 <= capacity, allow and increment. Otherwise reject.
Store updated values back to Redis with the configured TTL.

Diagram

Leaky bucket

The key insight: this algorithm does not allow bursts to hit your backend. The queue absorbs them, and the leak rate controls downstream pressure.

Worked example: burst absorbed, steady drain

At t=0, capacity is 50 and flowRate is 5 req/sec. A spike of 30 requests arrives. All 30 are accepted into the queue. Over the next 6 seconds, 30 requests leak out to the backend. If another burst of 50 arrives immediately, only 20 more are accepted before the queue fills and the rest are rejected. The backend still sees at most 5 requests per second the entire time.

Where it fits versus the other algorithms

Algorithm	Strength	Weakness	Best fit
Leaky Bucket	Smooths output rate, protects slow systems	Adds waiting and rejects when queue fills	Databases, queues, paid backends
Token Bucket	Allows bursts with average control	Can still allow short spikes	Public APIs
Fixed Counter	Fastest, minimal memory	Boundary spikes	Internal APIs
Sliding Window	Strictest enforcement	Highest Redis cost	Security and billing

Configuration in this project

yaml

rate-limit:
  fallback:
    algorithm: LEAKY_BUCKET
    capacity: 50
    flowRate: 5

capacity is the maximum queue depth per IP + endpoint. flowRate is the drain rate in requests per second. Increasing capacity makes the system more tolerant of bursts but also increases how long clients may wait before they stop being rejected. Increasing flowRate raises backend load, so tune it to what the downstream system can safely handle.

Redis state and TTL

Leaky Bucket uses a Redis hash:

text

key: rate-limit:{ip}:{endpoint}
queue_size: int
last_leak_time: unix_ms

TTL is controlled by rate-limit.redis.ttl. If it is too short, a quiet client can lose its queue state and a new burst may be treated as empty. If it is too long with very high cardinality keys, Redis memory grows unnecessarily.