← Back to Journal

leptons1618@anishgiri.dev:~/journal/the-thundering-herd-problem-and-how-sqs-helps-you-survive-it less post.md

#AWS#SQS#DISTRIBUTED_SYSTEMS#QUEUES#RELIABILITY

The Thundering Herd Problem and How SQS Helps You Survive It

Aug 18, 2026 · 6 min read

What happens when every consumer wakes up at once, and how AWS SQS visibility timeouts and backoff keep your pipeline from eating itself alive.

When a cache expires, every consumer that was waiting on it wakes up and hammers the same backend at the same time. That is the thundering herd problem. Picture 200 people trying to exit through one door because someone yelled “fire” in a room with no fire. It shows up most often in ingestion pipelines — records flowing from a queue into a relational database. Under load, failed batches would retry with no backoff, and a dozen consumer threads would all hit the database simultaneously. PostgreSQL would reject connections, the rejected batches would retry again, and the cycle would tighten until the pipeline ground to a halt.

Grab a coffee. Let me walk you through it the way I wish someone had walked me through it.

What the thundering herd actually costs

First lesson: the problem is not that retries happen. Retries are necessary in any distributed system. The problem is that they happen all at once. When 200 messages fail in the same second and all 200 consumers immediately retry, you get a burst of load that is worse than the original spike. The backend, already struggling, gets hit again before it has recovered. Each retry cycle is shorter and sharper than the last because the number of competing consumers has not changed but the interval between attempts has collapsed to zero.

The classic shape of it is a pipeline that copes fine at a moderate rate and then collapses at a higher one. The crash was not caused by the throughput itself. The database will usually sustain that same rate comfortably when the writes arrive spread out. The crash was caused by the retry pattern. A batch of 200 inserts would fail due to a transient connection timeout, all 200 records would immediately re-enter the queue, all 200 would be picked up in the next poll cycle, and the same connection timeout would fire again. The retry rate exceeded the fresh-message rate. The pipeline was eating its own tail.

Junior dev version of this bug:

while True:
    try:
        process(message)
    except ConnectionTimeout:
        retry_immediately()  # what could go wrong

Everything. Everything could go wrong.

Why a plain queue is not enough

A first instinct is to add a queue in front of the database. Put messages in, pull them out at a controlled rate. This works until you have multiple consumers. If five consumers poll the queue and all five receive messages at the same time, you have the same burst. The queue distributes load, but it does not spread it out in time.

Think of it like five roommates all deciding to shower before work at 8 AM. Congratulations, you now have no hot water and the same problem you started with.

The missing piece is a mechanism that prevents a message from being visible to other consumers while one consumer is processing it, and a mechanism that delays retries so they do not all land in the same window.

How SQS breaks the cycle

AWS SQS handles both of these with two features: visibility timeout and delay queues. Two features. That’s it. This is the rare AWS service where you do not need a diagram with 14 boxes to understand it.

Visibility timeout is the core mechanism. When a consumer receives a message from SQS, the message becomes invisible to other consumers for a configurable period. If the consumer processes the message successfully and deletes it, the message is gone. If the consumer crashes or times out, the message becomes visible again after the visibility timeout expires and another consumer can pick it up. This means that at any given moment, only the consumers actively processing messages are generating load. You do not get 200 consumers all hammering the database because 190 of them are holding messages that are invisible to the rest of the queue.

It’s basically “do not disturb” for messages. One consumer gets the message, everyone else politely looks away.

The default visibility timeout is 30 seconds. You can set it per queue or per message. For a database insert that takes 200 milliseconds, a 30-second timeout is generous. For a batch insert of 500 records that takes 3 seconds, it is still fine. The rule of thumb: the timeout should be longer than your processing time, so a slow consumer does not cause premature redelivery, but short enough that a dead consumer’s messages are reprocessed in a reasonable window.

# Too short: your slow consumer's message gets redelivered
# and now two consumers are doing the same work. Chaos.
visibility_timeout = 1

# Too long: your consumer dies and nobody notices for an hour.
# Also chaos, just quieter.
visibility_timeout = 3600

# Just right: comfortably longer than processing time.
visibility_timeout = 30

Delay queues solve the retry side. When a message fails processing, you can re-queue it with a delay. SQS will not make the message visible to consumers until the delay expires. If you use exponential backoff, first retry after 1 second, then 2, then 4, then 8, the retries spread out over time instead of landing all at once. A burst of 200 failed messages becomes a trickle: a few retries at second 1, a few more at second 2, a handful at second 4. The backend has time to recover between waves.

Your backend goes from “why is everyone yelling at me” to “ah, visitors, how nice.”

What this looks like in practice

Here is the pattern I would use with SQS for an ingestion workload. Follow along step by step:

  1. Producer writes to SQS. The producer does not talk to the database directly. It writes messages to a standard queue with a visibility timeout of 30 seconds. Your producer’s job is to write messages and mind its own business.

  2. Consumers poll in batches. Each consumer polls up to 10 messages per request (the SQS maximum for a single ReceiveMessage call) and processes them as a batch. With 10 consumers polling 10 messages each, you get 100 concurrent operations, not 1,000. Math is your friend here.

  3. On success, delete the message. DeleteMessage removes it from the queue. No retry, no reprocessing. Delete and move on, like a healthy adult.

  4. On failure, change the visibility timeout. Instead of immediately re-queueing, use ChangeMessageVisibility to set a per-message delay. First failure: set visibility timeout to 2 seconds. Second failure: 4 seconds. Third: 8. This is exponential backoff applied at the message level, without a separate retry queue.

def on_failure(msg, attempt):
    # attempt 1 -> 2s, attempt 2 -> 4s, attempt 3 -> 8s
    sqs.change_message_visibility(
        QueueUrl=QUEUE_URL,
        ReceiptHandle=msg['ReceiptHandle'],
        VisibilityTimeout=2 ** attempt,
    )
    # No, you may not set VisibilityTimeout=0.
    # We've talked about this.
  1. After N retries, route to a DLQ. SQS supports redrive policies. Configure a dead-letter queue that receives messages after a maximum receive count (e.g., 5). Messages in the DLQ stay there until you inspect and reprocess them manually. Think of the DLQ as the corner where you put the messages that keep misbehaving, so you can have a calm conversation with them later.

Standard vs FIFO

For most ingestion workloads, a standard queue is the right choice. Standard queues provide at-least-once delivery and best-effort ordering, which is sufficient when you are writing events to a database and the order of writes does not matter. Standard queues scale to nearly unlimited throughput.

FIFO queues give you exactly-once processing and order preservation, but they cap out at 300 transactions per second per API action (3,000 with batching). If you need ordering, say, processing state changes for a specific entity in sequence, FIFO is worth the throughput cost. If you are just ingesting events as fast as possible, standard is the better fit.

Quick decision guide:

if order_matters:
    use_fifo()   # slower, but everything happens in the right order
else:
    use_standard()  # fast, occasionally chaotic, like my desk

The part SQS does not solve

Now the honest talk, because a good senior does not just hand you a hammer and walk away. SQS manages delivery and retry timing. It does not make your database faster. If your backend genuinely cannot handle the throughput, spreading retries out will buy you time but will not fix the bottleneck. You still need to size your consumers, tune your batch sizes, and monitor queue depth.

What SQS does give you is a buffer that absorbs bursts and a retry mechanism that does not amplify them. The thundering herd problem is fundamentally about synchronization, every consumer waking up at the same time. Visibility timeout desynchronizes consumers by hiding in-flight messages. Delay queues desynchronize retries by spacing them out. Together, they turn a sharp spike into a manageable curve.

Set up the queue, tune the timeouts, watch your queue depth graph flatten out, and enjoy the rare feeling of a distributed system behaving itself.