The message that never becomes valid
Every queue eventually meets a message that is not going to work. A record is missing a field the schema requires. A payload is truncated. An upstream system starts sending a shape nobody anticipated.
The naive retry loop handles this badly, and it fails in a specific way: the message fails validation, gets republished, fails validation again, and goes round forever. Nothing crashes. Nothing alerts. You just watch a dead message consume a slot in your worker pool indefinitely.
Why adding backoff does not fix it
The first instinct is exponential backoff, and the instinct is right about the rate and wrong about the problem.
Backoff makes the loop slower. It does not make it end. A message that fails a structural check will fail it just as reliably on attempt 6 as on attempt 1 — it is not a transient failure, and no amount of waiting changes that. Worse, a slow loop of permanently-failing messages is harder to notice than a fast one, because the queue depth stops moving and everything looks calm.
What you need is not more patience. You need the message to be allowed to fail permanently, and a place for it to go when it does.
Count the attempts in the message
The retry count has to travel with the message. This is the detail that matters more than the backoff curve, and getting it wrong is common:
MAX_RETRIES = 5
def handle(msg):
attempt = int(msg.headers.get("x-attempt", 0))
if not validate(msg.value):
if attempt >= MAX_RETRIES:
quarantine(msg.key, msg.value, reason="validation", attempts=attempt)
return # give up on purpose
delay = backoff(attempt) # capped, jittered
republish(msg, headers={"x-attempt": attempt + 1}, delay=delay)
return
process(msg.value)
If you keep the count in consumer memory — a dict, a set, a field on your handler — it is lost every time the consumer restarts or gets rebalanced, and the message starts again from zero. The loop is then not just infinite, it is unknowably infinite: the message looks new every time. A header on the payload survives all of that, because it moves with the data.
The backoff curve matters less than the jitter
def backoff(attempt, base=1.0, cap=60.0):
return min(cap, base * (2 ** attempt)) * (0.5 + random.random() / 2)
The doubling is the part everyone quotes. The random factor is the part that actually matters, and for a reason that has nothing to do with any single message: without it, every message that failed at the same moment wakes at the same moment. You have replaced one thundering herd with a scheduled one, just on a longer fuse.
Full jitter, or at least base * 2^n * random(), spreads the retries across the window. The individual message waits slightly longer on average and the system behaves far better, which is the trade you want.
Quarantine is a destination, not a log
When a message hits MAX_RETRIES, send it somewhere real — a dedicated topic, with the original payload, the attempt count, the reason it failed, and a timestamp. Three things follow from treating it as a first-class destination:
It stops being invisible. A quarantine topic you can measure is a queue you can alert on. “DLQ depth above zero” is a page; “some messages are failing” written in a log at 3am is not.
It makes the failure diagnosable. The whole point of capturing the payload is that you can look at one and see why. “Schema validation failed” is useless. The actual record is the answer.
It creates a decision you have to make explicitly. And the right answer, most of the time, is to not automate it. Reinjecting a quarantined message the instant somebody fixes the validator will replay whatever caused the failure in the first place. A human should look at the payload, understand it, and then decide. Building the reinjection endpoint is easy; resisting the urge to put a cron in front of it is the actual work.
The trap waiting for you
All of this works until someone adds a required field to the schema.
The validator is strict, the deployment succeeds, and every message already in flight — and every message a producer emits before it picks up the new schema — now fails validation. If your retry budget is generous and your backoff is long, this does not look like a failure. It looks like a gradual, inexplicable increase in quarantine depth over several hours.
Two mitigations, both worth having:
Version the events. Put a version in the payload and keep the old validator alive alongside the new one, at least until the queues have drained. The cost is one field and some bookkeeping; the alternative is an incident.
Prefer a tolerant validator to a strict one. A validator that rejects unknown fields will break every time the producer grows. One that only checks the fields it cares about degrades gracefully instead.
The short version
Backoff without a give-up condition is an infinite loop with extra steps. Count the attempts in the message, cap them, quarantine what survives, jitter the retries so you do not synchronise your own recovery, and resist automating the reinjection.