LLM Retry Storms: How Backpressure Limits Queue Growth

I cap incoming work before tuning retries. Three retry layers allowing three attempts each can turn one request into 27 provider calls. Longer waits alone won’t stop the queue growing.

My approach is to:

• Bound queues and active requests, rejecting or deferring excess work within set limits.
• Honour Retry-After, then add capped exponential backoff with jitter.
• Give each request one retry owner, one attempt budget and one deadline across all layers.
• Retry only eligible failures, and prevent duplicate side effects when a timeout leaves the outcome unclear.
• Cap retry spending across the service, and check capacity again before each retry.
• Track queue age, attempts per request and completed work - not just falling error rates.

The rule I follow: limit the work, not just the retry pace. Then test overload and restore capacity gradually so recovery doesn’t trigger another wave.

::: @figure {Stop LLM Retry Storms with Backpressure} :::

Control Retry Timing with Retry-After, Backoff and Jitter

Once a failure is retryable, space out retries without letting queues grow.

Backoff spreads retries; backpressure limits admitted work. Use timing controls alongside bounded queues, concurrency limits and retry budgets. [1][2]

Use Retry-After as the Minimum Wait

Accept both delta-seconds and HTTP-date values. Start timing when the response arrives. For HTTP-date, calculate the delay from that receipt timestamp.

Use a synchronised system clock to interpret dates and a monotonic clock to measure elapsed waits. Account for clock skew and clamp negative results to zero. If the remaining deadline is shorter than the required wait, stop - don’t shorten the wait. Fall back to configured backoff when guidance is missing, malformed or unreliable. [4][7]

Use Capped Exponential Backoff with Jitter

For retry n, use B=min(Bcap, B0×2^n), starting at n=0. The goal isn’t just to slow retries down. It’s also to keep fewer retries queued at once.

A 1-second base with an 8-second cap yields 1, 2, 4 and 8-second windows, then holds at 8 seconds.

Full jitter samples uniformly within that window. The cap limits delay growth, not the number of attempts. [3][5]

Here, U means uniform random sampling, and A is a separately configured maximum offset. Use full jitter as the default for large, independently retrying clients. Choose equal or additive jitter when you need a minimum client-side delay.

Add Jitter After the Provider's Minimum Wait

Calculate the capped window B, then sample J=U(0,B). Without valid provider guidance, wait J. With a remaining provider wait R, wait W=R+J. Cap the client-side jitter window - not the provider’s minimum.

Schedule the retry only when W + execution allowance ≤ remaining deadline, the retry budget allows another attempt, and queue and concurrency limits allow admission. Otherwise, stop or defer.

Limit Queue Size and Concurrent Requests

Even well-spaced retries need admission control. Backpressure caps both waiting and active work. Backoff slows retries, but bounded queues and concurrency limits stop the backlog from growing. Apply those limits to deferred jobs, timers and retry counts too.

Reject or Defer Work When Queues Are Full

Set queue capacity using acceptable waiting time, throughput the system can maintain and the memory budget - not peak demand. Cap both item count and payload size, and expire work that cannot finish before its deadline. Spare queue capacity does not make a queue safe if its oldest request is already past its deadline.

Reject excess interactive requests with HTTP 429 for rate limiting or 503 for temporary service unavailability. Include Retry-After with either response when the client can retry later.

Defer background work only if the system supports asynchronous processing. Use bounded durable storage, deduplication, expiry and a dead-letter policy.

Once retry timing is bounded, control queue depth and active concurrency.

Adjust Admission Using Queue and Provider Signals

Monitor queue depth, oldest-queued age, active requests, provider 429 rates and retry volume together. Tighten admission when these signals worsen, and reduce concurrency when provider capacity is constrained. Even a short queue needs attention if its oldest item has waited too long.

Restore capacity gradually after a stable recovery window. Don’t release the whole backlog at once.

Budget tokens across replicas, not just per worker. Reserve estimated input and output tokens, account for request duration, and use shared limits or conservative per-replica allocations.

Release slots while waiting to retry. Treat each delayed retry as new work at admission: re-check its deadline, retry budget, queue depth and token availability. A timer firing must not bypass admission control.

Set Retry Budgets and Check Recovery

Backpressure limits queued work. Retry budgets limit the extra work retries create.

Set one retry policy per logical request, covering maximum attempts, an absolute deadline, maximum backoff and eligible error classes. Add a capped, service-wide retry token bucket: each retry spends a token, successful requests refill it gradually, and an empty bucket blocks further retries. Size the bucket against provider quotas and spare capacity. This caps extra work, but does not replace admission control.[8][10]

Enforce Deadlines Across Retry Layers

Pass the same deadline and attempt counter through the SDK, gateway, worker and fallback path. The deadline must cover queueing, time in deferred storage, Retry-After waits, backoff, network time and execution.[6][9]

Measure Retry Amplification and Queue Recovery

Once retry spending is capped, check whether the system is recovering.

Track retry amplification, retry-token depletion, completion rate and cost per completed logical request (£). Report rejected, expired and dead-lettered work separately. Otherwise, reduced load can look like recovery.

Count logical requests, initial attempts and retries separately. Retry amplification = total provider attempts ÷ logical requests. Fewer 429 responses alone do not prove recovery.[6]

Test Overload Controls and Provider Recovery

Test the same limits you enforce in production. Inject 429s, 503s, slow calls, invalid Retry-After headers and waits that exceed request deadlines. Test simultaneous failures across replicas, plus sustained arrivals above provider capacity.

Set pass criteria before testing: queue, deferred-storage, concurrency and retry limits must hold. Excess work must be rejected or expire within a known time. Verify that the backlog drains without a second retry wave.[6]

Conclusion: Limit Work Before Retrying

Cap admitted work first; tune retry timing second.

With admission capped, use Retry-After as the minimum wait. Then apply capped exponential backoff with jitter, stopping if the wait exceeds the remaining deadline.[11][1][3]

Once timing is under control, cap total retry spend with a single end-to-end attempt budget and deadline. Restore admission gradually, and only when queue age, retry amplification and provider errors have stabilised. These limits keep queues from growing without bound.

FAQs