LLM Rate Limits: Retry-After Headers and Backoff

When an LLM returns HTTP 429, I check whether waiting will help before retrying. For temporary throttling, I honour a valid Retry-After delay and add a little jitter. Without valid guidance, I use capped exponential backoff. If quota is exhausted, short retries won’t help.

My approach pairs retries with clear limits:

• Retry safely: use one retry owner, prevent duplicate actions, and cap attempts and total retry time.
• Control demand: use a bounded queue and shared limits for concurrent calls, requests and tokens.
• Check the results: test header parsing, overload and cancellation; track actual sends, queue waits, costs and deadline drops.

A queue buys time, not capacity. At 300 arrivals per minute and a 60-request-per-minute cap, the backlog grows by at least 240 jobs per minute. When demand exceeds capacity, I defer or reject work - not retry earlier than the server allows.

Rate Limits in LLM APIs Explained | HTTP 429, RPM, TPM & Exponential Backoff

Check HTTP 429 Responses and Retry-After Headers

Separate temporary throttling from a hard quota. Check the HTTP status, structured error body, Retry-After header and provider rate-limit headers together.[6][7] Use the error code, affected model or account, remaining request and token counts, and reset time to work out which limit applies. Header names and reset formats differ between services, so check the provider’s documentation. After confirming a 429 response, parse Retry-After into a usable delay.

Check whether the limit applies to request frequency, token volume or concurrent calls. Daily and monthly quotas need the documented reset; repeated short retries won’t help. Handle authentication, payload, permission and policy errors separately.[10][12]

Parse Retry-After Seconds and HTTP Dates

For temporary throttling, Retry-After can contain either a non-negative whole number of seconds or an HTTP-date.[8][9] Trim whitespace and try parsing seconds first. If that fails, use a standards-compliant date parser. Treat missing or malformed values as no retry guidance, not as a zero-second delay.

For a date, subtract the response-received time from the retry time, both in UTC, and use zero if the result is negative. Use timezone-aware timestamps. Where reliable, use the response Date header as the server-clock reference. Measure the wait with a monotonic timer, keeping malformed values separate from valid delays.

Wait at Least as Long as the Server Requests

Treat the parsed delay as a minimum wait, not a promise that capacity will be available. Add only small, positive jitter.[10] If the wait won’t leave enough time for another attempt before the deadline, defer or stop.

Log the raw header, parsed delay, error class, attempt number and action. Leave out API keys and sensitive content. Pass the parsed delay into the retry decision flow.

Build the Retry Decision Flow

::: @figure {LLM Retry Decision Flow: 429, Retry-After and Backoff} :::

Once you’ve parsed the delay, give one layer ownership of retries: either the client library or the application queue. Return successful responses immediately. For failures, treat permanent client errors as terminal, classify 429 as throttling, and retry only transient server or network failures allowed by the provider’s policy.

Next, check whether the operation is safe to repeat, calculate the delay, and check the attempt and deadline budgets before returning work to the queue. Outer layers must pass through the retry owner’s result and keep its attempt count unchanged. Only the retry owner decides whether - and when - to try again.

Check Retry Safety and Set Backoff

Retry a 429 only if the condition is temporary and the operation is safe to repeat. Check the response body and headers to tell temporary throttling apart from provider limits, including requests per minute, tokens per minute, quota exhaustion and account restrictions. If the server’s wait exceeds your maximum or remaining deadline, stop or defer the request.

Without usable server guidance, calculate ceiling = min(maximum_fallback_delay, base_delay × 2**attempt), starting at attempt zero. Then apply either full jitter, a random delay from 0 to ceiling, or equal jitter, half the ceiling plus a random delay up to the remaining half.

Limit Attempts, Waits and Duplicate Actions

Set limits for total attempts - including the initial request - total retry time and each fallback delay. Stop on success, cancellation or exhausted limits.

For operations with side effects, require an idempotency key, conditional write or reconciliation before retrying. Use the same key across attempts. Release the worker, semaphore permit or request slot before waiting, then rejoin the shared queue with the original deadline and attempt count.

Calculate Retry Delays in Python

The helper below decides the delay; it does not schedule queue work. After response classification and safety checks, it returns (should_retry, delay, reason).

It accepts decimal seconds and HTTP dates, and distinguishes missing or malformed guidance from waits that exceed policy limits. Full jitter applies only to fallback delays. Valid server waits act as a minimum, with any extra local delay kept within a bound.

import math
import random
import re
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def retry_delay(
    header, *, base_delay, maximum_fallback_delay,
    maximum_server_wait, attempt, jitter_bound,
    remaining_deadline, received_at=None,
):
    # Configuration: finite, non-negative delays; base_delay > 0;
    # attempt is a non-negative integer (0 means first retry).
    now = received_at or datetime.now(timezone.utc)
    value = header.strip() if header is not None else ""
    state = "missing" if header is None else "malformed"
    server_wait = None

    if re.fullmatch(r"[0-9]+(?:\.[0-9]+)?", value):
        server_wait = float(value)
    elif value:
        try:
            retry_at = parsedate_to_datetime(value)
            if retry_at.tzinfo is not None:
                server_wait = max(0.0, (retry_at - now).total_seconds())
        except (ValueError, TypeError, OverflowError):
            pass

    if server_wait is not None:
        if server_wait > maximum_server_wait:
            return False, None, "policy_exceeded"
        delay = server_wait + random.uniform(0, jitter_bound)
        reason = "server_guidance"
    else:
        try:
            ceiling = min(
                maximum_fallback_delay,
                math.ldexp(base_delay, attempt),
            )
        except OverflowError:
            ceiling = maximum_fallback_delay
        delay = random.uniform(0, ceiling)
        reason = f"fallback_backoff:{state}"

    if remaining_deadline <= 0 or delay >= remaining_deadline:
        return False, None, "deadline_exceeded"
    return True, delay, reason

Pass a timezone-aware received_at. Calculate remaining_deadline using a monotonic clock, leaving time for the next request - not just the wait. The retry owner must enforce attempt counts and schedule work; the helper does neither.

Manage Demand with Queues and Concurrency Limits

Backoff controls when a retry happens. Queues control whether new work enters the system.

Place incoming work in a bounded priority queue before calling the LLM. Store each job’s priority, deadline, cancellation state, retry count and estimated token demand. Cap the queue size to protect memory and make overload visible.

Combine Worker, Request and Token Limits

Consider Northstar Support, receiving 300 customer-support tickets per minute. It could use a 1,000-job queue, 20 workers and an eight-call concurrency cap. Assume its project allows 60 requests and 150,000 tokens per minute. Reserve 1,500 tokens per ticket: 1,200 prompt tokens plus up to 300 output tokens. With a 60-requests-per-minute cap, arrivals would outpace capacity by 240 tickets per minute.[1][14]

Use one semaphore for in-flight calls and one shared request/token budget across all workers. Afterwards, reconcile reported usage and return unused reservations only where the provider’s token-accounting rules allow.

Retries must pass through these same controls. Separate per-worker counters can, together, exceed a shared project quota.[5][14]

Handle Full Queues and Protect Deadlines

Retries must follow the same queue and deadline rules as new work.

A queue absorbs bursts; it cannot fix sustained overload. When it fills up, delay admission within a bounded timeout, reject excess work with a retryable status, or persist routine tickets for later processing. Put urgent production incidents ahead of routine enquiries, and preserve deadlines and cancellation across retries.

A 30-second ticket must not wait behind a 10-minute batch job.

Remove expired or cancelled work from the dispatch path. Persisted jobs also need an age limit and a dead-letter path.

After repeated 429s, reduce the shared dispatch rate rather than letting workers increase load independently. Keep delayed retries outside active slots. Increase throughput gradually after sustained success.[5]

Compare Request Dispatch Controls

Match each source of pressure to the control that handles it:

Test and Monitor Retry Behaviour

Once backoff and queue controls are in place, check that they reduce throttling rather than hide it. Validate Retry-After parsing, fallback backoff, bounded queues and concurrency limits.

Track Throttling and Queue Load

Track 429 rates by endpoint, model and operation, alongside Retry-After values, attempts per logical request, retry success and deadline drops. Compare physical sends with logical operations to spot retry amplification. Measure added latency, token use and cost per success, plus queue depth, oldest-job age, queue wait, in-flight calls, and request and token throughput.

Use structured logs with request_id, attempt, retry_after_seconds, backoff_seconds, queue_wait_seconds and outcome. Add throttle_source to distinguish server throttling from queue saturation or token exhaustion. Record cancellations separately from deadline expiry. OpenTelemetry recommends recording resend counts to distinguish one logical operation from its multiple sends.[15]

Set thresholds from your baseline: reduce dispatch when 429s exceed 2% for five minutes, and pause low-priority work when queue occupancy stays above 80%; resume below 60%.

Test Header Parsing, Overload and Cancellation

Test the parser on its own before running end-to-end overload checks. Use a fake clock and deterministic random source to test missing, malformed, negative, zero, fractional, whitespace-padded and oversized headers. Also test future and past HTTP dates, plus clock skew.

Check the earlier delay logic: valid server waits must never be shortened, and jitter must stay within its configured bounds. If the required wait exceeds the deadline or policy limit, stop or defer the job - don't shorten the wait and retry early. Missing or invalid values should trigger the documented backoff fallback.

Next, run repeated-429, queue-overload and cancellation tests across multiple workers to check how the queue and retry owner behave under pressure. Assert bounded attempts, shared request and token budgets, and gradual dispatch recovery. Count actual sends to verify a single retry owner; otherwise, application and SDK retry limits can multiply sends.[15][16]

Test cancellation during admission, waiting, slot acquisition and active calls, followed by shutdown. Pending waits must end promptly, reservations must be released, and cancelled or expired jobs must not retry.

Conclusion: Put Retry Limits into Practice

Retry temporary throttling only, and only when the action is safe to repeat. Do not retry authentication failures, invalid parameters, billing or quota exhaustion, or permanent failures. Treat Retry-After as the minimum wait. Otherwise, use capped exponential backoff with jitter. Set limits on both attempts and total retry time, and defer work rather than retrying early.[6][11][10]

Retries are just one control. Pair server-directed waits with a shared, bounded queue and separate controls for concurrency, requests and tokens. Track queue delay alongside retry overhead and processing cost.[13]

To make these controls work together in production, Hokstad Consulting can help integrate retry, queue and concurrency controls into your agent stack.

FAQs