LLM Retries vs Circuit Breakers: Handling API Errors

I use retries for brief faults and circuit breakers for repeated failures. My starting point is 1–2 retries for calls that are safe to repeat, with one deadline covering every attempt, wait and fallback.

Here’s how I keep failures from turning into more traffic and higher bills:

• Retry selectively: honour Retry-After for temporary 429s; use capped backoff with jitter for eligible 5xx errors. Stop on invalid requests or account faults.
• Stop repeated failures: use closed, open and half-open circuit states to control traffic and test recovery.
• Keep limits shared: put retries in one layer, cap concurrent calls and prevent duplicate work. A timeout does not prove the original call stopped.
• Switch only when approved: check fallback features, capacity and UK data-processing rules first. Keep the original deadline and budgets.
• Test recovery: track attempts, p95/p99 latency, circuit changes and cost per successful task in £.

Quick Comparison

My rule: retry briefly, stop when limits are reached, and never let a fallback reset the clock.

::: @figure {LLM API Errors: Retry, Break Circuit or Fall Back?} :::

Retries vs Circuit Breakers: How They Work

Retries and Circuit Breaker States

A circuit breaker tracks failure patterns across calls. A retry simply repeats a call after a transient failure that permits another attempt. Closed lets traffic through; open rejects calls without contacting the provider. Half-open allows limited probes after a cool-down: success closes the circuit, while failure reopens it.[9]

Rejecting calls quickly frees workers rather than leaving them waiting through repeated timeouts. Share breaker state across workers and cap half-open probes to prevent a surge during recovery. Treat circuit rejection as final, not as a reason to retry.[12] These rules help you decide when to stop retrying and hand off to an approved fallback.

Comparison Table: Recovery, Latency and Load

Check whether the SDK already retries. Centralise error classification and breaker state rather than duplicating policies across workers. The next table shows how to apply these differences to specific responses.

Error Decision Table: Timeouts, 429s and 5xx

Use the response body and provider guidance, not the status code alone. A 429 may apply to requests, tokens, concurrency, a model or an organisation-wide quota. Limit your response to the affected allowance: exhausting one quota should not block routes with independent capacity. A timed-out stream also needs different handling from a connection failure before any output arrives.

Circuit Breaker Pattern Explained: Timeouts, Retries and Fallbacks

Combine Limited Retries With Approved Fallbacks

Retries handle brief faults. Fallbacks take over only after the circuit breaker isolates a sustained failure.

Set Attempt Limits and Circuit Thresholds

Check the circuit before every retry. If it is open, stop retrying and use an approved fallback or fail fast. Set a hard attempt limit that includes the initial call, a timeout for each attempt, and capped back-off with jitter. Use one absolute deadline for calls, waits and fallback processing. Switching providers must not reset it.[11][15]

Define the failure threshold, observation window, minimum sample size, open duration, half-open probe limit and recovery criteria. Apply these settings to the affected provider, region, model or endpoint.

Give the fallback its own circuit, quota, concurrency and attempt limits. It must still stay within the original request’s shared attempt and time budgets. For actions with side effects, require idempotent behaviour or supported idempotency keys, plus application-level deduplication across providers. A provider-specific key does not prevent a replay across providers.[6]

With routing limits in place, check that the fallback can safely handle the same request.

Check Fallback Compatibility and Data Rules

Approve a fallback only after testing authentication, permissions, schemas, tool calls, structured outputs and streaming. Record differences in context limits, tokenisation, latency, pricing, retention and safety. Test representative prompts, long inputs and refusals. Valid JSON alone is not enough: its meaning must also work for the application.

Decide whether to return interrupted streams as incomplete or discard and regenerate them. Do not replay tool actions after a disconnect without deduplication.

Before routing UK personal data, check processing locations, contracts, subprocessors and retention terms. If sending data outside the UK constitutes a restricted transfer, establish a valid transfer mechanism, such as UK adequacy regulations or appropriate safeguards. Complete any required transfer-risk assessment. Safeguards can include the UK International Data Transfer Agreement or Addendum. Minimise or redact personal data before sending it. Complete these checks before any switch.[13][14]

Fallback Decision Table: When to Switch or Stop

Use these rules only after the fallback passes compatibility and data checks. Retry briefly, switch once, then stop when shared limits are reached.

Fail fast within limits; do not extend the deadline. For non-urgent work, use a durable queue with a deduplication key and bounded consumer retries. Otherwise, return a clear temporary-unavailability response. Log routing decisions, attempt numbers, circuit state, latency and cost.

Prevent Retry Storms and Test Recovery

Once fallback routing is defined, keep retries from multiplying load across layers.

Count attempts across the whole workflow, not just the application. Give one layer ownership of retries. Other layers should pass failures back or honour the shared deadline. Count unavoidable redeliveries against the same logical-request budget.[7][21]

This keeps a partial outage from triggering a surge of retry traffic.

Limit Retry Traffic and Concurrency

Use capped exponential backoff with full jitter: pick a random delay between zero and the current backoff cap.[10][18][16] Treat Retry-After as the minimum wait, even when it exceeds that cap. If the wait leaves too little time for another attempt, stop or defer.[8][19][20]

Set per-request, tenant and system-wide retry budgets. Combine a retry-to-initial-call ratio with an absolute rate ceiling.[17]

Cap in-flight calls separately for each dependency. Apply back-pressure and drop lower-priority work before queues age out. Reserve capacity for health checks and cancellations, and limit half-open probes. After recovery, increase concurrency in steps rather than releasing the entire backlog at once.

Pseudocode: Enforce Request Limits

The logic below shows how shared deadlines, budgets and circuit checks work together.

admit atomically checks circuit state, concurrency, quota and cost, then reserves one attempt from the shared budget. Each provider has its own circuit state.

provider = primary
while true:
    if deadline.expired() or shared_budget.exhausted():
        return controlled_failure()

    # safe_next_action: one approved fallback switch; no replay of
    # uncertain side effects without deduplication.
    permit = admit(provider, shared_budget, deadline)
    if permit.denied:
        action = safe_next_action(permit.reason)
    else:
        outcome = call_with_timeout(provider, deadline.remaining())
        permit.release_and_account(outcome)
        kind = classify(outcome)
        circuit(provider).record(kind)  # Only eligible faults count
        metrics.record(outcome)

        if kind.success:
            return outcome.response
        if kind.permanent:
            return controlled_failure(outcome)

        action = safe_next_action(kind, side_effect_status(outcome))

    if action.retry:
        wait = max(valid_retry_after_or_zero(), full_jitter(backoff()))
        if not deadline.fits(wait + minimum_attempt_time):
            return defer_or_return_failure()
        sleep(wait)
    elif action.approved_fallback:
        provider = action.provider  # Same deadline and shared budgets
    else:
        return defer_or_return_failure()

Measure Recovery, Response Times and Cost

Test these limits under both outage and recovery conditions.

Track attempts per request, retry success rate, retry rate and timeout/429/5xx rates. Record circuit transitions - including fallback switches and half-open recovery - and how long circuits stay open. Also measure fallback frequency and latency, queue age, end-to-end p95/p99 latency and cost per successful task in £, including failed attempts.

Test timeouts, throttling and outages at realistic concurrency, with queue and agent retries enabled. Check that traffic stays bounded, fallback circuits remain independent and probes stay limited. Verify that duplicate prevention works when an accepted operation times out.

Assess settings against availability, latency and cost targets - not maximum retry counts. Treat numerical thresholds as service-specific hypotheses until these tests support them.

Conclusion: Retry Briefly and Isolate Sustained Failures

Retry transient timeouts, 429s and 5xx errors only when requests are idempotent or deduplicated. Use circuit breakers to stop calls during sustained provider failure.

Keep retries brief. Switch once to a tested, approved fallback, staying within the remaining deadline, quota and cost budget, then stop. If output or data-processing rules prevent switching, return a controlled failure or defer the work.

Centralise retry decisions, jitter and back-pressure to keep recovery traffic within limits. Monitor whether these controls restore service rather than add traffic. Hokstad Consulting helps teams define and implement these AI reliability controls.

FAQs