AI Agent Backpressure: How Queue Limits Control Load

I use backpressure to keep AI agents from accepting more work than they can finish. I cap queued and running tasks, set one deadline from arrival to completion, and pause or reject work when capacity runs out. A queue absorbs bursts - it does not make workers faster.

My approach covers four checks:

• Bound the load: limit task counts, payload sizes, tool calls and retries across all workers and replicas.
• Choose what waits: reserve capacity for critical tasks; defer or drop lower-priority work under clear rules.
• Handle unfinished work: pass cancellation through each stage, remove expired tasks and prevent duplicate actions.
• Test the limits: track p95/p99 latency, queue age, completions and £ per completed workflow during bursts and dependency failures.

A full queue must change what the producer does. Otherwise, I’m just storing the overload somewhere else.

::: @figure {AI Agent Backpressure: From Admission to Completion} :::

Backpressure: When to Say No (AI System Design #18)

Set Queue Capacity and Waiting Limits

Bounded queues make overload an explicit choice: pause, reject or shed work. Set separate limits for executing tasks, queued tasks and total in-flight work, so queued + executing never exceeds the total limit. Active concurrency limits resource use during execution; queue capacity limits waiting work.

Pair task counts with a byte or token budget. Use estimated tokens, bytes or expected compute cost to account for context, expected output and tool fan-out. Reserve capacity at admission and release it when work finishes or is removed. Task counts alone cannot protect memory when payload sizes vary.[9]

Size Queues for Throughput and Latency

Size the queue around the waiting-time budget, not peak demand. Little’s Law (N = X × R) gives a baseline: in a stable system, average in-flight work equals throughput multiplied by response time. It does not guarantee p95/p99 latency.[12]

Measure representative workloads and track executing work separately from waiting work. Then size the backlog to fit an acceptable waiting-time budget. A larger queue only postpones rejection; it does not fix a capacity shortfall.[5][6]

Choose How Requests Enter the Queue

Once you have sized the queue, decide what happens when it fills. Choose blocking, timed admission or immediate rejection explicitly.

Blocking works only when you cap both the waiting duration and the number of waiting producers. Without those limits, the backlog simply moves into request threads, sockets or suspended agent tasks. Timed admission waits only until its admission deadline.[7][8][11]

Use One Deadline for Waiting and Execution

Use one deadline for every stage. Admission waiting happens before acceptance; queue waiting happens after acceptance but before execution. Start the deadline at ingress, then pass the remaining budget through admission, queueing, model/tool execution and result handling.

Measure admission waiting and queue waiting separately. Reserve time to send the response, and reject work if its remaining budget is too short to finish usefully.[8][9]

Pass cancellation through to queued records, workers and tool calls. Cancellation may not stop an external action already under way, so mutations need idempotency and status reconciliation. Remove expired tasks, release their reservations, and re-check cancellation before dispatch and between long steps.

Retry only transient failures. Use a small attempt limit, exponential backoff and jitter. Every retry must pass admission again and fit within the original deadline.[8][9][10][11]

Define Overload Responses and Task Priorities

Once queue and deadline limits are set, decide how the system will respond to overload.

Decide When to Pause, Reject, Defer or Shed Work

Base the response on queue occupancy, wait time, worker availability and task priority. Pause producers as the queue approaches its limit. Defer tasks only when they have an explicit asynchronous contract. Shed low-priority work before workers, memory or queue capacity reach their limits.

Task priority determines which work survives when capacity tightens.

Protect Critical Tasks and Limit Starvation

Prioritise customer-facing, safety-critical and compliance tasks. Shed optional work first.

Shedding discards work to reduce load. Priority-based shedding decides what to discard based on business and safety impact. Assign classes using workflow criticality, remaining deadline, estimated execution cost and safety or compliance consequences - not an unverified “urgent” label. Record the reasons in task metadata.

Reserve worker capacity for critical classes, cap background concurrency and schedule work fairly within each class. Ageing can increase waiting tasks’ effective priority within safety boundaries, reducing starvation during recoverable overload. If overload persists, background work may still need to be rejected or expired.

Return Clear Overload Statuses

Overload responses tell callers that queue limits have been reached. Use HTTP 429 for caller-specific limits and HTTP 503 for system-wide overload, maintenance or unavailable dependencies. Include Retry-After only when a later retry is likely to succeed.[13]

Expose deferred, rejected, shed and expired states, along with task IDs and reasons. For deferred work, provide a status endpoint or completion event. Retry actions that have side effects only with persisted idempotency controls.

These signals work only when workers and replicas share the same limits.

Implement, Monitor and Test Backpressure

Place a bounded priority queue between planning and execution, and use a semaphore to limit active tool calls. At admission, check the tenant allowance, task priority, queue capacity and request deadline. Each queued item should store its workflow ID, priority, enqueue time, deadline, cancellation token and estimated cost. Report every outcome to the calling workflow, and reject excess work at admission rather than letting the queue grow.[3][4]

Apply the policy across the admission path, queue and worker pool.

admit(task):
    if cancelled(task) or now() >= task.deadline:
        return report(task, EXPIRED)

    if !tenant_budget_available(task.tenant, task.priority):
        return report(task, REJECTED_TENANT_LIMIT)

    if !global_slots.try_acquire():
        if task.priority < SHED_THRESHOLD:
            return report(task, SHED_LOW_PRIORITY)
        if !queue.try_enqueue(task, priority=task.priority):   # bounded priority queue
            return report(task, REJECTED_QUEUE_FULL)
        return report(task, QUEUED)

    start_worker(task)
    return report(task, ADMITTED)

queue_worker loop:
    while running:
        task = queue.pop()
        if task == NONE:
            continue
        if cancelled(task) or now() >= task.deadline:
            report(task, EXPIRED)
            continue
        global_slots.acquire()
        start_worker(task)

execute(task):
    try:
        result = call_tool(task, timeout=remaining(task.deadline))
        report(task, COMPLETED, result)
    except Timeout:
        report(task, TIMED_OUT)
    except Cancelled:
        report(task, CANCELLED)
    except Error:
        report(task, FAILED)
    finally:
        global_slots.release()  # release should be idempotent

Before execution, atomically claim the task so retries or duplicate callbacks cannot run it twice. Make release idempotent too: retries, cancellation races and duplicate callbacks must not inflate capacity.

Every worker and replica must use the same reservation model for these rules to work.

Coordinate Limits Across Workers and Replicas

A local semaphore is not a global limit. Use a shared semaphore or partitioned budget to coordinate downstream capacity, alongside tenant allowances and bounded agent fan-out. The queue bound must also apply across replicas.

Pass cancellation through the planner, queued tasks, workers and child agents. For each reservation, record the owner, lease expiry and cancellation state. Keep cancel_requested separate from cancelled: a caller disconnect does not prove the tool has stopped. Hold the reservation until completion or verified lease expiry.[4]

Track Waiting Time, Completions and Cost

Once the path is in place, measure whether it behaves as intended.

Track queue depth, capacity and utilisation; oldest queued-task age; admission rate; queue-wait duration; active tool calls; and available slots. Also record rejection and shedding counts, retries, cancellation, timeout and failure rates, tool latency, and completions by priority.

Use p95 and p99 alongside averages, and compare deadline misses with your latency targets. If queue wait rises while execution time stays steady, arrivals are likely exceeding capacity. If execution time rises, look for a slow dependency.

Calculate £ per completed workflow by dividing model, tool, infrastructure and retry costs by the number of successfully completed workflows. Keep workflow IDs in traces, not metric labels, to avoid unbounded label growth.[16][17][18]

Test Overload and Find Bottlenecks

Test bursts, sustained load, slow dependencies, long jobs and mixed priorities. Repeat each scenario with mixed priorities and cancellation. Check that memory and queue length stay bounded, queued tasks expire at their deadlines, cancellation reaches workers, and high-priority tasks are admitted within the target latency.

Force worker failures and cancellation races to check that permits are released exactly once. Keep sending traffic after the dependency recovers to confirm that the queue drains without a retry storm.[15]

Conclusion: Limit Work and Plan for Overload

Once the limits and policies above are in place, check that they hold under load. Queues absorb bursts; they don’t add capacity. Limit queued and executing work, cap waiting time, and carry one deadline through queueing, execution and retries. When capacity runs out, pause producers that can honour backpressure, reject work that will miss its deadline, defer durable jobs or shed low-priority tasks.[3][14]

Retries need the same capacity and deadline rules. Route them through one layer, cap attempts and use one retry budget. Apply exponential backoff with jitter. Stop when the deadline or a full queue makes another attempt wasteful.[3][19]

Before deployment, cover these checks:

• Define latency and cost targets. Set queue, concurrency, per-tenant and downstream limits, with one deadline and bounded waits.
• Document when to pause, reject, defer, cancel or shed work. Return clear overload statuses with retry guidance, and reserve capacity for critical tasks.
• Track queue depth, wait time, execution time, completions, expiries, retries, token use and cost. Test burst traffic, sustained overload and dependency failures.

Tune the limits using measured p95/p99 latency, completions, rejections, expiries and cost, rather than queue depth alone.

FAQs