AI Agent Recovery: Replay and Duplicate Tool Actions

Replay can restore an AI agent’s progress, but it cannot stop a payment being taken twice. Before any state-changing tool call, save the approved request and a stable action ID. If the result is missing, check what happened rather than assume the request failed.

Safe recovery needs four controls:

• Stable identity: Keep the same request and idempotency key across retries. A £125 payment must not become a new payment just because a response was lost.
• Bounded retries: Limit attempts and elapsed time, and follow the provider’s rules. Key protection has a time limit.
• Outcome checks: Treat timeouts as unknown. Check provider records and pause for review when the result remains unclear.
• Duplicate protection and testing: Deduplicate webhooks and follow-on actions. Test crashes before and after provider acceptance, and give operators a clear audit trail.

The goal is one business result, not a promise that every tool call runs exactly once.

18. Retries Without Duplicate Actions: Idempotency for AI Agents

Save Action Identity Before Calling Tools

Save the authorised intent in durable storage before calling the tool. If replay reaches an external side effect with no recorded result, the action’s identity is the only safe anchor. It links the business request to every attempt and recovery step. If saving fails, do not call the tool. Once the record is committed, retry boundaries can apply without losing the original intent.

Hypothetical payment: A customer authorises £125.00 for order ORD-2048. Save the operation as payment-ORD-2048 before submission.[6][7]

Create Stable Idempotency Keys

Build the key from saved workflow, business entity and logical-operation identifiers. Or generate it once and save it with the action. Reuse that key across retries and replay; only a new authorised action gets a new key. A temporary tool-call ID will not do: the key must survive replay unchanged.

Check the endpoint’s enforcement scope, account scope, regional scope and retention rules. Before each submission, compare a canonical request fingerprint covering the amount, currency, customer, destination and order. Reject changed payloads under an existing key: £150.00 is not a retry of £125.00.[1][8]

Record Each Action’s Lifecycle

Save the core action fields, response metadata and an append-only attempt history. That way, retries cannot erase earlier evidence.

Also define how the receiver responds to duplicates: it should return the original result, an in-progress status, or a documented duplicate/conflict response. If the action is in progress, wait, poll or reconcile. Do not generate another key.

Use database transactions and uniqueness constraints on (tenant_id, operation_id) and (tenant_id, idempotency_key) to block duplicate local actions. Use a lease or compare-and-set claim to assign worker ownership, plus version checks to stop stale workers overwriting newer results. These controls coordinate your workers, not the remote provider.

Keep the saved identity available across crashes so retries can stop safely and uncertain outcomes can move into reconciliation.

Set Retry Boundaries and Resolve Uncertain Outcomes

::: @figure {AI Agent Recovery: Safe Retry and Reconciliation} :::

Once the action identity is saved, retry the smallest safe unit of work, not the whole workflow. On replay, use the saved command and recorded orchestration decisions. Do not reinterpret the request or issue a new command because the result is missing.[4][11] Replay should recover the action, not execute it again.

Define Recovery States and Retry Rules

Use a separate recovery state machine. An uncertain outcome moves to unknown → reconciling, never back to planned.[4][1]

Set one retry budget per business operation. Record each attempt’s number, layer, timestamp and reason. Use bounded exponential back-off with jitter, a deadline and an attempt limit. Retry only operations documented as safe. Validation errors and confirmed declines are not transient failures.[4][12][14]

Requests with unknown outcomes need a different path.

Reconcile Payments and APIs Without Idempotency

When an action reaches unknown, reconcile it before resending. For an uncertain payment, look up the original request using its payment, request or merchant reference. If it is still processing, wait for the provider’s final status. Reuse the original key only within the provider’s contract and retention window.[1][15][16]

Do not retry after key expiry without provider evidence or operator review.[15][16]

If the provider does not support idempotency, keep a local operation ledger. Record the business-operation ID, request fingerprint, intended effect, timestamps, attempt history, provider reference and final state.

Before retrying, query every provider lookup path and compare the results with the intended request. An empty search does not prove rejection. If the operation moves money, inventory or reservations, pause while its outcome remains unknown. A local uniqueness constraint cannot prevent a second remote action.

Apply the same rule to events that trigger follow-on work.

Prevent Duplicate Webhook and Downstream Effects

Webhook retries need the same recovery controls as tool calls: authenticate, record, deduplicate, then process. Verify signatures against the raw request payload where required.[13] Save the receipt first, then deduplicate using both the provider-scoped event ID and the business-operation ID.

Use an inbox for one-time local intake and an outbox for reliable downstream publishing. Give each downstream action its own stable identity and make it independently idempotent.

Store provider event timestamps and any available sequence information, object versions or status ordering. This prevents a late “processing” event from overwriting confirmed success. Inbox and outbox patterns support safe local processing; they do not guarantee remote exactly-once delivery.[10]

Test crash, timeout and duplicate-delivery paths before production.

Test Recovery Before Production

Once recovery rules are defined, test them under failure before release. Check external side effects, not just whether the workflow finishes. Prioritise tests by irreversible external impact: payments and refunds first, infrastructure changes that could cause outages next, then customer messages. For each action, record the maximum duplicate external effect, reversibility, provider support for idempotency and status queries, and expected recovery time.

Test Failures Before and After Provider Acceptance

Inject crashes before submission, after provider acceptance but before local persistence, and after persistence but before workflow acknowledgement. Add worker restarts, timeouts, duplicate deliveries and late responses. Treat timeouts as unknown, and check how recovery behaves in that state.[9] Verify that recovery reuses the same operation and key without duplicating the external effect. These tests should show whether the provider or your own ledger carries the recovery burden.

Test concurrent attempts, changed payloads under the same key and expired key retention. Reject or quarantine payload changes. When keys expire, require reconciliation rather than blind resubmission.

Test success, failure, processing and unknown responses. Require eventual completion only when the provider’s recovery contract supports a reliable status query, idempotency mechanism or durable webhook path. Otherwise, manual review may remain the correct result.

Give Operators a Clear View of Uncertain Actions

Use an access-controlled audit trail to show intent, attempts, timestamps, provider references, response classification and reconciliation results. Exclude credentials and sensitive payment data. Track duplicate-suppression count, unresolved-operation age and recovery time.

Test uncertainty lasting hours or days. Operators must be able to trace the original payment and fulfilment outcome through audit evidence. Give them a procedure to inspect provider evidence and reconcile or resume the original operation - not a button that silently submits another payment.

Review Agent Recovery with Hokstad Consulting

For a recovery review, Hokstad Consulting should inspect tool side effects, provider idempotency support, retry contracts and reconciliation paths. The findings should then guide deployment gates and integration controls.

Conclusion: Verify Safe Recovery Before Deployment

A successful replay does not prove recovery is safe. You also need durable action identity, idempotency, bounded retries and reconciliation. Use this pre-release checklist to check those controls before deployment and control external effects:

• Replay: Reuse recorded inputs and results without repeating completed external actions.
• Idempotency: Save the action ID and parameters before submission. Reuse them across retries only if the provider supports idempotency.
• Uncertain outcomes: Limit retry attempts and elapsed time, using back-off and jitter. Reconcile unknown results before retrying, and pass unresolved cases to an operator.
• Webhook handling: Verify signatures, deduplicate event IDs and protect downstream effects separately.
• Observability: Link workflow, action and provider references. Assign every unresolved case an owner, a deadline and an audit trail.

FAQs