AI Agent Checkpoints: What State Should Teams Store?

I treat an AI-agent checkpoint as a safe restart record - not a saved chat. It should show what the agent intended, what was approved, what completed and what still needs checking. If a £75 refund request times out, I check the provider’s records before retrying: no response does not mean no refund.

Here’s what I keep and test:

• Recovery state: task IDs, versions, accepted decisions, pending actions, approvals and tool-result status.
• Safe restart controls: durable writes before dispatch, stable idempotency keys, worker leases and checks that reject stale updates.
• Protected data: small records in the checkpoint, secure links to large files, no credentials or reasoning traces, and clear deletion dates.
• Proof of recovery: crash tests, approval checks, schema checks and named owners.

My rule: resume from the last confirmed transition, reconcile uncertain effects and stop when permission or required data is missing.

::: @figure {AI Agent Checkpoints: Safe Recovery Workflow} :::

19. Long-Running Agents: Checkpoints, Resume and Crash Recovery

Choose when and how to checkpoint

Once the recovery fields are fixed - task ID, pending action, approval record and tool result status - decide when the system must persist them. Define the recovery-point objective, recovery-time objective, tolerance for duplicate side effects and level of audit detail.

Set checkpoint frequency based on recovery targets, run duration, wait time, throughput and schema-evolution policy. These requirements tell you whether current-state records are enough or whether you also need event history, durable timers and reconciliation. Let recovery needs set the checkpoint boundary, not the storage product.

Save state at execution boundaries

Treat a checkpoint as current only after validation and completion of the durable write. Save state:

• After validating a decision and before taking an external action.
• Before publishing an approval request.
• When entering a timer, webhook or queue wait.
• After reconciling a tool result or business transition.
• Before releasing a lease or handing work to another worker.

Commit the transition and outbox event in one transaction. A separate publisher then sends the event and marks it as dispatched.

Per-message checkpoints suit simple queue consumers, but a worker failure can lose several internal decisions. Per-tool-call checkpoints give precise recovery around risky integrations, but add writes and schema complexity.

Per-step checkpoints are a practical default for multi-stage agents. They preserve completed decision or orchestration stages. A grouped execution step recovers as one logical unit, but each external action still needs its own intent record.

More frequent checkpoints reduce lost work, but increase write volume, storage costs, indexing pressure and migration overhead. Measure checkpoint latency and recovery duration rather than choosing a fixed interval.

Compare snapshots, journals and transactional state

Use snapshots for speed, journals for history and transactional state for current ownership. Durable workflow execution records completed steps and waits, so a worker does not need to stay active throughout a wait.[1][2]

Replay should reuse recorded outcomes, not repeat completed effects or assume a model will give the same answer. Store the accepted decision and relevant configuration, and validate any changed decision again. Neither a database commit nor queue publication proves that an external API action completed.

Resume from the last confirmed transition

After writing the checkpoint, resume only from the last confirmed transition. First, load the latest valid checkpoint and validate its integrity, schema version and workflow version. Then acquire a valid lease and identify confirmed progress, pending waits and uncertain effects. Reconcile those effects before resuming safe, authorised actions, then conditionally persist the next transition.[4]

Store lease ownership, expiry and a monotonically increasing fencing token. Require the expected state revision on every update; if no row matches, stop. Where supported, pass the fencing token to the receiving system so it can reject older workers. Lease expiry alone cannot stop a paused worker from returning after reassignment.[3][4]

Handle tool failures, retries and approvals

Checkpoint data must also record how the agent handles uncertain tool effects, retries and expired approvals.

Record each operation before dispatch

After storing recovery state, record each tool operation before it leaves the worker. A timeout does not prove failure. Track planned, locked, dispatched, confirmed, failed and unknown as separate states. Record the dispatch time and link attempts to the same operation, so recovery resumes from the last confirmed effect - not the last attempt.

The record must also define the approval and idempotency boundaries for safe retries. Store a stable idempotency key, its scope and expiry. Reuse it only when the arguments are identical, and check each integration’s support for deduplication and reconciliation.

Limit retries by both attempt count and elapsed time, using exponential back-off and jitter. Stop on invalid requests, and reconcile unknown outcomes before retrying.[5][6]

Tie approvals to exact actions

Bind each approval to the exact action using a canonical argument hash. Store its expiry, consumption and revocation status too. One approval covers one logical operation, not each retry.

After a long wait, recheck permissions, price, record version, eligibility and policy. Any material change requires a new operation and a new approval.

Log recovery and override decisions

If the status remains unclear, record the recovery decision before any manual override. Use append-only recovery logs for approvals, retries, reconciliation and overrides. Include the actor, reason, state change and a protected evidence reference.

Each override must identify the authorised decision-maker. It must not silently turn an uncertain outcome into success. Keep tokens, payment details and customer messages out of general logs.

On recovery, load the operation record and check the provider’s status using supported identifiers. Then mark the effect as confirmed, wait, or stop automatic retries for manual review.

Protect checkpoint data and limit retention

Classify data and control access

Checkpoints contain prompts, tool outputs and approvals, so protect them as carefully as workflow state. Keep checkpoint metadata separate from prompts, tool arguments, outputs and files. Store sensitive payloads separately and link them by object ID. Exclude credentials and personal data you don’t need; use references rather than raw payment or clinical data. Health data requires special-category safeguards and a lawful basis.[12] If a field is not required for safe recovery, do not retain it.[8][9]

Encrypt data in transit and at rest, including replicas and backups, using separately managed keys. Enforce tenant isolation across storage layers and key access paths. Apply role- and task-based access so recovery users can see only the workflows they need. Log reads, exports, restores and admin changes.[10] Tokenise identifiers only when the token is enough for correlation. Tokenised personal data still needs protection.[7][9]

Access controls don’t make restored content safe to follow. Treat restored documents and tool outputs as untrusted data, not instructions. Before use, validate their schema, provenance, tenant ownership and version. Keep recovered content separate from policy and system instructions, and quarantine anything malformed or unexpected.

Set retention periods by record type

Once data is isolated and validated, set retention periods by record type and recovery need. Keep active state for the workflow’s duration plus a documented recovery margin. Retain prompts, outputs and workspace files for less time than approvals and effect evidence, subject to legal, contractual and governance requirements. Record each category’s purpose, owner, trigger and deletion method.[7][11][14]

Track each object’s version and expiry alongside the checkpoint. Before data needed for recovery expires, retain a lawful, minimised recovery copy or pause the workflow. If that data expires without a copy, mark the checkpoint non-resumable rather than guessing or substituting newer data.

Delete or irreversibly anonymise expired data across replicas, indexes, caches and derived data stores. Document backup expiry, and reapply deletion rules before restored data becomes available. Keep legal holds separate and limited to named records, with an owner and review date.[9][13]

Conclusion: prove recovery works

Once state is minimised and protected, the final check is restoration under failure. A new worker must restore task IDs, prompt versions, tool-result references, pending-action records and approval records, then resume from the last confirmed transition.

Test crashes and deployment changes

Restore checkpoints in both a new process and a newer, schema-compatible deployment. Inject crashes before dispatch, after provider acceptance or completion but before the receipt is saved, and during approval waits. For each crash, check that task IDs, pending actions and approval records survive the restart.

Replay duplicate events and have a stale worker attempt an update. Confirm that stable idempotency keys and version checks block repeated effects and obsolete changes.

Test expired and revoked approvals, changed action parameters and unavailable referenced objects. Measure RTO, maximum lost work and duplicate side-effect count against agreed thresholds. Rerun these tests after material changes to schemas, tools, policies or deployments.

Assign owners for recovery controls

Name owners for checkpoint schemas and migrations, tool contracts, approval policies, retention schedules and recovery tests. Security owns reviews of encryption, isolation and access logging. Require a test report and explicit acceptance of any unresolved test failures before release.

The tests must show that a checkpoint restores safe state, not just data.

Release acceptance checklist:

• Schema migration/quarantine: versions migrate safely, or incompatible checkpoints are quarantined.
• Access control/encryption/logging: cross-tenant access is blocked; encryption and access logging work.
• Idempotency and approvals: keys survive retries; expired, revoked or mismatched approvals block dispatch.
• Provider reconciliation/object validation: unknown effects trigger reconciliation or review, not blind retries. Referenced objects are available, authorised and integrity-checked.
• Retention/deletion: retention jobs enforce deletion without removing active recovery evidence.
• Test evidence: the recovery scenarios above meet agreed thresholds.

FAQs