I trace an AI failure backwards - and test the suspected cause before blaming an agent. In the article’s hypothetical refund case, £150 exceeds a £100 approval limit by 50%. That tells me what went wrong, but not why.
My approach is to:
- Preserve the failure: save the request, output, versions and expected result.
- Follow the chain: connect agent hand-offs, tool calls and state changes to find the first broken contract.
- Test one change: replay safely, checking the suspected defect and the approval safeguard separately.
- Check the fix: test retries, concurrent writes and nearby cases, then add regression tests and monitor repeat failures.
The rule I follow: a better final answer is not enough. I need to check that the failed contract is repaired - not merely that another check hides the fault. With incomplete traces or a single successful replay, I keep the cause provisional.
::: @figure 
{How to Trace Root Causes Across AI Agents}
:::
Trace the failure backwards
Follow the output's dependencies
Use the recorded trace fields to work backwards through the workflow. Start with the final response span: identify the agent or coordinator that produced it, the model version, and any wrong or missing fields. For each affected field, list the upstream inputs that shaped it. Map those dependencies as a causal graph, then follow its edges backwards through specialist agents to the coordinator’s routing and merge decisions.
Use span links, message IDs and state-version references to connect shared-state values, cached results and asynchronous messages outside the parent branch. At tool boundaries, inspect arguments, raw responses and timeouts. At hand-offs, compare what was sent with what was received. Treat an agent’s explanation as a claim to check, not proof of its reasoning.[1][3][4] Once the data path is clear, check whether stale or out-of-order state supplied the wrong value.
Check event order and state versions
With the data path mapped, check when each dependency was read and written. Establish whether the faulty value entered before later retries or downstream writes. For every state read, record the exact version consumed, the span that wrote it, the previous version and whether the write committed. If you cannot establish the version or writer, mark that dependency as unknown.
Give each retry its own span, linked to the original attempt. Record its reason, idempotency key, back-off interval and acceptance result. Mark parallel branches as concurrent unless a causal link proves otherwise, and inspect the join that selected their results. For cancellations and timeouts, record the initiator, deadline and any partial writes.
Establish order using message-send and receive events, local sequence, and writes tied to reads of their versions. Use timestamps only to measure timing; arrival order is not execution order.[5][6]
Agentic AI Course: Stage 7 - Lesson 10: Trace a Complete Multi-Agent Run
Separate the first fault from later errors
Once you’ve mapped the event order, separate the first contract breach from the errors it caused. Label the earliest breach on the dependency path as the first visible contract breach. The defect may have started earlier but only become visible at that point.
If parallel branches or missing trace data prevent attribution, record the first failed event, span or state version. Keep the finding provisional until replay confirms it. Use this first divergence to choose which boundary to test next.
Classify each suspected defect
At each traced boundary, check for defects in factual values, payload structure, policy compliance, tool-result handling or staleness. Classify each defect by its origin:
- Input defect: The source material was already incorrect or incomplete.
- Introduced here: The current agent, tool or transformation turned valid data into invalid data.
- Inherited: The step passed on an earlier defect.
- Independent: The defect arose separately in another branch.
Compare expected and actual values at each boundary, checking both payload content and metadata.
Record a safeguard gap separately from the defect it should have stopped. An omitted policy field and a missing approval check are two different findings.
Missing telemetry is an evidence gap, not proof of what happened. No recorded approval-check span does not mean no check ran: the check may lack instrumentation.
Compare symptoms with the first contract failure
Use this distinction when reviewing refund failures.
Visible symptom First divergence Contributing fault Downstream consequence Refund issued without required approval Approval threshold omitted from the hand-off The receiving agent accepts incomplete context; a separate approval safeguard is missing or bypassed The refund agent treats the transaction as below the approval limit and issues it incorrectly Refund amount is wrong The transaction amount or currency is first altered in an upstream transformation The receiving agent trusts the altered value without validating it The policy decision is made against the wrong amount Explanation cites an inapplicable policy The policy lookup returns the wrong policy version or the hand-off omits policy identity The agent does not verify policy version or effective date The final response appears justified but rests on invalid policy context
Use that diagnosis to choose which agent boundary to validate in the next section.
Restore the £100 threshold while keeping every other variable fixed. If this consistently stops the incorrect refund, the omission is causal.
Test contracts at agent boundaries
Start at the first boundary where the trace diverged. Check whether it caused the first contract breach or simply passed on a symptom from elsewhere.
Validate hand-offs and tool results
Check both sides of each hand-off. Compare what the sender sent with what the recipient received. Check required inputs, identities, task scope, assumptions, permissions, status, schema version and any transformation. Record the trace ID, parent and child span IDs, message or request ID, sender and recipient, transport metadata, timestamps, retry number and schema version.[7]
For tool calls, check why the agent chose the tool. Compare the raw response with the parsed object the agent consumed. Check units, currency, types and error handling - not just the success status. Validate arguments before execution, and reject unexpected response schemas before parsing.[2][9]
If neither the hand-off nor the tool call caused the fault, move to shared-state writes.
Use this table to record candidate faults, not findings. Before attributing a downstream failure to a boundary, run a confirmation test that isolates the fault there.
| Boundary | Observable evidence | Suspected fault | Confirmation test |
|---|---|---|---|
| Agent hand-off | Sender/recipient IDs, trace ID, schema version, required fields, delegated goal, assumptions and validation result | Missing field, wrong recipient, scope drift or lost trace context | Replay the same payload through the recipient’s schema validator; compare it with a known-good contract fixture |
| Tool selection and arguments | Tool name/version, exact arguments, permissions, message or request ID and validation status | Wrong tool, invalid argument, excessive permission or stale parameter | Run the recorded arguments against a stub or sandbox; assert the expected tool, types, ranges and authorisation |
| Tool response parsing | Raw response, HTTP/status code, timeout, retry count, parsed payload and parse errors | Truncation, schema drift, bad type conversion or parser dropping a field | Feed the raw response to the parser independently; compare the parsed object with the agent input |
| Shared-state write | Writer identity, prior version, new version, field diff, commit order and conflict result | Lost update, overwrite after a newer write or field mutation without permission | Run concurrent writers; assert conflict rejection or the documented merge result |
| Retry to side effect | Idempotency key, remote request ID, commit status and retry history | Duplicate side effect after timeout or retry | Replay with the same idempotency key and verify one committed operation; test a timeout after remote commit |
Check state ownership and write order
Use the same trace to check who owns each field and which version each agent read. Identify the authoritative owner and permitted writers for every mutable field. Record proposed changes, the committed version and commit acknowledgement. Then check whether the recipient used the intended snapshot.[8]
Reject stale writes at storage. Require the submitted version to match the current version, or apply an explicit merge rule. Enforce permitted state transitions and record conflicts instead of silently overwriting data.
For retries with side effects, retain the idempotency key and remote commit result. A timeout may mean the response was lost - not that the operation failed. Emit structured rejection events that include the boundary, failed rule, expected and observed values, trace ID and schema version.
Test the suspected cause with controlled replay
Replay the failure with one change
Start at the first divergence, not the final symptom. Use the first suspected boundary from the previous section to replay only the affected slice.
Keep the original failure record immutable. Assign each replay its own run ID linked to the original trace. Preserve task inputs, prompts, model and agent versions, sampling settings, configuration, tool requests and responses, starting state, timestamps, retry history and scheduling details. Redact sensitive data, but retain stable identifiers or hashes. Begin with an unchanged baseline replay. If it diverges, record the first difference before testing a fix.
Next, change only the suspected fault. Restore the recorded history up to that point and rerun the dependent steps. Intercept external calls so the investigation cannot issue real refunds, orders or messages. Match each recorded tool response to the exact request. Reject unexpected requests rather than returning a mismatched fixture. Compare the two runs at the first divergence, through downstream state transitions and at the final authorised action.
Keep recorded model responses fixed when testing downstream code. If a change alters a model’s input, regenerate that downstream response and treat it as a variable. Repeat both runs, reporting the failure rate and trial count separately. Record the model version and sampling settings.
Missing responses, changing external data or an unknown execution order mean this is a partial replay, not an exact reproduction. Report separately whether the recorded history matched and whether the failure recurred.
Test retries, parallel runs and safeguards
Once the replay isolates the fault, test whether retries or overlapping runs create it. Isolate the affected branch and keep the payload fixed. Test no retries, the original retry policy and bounded retries using the same idempotency key.
Then keep retry settings fixed while testing sequential runs, reversed completion order and delayed reads/writes. Compare attempted actions, committed effects and state versions. A failure that occurs only during overlap points to concurrency. Duplicates that occur only with retries point to a separate retry-amplification fault.
Test the initiating defect and the safeguard independently.
For the refund investigation, restore the missing threshold without changing approval enforcement; then enforce approval while leaving the threshold missing; finally test both together.
Use deterministic assertions to check whether a request without authorisation reaches the payment tool. Keep model judgements in a separate evaluation. If either intervention blocks the action, record its separate role: restoring the threshold can remove the initiating fault, while approval enforcement contains its consequences. Treat one successful replay as evidence, not proof of causation.
Document findings and prevent repeat failures
Record findings and add regression tests
Write a causal record, not a symptom list.
Record the incident ID, UTC start and end times, affected workflow and agents, expected and actual outputs, deployment or configuration versions, trace ID, relevant span IDs, state IDs and event IDs, tool-call results and the sequence of hand-offs.
State the first contract breach in one sentence. Link every finding to that breach and the boundary where it first appeared. Separate contributing faults from downstream effects, and provide evidence for each link. Label conclusions confirmed, probable, possible or not supported. If evidence is missing, record what’s missing, who owns the follow-up and the review date.
Turn each finding into a fix owned by the component responsible for the failed contract. Add trace-linked regression tests for every boundary and the final outcome. Each test should identify its first failed assertion, rather than just flag an incorrect final answer. Run these tests in continuous integration and after changes to models, prompts, tools, schemas, routing or state stores.
Track repeat incidents per 1,000 runs, grouped by workflow version, agent, failure class, environment and first fault boundary. Measure how long it takes to identify the first failing span and resolve the incident, along with the proportion of uncertain incidents. Compare equivalent workloads before and after each fix. More tracing can reveal previously hidden failures; don’t mistake that for a drop in reliability.
Use the same incident record to set trace requirements before the next deployment.
Set trace requirements during agent design
Once you know the cause, build the required trace fields into the agent contract. Make trace continuity an acceptance criterion. Require context propagation across agent calls, queues, tools and state operations; spans for routing, hand-offs, approvals and final responses; and the relevant agent, model, prompt, schema, policy and state versions. Propagated trace and span IDs link work across processes, but they do not prove causation.[7][10]
Define boundary contracts, access controls, redaction, retention and deletion rules. Keep credentials and personal data out of trace payloads.[7][11] Add boundary-isolation tests that break one boundary at a time and check that the trace identifies it. Hokstad Consulting can help teams define these trace and boundary requirements.
These requirements feed the final verification step.
Verify the cause and the fix
Once the trace has isolated the suspect boundary, test the diagnosis through replay. Treat it as a replayable causal hypothesis, not a confirmed cause. If evidence is missing, keep the cause provisional.
First, reproduce the failure without the fix. Then replay with just one change, keeping inputs, state and recorded tool responses fixed. The replay must show that the upstream fault disappears, not just that the final answer gets better. A validator may block bad data while leaving the producer’s fault untouched.
Check the repaired contract and the containment control separately. Record the first breach, downstream faults and failed safeguards as separate items. Confirm both the first breach and the safeguard gap as separate findings. Then check that the fix restores the failed contract and that containment still works when you deliberately supply invalid data. For non-deterministic runs, report the failure rate across repeated trials. One successful run isn’t enough.
Before release, test adjacent cases and failure modes: valid neighbouring inputs, retries, duplicate delivery, concurrent writes and tool failures. Check that the cross-agent chain no longer passes the fault along, commits invalid state or duplicates irreversible actions. Monitor production for recurrence.
Verification checklist: Preserve → reconstruct → isolate the first contract failure → replay → fix → test boundaries and adjacent cases → monitor recurrence.
FAQs
How can I trace a failure when agent logs are incomplete?
Use manual root cause analysis rather than relying on automated hypotheses, which may be biased or wrong. Review the timeline or apply the Five Whys, then check each hypothesis against available logs, metrics and traces.
Look at infrastructure updates, deployment records and configuration changes made 15–60 minutes before the incident. Check whether symptoms appeared before each change so you don’t blame a recent change simply because of its timing. Test assumptions manually and resolve any conflicts between data sources.
What if several faults jointly cause an agent failure?
Separate the initial trigger from the errors it causes. Hokstad Consulting recommends using telemetry to group related failures by shared stack traces, error patterns and logs, rather than treating each signal on its own.
Use distributed traces to find the first faulty span in the request path. Check whether failures line up with recent deployments or configuration changes, then test your explanations against raw telemetry. This helps you avoid treating knock-on errors as separate faults.
How many replays do I need to confirm a cause?
There’s no fixed number of replays. Test your leading root-cause hypothesis against raw logs, metrics, traces and change events. To recover, resume from the last valid checkpoint and rebuild decisions from recorded values rather than generating them again.
If logs, metrics or traces disagree - or independent checks repeatedly return the same error - pause automated processes. Investigate manually using the Five Whys or timeline reviews.