How AI Agents Trace Root Cause in DevOps

AI agents help DevOps teams find likely causes of incidents faster by linking logs, metrics, traces, and change records in one timeline. In many cases, the first 15–30 minutes of an outage are lost to switching tools and checking data by hand. A good agent cuts that search time by ranking likely causes, showing the proof, and passing the result into the incident process for a person to review.

In plain terms, I’d sum it up like this: AI is good at narrowing the search, but people still decide the fix. That only works if your telemetry is clean, your deployment and config history is recorded, and your service map is up to date. Without that, the output can look certain and still be wrong.

Here’s the full picture in a few points:

• Start with the data. Use shared log fields, trace IDs, common metric labels, and OpenTelemetry across services.
• Add change history. Track deployments, config edits, feature flags, infra updates, and secret rotations with timestamps.
• Map dependencies and owners. The agent needs to know what talks to what, who owns it, and what downtime costs in GBP (£).
• Build a simple RCA flow. Spot abnormal behaviour, line it up with traces and logs, compare it with recent changes, then rank likely causes.
• Show proof, not guesses. Each hypothesis should include the suspected cause, supporting signals, affected scope, and a confidence score.
• Keep people in control. The agent can suggest a rollback or restart, but a human should approve any production action.
• Check cost as well as impact. A small latency issue can still drive cloud spend; both should feed triage.
• Learn from each incident. Feed confirmed causes back into the system so future triage improves over time.

A few numbers stand out:

• 60–80% of outages are linked to changes such as deployments or config edits.
• A latency jump from 250 ms to 1,200 ms can be enough for an agent to start tracing the failing span and matching logs.
• Confidence scores like 0.86/1.0 only matter if they are tested against past incidents.

If I were reducing the article to one line, it would be this: AI-assisted root cause analysis works best when it connects clean telemetry to recent changes, then hands a ranked answer to an engineer for approval.

::: @figure {AI-Assisted Root Cause Analysis: 4-Step DevOps Workflow} :::

From DevOps ‘Heart Attacks’ to AI-Powered Diagnostics With Traversal’s AI Agents

Step 1: prepare the data AI agents need

AI agents trace root cause by lining up logs, metrics, traces and change events on one timeline. AI-assisted RCA breaks down when those inputs are incomplete, inconsistent or split across tools. If the signals are patchy or formatted in different ways, the agent can point to the wrong cause or miss it altogether. Start with telemetry quality. Without that, the agent is guessing.

Standardise logs, metrics and traces across services

Use structured JSON logs with a shared schema: timestamp, service, environment, severity, error code, error message, and trace/span IDs. Those trace/span IDs matter because they let the agent join logs to traces instead of treating each signal as a separate thread.

For metrics, expose latency percentiles, error rates and saturation signals with the same environment labels across services. That way, the agent compares like with like instead of mixing production data with staging noise.

Implement OpenTelemetry for distributed tracing across all service boundaries, including calls between cloud services and on-premises components in UK data centres.

Once telemetry is in a common format, bring in change records so the agent can line up symptoms with releases.

Capture change events from deployments and configuration updates

Research keeps pointing to the same pattern: 60–80% of system outages stem from changes - deployments, configuration edits or infrastructure updates [1]. That makes change events a core signal in any AI RCA workflow, not a nice-to-have.

Record these events in the same observability platform as logs, metrics and traces:

• Deployments
• Config changes
• Feature-flag toggles
• Infrastructure updates
• Secret rotations

Store everything with UTC timestamps, then show incident timelines in UK time alongside UTC.

When change data sits next to telemetry in one platform, the agent can overlay change windows on the incident timeline and work out correlation scores. For example, it can flag that a checkout service deployment finished three minutes before a spike in 5xx errors on /checkout/submit.

Define service topology and ownership metadata

The last input is the service map. This tells the agent what depends on what. Keep an explicit graph of services, databases, queues and third-party APIs, including the direction and type of each dependency. Tag each component with environment labels, hosting type - cloud or on-prem - and provider.

Then add ownership and response details to each service:

• Team name
• On-call rotation
• Escalation path
• Business criticality tier
• Hourly impact (£) for downtime

Include SLO targets too, such as P99 latency under 300 ms and 99.9% monthly availability. That gives the agent a clear yardstick for severity, instead of leaving it to judge against made-up baselines.

Without service ownership and dependency data, the agent can't tell the difference between the service that failed and the services it took down with it.

Step 2: build the AI agent workflow for tracing root cause

Once the agent has standardised logs, metrics, traces and change events, it can line them up on a single timeline and reason across them. The workflow has four parts: ingest signals, filter noise, correlate evidence and rank causes.

If the input is poor, everything after that gets shaky fast. Clean telemetry is what lets the agent move from raw signals to a ranked list of likely causes that incident teams can review.

Correlate anomalies across telemetry and recent changes

The agent begins with the four golden signals: latency, traffic, errors and saturation. When one drifts from its normal range - say p95 latency on the checkout endpoint jumps from 250 ms to 1,200 ms - the next step is to use distributed traces to find the span in the request path where things start to slow down.

From there, pull the matching logs and metrics with the trace IDs tied to that slowed span. Then place recent change events on top of that timeline, looking at the 15 to 60 minutes before the breach. That time window often tells the story. A deploy, config edit or dependency update can suddenly stop looking harmless once it sits next to the telemetry.

Generate ranked hypotheses with evidence and confidence scores

The output should be short and easy to scan: a ranked list of candidate causes. Each hypothesis needs the same structure so responders can size it up without digging around.

• Cause statement: for example, Deployment regression in checkout-service v3.4.1
• Evidence: error rate rose from 0.2% to 7% straight after deploy; new NullPointerException in PaymentValidator log entries appeared; Git commit reference and deployment time in local UK time using the 24-hour clock, Europe/London
• Affected scope: which services, endpoints or customer segments are hit
• Confidence score: for example, 0.86 out of 1.0, plus a short reason such as multiple signals align in time and topology; no competing changes detected

That confidence score can't just be a nice-looking number. It should be calibrated against past incidents so that 0.85 keeps meaning high confidence, not high confidence on Tuesdays and guesswork on Fridays.

Choose the workflow pattern that fits your environment

There isn't one pipeline shape that's right for everyone. The best fit depends on the state of your observability, how well you track changes and how messy or simple your service estate is.

A sensible path is to start with anomaly-first. Add change-first when your change logging gets tighter. Bring in topology-first once tracing is dependable across the estate.

After the agent ranks the likely causes, send the summary to incident response for human approval.

Step 3: connect findings to incident response and human review

Once the agent ranks its hypotheses, send those findings straight into the incident workflow, not off to some separate dashboard. The result should land in the place your team already watches: the incident Slack or Teams channel, the PagerDuty alert payload, or the ServiceNow ticket.

Include the details responders need at a glance: affected services, start time, suspected cause, confidence, blast radius, evidence, and the runbook or owner to alert. Keep facts separate from the suspected cause, and attach the note to the incident record so it updates as new telemetry comes in.[4][5][7][8][9][10] That way, the on-call engineer has enough context to decide whether to step in.

Any action still needs human approval. An AI agent should not trigger a production change on its own. In advisory mode, the agent can suggest an action - for example, rolling back checkout-service to v3.4.0 - but an engineer still needs to check that the rollback target is healthy before anything happens. For higher-risk actions, require explicit approval and confirm the change window first.

Before an action runs, make sure the process checks role-based access, logs who approved it, records the model version, and records the outcome. That audit trail matters during post-incident review and for accountability.[2][3][6]

Technical severity on its own doesn't tell the whole story. A misconfiguration causing modest latency might also trigger aggressive autoscaling, which can drive infrastructure spend sharply upward. So the RCA review should cover both customer impact and cloud cost impact, with cost reported in GBP for a UK audience. That keeps cost inside the prioritisation decision instead of treating it as a separate issue.

Confirmed outcomes should then feed post-incident follow-up.

Step 4: handle model limits and feed results into post-incident work

Once a person has checked the findings, use those confirmed incident details to test the agent’s judgement and make the next investigation better.

Know the limits of AI-based root cause analysis

AI agents are only as good as the telemetry they can connect. If logs are missing, trace propagation is broken, or change records are incomplete, the system can produce hypotheses that sound sure of themselves and still be wrong.

One common trap is recent-change bias. A new deployment or config tweak is an easy thing to blame. But that doesn’t mean it caused the issue. Check when the symptom first appeared. If it started before the change, that change probably isn’t the cause.

Agents also have a hard time with new failure patterns. If logs, metrics, and traces seem to disagree, it’s time to step back and do manual RCA. That might mean using Five Whys or walking through a timeline review. Either way, every hypothesis needs to be checked against raw telemetry before anyone acts on it.

Use the agent to speed up triage. Then test the top hypothesis against raw logs, metrics, traces, and change events.

Compare AI-assisted RCA with manual RCA

Once the cause is confirmed, the job shifts from diagnosis to fixing what allowed the incident to happen in the first place.

Turn confirmed root causes into follow-up actions

Feed the confirmed cause back into the AI system. In practice, one confirmed root cause usually leads to a few follow-up outputs:

• An incident timeline for the post-mortem record
• Backlog tickets for the code or configuration fix
• Runbook updates so the next on-call engineer spots the pattern sooner
• Synthetic tests to catch the regression earlier
• Capacity or scaling changes exposed by the incident

That confirmation label helps future triage. It teaches the agent which signals were causal and which ones were just symptoms. Without this closed-loop feedback, the agent stays useful for triage, but it doesn’t get better at the failure modes that matter most.

Each well-handled incident makes the next one easier to diagnose.

FAQs