The Role of LLM Evals in Automated RCA

An agent investigating a production alert surfaces a hypothesis. It is well structured, cites real log lines, names a service, and reads like something a competent engineer would write. It’s great. And it’s wrong.

But in the moment, how can the on-call engineer know? How much confidence should they place in the automated root cause analysis? How would anyone tell this answer from a correct one?

That's the role of evals in automated RCA. Evals measure whether an AI system's investigations are actually right, scored against real cases with known outcomes, so confidence rests on a track record rather than on how an answer reads.

Key facts

• An eval scores an agent's investigation against known outcomes from real incidents, not against how good the answer sounds.
• A production eval measures four behaviors: correct root cause, abstention when evidence is thin, a valid causal chain, and calibrated confidence.
• Accuracy climbs fast at first, roughly a third of incidents within days, and the last stretch takes orders of magnitude more work.
• A model judging another model's answer measures coherence, which is exactly the property that diverges from accuracy.
• Autonomy decisions should rest on measured error rates per alert class, and every one of those numbers comes from an eval.

A coherent answer is not an accurate answer

Language models optimize for output probabilistically. Given partial evidence, a model produces the most internally consistent account available to it, and that account reads with the same confidence whether it rests on complete evidence or a third of it.

Let’s go back to that opening incident. The engineer works the theory, and forty minutes later the actual root cause turns out to sit three services upstream, in a dependency the agent never queried. The gap that produced the miss could live in any layer: thin context, shallow reasoning, or a model out of its depth. The lack of evals is what let it reach the engineer looking like a success. Nothing was measuring whether investigations like this one tend to be right, so a wrong answer had no way to look different from a right one.

In a production environment, this is deadly. A coherent but wrong answer in a code review costs a comment. During a live incident, it costs everyone who believed it the remaining time for the investigation, plus whatever the wrong action does to a system that was already degraded.

A better system prompt doesn't fix this because the model isn't poorly instructed. The problem is that nobody downstream can distinguish a well-evidenced finding from a well-written one. Closing that gap takes measurement, and measurement is what evals are.

Why are evals necessary for AI that drives to Automated RCA

An eval for automated root cause analysis is a reproducible test that scores an investigation against a known outcome. Eval discussion in the AI industry mostly stops at correctness, did the system name the validated cause, and that's also where a program should start. Resolve AI started there and added layers as the system matured, because the later layers are where systems that demo well fall apart. A mature program measures four things:

1. Whether the agent reached the correct root cause.
2. Whether it declined to answer when the evidence was insufficient.
3. Whether its causal chain holds.
4. Whether its stated confidence matched how often it was actually right.

What is the curve to define evals along for RCA?

Root cause accuracy for an agent investigating production alerts follows a pattern that is sufficiently consistent to plan against.

What matters most in this curve is how fast the early wins arrive. A capable engineer with tool access and a frontier model can get the common, single-service failures correctly diagnosed within a couple of days. That result reads like the start of a steady climb, but it's close to the ceiling of what the approach produces without serious investment, and without evals, nothing distinguishes the two situations at the time.

Teams that build this internally describe the same experience. Deceptively good results come quickly, and the last stretch takes orders of magnitude more work than the first. The reason has nothing to do with model quality. The hard middle of the curve is a specific set of investigation types:

• The root cause is a chain rather than a single service.
• The evidence is spread across systems that don't share identifiers.
• The correct answer is that the cause isn't observable from the available data.
• The failure being investigated is itself downstream of another failure.

None of these get tractable through better prompting. They get tractable when you can measure which of them you're currently failing.

5 things evals need to for AI that investigates production failures

A real eval platform for automated RCA does five jobs.

1. Positive evals confirm that the agent has identified the correct root cause. This is the obvious one, and it's where most eval work stops. It tells you how often the system is right and nothing about how it behaves when it isn't.
2. Negative evals confirm the agent abstains when evidence is insufficient. An agent that produces a confident hypothesis for every alert scores well on positive evals and is dangerous in production, because thin-evidence alerts are exactly where a wrong answer costs the most. Abstention has to be a scored behavior with its own test set, built from incidents where the correct output was some version of "the evidence doesn't support a conclusion, and here's what would be needed." Very few teams build this, and it's the clearest signal that an eval program is real.
3. Causal chain validation traces the full path rather than the surface signal. An agent that names the service where the symptom appeared hasn't found the root cause, and scoring it as correct teaches the system to stop at the first plausible point. The test set has to encode the chain, and partial credit has to reflect how far along it the agent actually got.
4. Confidence calibration checks that stated certainty maps to measured accuracy. High confidence should correspond to a high hit rate, and low confidence to a lower one. Calibration is what makes an agent's confidence usable as a routing signal. Without it, a confidence score is just a number the model generated, and using it as a threshold for autonomous action means granting autonomy based on something no one has verified.
5. Nightly regression runs validate every model and orchestration change before it reaches production. Evals that run when someone remembers to run them measure the system at unrelated points in time. Evals that run on a schedule and gate releases show you the effect of every change.

Why a generic LLM judge can't grade investigations

There are three basic ways to score an agent's investigations.

• Programmatic checks against ground truth. Compare the agent's conclusion to the validated cause of a labeled incident, and check verifiable signals along the way, like whether its queries executed and returned real data. This is the backbone when a labeled set exists.
• Human expert review. Senior engineers read investigations and grade them. It's accurate, and it doesn't scale. A nightly regression suite across hundreds of cases would consume the exact engineers the system is supposed to relieve, and two reviewers often disagree.
• LLM-as-judge. Another model scores the output against a rubric. It's cheap, it scales, and it's the industry default for open-ended output.

A real eval platform combines the first and third, anchored on labeled incidents. The trap is reaching for the third alone, because for production RCA a genericLLM-as- judge fails in a specific way. A LLM-as-judge without access to the same production evidence as the system under test can only assess whether the answer is coherent and well reasoned, and coherence is precisely the property that diverges from accuracy. The judge ends up measuring the failure mode instead of catching it.

How evals set the path to autonomy

Autonomy is an eval outcome. An agent earns the right to act on an alert class when its measured error rate on that class clears the bar, and it loses that right when the rate slips. The agent recommends now and acts later, once trust builds. Trust in that framing accumulates, and the move from human in the loop to human on the loop happens when someone senior is comfortable enough.

The alternative is a boundary defined as explicit policy the team sets, changes, and audits. That's only possible if you can state, with numbers:

• Which alert classes does the agent handle, and at what accuracy?
• How often it abstains correctly.
• What is the error rate for the specific action classes you're considering delegating?

Every one of those numbers comes from an eval. Staged that way, the progression has measurable checkpoints. Investigation with human approval is where most teams should operate now, and the value there is speed, consistency, and coverage rather than autonomy itself. Zscaler runs more than 150,000 alerts a month this way, and Resolve AI helped DoorDash Ads cut time to root cause by up to 87% inside this stage. The value arrived well before the autonomy did.

• Narrow autonomous remediation on reversible action classes comes next, earned by a known and measured error rate on those classes.

How evals make model upgrades safe

The orchestration underneath an RCA agent has a half-life of roughly three months:

• Frontier models ship, and relative strengths move.
• Routing that was right in March is leaving accuracy behind by June.
• Prompts tuned to one model's behavior transfer unevenly to the next.

Without a regression suite, every upgrade is a change made in the dark. The system might be better afterward. It might also have lost ground on an incident class nobody tests for, and the first sign will be an incident that goes badly for reasons no one connects to a model change six weeks earlier. Teams in that position tend to stop upgrading, which trades one form of decay for another.

This is where evals stop being a quality practice and become what lets the rest of the architecture move. Model orchestration is only tunable if changes are measurable.

How Resolve AI approaches evals

Resolve AI runs a domain-specialized eval framework using the components described on this page.

• It's built on datasets Resolve AI creates itself, synthetic incidents generated in simulated environments alongside internally labeled cases. Customer data is never used to build them.
• It scores investigations on accuracy, abstention, causal chains, and calibration.
• It runs nightly and gates every model and orchestration change before it ships.

That's what lets the platform absorb new model releases, task by task, without regressions, and it's the basis for staging autonomy with customers as an explicit, measured policy.

Salesforce reduced MTTR by roughly 60% by having Resolve AI operate this way. Book a demo to see the eval framework behind the agents.

FAQ

What is an eval for automated RCA?

A reproducible test that scores an agent's investigation against known outcomes from real incidents. A production-grade eval measures four behaviors: whether the agent found the correct root cause, whether it abstained when evidence was thin, whether its causal chain holds, and whether its confidence matched its actual hit rate.

Why can't an LLM judge grade investigations?

A judge without access to the production evidence can only assess whether an answer is coherent and well reasoned, and coherence is the property that diverges from accuracy. Grading an investigation requires telemetry, deployment history, topology, and a confirmed cause.

What are negative evals?

Tests that score abstention. They're built from incidents where the correct output was that the evidence doesn't support a conclusion, and they catch the agent that produces a confident hypothesis for every alert. Very few teams build them, which makes them the clearest sign an eval program is real.

What is confidence calibration in automated RCA?

A check that the agent's stated certainty maps to measured accuracy, so high confidence corresponds to a high hit rate. Calibration is what makes confidence usable for routing decisions and autonomy thresholds, rather than an unverified number.

Do evals matter if you buy a tool instead of building one?

Yes, they're how you see past the demo. Ask the vendor for their accuracy figure on the test set, the labeling method, and the abstention rate, and ask how often the evals run and whether they gate releases.