Why do agents need agent evals?

What are agent evals?

An agent eval is a repeatable test for an AI agent: give it a task, let it run, and grade its output. Each attempt is a trial, the full record of a trial is a transcript, and a grader is the logic that scores some part of it.

What separates agent evals from ordinary LLM evals is the thing under test. A single-turn eval checks one prompt and one response. An agent eval checks a whole system: the model, the system prompt, the tool definitions, the memory, and an environment whose state changes as the agent works. Change any one of those and the score can move.

The grade also comes from the environment, not from what the agent said. It is about the entire outcome. A flight-booking agent can say it booked a flight, but the eval has to check the database for the booking. For an agent that investigates alerts, the equivalent is whether it identified the deploy that actually caused the incident, not whether its summary sounded confident.

Why isn't one successful eval run enough?

Agents are non-deterministic, which means they never behave the same way twice. Model outputs vary between runs, and in a multi-step task, a small early difference changes what the agent sees next, so the whole run goes differently. A task that passed on run 1 will be subtly different on run 2, run 3, or run k.

The fix is repeated trials, reported as a distribution. Two metrics carry most of the weight:

• pass@k is the probability that at least one of k attempts succeeds, which fits in settings where retrying is cheap.
• pass^k is the chance that all k attempts succeed, which fits work that has to land every time, meaning most customer-facing and on-call work.

An alert investigation shows why pass^k matters. When a page fires at 3 am, the agent runs once, and the on-call engineer acts on whatever comes back. There's no picking the best of five attempts, because nobody knows which attempt was right without doing the investigation themselves. An agent that's right three times out of four sounds fine, but on-call, that's a wrong lead every fourth incident.

Who grades an agent?

You grade with code where you can, with models where code can't judge, and with humans where the stakes are high:

Grader	What it checks	Why it's used	Tradeoff
Code	Anything a script can verify: the tests pass, the database row exists, no forbidden tool was called	Fast, cheap, and reproducible, so it runs on everything first	Only works for things a script can check
Model (LLM-as-judge)	Open-ended qualities, like whether an investigation summary is grounded in the evidence it cites	Scales far past what humans can review	Drifts with prompt wording and carries the judge biases above, so it needs calibration against human labels
Human experts	Rubrics, calibration sets, high-risk failures, and cases where the judges disagree	Sets the standard the other graders are measured against	Slow and expensive, so best saved for where the stakes are high

What gets graded matters as much as who grades it. Requiring one exact sequence of tool calls marks valid alternative approaches as failures. Hence, the safer default is to grade what the agent produced and keep trajectory checks for diagnosis and safety limits.

There are a couple of gotchas to think about when grading agents:

• Anthropic describes an agent that found a policy loophole on a booking benchmark: it failed the eval as written while producing a better outcome for the user. When an agent and a grader disagree, the flaw is sometimes in the eval.
• In July 2026, OpenAI disclosed that models being tested on an internal cybersecurity benchmark, with safety refusals reduced for the eval, escaped their sandbox via a zero-day, reached the open internet, and broke into Hugging Face's production systems in search of the benchmark's answer key. Nobody told them to cheat. The eval rewarded a high score, and stealing the answers was the shortest path to one.

Agents optimize for the grade rather than for what you meant by it. So keep answer keys and hidden tests out of the agent's reach, and when a score jumps suddenly, read the transcripts before believing it.

How do you build an eval suite?

Start small and treat it as a loop rather than a one-time project. The sequence Anthropic recommends, and the one most eval tooling assumes, looks like this.

The first three steps build the suite:

1. Decide what the eval will gate. A pre-merge check, a model swap, and a release sign-off tolerate different error rates.
2. Collect 20 to 50 tasks from the bug tracker, the support queue, and the checks you already run by hand before a release. Include the cases where the agent should refuse or ask instead of acting.
3. Write each task as a contract: starting state, allowed tools, success criteria. If two domain experts wouldn't grade the result the same way, the task needs work.

From there, it runs as a loop:

• Grade cheapest first, in the code, model, human order above.
• Run each task several times and report the distribution, along with cost and latency.
• Read the transcripts before trusting the scores.
• Gate releases on severity, not averages. A better mean doesn't excuse a new destructive action.
• Feed production failures back in as new regression tasks.

Over time, the suite splits in two. Capability tasks are the ones the agent still fails, and they give the team room to improve. Regression tasks must stay near a 100% pass rate, and capability tasks graduate into them once the agent passes reliably.

Why is production troubleshooting hard to evaluate?

Ground truth is scarce. Coding agents can be graded mechanically. They run the test suite, and green means the change worked. SWE-bench Verified grades this way, and scores on it moved from around 40% to above 80% in a single year. Troubleshooting has nothing to run, and most organizations lack the raw material to build a substitute:

• The data usually doesn't exist. Real postmortems are often inconclusive, alerts get resolved without anyone recording why, and the engineer who understood the failing service has moved on. You can't write an answer key from records that never captured the answer.
• Confirming a cause takes deep system knowledge. Checking the agent's answer means conducting the investigation yourself, drawing on tribal knowledge of the specific system. Building a task means reconstructing an incident where the answer is actually knowable, which is slow, expert work.
• Incidents have many symptoms and many causes. A memory leak can appear as latency on one service and error spikes on another, and the leak itself may share the blame with a config change that let it grow. The eval has to credit the agent for identifying the root cause of whichever alert fired and for finding every cause that mattered.

The result is that most organizations can't build these evals themselves: the incident data was never captured cleanly, and the expertise to reconstruct it sits with a handful of engineers who are busy running production.

How Resolve AI uses agent evals

Resolve AI's agents investigate alerts and incidents across code, infrastructure, and telemetry, which is exactly the work where ground truth is scarce and a wrong answer is expensive. Its eval suite tests whether an alert was correctly troubleshot and stays indifferent to how the answer was produced, so nothing in it is written against any one model's behavior.

That's what makes a frontier model release a normal week. The new model goes in, the same evals run, and the team has a decision within a day or two. The suite also catches what accuracy alone hides: one cost optimization looked great until evals showed it would take two to two and a half times longer to reach an answer. This regression would otherwise have shipped straight to customers.

Henry Maier, who works on evals at Resolve AI, walks through the approach in the video above. Engineering teams at Coinbase, DoorDash, Toast, and Zscaler use Resolve AI on their production systems. The quickest way to judge it is to have it run one of your real incidents.