Why internal agent projects stall after the first good demo

Anything you do for the first time gives you a large improvement. That is the part that misleads people.

Why do internal agent projects show strong early results?

Because the first pass at any system captures the easy gains, and there are a lot of easy gains available. A team assembles a harness, points a capable model at their environment, tunes it by hand against cases they recognize, and the improvement over doing nothing is large and immediate.

The initial result is real. What it does not tell you is anything about the slope from here.

What causes the plateau?

Running out of a way to tell whether the next change is an improvement. Early tuning works on judgment, which is fine while the problems are obvious enough that any competent engineer can see them. Once the obvious failures are fixed, the remaining changes are ones where opinions differ and nobody can settle the argument.

At that point the team is still working, still shipping changes, and no longer improving in any measurable direction. The plateau is not a capability ceiling in the model. It is the absence of a signal to climb.

Why isn't dropping a better model into an existing harness enough?

Because a fixed agent harness with a new model inside it does not reliably work better. Swapping the model changes the behavior of a system that was tuned around the previous one, and the parts of the harness that were compensating for the old model's weaknesses are now doing something else.

Making a model perform in a real domain requires a loop around it rather than a slot to drop it into. That loop is what turns a capable model into a system that gets better over time, and building the loop requires evals.

Why is building evals a machine learning problem?

Because the failure modes are the ones that machine learning practice exists to catch. An eval set can be wrong, meaning it measures something other than what you care about, and it will still produce numbers that go up. An eval set can be too easy, in which case every change passes and you learn nothing.

Both failures are invisible from inside. The set produces a score, the score improves, and the team believes it is making progress. Knowing how to construct a set that discriminates between a good change and a bad one, and knowing how to improve against it without overfitting to it, is domain expertise rather than a task a strong engineer picks up on the way to something else.

What's the alternative to evals in practice?

Tuning by feel, which works exactly once. Improving a system on judgment produces a strong first result for the same reason any first attempt does, and then flattens, because judgment cannot resolve the close calls that make up everything after the first round.

Resolve AI's model training and orchestration decisions are made against eval results rather than against impressions. The reason is not rigor for its own sake. It is that there is no other way to keep improving once the easy wins are gone.

Can building the eval set be automated?

Substantially, which is what makes it viable at the volume required. The work is finding signals that can be turned into evaluation cases without a person in the loop for every one, and Resolve AI now generates a large share of its cases this way, with the time cost per case reduced to hours.

The automation is not the interesting part on its own. It matters because eval coverage has to keep pace with the system, and a process that requires an expert to hand-build every case caps how fast the system can improve regardless of how good the team is.

What does a team need before it can improve an agent at all?

Evals, the environments to run them in, and people who know how to build both for this specific domain. A team that has a capable model and an integration layer but none of these can ship something that works and cannot answer the question of how to make it work better.

That question tends to arrive later than expected and all at once. It shows up as a specific use case where the agent is wrong, no clear reason why, and no mechanism to determine whether an attempted fix helped.