Building agents for on-call and incidents?

Guardrails and governance: holding an AI agent to the standard of a high-quality engineer

Claire Yin, Member of Technical Staff at Resolve AI, on what it takes to let an agent take real action in production, where evals end and guardrails begin, and why there’s no trade-off between safety and efficiency.

Why do agents need guardrails?

You can only predict so much of how an agent will act. The same task won't always follow the same path: the model varies, the environment changes, new context shows up. Guardrails exist for exactly those unpredictable situations. Whichever path it takes, the agent can't do anything destructive or harmful to the system.

What's the difference between evals and guardrails?

Think of the agent like any engineer you'd bring onto the team. There are two different things you do with a new engineer. You train them and put limits on what they can do.

  • Evals are the training part. They're how the team checks the agent's performance and works to make it more predictable, much like you'd develop a person over time.
  • Guardrails are the limits. Training only goes so far, because even a good engineer sometimes does something just wrong. A guardrail prevents that action from occurring at all, regardless of how well or poorly the agent performs.

The limits used depend on the situation. In some cases, the agent is normally allowed to take the action, but in certain scenarios it is stopped, much like a new engineer might be trusted to restart a service in staging but not in production. Other times, the agent simply doesn't have access, so the guardrail is absolute.

What standard does Resolve AI hold before letting an agent act in production?

An agent is like a high-quality engineer. If the agent is going to take actions as an engineer would, it is held to the same standard as an engineer. That means traceability and auditability, with nothing executing in the dark. Every action leaves a record of what ran and where, the same trail you'd expect from a person making the same change.

That holds even after the agent hands off to a person. A common pattern is the agent suggesting an action and an engineer clicking the button to run it. At that point the agent has technically stopped, and the human is the one acting, but the person wouldn't be doing anything without the agent. So the audit trail extends to wherever Resolve AI touches. The action remains traceable to both Resolve AI and the user, to the agent who proposed it, and to the person who approved it.

Is there a trade-off between safety and efficiency?

No. Resolve AI moves backward from safety, and efficiency comes later. There's no balancing act being run. Safety is the starting point, and efficiency gets recovered afterward rather than traded away up front. That's also why guardrails are easy to undervalue. When one is doing its job, nobody hears about it.

Why is explainability part of safety?

Knowing who took an action isn't enough on its own. When something goes wrong, the question that matters is why it happened, and AI can feel like a black box exactly when you need that answer. The standard is the same one you'd apply to a person. You expect an engineer to explain their actions, so agents are held to the same standard of accountability. In practice, that means you can go back to Resolve AI after the fact and ask it why it did what it did.

Why does an external tool need stronger guardrails than an internal one?

Because the consequences land in someone else's production. A team building agent tooling internally answers to itself. If something goes wrong, the risk was theirs to take, and the damage is theirs to fix. Resolve AI operates inside customers' environments, so that latitude doesn't exist, and the guardrails have to be inherently stronger than what an internal team would accept for itself.

The difference is most apparent when a new model comes out. Each release blurs the line between efficiency and safety. The model can do more, so the pull is to let it, and an internal team more or less has to move with that line as it shifts. Resolve AI also moves the line, but on its own terms. The investment in quality and evals means the team knows exactly how to move it, and it happens in a slow, progressive manner instead of at the pace of model releases.

What earns an agent the trust to change a customer's environment?

Everything Resolve AI can do is defined up front in terms of the actions it's allowed to take, which keeps its behavior controllable and tractable. There's no action available to the agent that nobody decided to give it. However unpredictable a given path gets, it can only ever end in an action from that defined set.

The gate matters because customers are granting two different things, and they're not the same size. Reading from production means being trusted with what the customer's systems reveal. Executing changes means being trusted to run those systems. The engineer standard, the guardrails, the audit trail, and the bounded action space all exist to make it reasonable for a customer to let an agent change their environment rather than just look at it.