Building agents for on-call and incidents?

Vendor and tool agnosticism: why a root cause agent can't live inside one vendor

Swarit Joshipura, Member of Technical Staff at Resolve AI, on the middle-of-the-night page for a service ten hops upstream, why Resolve AI queries data live instead of ingesting it, and what makes observability integrations so hard to build.

Why does a page for an upstream service leave the on-caller stuck?

It's the middle of the night, and you've been paged for a service that sits ten hops upstream of yours. Somebody else's code change broke something in somebody else's service, and the alert landed on you. You have no idea what's going on, and in another world you'd now have to dig into something you know nothing about.

This is the situation Resolve AI is built for. It takes the signal, correlates it against its very large knowledge graph, pulls the relevant code change, and tells the on-caller the right thing to do about it. And it does that autonomously, before the person paged has had to learn a service they've never touched.

Why can't a root cause agent live within a single vendor?

Data is siloed so the evidence never lives in one place. One signal is in Grafana, another in Splunk, and the infrastructure logs go to a third provider. The correlations don't respect those boundaries. A signal in one tool correlates with a signal in another, which correlates with some code change that went in. The root cause lies at the end of that chain, and the chain spans vendors.

A human engineer handles this by having access to all three tools, and the agent needs the same thing to correlate signals and root cause effectively. Sitting above the data sources rather than inside one of them is what lets it synthesize the information, and it can do that much faster and more efficiently than a person clicking between tabs.

Why doesn't Resolve AI ingest the data?

Many tools pull in all the data and run semantic search on it. This ingestion process creates two problems.

  • Everything is stale. The copy starts aging the moment it's made.
  • You've effectively become a telemetry provider, holding a second copy of data that already has a home.

Resolve AI queries the data in real time instead. The agents craft queries in real time against the sources themselves, just as a human would. Nobody downloads Splunk before investigating an alert. They ask Splunk the question and read the answer, and the agent does the same.

How does the agent know which tool to reach for?

Through a purpose-built agentic harness. The agent is very aware of its goal, which is to root cause, and it's aware of the integrations it has and the tools inside those integrations at runtime. When an alert fires, it knows what it's connected to and how to use it.

That awareness is what makes live querying workable. Rather than browsing every source and hoping something turns up, the agent uses the right tool and query because it knows what each connection can answer. The whole thing is optimized to find a needle in a haystack extremely efficiently.

Why are observability integrations so much harder to build?

Building a connector to Slack or HubSpot is easy. Observability is where the complexity grows. Grafana isn't one source, since it can have Prometheus, ClickHouse, and Tempo all feeding into it. Splunk can have many, many indexes, which raises the question of how to query them efficiently. And every integration has to be maintained, evaluated, and instrumented once it exists.

The integrations are only half of it. The other half is the multi-agent orchestrator that knows which integration to use and how to use it at runtime, which is its own hard problem sitting on top of the connectors themselves.

Both halves serve the same goal. The agent should look as much like an SRE or an engineer as possible, and that's what the team is trying to replicate. An engineer with access to every tool, who knows which one to open and what to ask it, is the standard by which the whole system is measured.