What engineering leaders are learning at the AI frontier

Last year, we held our inaugural Engineering Leadership Forum where the conversations held then now feel rudimentary and revolved around getting engineering orgs ready to adopt AI. As pointed out by my colleague Manveer Sahota, “many senior engineers [continued] to see AI only as an assistant in writing more code rather than as a transformative force for operational resilience.” Jump to today and there isn’t a single person who disagrees that AI is fundamentally transforming the way software is produced and run in production.

At this year’s forum, we had the opportunity to sit down with three engineering leaders from Toast, Zscaler, and Motive for a live panel, and it was clear how much the conversation has shifted in just a year. Nobody on stage was debating whether engineers should use AI. The conversation centered on how much work to hand over, and what engineers should focus on once AI gives their teams more capacity.

Here’s what we learned.

Trust is paramount

Trust was a recurring theme this year, and there is still an extreme range in the amount of trust engineering leaders are willing to place in AI agents. Prashant Ramarao, Head of Platform Engineering at Motive, is taking a more deliberate approach, keeping humans in the loop until agents prove they can be trusted over the long term. Steve Brewer, SVP Engineering at Toast, and his team are taking a more ROI-driven approach. As AI continues to improve at tasks like code review, Steve sees a point where it will surpass humans and simply make more sense, both economically and from an accuracy perspective, to hand the work over. He expects that same shift to happen across different parts of the software engineering job as AI continues to improve.

There is also the question of trusting the vendors providing these agents. Steve emphasized that companies need to be careful with the vendors they put their bets on and make sure their incentives are aligned with the outcomes of the business. This becomes increasingly important as those vendors take on more work across the engineering organization.

The best AI tools remove work from engineering teams

Each leader had a different red herring metric they told the audience to avoid focusing on, but the one thing they all agreed on is that the best AI tools are the ones that take time-intensive work from an engineer’s hands and allow them to focus on more customer-impacting work.

We’re already seeing what that looks like in practice. AI is taking on work like refactors, runbooks, incident investigation, and other tasks that have historically taken up engineers’ time. Jim Tsiamis, VP and Lead of SRE at Zscaler, wants fewer of his engineers on-call for low severity incidents and more of them focused on bigger ticket items like rearchitecting their infrastructure and making their systems more reliable. At Toast, AI has enabled engineers to solve issues on-demand instead of just finding and declaring them and adding them to the backlog. Steve has even observed a renewed interest across his organization as engineers who once felt like task-doers now feel like innovators as they are able to go out, find, and solve problems on their own.

The same principle is shaping how these leaders think about what to build internally versus what to buy. Jim summed up Zscaler's approach as "buy the commodity, build the differentiator." The commodity here is the work itself. Investigating a low-severity alert is necessary, but it doesn't make Zscaler's product any better than a competitor's, no matter how hard the problem is to solve. If there is an established solution that can take that work off your engineering team's plate, buy it and let your engineers spend their time on the problems that are actually specific to your business. As Steve put it, Toast doesn't sell developer hours.

"Any vendor that can come in and help us move faster, innovate better, and improve our margins all while helping us deliver a better product for our customers, that's a good partner." - Steve Brewer, SVP Engineering, Toast

Buying more doesn't mean building less, though. At Motive, AI has pushed Prashant's team to build more internally. He pitched it to his CTO with a simple analogy: if you want a Ferrari, you need a good road for it to drive on. For Motive, that means a major emphasis on its platform and infrastructure teams and making sure they're building the right foundation for AI agents to operate effectively. An agent is only as effective as the systems it runs on, and that road is something only Motive's own engineers can build.

With the rate at which new AI tools and capabilities are coming out, staying focused on those problems is increasingly important. Steve cautioned against seeing what another company is doing with AI and assuming it should be applied to your organization too. Just because there’s an interesting blog demonstrating some incredible AI capability doesn’t mean it’s the right area for your team to focus on. Start with the problems that are most important to your organization and figure out where AI can help.

Advancements in AI aren’t decreasing the value of engineers

Despite the sentiment over the last few years that AI might replace the software engineering job, our three guest leaders have each experienced the opposite outcome. As AI tooling has strengthened and offloaded real work from their engineering teams, they are starting to hire more engineers than before AI.

When posed with the question, "are you hiring more agents or more engineers?" all three pointed back to the work they now want their engineers spending more time on. Jim wants his production engineers focused on making Zscaler's systems more reliable and resilient rather than solving incidents or going through logs. For Prashant, it comes down to where his SREs spend their time: "We don't want our SREs sitting in incident rooms trying to debug incidents. We want them looking at the next architecture we can implement to increase our reliability. That's way more important to us." Steve sees agents making his engineers more valuable by allowing them to solve more important customer problems instead of spending their time on manual work and maintenance.

As AI takes on more of the work that has historically consumed engineers' time, these leaders see more opportunities for their engineers to solve higher-order problems, and they are hiring accordingly.

What will the next year bring us?

Last year, we were asking how engineering organizations should prepare for AI and what its impact on software engineering might look like. A year later, many of those questions are already being answered in practice. AI is taking real work off engineers' plates and leaders are getting more comfortable handing it over. At our customers like Toast, Zscaler, and Motive, that shift has given engineers more time for the work their leaders care about most, and all three are hiring more engineers than before. If the progress between our first two forums is any indication, next September's panel will have even more to share, and we're already looking forward to it.