Hero of prod: Pedro Sardela

That's the motto we live by as Red Flag Engineers at Toast, where more than 180,000 restaurant locations run on our POS and roughly $240 billion in payments flows through it every year.

What makes the role unique is that we do not own a single product end to end. We own the escalation process itself. We step in when support cannot resolve an issue on its own, investigate what is happening, and decide whether we can address it ourselves or need to bring in the domain team that owns the service. That means acting as the generalist in the room, narrowing the scope, providing context, and helping move mitigation forward quickly, all without being the deepest expert on every service involved. In many cases we are also asked to operate a bit like an engineering team, running the services needed to support those fixes. When we are doing the job well, no one is really thinking about us, not the restaurants, not the customers, and not even the engineering teams we work alongside.

My team goes through roughly 500 tickets per month, and we're a relatively small group handling that volume. Historically, every one of those tickets required human review and manual handling, which meant a lot of repetitive operational work. As AI started accelerating how teams build and ship, it made the limits of that model more obvious, not necessarily because ticket volume suddenly changed, but because the pace and complexity of the work made a fully manual process harder to sustain. At that scale, the challenge stops being just how to work the queue faster and becomes how to build a system that can handle more of that process with less human touch.

But then AI became a real part of how we thought about the work. It was rough at first. My first attempt (over 18 months ago) at building an agent for the queue was bad enough that it turned me off for a month. But over time, both the tools and our understanding of how to use them got better, and now we're at a point where we can actually rely on AI for scaffolding and helping shape the workflow.

From coding tools to R&D tools, to using Resolve to help our investigations out on the Red Flag team, there's AI tooling every which way at Toast. A lot of that investigation work my team does was manual and meant digging through Splunk, doing forensic analysis on logs and browser reproduction files, and trying to understand the code domain by hand. Now we hand off more of that to Resolve whether it be a complex ticket and I need a one-shot overview immediately, or when I'm a little underwater on backlog and need something to kickstart the investigation, or even when a Red Flag arrives with an investigation our customer-response team already kicked off upstream. The impact is that we now are never left guessing, and we can, with reasonable confidence, assess the Red Flag properly before it goes to the domain owner, so they're getting a focused scope and evidence instead of a shrug.

I think there's been a real mindset shift in how we think about the job. What used to be 80% doing the work and 20% steering is becoming 80% steering and 20% handling edge cases, especially now that we're more enabled and more confident relying on AI tooling for scaffolding and workflow support. In that model, the point of spending time on the strange issues isn't just to resolve them in the moment, but to feed those learnings back into the broader system and improve the process. The goal is that we don't spend next week's 20% dealing with the exact same kind of ticket again.

That is really where I think the future is heading. The goal is not to take people out of the loop. It is to let systems absorb more of the repetitive work so we can spend our time steering, making the judgment calls, and improving the process itself. I don't think fully autonomous production is close, and not only for technical reasons. Compliance, regulation, and plain trust all take time. We're always shipping (or breaking) something, and no system we build today will anticipate everything next quarter throws at it. It's a bit like Excel. However good the tooling gets, there is always going to be a need for human flexibility. That's what a human is for.