Hero of prod: Pedro Sardela

That's the motto we live by as Red Flag Engineers at Toast, where more than 180,000 restaurant locations run on our POS and roughly $240 billion in payments flows through it every year.

Owning the escalation

What makes the role unique is that we do not own a single product end to end. We own the escalation process itself. We step in when support cannot resolve an issue on its own, investigate what is happening, and decide whether we can address it ourselves or need to bring in the domain team that owns the service. That means acting as the generalist in the room, narrowing the scope, providing context, and helping move mitigation forward quickly, all without being the deepest expert on every service involved. In many cases we are also asked to operate a bit like an engineering team, running the services needed to support those fixes. When we are doing the job well, no one is really thinking about us, not the restaurants, not the customers, and not even the engineering teams we work alongside.

Outgrowing the queue

My team goes through roughly 500 tickets per month, and we're a relatively small group handling that volume. Historically, every one of those tickets required human review and manual handling, which meant a lot of repetitive operational work. As AI started accelerating how teams build and ship, the limits of that model got more obvious. Ticket volume didn't suddenly change. The pace and complexity of the work just made a fully manual process harder to sustain. At that scale, the challenge stops being just how to work the queue faster and becomes how to build a system that can handle more of that process with less human touch.

That's when AI became a real part of how we thought about the work. It was rough at first. My first attempt (over 18 months ago) at building an agent for the queue was bad enough that it turned me off for a month. But over time, both the tools and our understanding of how to use them got better, and now we're at a point where we can actually rely on AI for scaffolding and helping shape the workflow.

Handing off the digging

From coding tools to R&D tools, to using Resolve to help our investigations out on the Red Flag team, there’s AI tooling every which way at Toast. A lot of that investigation work my team does was manual and meant digging through Splunk, doing forensic analysis on logs and browser reproduction files, and trying to understand the code domain by hand. Now we hand more of that off to Resolve, whether it be a complex ticket where I need a one-shot overview immediately, or when I'm a little underwater on backlog and need something to kickstart the investigation, or even when a Red Flag arrives with an investigation our customer-response team already kicked off upstream. The impact is that we're rarely left guessing, and we can, with reasonable confidence, assess the Red Flag properly before it goes to the domain owner, so they're getting a focused scope and evidence instead of a shrug.

What a human's for

I think there's been a real mindset shift in how we think about the job. What used to be 80% doing the work and 20% steering is becoming 80% steering and 20% handling edge cases, especially now that we're more enabled and more confident relying on AI tooling for scaffolding and workflow support. In that model, when we spend time on the strange issues, we feed those learnings back into the broader system and improve the process. The goal is that we don't spend next week's 20% dealing with the exact same kind of ticket again.

That is really where I think the future is heading: letting systems absorb more of the repetitive work while people stay in the loop, steering and making the judgment calls. I don't think fully autonomous production is close, and not only for technical reasons. Compliance, regulation, and plain trust all take time. We’re always shipping (or breaking) something, and no system we build today will anticipate everything next quarter throws at it. It's a bit like Excel. However good the tooling gets, there is always going to be a need for human flexibility. That's what a human is for.