Why context engineering is important for AI in production

In production, those decisions matter more than the choice of model, because most failures of AI systems in production are context failures. The model reasons competently over what it sees. It just sees the wrong things, too many things, or things that contradict each other. The gap between a GenAI demo and a dependable agent is mostly a context gap.

What is context engineering?

Context engineering is the practice of controlling everything a large language model sees when it generates a response: the system prompt, the message history, retrieved documents, tool outputs, and any external data the application pulls in. In production, that practice decides more about an AI system's accuracy and cost than the choice of model does.

Why can't you just use a bigger context window?

If the evidence might be anywhere, why not send everything? Attention is why that fails. A transformer reads its context by having every token attend to every other token, and the attention weights are normalized so that they sum to a fixed budget. That budget doesn't grow when the window does. Stuff ten thousand tokens of dashboard output into the prompt, and the deploy diff that matters now competes with all of it for the same finite attention.

The dilution hits reasoning first. An investigation is a chain of steps read from the window: connect the symptom to a service, the service to a deploy, the deploy to a cause. Holding a chain like that together means attending to several facts at once, and a link the model attends to weakly is a link it effectively doesn't have. That's why degradation shows up soonest on tasks that connect facts rather than look one up.

Two properties of the architecture compound the dilution:

• Pairwise relationships grow with the square of sequence length, so a window twice as long gives the model four times as much to relate.
• Training data skews short, which means long-range recall is precisely what models have practiced least.

Anthropic's engineering guidance describes the result as a finite "attention budget" that every new token draws against.

The behavior shows up in measurement. Lost in the Middle, a 2023 study led by Stanford researchers, found LLMs recall information at the start and end of a long input far better than information buried in the middle, exactly where a mid-stream log line tends to sit. Chroma's context rot report tested 18 LLMs, including GPT-4.1, Claude 4, and Gemini 2.5, and found reliability decays as input length grows, even on tasks as simple as repeating text.
The dilution shows up in behavior before anything fails outright. Bury the actual instruction under pages of background and the model's focus stretches thin, so it confidently ignores the formatting rules or negative constraints you specified. And because generation is next-token prediction over everything in the window, irrelevant context adds statistical noise to that prediction, nudging the model toward patterns that have nothing to do with the task.

Inference takes the second hit. Every token in the window bills on every call, so an application that stuffs its context pays for the same bloat thousands of times a day. And since attention work grows with the square of input length, time to first token stretches too, which is how a thorough agent ends up feeling sluggish.

Pointing the biggest LLM at all of production demos well once, then collapses on cost and context rot at real volume. A bigger context window raises the ceiling on all of these problems without fixing any of them.

What makes production the hardest place to get context right?

Production violates the assumptions that make context easy elsewhere:

• The information isn't self-contained
• The state changes while the task runs
• No single source holds the full picture

Logs, metrics, traces, code, and infrastructure state are effectively unbounded, so the real question is never whether to gather telemetry but what to query, at what resolution, and what to discard.

The knowledge required to resolve an incident is scattered by design. Symptoms surface in observability tools; causes hide in repos and deployment history; and the load-bearing facts live in runbooks, Slack threads, prior incidents, and senior engineers' heads. Ownership boundaries scatter it further, since every team documents its own slice its own way. And most of it sits in formats built for human eyes, dashboards that render charts and APIs that paginate for applications, rather than for machine reasoning.

AI agents compound the volume problem on their own. Every tool call, every retrieved document, and every conversational turn stacks onto the next LLM inference so that a ten-step investigation can accumulate context far beyond the original alert.

A generative “AI SRE” demo escapes all of this. It runs once, on curated input, with a person watching. A system running thousands of investigations a day gets no such grace, which is why AI engineering effort in production keeps shifting from prompt phrasing toward context engineering. Teams that ship reliable agents in production treat context as a designed system rather than an accumulation.

What does badly engineered context do to an agent?

It produces shallow reasoning. A model that can't assemble the right evidence rarely admits it. It narrates. Frontier LLMs are tuned for coherent answers, and coherence is cheap, so, given thin or noisy context, the model anchors on its first plausible theory, backfills a tidy story around it, and pages the wrong team with a confident root cause that the evidence never supported.

The failure patterns are consistent enough that Drew Breunig cataloged them:

• Context poisoning. A hallucination or error enters the context and gets referenced again and again. One invented config value, in turn, quietly steers every turn after it.
• Context distraction. The context grows so long that the model relies on it rather than on its training, rerunning the same queries instead of planning new ones.
• Context confusion. Superfluous content drags down response quality, an overload of MCP tool definitions being the classic case.
• Context clash. Accumulated information contradicts itself, like metrics gathered before and after the deploy, and the model can't recover.

What does context engineering actually involve?

Deciding, at every step, what earns a place in the window. In a typical agentic system, the LLM's input is assembled from several sources at once:

• System prompt. The standing system instructions that define behavior, constraints, and tone.
• User prompt. The current request, plus anything attached to it.
• Conversation history. The short-term memory of the session, which agents often treat as working memory. It grows with every turn.
• Long-term memory. Facts, preferences, and outcomes persisted across sessions and recalled when relevant. Agent memory systems handle the storage and retrieval.
• Retrieved information. Documents are retrieved via retrieval-augmented generation (RAG), usually by matching embeddings in a vector database.
• Tool definitions and outputs. Schemas describing what the model can call, plus the external data those calls return. The Model Context Protocol (MCP) standardizes how tools expose both.
• Output schemas. Instructions for structured output, often a JSON object definition, the response has to match.

Every one of these competes for the same token budget. The context window limit is fixed, so anything you add crowds out something else, and context engineering is the craft of deliberately spending that budget.

How do production agents get context just in time?

Not by retrieving it all up front, because much of it doesn't exist yet. Classic RAG pre-indexes a corpus, matches a query against embeddings, and loads the results before the model runs. That suits the stable half of production knowledge: runbooks, past incidents, architecture docs. The volatile half, live metrics, pod states, and the deploy that shipped eleven minutes ago, can only be fetched at investigation time.

Anthropic's guidance calls the second mode just-in-time context: the agent holds lightweight references and runs targeted queries as the investigation narrows, with each result deciding the next one.
Production leans on this mode unusually hard. An investigation draws on code, infrastructure state, telemetry, and team knowledge at once, many times the context of a coding task confined to a single repo, and the agent works that spread with PromQL, log queries, and kubectl. Reranking what comes back keeps the strongest evidence near the edges of a small window rather than lost in the middle of a huge one.

How does context layout change the economics?

Order matters to the cache. Inference providers store the computed state of a prompt's opening tokens, so requests that share an identical prefix skip most of the work. Anthropic prices cached reads at a tenth of fresh input tokens, with cost reductions up to 90% and latency reductions up to 85% on long prompts. At thousands of investigations a day, that's the difference between a rounding error and a line item.

The layout rule falls out directly: stable content first, volatile content last. Standing instructions and tool definitions form the cacheable prefix, while the evidence for this hypothesis and this turn's tool results ride in the tail. An agent that interleaves the two breaks its own cache on every call and pays full price for the same tokens all day.

How is context engineering different from prompt engineering?

Prompt engineering optimizes the instructions you write. Context engineering manages everything the model receives alongside it, which, in an agent that runs for hours, makes the user prompt a small fraction of what the LLM processes. The rest is system instructions, tool definitions, conversation history, and retrieved evidence, and that's where the quality problems actually live.

The term itself took over in June 2025, as generative AI applications grew from single prompts into agentic systems. Shopify CEO Tobi Lütke posted that he preferred it to prompt engineering because it names the actual skill: giving the model everything it needs to plausibly solve the task. Andrej Karpathy agreed a week later, calling it "the delicate art and science of filling the context window with just the right information," and Anthropic's guidance now treats prompt engineering as one part of that bigger job.

The unit of engineering keeps widening. Prompt engineering tuned a single instruction. Context engineering curates everything around one inference. The newest layer is the loop: an agent calling tools, reading results, updating its hypothesis, and going again, with context rebuilt on every pass. Each discipline contains the one before it, which is why teams building serious agents now engineer the whole loop, and in production, the engineered loop is called a harness.

Context engineering inside a production harness

Resolve AI treats context engineering as a dedicated layer of its agentic harness, alongside model orchestration, causal reasoning, and governed actions. The platform maintains a live knowledge graph of services, dependencies, deployments, and team knowledge, so the scattered tribal knowledge that normally lives across tools and heads becomes structured, up to date, and retrievable. Its agents then pull only the evidence the active hypothesis needs rather than replaying raw telemetry into the model.

That design counters shallow reasoning directly: sharper evidence means less confident guessing, and bounded retrieval keeps the cost per investigation flat whether a team runs ten a week or thousands. It's a big part of how AI for production systems differs from a chatbot pointed at your dashboards. On DoorDash's ad platform, Resolve cut time to root cause by up to 87%. Salesforce reported a roughly 60% drop in MTTR and about 70% faster alert triage.

If you're weighing how much context infrastructure to build yourself, it helps to watch a system that has already done the engineering work run one of your real incidents. See it in action.