Most AI agents don't fail in production, they fail on the way there. A CTO's recipe book for context engineering, evals, durability, and the team design that gets agents shipped.
Leonardo Piñeyro
CTO
Most AI agent projects do not fail in production. They fail on the way there, quietly, in the gap between a demo that wows the boardroom and a system that survives real traffic on a Tuesday morning.
The numbers back this up. By early 2026, practitioners at QCon London cited reports that as many as 80% of firms see no tangible benefit from their AI initiatives.1 Analyst syntheses put the failure rate of multi-agent pilots at roughly 40% within six months.2 That is not a model problem. Frontier models are extraordinary. It is an engineering and organizational problem, and it is solvable.
Here is the reframe that changes everything for a technical leader weighing where to spend the next million dollars of build budget: an AI agent is a distributed system and an organizational commitment first, and a clever prompt a distant last. The teams shipping agents that actually move KPIs treat context engineering, evaluation, durability, and team design as the real work. The prompt is the easy part.
This is the recipe book we wish every CTO had before the first pilot. It is opinionated, grounded in what current sources from Anthropic, OpenAI, and Google recommend, and shaped by what we have shipped for clients. Read it as a sequence of decisions, each one cheaper to get right early than to fix in production.
A demo proves an agent can do a task once. Production asks it to do the task ten thousand times, cheaply, safely, and in a way you can audit when a customer complains. Those are different problems.
The failures are rarely dramatic. They are death by a thousand cuts. A misrouted support ticket here. A contract clause the agent missed there. A token bill that triples the month a feature goes viral. An agent that loops on a malformed input until someone notices the cloud invoice. Each cut is survivable. Together they erode the trust and the unit economics that justified the project.
The good news: the levers that prevent this are well understood in 2026, and they are mostly not about the model. They are about how you feed the model, how you measure it, how you contain it, and who owns it. Let us walk through them in the order you should make the decisions.
The most reliable advice across every credible source in this space is the same: start with the simplest thing that works, and add complexity only when evaluation proves you need it. Anthropic calls the principle building effective agents through "minimum viable complexity."3 OpenAI's practical guide makes the same point from the other side: validate that your use case actually needs agentic reasoning before you build an agent at all.4
The distinction that matters is between a workflow and an agent. A workflow is an LLM orchestrated through predefined code paths: classify, then route, then summarize. An agent is an LLM that directs its own tool use in a loop, deciding what to do next based on what it just saw. Workflows are cheaper, more predictable, and far easier to debug, because you only pay for model reasoning at decision points you chose. Agents are for problems where the path genuinely cannot be predicted in advance.
For a leader, this is a budget decision before it is a technical one. Agentic reasoning costs real money per step, and most business processes are stable enough to encode as a workflow. The expensive mistake is reaching for an autonomous agent when a deterministic pipeline with one LLM call would have been faster, cheaper, and auditable. Reserve agents for the ambiguous, multi-step, high-value workflows where rules-based automation has actually failed.
The rule of thumb: if a competent engineer can draw the decision tree on a whiteboard, build the workflow. If they cannot, you may have a real agent use case.
Here is where context engineering comes into the picture, and it is the single highest-return investment you can make in an agent.
Every model has a finite context window, the working memory it can attend to at once. It is tempting to treat that window as free space and stuff it with everything that might be relevant: the full document, the entire chat history, every tool result. This backfires. Performance degrades as the window fills, a phenomenon Anthropic and others now call "context rot."5 More context is not better context.
Context engineering is the discipline of finding the smallest set of high-signal tokens that produce the outcome you want. It is the evolution of prompt engineering for the agent era, and it covers everything the model sees: the system instructions, the retrieved documents, the tool definitions, the running memory, and what you choose to leave out.
In practice it means a handful of concrete techniques. Compaction, where you summarize a long conversation as it approaches the window limit and reinitialize with the summary plus the most recently used files. Structured note-taking, where the system writes progress to external storage and retrieves it on demand instead of holding everything in active memory. Sub-agent isolation, where each helper works in its own clean window and passes back only a summary. And system prompts written at the right altitude: specific enough to guide behavior, general enough that they do not shatter the first time reality varies.
A concrete example makes it tangible. Picture an agent that reviews supplier contracts. The naive build dumps the entire 80-page agreement plus every past email thread into the window and asks for a risk summary. It is slow, expensive, and it misses things, because the signal is buried in noise. The engineered build retrieves only the clauses that match the risk policy, pairs each clause with the specific standard it should be measured against, and returns a citation for every flag. Same model, same question. The difference in accuracy and cost is the difference between a tool the legal team trusts and one they quietly ignore.
For your business, this is the lever that quietly determines cost, latency, and accuracy all at once. A well-engineered context is smaller, so it is cheaper per call and faster to first response. It is also more accurate, because the model is not distracted by noise. We have seen context engineering move an agent from "impressive but unreliable" to "boring and dependable" without touching the underlying model. That move is the whole ballgame.
Multi-agent architectures are the most over-applied pattern in the field. They are genuinely powerful for a narrow band of problems and a costly mistake everywhere else.
The evidence is striking. Anthropic's own multi-agent research system, with an orchestrator delegating to parallel workers, beat a strong single agent by a wide margin on internal research tasks, but it consumed roughly fifteen times the tokens of a normal chat interaction, and token usage alone explained about 80% of the performance difference.6 Separately, benchmarking synthesized by Beam.ai found a single agent matched or beat multi-agent systems on roughly two thirds of tasks when given the same tools and context, with multi-agent adding only a couple of percentage points at roughly double the cost.2 Treat those specific figures as vendor and analyst estimates rather than settled fact, but the direction is consistent and it matters: multi-agent is expensive, and it only wins when the task genuinely splits into independent parallel threads.
When you do need multiple agents, the production default in 2026 is the orchestrator-worker pattern: one capable lead model decomposes the job and delegates to cheaper specialist workers, with the orchestrator owning the canonical state. Using a strong model for the lead and cheaper models for the workers is a standard way to cut cost by a meaningful margin. The patterns that get teams into trouble are the open-ended ones, where agents talk to each other freely. Coordination complexity grows roughly with the square of the agent count, so a ten-agent mesh has around forty-five communication paths and becomes nearly impossible to observe or debug.
The discipline here protects your budget: cap the number of specialists, enforce hard step budgets from the orchestrator rather than trusting agents to self-limit, set a mandatory iteration ceiling, and run an independent verification pass. If a single agent already passes your evaluations, ship the single agent.
Whatever the topology, an agentic workflow that runs unattended in production needs containment built in from the start. The failure modes are well catalogued: context fragmentation across steps, coordination overhead, cost blowup, and the nastier ones, like a hallucination from one step being treated as ground truth by the next, or a retry loop with no termination condition burning tokens until someone pulls the plug.
The controls are not exotic. Hard step budgets enforced by code, not by asking the model nicely. Explicit termination conditions and a maximum iteration ceiling. Source citation so every claim can be traced. And human-in-the-loop approval gates for any action that is hard to reverse or touches money. The academic backbone here is worth knowing: a 2025 study introduced a taxonomy of fourteen distinct failure modes for multi-agent systems, built from over 1,600 annotated execution traces.7 The lesson for leaders is that "the agent will figure it out" is not an operating model. Bounded autonomy is.
The framework market has converged, which is good news. Tool calling is now commoditized; the real differences are in state management, durability, and observability. You do not need to agonize over this choice. Pick on two axes: your team's primary language, and how much of the system you want the framework to own.
A practical map of the current landscape:
The worst outcome is adopting a framework that fights your architecture, for example forcing a delegation problem into a graph tool, or building a complex stateful workflow on a framework with no persistence. Provider-native SDKs give tighter model integration at the cost of some lock-in. For most enterprises, the right move is to choose by language, prototype fast, and avoid betting the system on any one vendor's roadmap.
An agent is only as useful as the tools and data it can reach, and in 2026 the way you connect those has standardized around two complementary layers.
The Model Context Protocol (MCP), introduced by Anthropic and now governed under the Linux Foundation, is the standard for connecting agents to tools and data.8 It has seen rapid adoption across Anthropic, OpenAI, Google, Microsoft, and AWS, with thousands of public servers available. For most teams, MCP is now table stakes for production tool integration. The second layer, A2A (Agent2Agent), handles agent-to-agent coordination across vendors and organizations. The practical rule: start with MCP for tools, and add A2A only when you have genuine cross-organization agent coordination.
For a leader, the protocol layer carries a real risk you should name out loud: third-party MCP servers are third-party code with access to your systems. Treat them exactly as you would any external dependency. Review them, pin versions, and audit what they can touch. Documented attack vectors include tool poisoning, where a malicious tool description manipulates the agent, and the field has already seen high-severity disclosures. The model is not a security boundary. Your architecture has to be.
The deeper point for your roadmap is portability. Open standards like MCP, plus a clean separation between your business logic and the model layer, are what keep next quarter's model upgrade from becoming a rewrite.
If there is one practice that separates teams who ship reliable agents from teams who ship anxiety, it is this: they build the evaluation before they build the agent.
Eval-driven development treats the evaluation suite as the working specification. The reasoning is blunt: if you cannot express what "correct" looks like as a repeatable test, you are not ready to build the thing. The recipe is a golden set of fifty to eighty on-spec cases built with a domain expert, a grader (often a capable model scoring outputs against a rubric, an approach validated by methods like G-Eval9), and a harness that runs in your CI pipeline so a regression blocks the merge. For retrieval-heavy agents, isolate retrieval quality from generation quality so you can tell which half broke.
The decision-maker takeaway is that agent evaluation is your quality contract and your risk control in one artifact. It is what lets you upgrade a model next quarter without praying. It is what turns "the agent seems worse this week" from a vibe into a number. Set a realistic baseline, gate every change against it, and feed real production failures back into the golden set so your evaluation stays representative. Budget for this work explicitly. Teams that skip it pay for it later, with interest, in production incidents.
This is the stretch where most of the engineering, and most of the budget discipline, actually lives. Taking a working prototype to a production system means adding the unglamorous machinery that keeps it cheap, safe, and observable.
Durable execution is now the mainstream answer to reliability. Long-running agents need checkpoint-and-replay so that a crash resumes from the last completed step rather than starting over or corrupting state. Platforms like Temporal, Inngest, Restate, and DBOS, along with built-in checkpointers in frameworks like LangGraph, handle this. What durable execution does not fix is also worth saying plainly: it does not stop hallucinations, runaway loops, or evaluation drift. Those need their own controls.
Model routing is the highest-leverage cost lever. Most production traffic never needed a frontier model; routing easy requests to cheap models and reserving the expensive ones for hard cases can cut model bills by anywhere from 40% to 85%, because the price spread between model tiers runs to roughly 100x. Prompt caching and per-session token caps further protect the budget and stop runaway loops.
Layered guardrails wrap the whole thing: deterministic input filters, output validation, risk ratings on each tool by how reversible and how costly its actions are, and human approval gates for the high-risk ones. Prompt-level instructions alone do not survive a determined injection, so the defenses have to be architectural, including running tool execution in isolated, least-privilege sandboxes and making every action idempotent so a duplicate never charges a customer twice.
Observability and kill switches close the loop. You want structured tracing across every model call, tool invocation, and memory operation, with the parent-child relationships preserved across handoffs, plus a global switch to suspend autonomy instantly if something goes wrong. Pair the traces with a drift-monitoring loop: stream the logs, flag low-confidence cases for human review, and feed the corrected ones back into your evaluation set so it keeps tracking reality. Roll out through shadow mode (the agent runs in the background, its outputs compared against humans, nothing shipped) before assist mode (the agent drafts, a human confirms) before any autonomy.
The business framing for all of this machinery is one word: risk. Evaluation, guardrails, and observability are how you cap the downside of a probabilistic system before it touches a customer.
One more pattern earns its keep with enterprises carrying legacy systems. Rather than a risky rewrite, wrap the existing system: the agent reads through and writes back through your current APIs, so all the validation and business logic you already trust stays in place. It is the agent-era expression of the well-known strangler-fig integration pattern.10 You surround the old system instead of replacing it.
Here is the finding that most surprises technical leaders, and the one with the highest leverage: the bottleneck is usually the org chart, not the model.
Software architecture mirrors the communication structure of the team that builds it. That is Conway's Law, and it is back in force for the AI era. The teams that succeed build a small amount of organizational maturity before they scale: clear ownership, a shared vocabulary between the model engineers and the product engineers, and governance for what an agent is allowed to do on its own.
Two pieces of sequencing advice repay attention. First, build your data infrastructure and hire data engineers before you hire data scientists, because a data scientist without clean, accessible data spends most of their time wrangling instead of modeling. Second, put your model metrics and your business metrics on the same dashboard. Accuracy, latency, and hallucination rate next to adoption, retention, and task-time saved. When those live on separate screens, the model team optimizes a number the business does not feel, and the business chases an outcome the model team cannot see. Most organizations eventually evolve toward a hub-and-spoke structure: a central platform team that sets tooling and guardrails, with applied teams embedded in the products. You do not need that on day one. You do need to know it is where you are heading.
Principles are cheap. Here is how this plays out when the system has to work for real clients. (Details are anonymized.)
A global hospitality operator with more than 250,000 employees. This client needed two agents in document-heavy, compliance-sensitive corners of the business: a legal contract review copilot and an HR screening assistant. The temptation was full autonomy. The right answer was trust calibration: bounded autonomy, earned in stages. We built the contract copilot in assist mode, where it reads a contract against the company's own policies and flags deviations with a justification trail and an inline pointer to the exact source clause, and a lawyer makes the call. The heavy lifting was context engineering, feeding the agent the right slices of policy and precedent rather than everything, plus guardrails and a clean audit trail. The screening assistant followed the same shadow-then-assist path. The business outcome was not "we replaced the lawyers." It was experts spending their time on judgment instead of on the first pass, with every recommendation traceable.
A healthcare software startup. The founder had vibe-coded a working prototype, the kind of thing that demos beautifully and breaks the moment a second user shows up. Our job was the unglamorous part: turn it into something a regulated healthcare customer would trust. We hardened it with evaluation, durability, and the production machinery above, and took it from founder prototype to production in four months, with the first paying client onboarded by month six. The lesson that generalizes: the distance between "interesting demo" and "system a customer pays for" is mostly engineering discipline, and it is shorter than most teams fear when the right practices are in place from the start.
If you take one thing from this recipe book, let it be the sequence, because the order is where the money is saved.
Decide whether you even need an agent; most of the time a workflow wins. Start with a single agent, and write the evaluation before the agent. Invest early in context engineering, because it is the cheapest lever with the largest effect on cost, latency, and accuracy at once. Harden for production with durable execution, model routing, layered guardrails, and observability before you scale. Go multi-agent only when the task genuinely splits into parallel threads and your evaluations prove the extra cost pays for itself. And treat the organizational work, ownership, shared metrics, data infrastructure first, as part of the build, not an afterthought.
Two principles sit underneath that sequence. Budget for the production stack, not just the model: the demo is maybe 20% of the work, and the evals, guardrails, observability, and context engineering behind it are the other 80%, which is where the ROI is won or lost. And start narrow enough to prove it on a real metric: one agent, one workflow, one number that matters, whether that is a cost reduction, a cycle-time cut, or a conversion lift. Earn the right to expand by measuring it.
The gap between an AI demo that impresses and a production system that moves your KPIs is smaller than most teams think. It is not about a better model. It is about context engineering, evaluation, durability, and the discipline to add complexity only when you have earned it.
That is the part we love. If you are mapping where agents fit on your 2026 roadmap, or you have a prototype that needs to become a product, we would be glad to compare notes. Schedule a conversation with our team.
QCon London 2026: Team Topologies as the Infrastructure for Agency with AI (InfoQ). The "80% see no tangible benefit" figure is cited at the conference and should be read as a directional estimate. ↩
6 Multi-Agent Orchestration Patterns for Production (Beam.ai, 2026 synthesis). The ~40% pilot-failure rate and the single-agent-vs-multi-agent comparison are analyst and vendor estimates synthesizing benchmarks including Princeton NLP work; verify against primary benchmarks before quoting. ↩ ↩2
Building Effective Agents (Anthropic). ↩
A Practical Guide to Building Agents (OpenAI). ↩
Effective Context Engineering for AI Agents (Anthropic). ↩
How We Built Our Multi-Agent Research System (Anthropic). The 90%+ improvement and ~15x token figures are from Anthropic's internal research-task evaluation and may not generalize. ↩
Why Do Multi-Agent LLM Systems Fail? (Cemri et al., arXiv 2503.13657; NeurIPS 2025 spotlight). ↩
Model Context Protocol (Anthropic / Linux Foundation). ↩
G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment (Liu et al., arXiv 2303.16634; EMNLP 2023). ↩
StranglerFigApplication (Martin Fowler). ↩
CONTACT US
Work with Pento to turn promising AI experiments into systems that perform reliably in production, with the right architecture, delivery model, and engineering support.
