“It's harder to keep agents running than to build them”—Designing Agentic Infra to break through the 5% success rate barrier
Generative AI adoption in enterprises is not a “magic wand.” Episode 14, “Agentic infra is the problem you're probably not thinking about,” repeatedly highlighted that the bottleneck for field deployment lies not in the “model itself,” but in the “infrastructure required to keep agents running (orchestration, evaluation, human-in-the-loop, data connectivity).” This article organizes the structural causes of failure and practical design principles based on the discussions in the program.
1. The reason why 95% of PoCs are “all show and no go”
Regarding the MIT topic mentioned at the beginning—"95% of enterprise GenAI pilots fail." The guests cited the “extremely large denominator (number of trials)” and “excessive expectations” as the primary causes.
The essence: What you build in a PoC is a “smooth-looking demo.” However, to move to production, you must overcome the “unassuming hurdles” of operations, auditing, and reliability.
Quote: "Models get better every year, but we need an additional layer to bridge the gap with the 'specific work' that companies truly need."
1-1. The pitfall of “AI Work Slop”
The “content-free, perfectly formatted output mass-produced by AI” pointed out by BetterUp and Stanford.
Statement: “Expansion is noise; compression is value.” The ‘telephone game’ of mindlessly generating and re-summarizing long texts erodes the core of decision-making.
2. Context Engineering: Accuracy is determined by the “context provided”
Point of discussion: "Context rot is often more serious than hallucinations."
Typical example: Even when the “correct answer” exists in the internal search index, the agent retrieves low-quality documents due to an incorrect query, causing performance to drop below that of a web search.
-
Key point:
Infinite information is poison. Necessary and sufficient context must be “compressed” and passed along.
Agree on the definition of quality first (“What criteria must this output satisfy?”). Everything should be reverse-engineered from those evaluation metrics.
2-1. Single vs. Multi-agent
Multi-agent systems are attractive, but they quickly become complex due to overlapping domains and write conflicts.
-
Guiding Principles:
Parallelizing read operations is relatively safe.
For write operations, a supervision layer (orchestrator + human approval) is mandatory.
Deciding how much internal state to expose to the user is a key point in product design.
3. What is “Agentic Infra”?
The definition of the performers is practical. It only becomes operational once you have the necessary components: scratch space to support planning → execution → learning → ToDo updates → sharing, long-running execution, pause/resume capabilities, human escalation, permission/secret management, and an evaluation infrastructure.
Quote: “A ‘raw agent’ uploaded to a server dies the moment it escalates.” The mechanism for state persistence and restarting is key.
3-1. Making Human-in-the-Loop (HITL) a “first-class citizen”
The more difficult the problem, the more human approval and division of labor are involved. Design multi-stage approvals and cross-organizational handoffs as part of the business logic.
Important implication: “If a human is not needed, the task might not be business-critical.” Focus on projects that assume HITL.
4. Two layers of data: Stored data and behavioral data
“Data held by a company” is not just stored data (DB, DWH). The “ways of doing things” in the minds of the staff (judgment habits, approval criteria, avoidance principles) are the true source of differentiation.
Policy: Have agents work closely with the field to collect behavioral logs and learn the “company’s way of decision-making.”
Furthermore, you need “glue” to bond statistical optimization (such as demand forecasting) with semantic overrides (judgment of exceptional events).
5. The scale barrier: Is it harder to expand than to build?
-
The answers to this hot take were divided.
Position A: “The act of ‘building’ a good agent itself remains unsolved”.
Position B: “When you scale, assumptions collapse, and unknown bugs erupt due to an explosion in surface area”.
Practical implication: Increase the surface area incrementally and support it with an evaluation layer. The stronger the evaluation, the more safely you can experiment.
6. Implementation principles: Where to start and what to discard
1) Start by amplifying star talent
"Augmentation, not replacement." Turn the decision-making of top performers into a "textbook" and correct errors with HITL.2) Start with measurable KPI tasks
Examples: "Churn rate reduction," "First response time," "Billing discrepancy detection." Confirm wins with numbers, then expand the scope.3) Minimize plumbing
Offload secret management, long-running jobs, human approval, and audit logs to a common infrastructure.4) Context design with a focus on compression
No "dumping everything in." Use the bare minimum plus reference procedures in prompt templates.5) Lock in the evaluation-to-improvement loop
Offline reproduction tests (prompt/tool/data diffs) + online monitoring (drift, approval rates, recovery rates).
7. Summary: "AI value" resides in the infrastructure
Generation and search are just the entrance.State management, human intervention, long-term execution, evaluation, and permissions— the agent infrastructure that glues these together is what turns PoCs into production systems.
Closing with a quote: "There is only one thing to focus on this sprint—modeling your business logic with the highest quality. Move the rest of the plumbing to a common foundation."