# Building AI Agents That Survive Production Haytham Abuelfutuh, Union.ai | AI Agents 2026 | 31:59 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=swO5svhBhQ4 Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/building-ai-agents-that-survive-production Published: 2026-05-14 Tags: agents, caching, guardrails, reliability ## TL;DR - Production agents should be designed to tolerate failures and recover from them instead of trying to eliminate every failure. - A platform for agents should let developers write dynamic native Python, recover durable sessions, and safely run untrusted generated code. - Dragonfly moved its product-catalog agent from a laptop prototype to a production system indexing more than 250,000 products in one sitting. ## Summary Haytham Abuelfutuh argues that production agents will fail, so teams should make those failures cheap to handle. He presents three design principles for an agent platform: Dynamic, Durable, and Defended. Dynamic means developers can write ordinary Python with loops, branching, and error handling instead of fitting an agent into a constrained DSL. Durable means the platform records actions, caches non-deterministic LLM and tool calls, retries work after infrastructure failures, and resumes long-running sessions without repeating questions to users. Defended means generated code runs in a network-isolated environment separate from the host, with retries and a human bailout when the agent cannot recover. He illustrates the approach with Dragonfly, whose four-tier product research system indexes more than 250,000 products. The talk ends with questions about Union.ai's Go and Rust services and moving infrastructure control closer to developers. ## Key ideas ### A good agent remembers a long-running conversation after failures [05:28](https://www.youtube.com/watch?v=swO5svhBhQ4&t=328s) Haytham Abuelfutuh uses an 18-year-old travel-agent experience as the standard for an agent session. Over several weeks, the human agent handled emails, phone calls, and dropped calls without asking him to repeat what he had already said. A production agent should preserve that continuity. It should remember the user's destination and budget, recover from a dropped connection, and continue from the same point. Haytham also wants sessions that can remain open for a long time without losing their context or forcing users to start again. ### Production failures include infrastructure and service problems beyond the agent's logic [08:08](https://www.youtube.com/watch?v=swO5svhBhQ4&t=488s) Teams spend much of their time fixing semantic and logical errors, such as getting a travel agent to perform the intended task. Once the agent leaves a laptop, other failures appear. Processes crash, memory runs out, tools fail or stop scaling, networks break, and external APIs throttle requests. A restart can also produce corrupt context and a poor user experience. Haytham separates these operational failures from the agent's underlying idea. His recommendation is to expect them and build an agent that knows how to react when they occur. ### Dynamic agents should run ordinary Python instead of a restrictive DSL [09:49](https://www.youtube.com/watch?v=swO5svhBhQ4&t=589s) The first design principle is Dynamic. Haytham says developers should write the agent in the form that matches their thinking, using normal Python functions, loops, try/except blocks, and asynchronous execution. The platform should not require a special declaration for every branch or loop. A constrained DSL can make agent development less natural and limit how the agent interacts with users. Dynamic execution also lets the agent change its behavior as it works, while the platform still manages the execution around it. ### Durability requires infrastructure-aware retries and recovery [10:33](https://www.youtube.com/watch?v=swO5svhBhQ4&t=633s) The platform should know what infrastructure an operation needs because some failures cannot be caught inside the function. If a spot machine disappears or a process runs out of memory, the function cannot handle the event after the process is gone. Haytham proposes declaring memory and CPU requirements in the code, then allowing the system to retry the work on the same or different hardware. The same approach applies to external tools and code. If an operation needs more memory than expected, the platform can change the allocation and run it again. ### Crash recovery should replay recorded actions without repeating non-deterministic calls [15:58](https://www.youtube.com/watch?v=swO5svhBhQ4&t=958s) Agent sessions can last for weeks, so a crash is likely during a session. The platform should record each action and recover from the recorded state rather than rerunning every step. This matters for LLM calls because the same prompt can produce a different result days later. It also matters for user input. If the agent already asked for a destination or budget, recovery should not ask the user the same question again. Recording actions gives the resumed agent a consistent path through the earlier work. ### Cross-session caching depends on whether an output changes frequently [17:58](https://www.youtube.com/watch?v=swO5svhBhQ4&t=1078s) Durability also applies across multiple users and agent executions. Haytham says a platform should let developers choose which calls can use a shared cache. A web search for the same term on the same day may be suitable for reuse. An LLM-generated report may need to run again instead. Shared caching can reduce token use and make later executions faster, but the decision belongs to the developer because different calls have different freshness and variability requirements. ### Defended agents isolate generated code and involve a human after failed retries [20:22](https://www.youtube.com/watch?v=swO5svhBhQ4&t=1222s) An agent may decide that its supplied tools cannot complete a task and generate custom code. That code should run in a secure, network-isolated environment that is separate from the host and its running processes. Haytham describes retrying generated code by sending the error and code back to the LLM, then trying again. If the attempts still fail, the agent should ask the user what to do next or offer choices. This human-in-the-loop bailout gives the session a controlled way to proceed or stop. ### Dragonfly uses four agent tiers to research a catalog of more than 250,000 products [24:03](https://www.youtube.com/watch?v=swO5svhBhQ4&t=1443s) Dragonfly helps users compare software products by crawling, cataloging, and indexing a catalog of more than 250,000 products. Its production architecture has four tiers: catalog agents, coordinators, researchers, and tools. Coordinators assign product research to researchers, which use web search, indexes, documentation, and other tools. A coordinator can notice that researchers are doing the same work and stop one of them. The system traces and audits this activity, giving Dragonfly records to inspect and use when improving its agents. ## Notable quotes - Haytham Abuelfutuh: "Agents do fail and they will fail in production." (07:37) - Haytham Abuelfutuh: "You should be building agents that tolerate that, that know how to react when failure is happening because they will happen." (07:57) - Haytham Abuelfutuh: "You want to make sure we walk out of here probably if there's one thing I would love everyone to agree with is that failures will happen and it's not about making agents never fail." (19:02) - Haytham Abuelfutuh: "You want the recovery from failures to be cheap." (19:19) - Haytham Abuelfutuh: "The system in the back is all built on Go and Rust services." (28:32) ## Tools & references mentioned - MLOps Community - Union.ai - Flyte - Dragonfly - Pydantic Monty - Python - Go - Rust - Erlang ## Who should watch - You have an agent that works on a laptop but fails when it encounters real traffic, memory limits, throttled APIs, or disappearing machines. - Your team needs long-running sessions that can resume without rerunning LLM calls or asking users to repeat earlier answers. - You are deciding how to let agents generate code while keeping execution isolated and giving users a controlled fallback. ## Related talks - [Decomposing the Agent Orchestration System: Lessons Learned](https://mlopstalks.com/talks/decomposing-the-agent-orchestration-system-lessons-learned) (Niels Bantilan, Union.ai, 30:13) - [How to build agents that take ACTION](https://mlopstalks.com/talks/how-to-build-agents-that-take-action) (Alex Salazar, Arcade, 28:59) - [Building AI that Doesn't Break](https://mlopstalks.com/talks/building-ai-that-doesnt-break) (Elliot Gunton, Pipekit & Qian Li, DBOS, Inc. & Alan Nichol, Rasa, 1:01:03) - [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03) - [The Next Evolution of AI Agents](https://mlopstalks.com/talks/the-next-evolution-of-ai-agents) (Alon Horev, Vast Data, 13:38)