Agent infrastructure must preserve state and provide enough context for agents to recover from failures across the stack.
2
Replay logs, shared caching, and intermediate state persistence make failures cheap by avoiding unnecessary recomputation.
3
Secure sandboxes and infrastructure-aware feedback let agents adjust resources, fix code, and recover inside their own execution loop.
Summary
Niels Bantilan explains what Union learned while building Flyte 2.0 and productionizing agentic systems. His concern is the infrastructure around an agent, especially failures caused by resource limits, preemptions, networking, lost state, and parallel work. He proposes six design principles: use a general-purpose language such as Python, add durability and observability hooks, make failures cheap, expose infrastructure as context, give agents self-healing utilities through sandboxes, and keep a human as the final recourse. The implementation rests on replay logs, cross-run caching, and intermediate state persistence. These let an agent resume from a failed step without repeating completed work. Bantilan also shows how agents can respond to out-of-memory errors, update dependencies, and fix code in sandboxes. A Dragonfly case study used this approach for long-running research agents with many concurrent runs. Bantilan recommends starting with one agent and, when needed, one layer of sub-agents.
Agent infrastructure fails at several layers and can erase the context needed for recovery
Bantilan says a working agent can still fail in production because its tools need secure, least-privilege access and its workloads may need changing amounts of compute. A machine learning agent might begin on a small CPU machine, then need to train on a multi-gigabyte dataset. Parallel tool calls and sub-agents can create resource contention. Containers can be killed by a scheduler, and spot instances can be preempted. These failures can cause memory loss or corruption, removing the context the agent needs to recover. Agents therefore need visibility into infrastructure, networking, logical, semantic, and tool-execution failures.
Durability means resuming work without repeating completed steps
Bantilan defines three building blocks for durable execution. A replay log records the state of an agent and its subtasks at a granular level, including intermediate outputs. If the agent completes two tool calls and fails on the third, it can resume from the third call. Global caching shares reusable work across agents and runs, while run-level replay protects one execution. Intermediate state persistence stores agent outputs and tool results in object storage through durability hooks, without requiring each application to write its own serialization and deserialization code.
General-purpose code keeps agent construction understandable to people and models
Bantilan recommends plain Python, TypeScript, JavaScript, or another general-purpose language understood by the model. In his example, Flyte 2 works with plain Python, so developers and agents can use ordinary loops, conditionals, fan-out, asynchronous code, and try-except blocks. A framework remains optional. He argues that this avoids surprising domain-specific language behavior and makes the orchestration code easier to inspect. Functional hooks can then add tracing, checkpointing, and persistent intermediate state without changing the basic programming model.
Exceptions can carry infrastructure context into the agent loop
Bantilan treats exceptions as a delivery mechanism for failure information. A task environment can specify the container image, dependencies, resource requests, and other execution details, while tasks run in isolated Kubernetes pods. If code written by an agent causes an out-of-memory error, the agent can receive that error, adjust the requested resources, and send the change back through the loop. Failed runs can also become training data or additional context. This turns infrastructure failures into signals the agent can use rather than events that always terminate execution.
Sandboxes give agents controlled ways to repair orchestration and application code
Bantilan describes a code mode sandbox where an agent writes Python using a restricted toolbox. The code cannot perform arbitrary input and output, network calls, or extra imports. The orchestrator runs the generated program and returns errors so the agent can revise it. He mentions Pydantic Monty as a project used for this approach. A stateless code sandbox allows a one-shot program to use selected libraries, limited network access, and limited filesystem operations. The agent can also write unit tests for the generated code.
Human input remains the fallback when the agent lacks the needed context
Some failures come from a poor prompt, an unsuitable system instruction, or missing information. Bantilan says the agent should then stop after its iteration budget and request more context. A user can provide text, upload a file, or supply a zip archive containing additional Markdown files. The agent can then be called again with that material. This is the final recourse in his design, after retries, infrastructure-aware recovery, and sandbox-based repair have been exhausted.
A long-running research system benefits from tiers, caching, and checkpoint recovery
Bantilan describes Dragonfly's automated solutions architect, which builds a living knowledge graph of software products. Each agent call involved about 200 steps, and each product involved about 100 LLM calls. The system used an agent driver, research coordinators, researchers, and a tool layer. Cross-run caching avoided repeating LLM calls for the same research prompt. Coordinators grouped semantically similar research threads to reduce duplicate work. Checkpoint recovery made spot-instance interruptions mostly harmless, while tracing provided an audit record for every LLM and tool call.
Most applications should begin with one agent and add only a shallow sub-agent layer
In the question period, Bantilan says the four-tier architecture was specific to Dragonfly's scale and nondeterministic research workload. For internal use cases, he generally prefers one agent with tools, or one agent plus one layer of sub-agents. The sub-agents can handle separate tasks, but deeper hierarchies are usually unnecessary. He recommends starting with one agent and adding a sub-agent layer only when the work requires it, such as when a coordinator must manage many research threads.
"The problem isn't that agents fail, it's that recovering from failure is challenging without the full context of how infra, networking, logical, semantic layers, all of these interact so that the agent can figure out how to dig itself out of the hole it's found itself in."07:09
Who should watch
You are building agents that need access to organizational data, databases, or variable amounts of compute.
Your agent runs for many steps and you need to recover from crashes, retries, preemptions, or repeated tool calls without losing state.
You are deciding how many agent tiers to introduce and want an example of when a shallow hierarchy is enough.
