Production AI agents must handle changing outside events, slow model calls, and many ways to fail.
2
Customer support agents need curated, time-aware knowledge because company documentation is often incomplete, private, or outdated.
3
Gradient Labs uses state machines, behavioral agent code, event logging, and durable execution to control production workflows.
Summary
Neil describes what Gradient Labs has learned while running an AI agent for complex customer support. He separates the production problem into external integrations, model calls, knowledge and tools, and the agent workflow itself. Agents have to react when customers send new information, abandon chats, or trigger events while earlier work is still running. Successful model API calls can still produce bad answers, and knowledge bases need careful curation because documentation often contains private information or misses facts held by employees. Gradient Labs models its agents as state machines, with deterministic and agentic states, while keeping the code between the states focused on behavior. It logs decisions, uses Temporal for durable execution, and abstracts model providers without hiding costs or completions. For testing, the team runs simulations against production-grade traffic in an isolated environment. The talk is honest about how much more work is involved than adding retrieval augmented generation to a support workflow.
AI agent production adds failure handling to ordinary correctness
Neil compares three kinds of systems. MLOps infrastructure asks whether things work well, while traditional machine learning also asks whether the system makes good decisions. AI agents add a third concern: the many ways the system can go wrong. A support agent may need to stop or change work when a customer sends new information, or reach back out after a customer abandons a chat. Neil describes this as managing fast and slow race conditions between the agent and the outside world. Even a workflow with only two or three LLM calls can be much slower than a database request, so timing becomes part of the design.
A completed API call can still return a poor completion, which Neil calls a call that fails successfully. A production support agent also has different timing needs. Answering a customer is time-sensitive, while checking whether knowledge is still current or processing new documentation can happen asynchronously. Rate limits add another constraint for smaller companies. Neil says abstractions are difficult because prompt performance depends strongly on the model being called. The model and the prompt cannot be cleanly separated in the same way that application code might separate a generic API client from its implementation.
Neil warns that putting all company documentation into a vector database is close to a guaranteed bad outcome. Internal documents often assume background knowledge, and they may mix information that can be disclosed to customers with information that must remain private. The larger problem is missing information. Much of what a company knows remains in employees' heads and never reaches documentation. He also points out that high-quality answers can make approximate nearest-neighbor search insufficient, and that a changing knowledge base makes debugging difficult. To understand an answer produced days earlier, the agent needs a point-in-time view of the knowledge it used.
Public demos can let an agent search the web, change a repository, or open an issue. A company agent often cannot call private or sensitive APIs during evaluation. That creates a testing problem because the team needs to assess behavior without granting unrestricted access to the real tools. Tool results also change while an interaction is in progress. Neil gives the example of retrieving a customer's fraud status and then receiving new information that causes the fraud engine to change it. The agent therefore has to reason about the freshness and meaning of returned data, rather than treating every tool response as permanent truth.
State machines give customer support agents controlled flow
Gradient Labs models its support agents as state machines. Neil compares this with a conversation: one person enters a listening state, then the other takes a turn, with the process continuing until the conversation ends. Real agents have many more states, some deterministic and some agentic. Inputs from the outside world, timers, and signals that conditions have changed control movement between states. This lets the people working at the higher level think about the agent's behavior without handling every lower-level detail. A simple example classifies a customer's conversation for clarity and asks for clarification when the conversation is unclear.
The agentic code should sit between behavior and infrastructure
Gradient Labs chose not to adopt an LLM framework because the team wanted to perform deeper changes and felt the ecosystem was too early. Its preferred developer experience is a middle layer where engineers describe what the agent should do without thinking directly about individual states or raw LLM calls. The code should still be understandable when read. Every decision returns an event, while lower-level model calls run through a durable execution engine. This removes retry handling from the agent engineers' work and allows the system to use models from different providers while recording completions, costs, and outcomes.
Testing uses production-shaped simulations in isolation
Neil says traditional A/B rollouts and shadow deployments are feasible for conventional machine learning systems, but agent testing needs several forms of evaluation. Gradient Labs creates a pull request version of an agent, runs thousands of simulations on production-grade traffic, and then evaluates the results manually or automatically. The simulations use production data in an isolated evaluation environment rather than exposing a new agent directly to live customer interactions. This gives the team the varied edge cases found in real traffic while keeping the evaluation separate from the live production environment.
Customer-specific constraints belong in the agent framework
Different companies need different controls over language, handoffs, conversation closure, supported channels, timers, and customer intents. If an agent speaks Italian but the human support team cannot, handing the conversation off creates a practical failure. A bank may allow the agent to discuss some topics while prohibiting it from handling fraud questions. Neil says the framework must expose both functional controls, such as timer values, and behavioral controls, such as topics the agent may or may not discuss. These requirements become more important as the agent is used by higher-risk companies.
"The biggest thing that we found in running our AI agent for the last few months is that the even larger problem than just trusting company documentation is the one of missing information."Neil11:51
Who should watch
You are building a customer support agent and need to understand the operational work beyond prompting and retrieval.
Your team is moving an LLM prototype into production and needs a concrete list of timing, data, tool, and testing problems.
You work on agent infrastructure and want an example of separating state-machine control, behavioral code, and durable model execution.
