# Sub-Agent Architectures: What You Can Leverage Sidd Seethepalli, Vellum | MLOps Community | 14:48 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=hIMvuMVLTy8 Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/sub-agent-architectures-what-you-can-leverage Published: 2025-11-26 Tags: agents, evals, observability, tool-use, tracing ## TL;DR - Sidd Seethepalli argues that tool design determines whether an agent calls the right tool with the right arguments and can recover from errors. - Vellum uses both informal "vibes based testing" and rigorous test suites, depending on whether the team is exploring behavior or preventing regressions. - Agents can turn text output into interactive interfaces, including visible thinking steps, tool calls, status indicators, and buttons. ## Summary Sidd Seethepalli explains why Vellum moved from a drag-and-drop agent editor to a natural language agent builder. He shares practical lessons from building and testing the new system. Tool design needs to start from the actions the model should take, rather than exposing raw APIs. Clear names, simplified arguments, grouped API calls, and descriptive errors help the model choose tools and recover from failed calls. Vellum also uses short knowledge-base summaries so the model can retrieve detailed documentation when needed. Testing involves a trade-off between speed, regressions, and test-writing effort. Informal testing works well during early exploration and for judging the user experience, while rigorous tests help measure small changes and prevent lost capabilities. Sidd also argues that text output can become a richer interface with buttons, status updates, and expandable details. Vellum's next steps include real-time workflow updates, better mobile use, and background agents that can handle several changes at once. ## Key ideas ### Trying small changes is a practical way to improve an agent [02:33](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=153s) Sidd says builders should try an idea whenever they think it might improve an agent, even when conventional wisdom disagrees. They can "hill climb" toward a useful system by testing many changes quickly. He also advises builders to think from the model's point of view. Tool descriptions, arguments, and workflows should be judged by how an LLM is likely to interpret them. His example is that prompting yourself with "what we usually do in that situation is" can produce a made-up continuation, which he uses to illustrate how next-token prediction can shape model behavior. ### Tool design should expose simple actions instead of raw APIs [03:43](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=223s) The agent's tools affect whether it calls nothing, calls too many tools, chooses the wrong tool, or sends bad arguments. Sidd recommends starting from the higher-level actions the model needs to take, then representing those actions as simply as possible. Several API calls in sequence can become one coherent tool. Arguments can be omitted or simplified. Similar tools can be combined with an argument that selects the desired operation. Errors also need to explain whether a request timed out or had a bad argument, so the model has enough information to retry or correct itself. ### A small knowledge base can keep long documentation out of the system prompt [10:16](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=616s) Vellum has about 100,000 tokens of documentation covering its product and SDK. Loading all of it into the system prompt would be slow and expensive. Instead, the team created a knowledge base of about 30 documents and put short summaries in the system prompt. The model can call a knowledge-base tool to identify the relevant entry and read more about a specific need, such as which node to use while building an agent. Sidd presents this as a simpler approach than placing all documentation in context or building a more complex retrieval system. ### Testing requires a trade-off between speed, regressions, and test-writing effort [07:12](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=432s) Sidd describes a three-way tension in software testing. Teams can move quickly, avoid regressions, and write many tests, but they cannot fully maximize all three at once. For a personal project that will not change much, skipping tests may be reasonable. Informal "vibes based testing" helps at the start, when a broken experience is easy to distinguish from a working one, and it is useful for judging UI and UX. A rigorous suite becomes more useful when improvements are small, such as distinguishing an 85% pass rate from an 88% pass rate, or when the team needs to stop capabilities from regressing. ### Execution traces show what users actually experience [09:52](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=592s) Sidd recommends storing executions and reviewing them even though it can feel uncomfortable, like watching a recording of yourself giving a talk. User data is valuable, and discarded inputs may be needed later. In one bad Vellum agent-builder execution, the system produced import and type errors, exposed many thinking steps and tool calls, and gave the user a poor response. The team reviewed that execution, found about 20 takeaways, and fixed them. He describes this inspect, learn, and fix cycle as a major part of their development process. ### Text output can become an interactive user interface [11:05](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=665s) An agent does not have to show users only the text produced by a model. Vellum parses text into thinking blocks, tool calls, completion states, and interactive elements. Its agent builder can show a Slack connection control instead of asking users to type every action. Sidd recommends reasonable defaults with deeper visibility available to users who want it. Blocks can be collapsed by default, and interfaces can use buttons or other interactive elements when typing would be unnecessary. He calls this direction "text in UI out." ### Agent builders are moving toward real-time and background work [12:06](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=726s) Vellum's current builder can take roughly 30 seconds to 2 minutes to produce a workflow or agent. Sidd wants users to see more real-time updates as the workflow changes. He also sees mobile use as an important future direction, although Vellum's current mobile experience is poor. Another distinction is between foreground and background agents. The current experience handles one workflow or agent change at a time. A background model could start several changes and let the user return to inspect them later. ### Deployment can expose the same agent through several interfaces [13:21](https://www.youtube.com/watch?v=hIMvuMVLTy8&t=801s) Vellum provides tracing and visibility while agents are being built. When an agent is ready to deploy, users can export executable SDK code into their own codebase, use an API endpoint that runs on Vellum's servers, or publish an AI app with one click. Sidd presents these as different ways to move from an agent built in the natural language interface to something that users or an existing application can access. ## Notable quotes - Sidd Seethepalli: "If you think something might work, you should just try it." (02:33) - Sidd Seethepalli: "You have to set the model up for success when it comes to your tools." (04:21) - Sidd Seethepalli: "Storage is cheap. Tokens are more expensive, but user data is invaluable." (10:16) - Sidd Seethepalli: "A paradigm that I think is really interesting is text in UI out." (12:06) ## Tools & references mentioned - Vellum - Slack ## Who should watch - You are building an agent that often chooses the wrong tool, sends poor arguments, or cannot recover from tool errors. - Your team is deciding when informal testing is enough and when you need regression tests and execution traces. - You are designing an agent interface and want to turn model output into buttons, status indicators, or other interactive elements. ## Related talks - [AI Agents: The Future of Productivity, or Just a Fad?](https://mlopstalks.com/talks/ai-agents-the-future-of-productivity-or-just-a-fad) (Sam Partee, Arcade AI, 35:18) - [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07) - [The Agent Landscape - Lessons Learned Putting Agents Into Production](https://mlopstalks.com/talks/the-agent-landscape-lessons-learned-putting-agents-into-production) (Paul van der Boor & Floris Fok, Prosus Group, 1:08:41) - [9 Commandments Building AI Agents](https://mlopstalks.com/talks/9-commandments-building-ai-agents) (Paul van der Boor & Dmitri Jarnikov, Prosus Group, 1:20:34) - [Building Agentic Tools for Production](https://mlopstalks.com/talks/building-agentic-tools-for-production) (Sam Partee, Arcade AI, 23:55)