# Omnigent: Composition, Control, and Collaboration for AI Agents

Denny Lee, Databricks | MLOps Community | 58:17
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=MQqV-v5HqaU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/omnigent-composition-control-and-collaboration-for-ai-agents
Published: 2026-07-15
Tags: agents, guardrails, memory, multi-agent

## TL;DR
- Denny Lee uses debating agents to research whether Taiwan can support high-quality matcha production, including regions, elevation, soil, farmers, and processing infrastructure.
- Omnigent is a meta-harness that keeps agent memory and configuration portable across models and harnesses, while adding forking, policies, budgets, and team sharing.
- AI teams need both developer-level token visibility and central governance, with automatic model selection and evaluation helping control cost without forcing every task onto the most expensive model.

## Summary
Denny Lee describes Omnigent as an open-source layer above models and agent harnesses. He wants developers to keep their context, skills, memory, and configuration while switching between systems such as Claude Code, Codex, Pi, and other models. His matcha research in Taiwan is the practical example: two agents debate regions, soil, elevation, language sources, farmers, and processing requirements, which produces questions he would not have thought to ask. Lee also argues that agent interfaces need conversation forking, because a user should be able to branch a thread while preserving its original context. Managing agents has a span-of-control problem, since people lose track when they supervise too many parallel threads. Omnigent addresses this with portable context, approval policies, budgets, shared sessions, and model flexibility. Lee connects token spending to the earlier shift from CapEx to OpEx, arguing for visibility at the developer level alongside central controls. He expects classic machine learning and evaluation to help choose models automatically.

## Key ideas
### Debating agents can turn a personal obsession into structured research
[01:22](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=82s)
Lee uses Omnigent's Poly workflow to make two agents debate his plans for growing matcha in Taiwan. He gives them background through agent memory and skills files, then asks them to investigate regions, farmers, distributors, elevation, soil pH, and processing infrastructure. The research revealed a practical constraint: matcha leaves oxidize quickly and need immediate steaming and drying, while some Taiwanese tea-growing areas do not yet have that infrastructure. He also uses models for traditional Chinese sources and handwritten material, where language and token requirements create additional problems. The agents help him investigate a business he might otherwise never have had the time or language ability to explore.

### Agent debates are useful because they generate missing questions
[12:06](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=726s)
Lee says the goal of the debate is not to decide whether he should pursue an idea. He expects to pursue it and wants to understand how hard the obstacles will be. By giving both agents the same background and letting them challenge one another, he gets questions about which region to use, its elevation, soil quality, processing methods, and the characteristics a Taiwanese matcha might have. Demetrios describes the benefit as discovering questions that were absent from the original plan. Lee agrees that the process gives a person with strong curiosity but limited subject knowledge a way to find better questions and investigate them.

### Conversation forking should be a standard agent interface
[14:11](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=851s)
Demetrios describes a side-chat widget that reads selected text from an agent conversation or a web page and opens a separate thread. Lee connects this to a feature in Omnigent that lets him fork a conversation while retaining the original context. During his matcha research, he split one thread into Japan and Korea research and kept another focused on Nanto County in Taiwan. Both branches inherited the starting context, but they could develop independently. He argues that ordinary chat windows, including terminal-based agent interfaces, make users lose context as they open more windows and mix unrelated lines of work.

### People lose track when they manage too many agents
[20:38](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=1238s)
Lee applies the military span-of-control idea behind the two-pizza rule to agent work. A manager can effectively handle about five to seven direct reports before losing track of what is happening. He treats agents as junior developers and says the same limit applies when supervising an agent fleet. Forking can help isolate work, but it can also increase the number of parallel threads and make context loss worse. This is why Omnigent focuses on keeping context and memory portable, rather than assuming that simply adding more agents will solve the problem.

### A meta-harness keeps models and harnesses replaceable
[23:44](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=1424s)
Lee says models and harnesses change too often for teams to rebuild their workflows each time. Omnigent sits above them as a meta-harness, carrying configuration, context, and agent memory across different systems. He gives examples of using one model to write code, another to generate tests, and a cheaper model for routine maintenance. The point is not to claim that Omnigent is a better harness than every other option. It provides a composition layer where each model or harness can do the work it handles well. Lee says the project was built with this flexibility in mind because even models inside Databricks can vary by task and by team.

### Policies and budgets belong around the agent workflow
[36:34](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=2194s)
Omnigent includes policies for commands, approvals, and spending limits. Lee describes a rule that can require approval before an agent runs a shell command or exceeds a specified budget. These controls sit outside the prompt, so a user does not have to rely on the model to follow an informal instruction. He also describes shared sessions through an interface similar to a Google Doc. A team member can inspect a debate or agent run, continue it, and add new requirements. This turns an individual agent workflow into something colleagues can review and extend.

### Token economics repeats the move from CapEx to OpEx
[39:34](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=2374s)
Lee compares token spending with the earlier move from capital expenditure to operating expenditure. When infrastructure moved to the cloud, developers had to understand the cost of the resources they created, while central teams still needed governance. He argues that agent usage needs the same balance. Developers should see what they are spending, rather than having only finance teams or executives view dashboards. Central systems should still set limits and route or restrict expensive models. The model choice is difficult because some engineers may spend heavily for good reasons, while others may use expensive models for simple work without realizing the cost.

### Automatic model selection may need ordinary machine learning
[45:22](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=2722s)
Lee thinks model selection should eventually happen automatically, with the system explaining why it chose a particular model and retrying with a stronger one when necessary. He does not assume that another language model must make this decision. Task complexity and other features may be enough for conventional machine learning to route work to a cheaper or more capable model. He also ties this to the renewed focus on evaluation. A debate can give a cheaper model a chance to work, while evaluation checks whether the result meets the required standard. The system then has a basis for deciding whether to retry or accept the output.

### Agents still need accountable human owners and stateful memory
[30:47](https://www.youtube.com/watch?v=MQqV-v5HqaU&t=1847s)
Lee says that code remains the responsibility of the person who submits it, even when an agent helped create it. The same applies to business decisions such as choosing a region for a matcha project. He argues that users need to supply context, background, configuration, and suitable loops instead of expecting a zero-shot prompt to produce useful work. He also places databases at the center of agent systems because agents need cheap, persistent state. He points to Postgres, Lakebase, and Neon as examples of the renewed interest in database-backed memory. In his view, portability and state are long-standing database problems applied to agent workflows.

## Notable quotes
- Denny Lee: "The idea is that if you've got all these different things floating around, you will lose context." (23:23)
- Denny Lee: "The whole point is that I'm just using Omnigent so I didn't care." (28:27)
- Denny Lee: "You actually need both. You need to give the developers the tools to understand what they're doing and what the spend is, but you also need to have some central governance story so you can understand what's going on." (42:08)
- Denny Lee: "It is still the individual, the user, that is the one that submitted the code." (31:10)
- Denny Lee: "We still need something called state. Well, that state is databases." (56:31)

## Tools & references mentioned
- Omnigent
- Databricks
- Claude Code
- Codex
- Pi
- Anthropic
- OpenAI
- DeepSeek
- Postgres
- Lakebase
- Neon
- Delta Rust
- Apache Arrow
- DataFusion
- Apache Iceberg
- FinOps X
- Uni Catalog
- Uni Gateway
- MCP
- Poly
- MLflow

## Who should watch
- You are building coding-agent workflows and need to switch models or harnesses without rebuilding your context and configuration.
- Your team is trying to control token spend while allowing developers to choose the tools that fit their work.
- You want practical ideas for agent interfaces, including conversation forking, debates, approvals, shared sessions, and persistent memory.

## Related talks

- [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07)
- [Beyond the Gold Standard: Evaluating and Trusting Agents in the Wild](https://mlopstalks.com/talks/beyond-the-gold-standard-evaluating-and-trusting-agents-in-the-wild) (Sanjana Sharma, Prosus, 24:45)
- [Architecting Modern AI Systems: Platforms, Agents, and Integration](https://mlopstalks.com/talks/architecting-modern-ai-systems-platforms-agents-and-integration) (Allen Roush, BuzzHPC & Frédéric Bénard, Mila & Shuo Wang, Bell Canada, 57:00)
- [Architecting Modern AI Systems](https://mlopstalks.com/talks/architecting-modern-ai-systems) (, 56:58)
- [Governance for AI Agent Deployment](https://mlopstalks.com/talks/governance-for-ai-agent-deployment) (Spencer Reagan, Airia, 54:18)