# Open vs Closed Source Agent Infra?

Adel El Hallak, NVIDIA | Coding Agents Conference 2026 | 30:45
Hosted by Ben Epstein

Source: https://www.youtube.com/watch?v=TTAwep2uwto
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/open-vs-closed-source-agent-infra
Published: 2026-02-28
Tags: agents, debugging, observability, open-source

## TL;DR
- Open-source agent frameworks are useful for learning what works, but teams should start with the lowest abstraction they can understand and debug.
- Teams often begin with frontier models to prove a use case, then move to open models when privacy, compliance, scale, or control becomes important.
- Large companies need shared infrastructure for operational concerns while still allowing teams to experiment with different agent frameworks.

## Summary
This panel discusses how teams should choose between open and closed agent infrastructure. Laurel recommends experimenting with several open-source frameworks to learn where agents succeed and fail, while starting with a closed model unless the team already knows how to host models. Adel describes a similar progression: use frontier models to prove the use case, then consider open models for compliance, privacy, and production control. The panelists warn that highly abstract frameworks can make debugging harder, especially when many agents exchange messages. They prefer simple frameworks and clear evaluation data. Adel explains NVIDIA's NeMo Agent Toolkit, which works across frameworks and emits OpenTelemetry traces instead of forcing teams into one observability product. The discussion also separates sandbox experimentation from production standards. Large organizations can standardize Kubernetes, artifact storage, privacy controls, and tracing while leaving room for exploration. Low-code agents are more suitable for internal workflows than customer-facing systems, where teams need much higher precision.

## Key ideas
### Open-source frameworks help teams learn where an agent works before they commit
[05:46](https://www.youtube.com/watch?v=TTAwep2uwto&t=346s)
Laurel argues that open agent platforms are a useful starting point because teams can try several approaches quickly. She says there may be five to 20 common platforms, and that working with them for a couple of weeks can reveal what fits a particular use case. Her recommendation is to treat the early stage as an experiment: identify where agents fail, what they handle well, and what a production system would need to do better. She recalls customers who had tried LangChain and CrewAI before evaluating another product, which gave them a concrete baseline for comparison.

### Teams can prove the use case with a closed model before moving to open models
[08:11](https://www.youtube.com/watch?v=TTAwep2uwto&t=491s)
Adel says teams should first demonstrate that an agent can solve the business task, often using frontier models such as Anthropic or OpenAI systems. He describes a later transition driven by production requirements. NVIDIA could not send prompts elsewhere for compliance reasons, so the team had to deploy some open-source models internally. He connects the move to open models with privacy, scale, and control. The panel treats model choice as a separate decision from the choice of agent framework.

### Highly abstract frameworks can hide the behavior teams need to debug
[10:47](https://www.youtube.com/watch?v=TTAwep2uwto&t=647s)
Ben says the claim that open-source software can always be fixed is too optimistic. Some frameworks make it easy to start, then leave teams stuck behind abstractions they do not understand. Laurel agrees that teams should use the lowest abstraction that gives them useful building blocks. She describes a simple agent as state plus a loop that keeps choosing actions until the state is complete. Her team eventually abandoned an internally built framework and chose a simpler production framework rather than rebuilding every component from scratch.

### A capable model with curated tools can be easier to maintain than many communicating agents
[12:45](https://www.youtube.com/watch?v=TTAwep2uwto&t=765s)
Laurel explains that earlier frameworks used multiple agents and communication protocols as a substitute for reasoning. The arrival of reasoning models changed that trade-off for her team. Their current preference is usually one orchestrator, a strong model, and a carefully selected set of tools. Eight agents exchanging messages can become difficult to debug because one mistake in the message history can confuse the whole system. Simpler frameworks still provide model-call and communication primitives, while avoiding a large amount of hidden behavior and boilerplate.

### Shared agent infrastructure can support many frameworks without forcing one choice
[14:23](https://www.youtube.com/watch?v=TTAwep2uwto&t=863s)
Adel says NVIDIA found internal teams building agents with LangGraph, CrewAI, AutoGen, Semantic Kernel, and plain Python. Rather than stopping that experimentation, NVIDIA built the NeMo Agent Toolkit to work with different frameworks. The toolkit adds interoperability, system-wide observability, and profiling across agents. Adel says this makes it possible to collect traces from a system built from several frameworks and then make decisions about how to run it more efficiently. He also notes that data location can influence framework choices.

### Open telemetry keeps observability portable across an organization's existing tools
[18:11](https://www.youtube.com/watch?v=TTAwep2uwto&t=1091s)
Adel says the NeMo Agent Toolkit emits OpenTelemetry traces rather than requiring teams to abandon their existing logging and observability products. NVIDIA uses Datadog, Weights & Biases, and LangSmith internally, so he does not want to force teams onto one platform. Olga describes a similar approach in her organization, where users can choose among solutions as long as they understand the trade-offs around compatibility, cost, scale, and fine-tuning. Laurel says her small company stores JSON logs in a lakehouse and queries them with SQL.

### Large organizations need standard production building blocks alongside flexible experimentation
[21:21](https://www.youtube.com/watch?v=TTAwep2uwto&t=1281s)
Adel separates a sandbox for trying frameworks from the process of operationalizing an agent. NVIDIA standardizes some shared infrastructure, including Kubernetes for scale and orchestration and JFrog Artifactory for artifacts, while allowing teams to explore different frameworks in the sandbox. Production also requires privacy controls. Adel says some teams cannot have their prompts or traces collected, so simply sending all traces to a data lake is not acceptable. The production path therefore includes compliance and privacy decisions that do not appear in a quick prototype.

### Low-code agents fit internal workflows more readily than high-precision customer systems
[27:07](https://www.youtube.com/watch?v=TTAwep2uwto&t=1627s)
Laurel says low-code builders can be useful, but many users would rather describe a problem and have it handled than configure a system prompt themselves. She prefers workflow builders and templates because they are easier to understand than a blank text box. Olga supports internal agent building when users receive guidance and governance, but is more cautious for systems that affect end users. Ben frames the decision around failure tolerance: internal tools can work at lower precision, while customer-facing agents need much higher scores on an evaluation set before release.

## Notable quotes
- Laurel: "The community kind of latched on to this and realized it was something they could quickly build and iterate on." (06:09)
- Laurel: "If you have a good model with curated tools, it's going to do far better than a very advanced framework." (12:45)
- Adel El Hallak: "We force the whole ecosystem, all we do is spit out OpenTelemetry traces." (18:11)
- Adel El Hallak: "You have to differentiate between a sandbox that allows for exploration versus how you operationalize those agents." (21:21)
- Ben Epstein: "If you're doing internal tooling, it's really fun and really powerful to give nontechnical people ways to automate every part of their job that is automatable." (29:45)

## Tools & references mentioned
- Demetrios Brinkmann
- NVIDIA
- GrottoAI
- Anthropic
- OpenAI
- Llama 3
- DeepSeek
- LangChain
- CrewAI
- LangGraph
- AutoGen
- Semantic Kernel
- NeMo Agent Toolkit
- OpenTelemetry
- Datadog
- Weights & Biases
- LangSmith
- BAML
- ZenML
- Kubernetes
- Red Hat
- JFrog Artifactory

## Who should watch
- You are deciding whether to start an agent project with a hosted model, an open model, or an open-source framework.
- Your team has several agent frameworks in use and needs shared tracing, deployment, privacy, or compliance standards.
- You are building internal automation or customer-facing agents and need to set different expectations for precision and support.

## Related talks

- [Graduating from Proprietary to Open Source Models in Production](https://mlopstalks.com/talks/graduating-from-proprietary-to-open-source-models-in-production) (Philip Kiely, Baseten, 23:16)
- [How AgentOps Enables Observability](https://mlopstalks.com/talks/how-agentops-enables-observability) (Nehil Jain, Stealth AI Startup & Adam Becker, MLOps Community & Valdimar Eggertsson, Snjallgögn (Smart Data inc.) & Binoy Perera, MLOps Community, 49:48)
- [Too much lock-in for too little gain: agent frameworks are a dead-end](https://mlopstalks.com/talks/too-much-lock-in-for-too-little-gain-agent-frameworks-are-a-dead-end) (Valliappa Lakshmanan, 35:37)
- [Architecting Modern AI Systems: Platforms, Agents, and Integration](https://mlopstalks.com/talks/architecting-modern-ai-systems-platforms-agents-and-integration) (Allen Roush, BuzzHPC & Frédéric Bénard, Mila & Shuo Wang, Bell Canada, 57:00)
- [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07)