# Real World AI Agent Stories

Zach Wallace, Nearpod Inc. | MLOps Podcast | Episode 283 | 47:08
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=GDw29ThkqjM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/real-world-ai-agent-stories
Published: 2025-01-14
Tags: agents, cost, data-engineering, evals

## TL;DR
- Zach Wallace says a reliable agent system starts with a data platform that puts well-defined, current data where models can use it.
- Nearpod builds narrow, specialist agents and an agent registry so teams can combine reusable capabilities instead of asking one agent to handle everything.
- Most of the work comes after the proof of concept, with evaluation, debugging, cost control, and collaboration with subject-matter teams taking far longer than the initial build.

## Summary
Zach Wallace describes how Nearpod moved from a fragmented data architecture toward a system built with dbt, Redshift, Snowflake, S3, and domain-oriented data products. He connects that work directly to AI agents because model responses need current, well-defined data. Nearpod began with agents for generating questions for teachers, while keeping legal, curriculum, and other subject-matter teams close to development. Zach describes agents as narrow consultants, each with a specific job, that can be combined through deterministic or non-deterministic orchestration. He says a proof of concept can be built quickly, but evaluation and debugging take most of the effort. Nearpod uses custom evaluations, CI/CD, telemetry, and cost estimates to decide whether agents are ready for production. The conversation also covers agent registries, decommissioning short-lived agents, legacy code, and the limits imposed by regulation and data quality.

## Key ideas
### Nearpod's data platform turns scattered sources into defined data products
[01:43](https://www.youtube.com/watch?v=GDw29ThkqjM&t=103s)
Zach Wallace says Nearpod had data spread across many systems, including Aurora databases and Dynamo, with no consistent way to transform it or prepare reports. The team uses dbt Core to process data and Redshift's zero-ETL capability to move updates from disparate systems into Redshift. From there, data can go to Snowflake or S3. A data product is defined as the intersection of data and its definition. For example, the times a user logs in or performs an action can form a user-usage data product when the meaning is clearly specified. Nearpod is working toward a data mesh, though Zach says it currently supports batch processing rather than streaming data products.

### The data platform is the foundation for useful agents
[10:28](https://www.youtube.com/watch?v=GDw29ThkqjM&t=628s)
Zach rejects the idea that agents are a separate direction from the data platform. He says the data platform is the first step because models need information about users, systems, and architecture to produce reliable answers or suggestions. Nearpod operates in education, where AI systems must account for students, parents, legislation, language differences, and cultural sensitivities. Zach says large language models can translate in some cases, but they are weaker at adapting language to cultural context and recognizing culturally sensitive topics. Nearpod therefore started with a bounded use case, generating questions to reduce the time teachers spend preparing classroom material.

### Nearpod uses specialist agents instead of one general-purpose consultant
[13:03](https://www.youtube.com/watch?v=GDw29ThkqjM&t=783s)
Zach describes the early agents as being like three-year-old consultants. They can produce a first result, but they need subject-matter experts to define what good work looks like. Nearpod wants separate agents for tasks such as input validation and question generation, with legal and curriculum teams involved where their knowledge is required. The company is building an agent registry so engineers and other teams can find existing agents and combine them into products. In the e-commerce example, separate agents could identify a user's interests, understand which products the business wants to sell, and combine those results into a user profile. Zach says narrow task definitions make the system easier to understand and reuse.

### Narrower tasks reduce model cost and improve reliability
[22:38](https://www.youtube.com/watch?v=GDw29ThkqjM&t=1358s)
Demetrios Brinkmann asks about limiting the scope of both agents and tools. Zach says specialized agents usually need fewer tokens and can process their context faster. He agrees that a smaller scope can reduce cost and improve reliability. Nearpod uses multiple agents because their separate responsibilities make the process easier for engineers to understand and adjust. The agents can be connected through deterministic orchestration when the steps are known, or through a non-deterministic orchestrator that chooses which tools to call. Zach's e-commerce example uses agents for user interests, business priorities, and a later step that combines or summarizes their outputs.

### The proof of concept is fast, while quality work takes most of the schedule
[24:40](https://www.youtube.com/watch?v=GDw29ThkqjM&t=1480s)
Zach says building a proof of concept and making it available to users takes about 10 to 20 percent of the project time. The remaining 80 to 90 percent goes into assessing quality, finding false positives, and adjusting prompts or code. A false positive occurs when an agent appears to complete a task correctly but has actually made a mistake. This changes how teams need to work. Engineers must communicate exactly what each agent is supposed to do, and departments need a shared view of the agent's behavior. Zach says Nearpod has not reached thousands of agents yet, but he expects agent counts to grow substantially.

### Nearpod uses evaluations as the main production safety check
[39:48](https://www.youtube.com/watch?v=GDw29ThkqjM&t=2388s)
To release non-deterministic software, Nearpod treats its evaluations as the single source of truth. Zach says the company has thousands of evaluations for agents, similar to tests in test-driven development. The team creates prompts that could occur in production and adds them to the evaluations, with particular care around sensitive topics and different legislative requirements around the world. The evaluations run through CI/CD and provide confidence levels for each agent. Zach says Nearpod can get results in the 98 to 100 percent range on its evaluations while using lower-cost models. He still accepts that non-deterministic systems carry a residual risk that testing cannot remove.

### Agents can be priced, reused, and retired like short-lived software components
[30:30](https://www.youtube.com/watch?v=GDw29ThkqjM&t=1830s)
Nearpod built a custom evaluation framework for Python and TypeScript and connects it to internal cost and telemetry data. The team estimates production cost from model usage, expected users, and the calls made by each agent. Zach describes an ongoing optimization loop that considers token size, number of calls, prompt wording, quality, and money. He also compares agents to consultants. Because they can be built quickly, a team can retire one after a small amount of work rather than keeping a project alive after months of investment. Zach imagines an agent registry where users can see the cost of individual agents and estimate the combined cost of a proposed workflow.

### Agents change how departments work together on product development
[35:43](https://www.youtube.com/watch?v=GDw29ThkqjM&t=2143s)
Zach says agents make it possible to give other departments a working proof of concept quickly, rather than keeping product ideas inside a long handoff from product teams to engineers. Nearpod can build an initial version in hours, then ask curriculum development, sales, marketing, or other groups what the system misunderstood. This brings subject-matter knowledge closer to the code and gives engineers actionable feedback before they spend months refining the wrong behavior. Existing monoliths still require context from software engineers, but Zach expects agents to help replace or update technically difficult parts of older codebases where they can address the same user need.

### An agent is a model that can perform a job through tools and process
[46:07](https://www.youtube.com/watch?v=GDw29ThkqjM&t=2767s)
Zach says traditional machine learning, large language models, and other models provide the brain power for specific tasks. An agent uses that capability to perform a job. In his view, agents and conventional machine learning are not mutually exclusive. The distinction concerns how a model is connected to actions, tools, context, validation, and other steps. This framing also explains why the Nearpod team breaks work into small specialist agents. Each one handles a defined part of a larger process, while orchestration determines how those parts are combined and how the final result is produced.

## Notable quotes
- Zach Wallace: "The data platform is really the first step, it's getting the data in places that now you can utilize that data for better quality responses or better quality LLM responses." (10:56)
- Zach Wallace: "We need to get those subject matter experts closer to the code, closer to the development cycle." (15:23)
- Zach Wallace: "It takes 80, 90% of your time to debug that." (25:39)
- Zach Wallace: "Our evals were the single truth to our system." (40:12)
- Zach Wallace: "ML or LLMs, any models, are the brain power. Agents are what are able to perform the job for you with that brain power." (46:22)

## Tools & references mentioned
- Nearpod Inc.
- MLOps Community
- dbt Core
- Redshift
- Snowflake
- S3
- Aurora DB
- Dynamo
- Apache Spark
- data mesh
- LLMs
- OpenAI
- ChatGPT
- RAG
- Mixpanel
- HubSpot
- Salesforce
- Meta
- Arsenal
- HIPAA
- Silicon Valley

## Who should watch
- You are building an agent system and need a practical way to split work between specialist agents, orchestration, validation, and data services.
- Your AI proof of concept is easy to build, but evaluation, production risk, model cost, and monitoring are becoming the larger engineering tasks.
- You work in education, health, government, or another regulated area where subject-matter experts and legal requirements must shape the agent before release.

## Editor's note

Zach Wallace says the data platform has to come first because agents need current, well-defined data. That also creates a need to trace how each data product was produced. ZenML records pipeline steps, inputs, outputs, and code version for every run, so teams can connect an artifact or model to the data and code behind it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Before Building AI Agents Watch These Hard Earned Lessons](https://mlopstalks.com/talks/before-building-ai-agents-watch-these-hard-earned-lessons) (Nishikant Dhanuka, Prosus Group, 52:37)
- [Hundreds of Users Love Our Data Analyst AI Agent](https://mlopstalks.com/talks/hundreds-of-users-love-our-data-analyst-ai-agent) (Ioannis Zempekakis & Donné Stevenson, 29:00)
- [The Coming Revolution of AI Agents](https://mlopstalks.com/talks/the-coming-revolution-of-ai-agents) (Jazmia Henry, Iso AI & Rogerio Bonatti, Microsoft & Julia Kroll, Deepgram & Joshua Alphonse, PremAI, 28:59)
- [How AI Agents Are Transforming Data Analytics](https://mlopstalks.com/talks/how-ai-agents-are-transforming-data-analytics) (Ines Chami, Numbers Station, 24:26)
- [Using Agents in Production: Past Present and Future](https://mlopstalks.com/talks/using-agents-in-production-past-present-and-future) (Euro Beinat, Prosus, 23:12)