# Building Robust AI Systems with Battle-tested Frameworks

Vaibhav Gupta, Boundary ML & Charles Frye, Modal | MLOps Mini Summit 2025 | Episode 10 | 1:08:10

Source: https://www.youtube.com/watch?v=EUnvnroWEtQ
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-robust-ai-systems-with-battle-tested-frameworks
Published: 2025-04-17
Tags: model-serving, reliability, structured-outputs, testing

## TL;DR
- Vaibhav Gupta argues that LLM applications need software-style reliability, with schemas, programmatic checks, fallbacks, tests, and human review for cases that cannot be resolved automatically.
- BAML treats LLM calls as typed functions that transform inputs into structured outputs, giving developers prompt previews, raw API visibility, streaming parsing, and fast test iteration.
- Charles Frye presents Modal as Python infrastructure for running functions on cloud CPUs and GPUs, with autoscaling, deployment, and resource allocation handled without manually maintaining a cluster.

## Summary
Vaibhav Gupta opens by arguing that an LLM application cannot remain at the 80% accuracy level that is acceptable for a demo. Developers need to catch failures before customers see them, much as compile-time checks catch errors in ordinary software. He demonstrates BAML, Boundary's language for defining typed LLM functions, schemas, validations, fallbacks, prompt previews, and tests. His approach uses programmatic checks to detect inconsistent extracted data, then routes errors to another model or a person. Production examples become regression tests over time, creating a data flywheel rather than assuming perfect accuracy on day one.

Charles Frye then introduces Modal as remote procedure calling for cloud infrastructure. A Python decorator can move a function onto cloud CPUs or GPUs, run many inputs in parallel, expose it as a service, and deploy new versions without writing Dockerfiles or managing Kubernetes. He explains how fast autoscaling can improve GPU utilization and response times compared with fixed clusters. The discussion also covers running smaller open models, Modal's integration with Prefect, and the remaining startup costs around serving models such as vLLM.

## Key ideas
### LLM applications need software-level reliability before they reach customers
[03:21](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=201s)
Vaibhav Gupta compares an LLM application that fails 5% of the time with ordinary software that fails 5% of the time. A website may expose a deployment-time error to customers, while a compiler can catch a missing token before deployment. He argues that LLM developers now need similar protection around model calls because APIs can fail, prompts can change meaning through formatting mistakes, and ordinary exception handling is not enough. His target is an application that catches failures during development or routes them safely instead of exposing a broken result.

### Typed schemas turn an LLM call into a component that can be checked
[07:27](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=447s)
BAML defines an LLM function with explicit inputs and outputs. Vaibhav shows an invoice extraction flow that produces a schema while streaming, then uses known relationships such as quantity multiplied by rate equaling amount. Additional checks can compare line-item amounts with the subtotal and compare the subtotal plus tax with the total. These checks do not require another model. They detect that the expected guarantee is false, whether the source document is wrong or the extraction is wrong, so the rest of the system can decide what to do.

### Failures should move through separate software paths instead of one giant prompt
[21:36](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=1296s)
When a validation detects a discrepancy, the application can email a person, ask a model to classify the error, or apply a suggested correction. An extraction error can be corrected automatically, while a document error can go to a human for review. Vaibhav says the model should be one function in a larger system of conditions, loops, validations, and workflows. He also recommends giving classification prompts an option to say that no error exists, since a model can otherwise agree too readily with a leading question.

### Production examples become regression tests that improve accuracy over time
[24:35](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=1475s)
Vaibhav describes reliability as a process rather than a property available on day one. Teams can test PDFs and other examples, collect failures from production, and add those cases to their test set. Each new edge case gives the team another regression test and makes the pipeline more accurate over time. If accuracy is required immediately, the team has to collect representative data and pay that upfront cost. His criticism is aimed at the gap between teams that can build machine learning experiments and teams that can maintain shippable software.

### BAML shortens prompt development by making requests inspectable and executable
[11:50](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=710s)
BAML provides a live prompt preview for test cases, token-level visibility, raw web requests, fallback configuration, and a run-tests button. Vaibhav shows an OpenAI fallback that retries one model and then moves to another model, with the underlying request visible for each provider. The output can be parsed into a typed data model while it streams, even when the model output resembles JSON without being valid JSON. Developers can change a field from a string to a structured type and see the generated client types update.

### Schemas should change with the use case and can be generated once by a model
[16:16](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=976s)
A resume schema for a senior engineer may focus on business impact, while a schema for a graduate student may focus on research and an H-index. Vaibhav says forcing irrelevant fields into the prompt can make the model hallucinate. Charles Frye adds that a model can generate several candidate schemas once, after which a human can choose and validate one before using it across many documents. Filling an approved schema is easier than asking a model to invent a new schema for every inference.

### Modal moves Python functions to cloud hardware with small code changes
[30:28](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=1828s)
Charles Frye presents Modal as remote procedure calling for code that works locally but needs cloud resources. A Python decorator marks a function for Modal, and the SDK sends it to the cloud without requiring a Dockerfile or YAML configuration for the basic case. The same function can run on GPUs, handle many inputs, or become a deployed service. He demonstrates changing a function from squaring a number to cubing it, deploying a new version, and having later calls use the new behavior without a broken intermediate service.

### Fast autoscaling can reduce both GPU waste and user queues
[47:45](https://www.youtube.com/watch?v=EUnvnroWEtQ&t=2865s)
Frye contrasts fixed GPU provisioning with automatic allocation. A fixed cluster can sit mostly unused during quiet periods, then leave users waiting when demand spikes. Slow infrastructure can create a delayed response to demand, because new replicas take minutes rather than seconds to start. Modal's approach adds instances as inputs queue and removes them when work ends. Frye connects this to higher utilization and better service quality, while noting that GPU availability can still vary by model, cloud, region, and demand.

## Notable quotes
- Vaibhav Gupta: "We spend tens of thousand dollars in our companies trying to get to an AI stage that works, but we all get stuck at like because it's 80% good." (03:08)
- Vaibhav Gupta: "We treat an LLM like a calculator. A calculator takes two numbers in and produces a number out." (11:07)
- Vaibhav Gupta: "The point is, you know the guarantee that you're expecting is not correct." (21:36)
- Charles Frye: "The goal being, we want you to focus on the actual business logic on the actual problems that your code is solving, not the hardware infrastructure and nuts and bolts that it needs in order to do that." (35:57)
- Vaibhav Gupta: "You can have a human or an LLM function that takes in the error and the configuration and tells you what the suggested fix is." (24:56)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- Boundary
- BAML
- Modal
- Prefect
- OpenAI
- Anthropic
- Claude
- Gemini
- Llama
- Cursor
- Python
- TypeScript
- Java
- React
- FastAPI
- AWS Lambda
- Kubernetes
- Docker
- Hugging Face
- PyTorch
- vLLM
- DeepSeek
- Flux
- Llama 3.3
- Nvidia SMI
- Nvidia
- H100
- L40S
- A100
- EC2
- EKS
- UV
- ZenML
- Dagster
- Airflow
- Polars
- DuckDB
- Iceberg
- Delta
- S3

## Who should watch
- You are shipping an LLM feature that works in a demo but still produces malformed outputs, inconsistent extractions, or failures that customers can see.
- Your team wants typed LLM outputs, test cases from production failures, and explicit routing to a fallback model or a person.
- You run Python workloads that need GPUs or burstable compute and want to compare a serverless approach with fixed clusters and manually managed infrastructure.

## Related talks

- [The Emerging Toolkit for Reliable, High-quality LLM Applications](https://mlopstalks.com/talks/the-emerging-toolkit-for-reliable-high-quality-llm-applications) (Matei Zaharia, Databricks, 31:01)
- [Impact of LLMs on the Tech Stack and Product Development](https://mlopstalks.com/talks/impact-of-llms-on-the-tech-stack-and-product-development) (Anand Das, Bito, 55:31)
- [Age of Industrialized AI](https://mlopstalks.com/talks/age-of-industrialized-ai) (Dan Jeffries, AI Infrastructure Alliance, 29:56)
- [Building LLM Applications for Production](https://mlopstalks.com/talks/building-llm-applications-for-production) (Chip Huyen, Claypot AI, 35:23)
- [Boosting LLMs: Performance, Scaling, and Structured Outputs](https://mlopstalks.com/talks/boosting-llms-performance-scaling-and-structured-outputs) (Tom Sabo, SAS & Matt Squire, Fuzzy Labs & Vaibhav Gupta, Boundary ML, 1:01:24)