# How to Make AI Agents that ACTUALLY WORK

Patrick Marlo, Google | MLOps Community | 25:43

Source: https://www.youtube.com/watch?v=3y17VO3C8jo
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-to-make-ai-agents-that-actually-work
Published: 2024-12-06
Tags: agents, evals, guardrails, prompt-engineering

## TL;DR
- Production agents need system design around the model, including prompts, grounding, orchestration, API integrations, CI/CD, and analytics.
- Meta-prompting uses one AI system to generate or improve prompts for a target agent, with evaluations feeding back into later prompt versions.
- Agents need layered safety controls and evaluations at every stage of a pipeline, especially when changes can affect tool calls, retrieval, or generation.

## Summary
Patrick Marlo shares lessons from building generative AI agents with developers, customers, and partners. He argues that an agent in production is a software system, not only a language model. The surrounding work includes prompt design, grounding, orchestration, APIs, CI/CD, analytics, safety controls, and evaluation. He explains meta-prompting, where one model creates or improves prompts for another agent, either from a seed prompt or from production evaluation results. He recommends multiple safety layers, including input filters, conversation limits, output controls, error handling, retries, and response caching. His strongest recommendation is to build evaluations from the start. A golden dataset defines expected responses and tool calls, then compares them with runtime behavior. For multi-stage retrieval systems, each stage needs its own evaluation so teams can identify whether problems come from query rewriting, retrieval, reranking, or summarization.

## Key ideas
### A production agent is a software system around a model
[04:44](https://www.youtube.com/watch?v=3y17VO3C8jo&t=284s)
Patrick says teams often focus on which language model they are using, such as Gemini, GPT-4, or another model. In production, the model is only one part of the system. Teams also need grounding, tuning, prompt engineering, orchestration, API integrations, CI/CD, and analytics. He expects many models to become fast, capable, and inexpensive over time. The remaining difference will come from the ecosystem built around the model. His point is practical: agent development requires the same system design and software development life-cycle thinking used for other production systems.

### Meta-prompting can create higher-fidelity prompts
[06:38](https://www.youtube.com/watch?v=3y17VO3C8jo&t=398s)
Meta-prompting uses AI to build AI. A meta-prompting system generates a prompt for a target agent, and the target agent produces responses that can later be evaluated. Patrick compares a short handcrafted prompt with a longer version produced by an optimizer. The semantic intent remains similar, but the generated version describes the tasks in more detail and structures the instructions more clearly. He says this helps when a developer is building an agent for the first time, especially when the developer is not confident in prompt engineering or does not consider themselves a strong creative writer.

### Seed prompts provide a practical starting point for agent development
[09:09](https://www.youtube.com/watch?v=3y17VO3C8jo&t=549s)
Patrick describes a seed-based approach to meta-prompting. The developer writes a system prompt for the meta-prompting model, then supplies a seed prompt containing details about the company, the user, and the task. The meta-prompting system turns that seed into prompts for the target agent. The developer can repeat the loop and refine the generated prompts before putting them into the target system. This gives teams a way to start with a basic description and have a language model expand it into instructions with more detail and structure.

### Production feedback can drive prompt optimization
[10:45](https://www.youtube.com/watch?v=3y17VO3C8jo&t=645s)
The second meta-prompting approach starts after an agent is in production. The team collects responses, evaluates them against measures such as coherence, fluency, semantic similarity, and tool-calling performance, then sends the existing prompt and evaluation results back to the meta-prompting system. The optimizer can propose changes aimed at improving a chosen measure or reducing losses in tool calling. Patrick presents this as a loop that can be repeated as the team learns more about agent behavior. He names DSPy, AdalFlow, and Vertex Prompt Optimizer as examples of systems that can support this kind of work.

### Safety requires controls before and after the agent
[12:27](https://www.youtube.com/watch?v=3y17VO3C8jo&t=747s)
Patrick warns that a UI with prompt engineering as its only defense is insufficient for a public agent. Attackers may try prompt injection or other ways to manipulate the system. Input filters can check language, categories, and session limits before the request reaches the agent. He notes that some attacks develop over many turns, so limiting a conversation to a defined number of turns can reduce that risk. On the way out, teams need error handling, retries, and controlled JSON outputs. He describes these as multiple layers rather than a single defensive measure.

### Caching can improve an agent without involving the model
[14:50](https://www.youtube.com/watch?v=3y17VO3C8jo&t=890s)
Patrick says teams sometimes overlook caching because they want to build with the newest technology. If the same query appears repeatedly, the system can cache its response and return it without invoking the agent. This reduces token costs and improves response speed. He frames the decision around the quality of the outcome rather than whether the latest model was used. Production analytics can then show what users ask about and provide signals for updating prompts, input filters, and output filters.

### Golden datasets turn agent quality into something measurable
[16:12](https://www.youtube.com/watch?v=3y17VO3C8jo&t=972s)
Patrick calls evaluations the most important action a team can take if it does nothing else. A golden dataset, also called expectations, defines the desired behavior for an agent. It can specify that a user message should produce a particular response, or that a certain request should call a tool with particular inputs before returning an answer. Runtime responses are compared with these expectations and scored with measures such as semantic similarity, tool calling, coherence, and fluency. The expectations stay mostly fixed while the agent changes, making regressions visible after new features or prompt changes.

### Each stage of a retrieval pipeline needs its own evaluation
[19:24](https://www.youtube.com/watch?v=3y17VO3C8jo&t=1164s)
Patrick describes a multi-stage retrieval pipeline with query rewriting, vector retrieval, reranking, and final summarization. Evaluating only the final output can show that quality has declined without explaining why. If a new model is used throughout the pipeline, the team cannot tell whether the problem comes from rewriting, retrieval, reranking, or summarization. Evaluating each stage separately helps locate the loss. The team might then keep the new model in some stages and restore the previous model in another stage, such as summarization, to recover the best overall result.

## Notable quotes
- "Production is not just a simple model." (04:44)
- "The TLDR around meta prompting is essentially you're using AI to build AI." (06:38)
- "If you don't do anything else, implement evaluations." (16:12)
- "The only thing that matters when you're putting these systems into production is the outcome that they achieve." (14:49)
- "The best way to manage versions of Agents is kind of treat them the same way that you would in a traditional software development life cycle with CI/CD." (24:30)

## Tools & references mentioned
- Google
- Vertex AI
- Gemini
- Dialogflow CX
- Vertex Agents
- Vertex Search
- LangChain
- DSPy
- AdalFlow
- Vertex Prompt Optimizer
- Claude
- OpenAI
- Rapid Eval SDK
- Scrappy
- Kaggle
- Dolly Llama

## Who should watch
- You are building an agent that works in a demo but becomes difficult to control after adding tools, retrieval, or new prompts.
- Your team needs a practical method for improving prompts without relying entirely on manual prompt writing.
- You are changing a retrieval or agent pipeline and need to find which stage causes quality regressions.

## Related talks

- [Meta-Prompting: The Hack That's Changing Production AI](https://mlopstalks.com/talks/meta-prompting-the-hack-thats-changing-production-ai) (Aman Khan, Arize, 19:28)
- [How to Optimize AI Agents in Production](https://mlopstalks.com/talks/how-to-optimize-ai-agents-in-production) (, 27:06)
- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)
- [Building Reliable AI Agents](https://mlopstalks.com/talks/building-reliable-ai-agents) (Travis Fischer, 17:43)
- [The Science of Improving AI Agents](https://mlopstalks.com/talks/the-science-of-improving-ai-agents) (, 26:43)