# Beyond Chatbots: How to build Agentic AI systems with Google Gemini

Philipp Schmid, Google DeepMind | Agents in Production 2025 | 27:17

Source: https://www.youtube.com/watch?v=MZbNrW0_1W8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/beyond-chatbots-how-to-build-agentic-ai-systems-with-google-gemini
Published: 2025-09-29
Tags: agents, context-engineering, evals, tool-use

## TL;DR
- Agents use an LLM to decide an application's control flow, rather than following only a fixed sequence of steps.
- Agent evaluation must measure reliability across repeated runs because agents act in environments and can call an unpredictable number of tools.
- Gemini provides AI Studio, a Gemini SDK with automatic function calling, built-in tools, and integrations with several agent frameworks.

## Summary
Philipp Schmid explains how AI systems have moved from text completion and instruction following toward agents that can reason, call tools, and work through multi-step tasks. He defines an agent as a system where an LLM decides the application's control flow. Tools provide actions, while short-term and long-term memory provide context. He compares agents with fixed workflows and describes reflection, tool use, orchestration, and multi-agent handoff patterns. Evaluation is harder because agents act in real environments, have variable costs, and need task-specific tests. Schmid argues that reliability across repeated runs matters more than a single successful result. He then shows how developers can start with Gemini through AI Studio, the Gemini SDK, automatic function calling, Google Search, and open-source integrations. In the discussion, he recommends choosing frameworks based on existing experience and focusing developer productivity measurements on shipped features and business value rather than lines of code or pull requests.

## Key ideas
### Agents developed from completion models into systems that take actions
[01:28](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=88s)
Schmid describes a progression from GPT and GPT-2 completing text, through instruction tuning and multi-turn conversations, to models that call functions. With reasoning models, the response may take minutes or hours rather than seconds. The user provides a goal and a set of functions, then expects the model to perform several actions before returning an answer. Earlier work on browser agents used reinforcement learning to teach models where to click, while later tool-use work taught models when to call tools instead of continuing text.

### An agent lets the LLM decide the application's control flow
[04:32](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=272s)
Schmid defines an agent as a system that uses an LLM to decide the control flow of an application. The distinction is decision-making. A sequence of LLM calls is a workflow when its steps are predetermined. An agent decides whether to call a tool, generate a response, or take another action. He describes the model as the agent's reasoning system, tools as its hands or senses, and context and memory as information supplied during the loop.

### Memory divides into conversation context and information from earlier interactions
[05:13](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=313s)
The agent loop continues until the user's request is served. Schmid separates short-term memory from long-term memory. Short-term memory contains the current conversation. Long-term memory includes information and preferences from prior conversations or a user's experience with the agent. Both are external information that is provided to the agent. This distinction matters when designing what the model should receive during a particular task.

### Agent patterns are composable building blocks for software development
[06:54](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=414s)
Schmid presents reflection, tool use, orchestration, and multi-tool or multi-agent handoff patterns. Reflection generates an answer, reviews it, and creates a revised version. Tool use gives the model a JSON definition and lets it decide whether to call a function. Orchestration has the model make a plan and delegate tasks to other agents or model calls. Handoff passes a shared context between agents, such as moving from flight booking to hotel booking. These patterns can be combined and tested separately.

### Agent evaluation must measure reliability over repeated runs
[11:30](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=690s)
A conventional question-answering system has an input and an output that can be checked, with relatively predictable token costs. An agent may send emails, call several tools, and run through an unknown number of loop iterations. Its cost is therefore harder to predict. Evaluation is also task-specific because agents are built for particular domains. Schmid says teams should measure both capability, which asks whether an answer is correct, and reliability, which asks how often the system is correct across many runs.

### Gemini makes function-calling agents accessible through AI Studio and its SDK
[14:30](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=870s)
Developers can start in AI Studio, where they can experiment with Gemini models, prompt with text, and use built-in tools such as Google Search. For code, the Gemini SDK can turn a Python function's docstring and input parameters into a JSON schema. Gemini can then choose whether to call the function, while the SDK handles the function-calling loop and returns the tool result to the model. Schmid presents this as a simple route to a weather agent and similar systems.

### Framework choice should follow the team's existing environment and experience
[24:44](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=1484s)
In the discussion, Schmid says Gemini is integrated with frameworks such as LangChain, LangGraph, CrewAI, and LlamaIndex so that teams are not forced to abandon existing workflows or provider choices. Developers with experience in a framework should use the tool they know best, because that can help them ship faster. Teams without a requirement can compare the abstractions and examples in different frameworks, or implement their own workflows with functions, conditions, and structured outputs.

### Context engineering requires giving the model the right information at the right time
[19:19](https://www.youtube.com/watch?v=MZbNrW0_1W8&t=1159s)
Schmid says prompts are only one part of an agent's context. Tools, external information, and the desired output format also matter. Teams need to define success, add guardrails, and test what happens when the agent makes a harmful decision, such as selling a car for an incorrect price. He warns against filling the context window with everything available. Evaluations let developers improve the supplied policies, data, and instructions in a controlled way.

## Notable quotes
- "We define an agent as a system that uses an LLM to decide the control flow of an application." (04:51)
- "A perfect agent will most of the time not only just use one pattern, it could be some kind of orchestration pattern with a tool use agent with a reflection agent or with something additional." (10:49)
- "Reliability of agents is much more important than it was compared to traditional LLM use cases because we want our agents to be successful on all of the runs and not only just for one case." (14:30)
- "You don't want to just constantly be stuffing everything as much as you can into the context window." (21:05)
- "If you have experience using Anthropic with CrewAI, use Gemini with CrewAI." (25:25)

## Tools & references mentioned
- Google DeepMind
- Gemini
- Gemma
- GPT
- GPT-2
- ChatGPT
- Claude
- OpenAI
- Meta
- ReAct
- AI Studio
- Google Search
- Gemini SDK
- LangChain
- LangGraph
- CrewAI
- LlamaIndex
- Vercel
- Browser Use
- Browserbase
- SlideLM
- Gemini CLI
- GitHub Copilot
- Databricks

## Who should watch
- You are building an LLM application that needs tool calls, multi-step reasoning, or handoffs between specialized agents.
- Your team is deciding whether a fixed workflow is enough or whether the model should choose the next action.
- You need an evaluation approach for an agent whose cost, tool calls, and success rate vary across runs.

## Related talks

- [AI Agents: The Evolution of Our Digital Friends](https://mlopstalks.com/talks/ai-agents-the-evolution-of-our-digital-friends) (, 10:16)
- [Agentic Relationship Management](https://mlopstalks.com/talks/agentic-relationship-management) (Ashe Magalhaes, Hearth AI, 10:01)
- [AI Agent Development Tradeoffs You NEED to Know](https://mlopstalks.com/talks/ai-agent-development-tradeoffs-you-need-to-know) (Sherwood Callaway, 11X, 57:07)
- [The Future of AI: Building Multi-Agent Systems](https://mlopstalks.com/talks/the-future-of-ai-building-multi-agent-systems) (, 17:30)
- [Sub-Agent Architectures: What You Can Leverage](https://mlopstalks.com/talks/sub-agent-architectures-what-you-can-leverage) (Sidd Seethepalli, Vellum, 14:48)