Agents use an LLM to decide an application's control flow, rather than following only a fixed sequence of steps.
2
Agent evaluation must measure reliability across repeated runs because agents act in environments and can call an unpredictable number of tools.
3
Gemini provides AI Studio, a Gemini SDK with automatic function calling, built-in tools, and integrations with several agent frameworks.
Summary
Philipp Schmid explains how AI systems have moved from text completion and instruction following toward agents that can reason, call tools, and work through multi-step tasks. He defines an agent as a system where an LLM decides the application's control flow. Tools provide actions, while short-term and long-term memory provide context. He compares agents with fixed workflows and describes reflection, tool use, orchestration, and multi-agent handoff patterns. Evaluation is harder because agents act in real environments, have variable costs, and need task-specific tests. Schmid argues that reliability across repeated runs matters more than a single successful result. He then shows how developers can start with Gemini through AI Studio, the Gemini SDK, automatic function calling, Google Search, and open-source integrations. In the discussion, he recommends choosing frameworks based on existing experience and focusing developer productivity measurements on shipped features and business value rather than lines of code or pull requests.
Agents developed from completion models into systems that take actions
Schmid describes a progression from GPT and GPT-2 completing text, through instruction tuning and multi-turn conversations, to models that call functions. With reasoning models, the response may take minutes or hours rather than seconds. The user provides a goal and a set of functions, then expects the model to perform several actions before returning an answer. Earlier work on browser agents used reinforcement learning to teach models where to click, while later tool-use work taught models when to call tools instead of continuing text.
An agent lets the LLM decide the application's control flow
Schmid defines an agent as a system that uses an LLM to decide the control flow of an application. The distinction is decision-making. A sequence of LLM calls is a workflow when its steps are predetermined. An agent decides whether to call a tool, generate a response, or take another action. He describes the model as the agent's reasoning system, tools as its hands or senses, and context and memory as information supplied during the loop.
Memory divides into conversation context and information from earlier interactions
The agent loop continues until the user's request is served. Schmid separates short-term memory from long-term memory. Short-term memory contains the current conversation. Long-term memory includes information and preferences from prior conversations or a user's experience with the agent. Both are external information that is provided to the agent. This distinction matters when designing what the model should receive during a particular task.
Agent patterns are composable building blocks for software development
Schmid presents reflection, tool use, orchestration, and multi-tool or multi-agent handoff patterns. Reflection generates an answer, reviews it, and creates a revised version. Tool use gives the model a JSON definition and lets it decide whether to call a function. Orchestration has the model make a plan and delegate tasks to other agents or model calls. Handoff passes a shared context between agents, such as moving from flight booking to hotel booking. These patterns can be combined and tested separately.
Agent evaluation must measure reliability over repeated runs
A conventional question-answering system has an input and an output that can be checked, with relatively predictable token costs. An agent may send emails, call several tools, and run through an unknown number of loop iterations. Its cost is therefore harder to predict. Evaluation is also task-specific because agents are built for particular domains. Schmid says teams should measure both capability, which asks whether an answer is correct, and reliability, which asks how often the system is correct across many runs.
Gemini makes function-calling agents accessible through AI Studio and its SDK
Developers can start in AI Studio, where they can experiment with Gemini models, prompt with text, and use built-in tools such as Google Search. For code, the Gemini SDK can turn a Python function's docstring and input parameters into a JSON schema. Gemini can then choose whether to call the function, while the SDK handles the function-calling loop and returns the tool result to the model. Schmid presents this as a simple route to a weather agent and similar systems.
Framework choice should follow the team's existing environment and experience
In the discussion, Schmid says Gemini is integrated with frameworks such as LangChain, LangGraph, CrewAI, and LlamaIndex so that teams are not forced to abandon existing workflows or provider choices. Developers with experience in a framework should use the tool they know best, because that can help them ship faster. Teams without a requirement can compare the abstractions and examples in different frameworks, or implement their own workflows with functions, conditions, and structured outputs.
Context engineering requires giving the model the right information at the right time
Schmid says prompts are only one part of an agent's context. Tools, external information, and the desired output format also matter. Teams need to define success, add guardrails, and test what happens when the agent makes a harmful decision, such as selling a car for an incorrect price. He warns against filling the context window with everything available. Evaluations let developers improve the supplied policies, data, and instructions in a controlled way.
"A perfect agent will most of the time not only just use one pattern, it could be some kind of orchestration pattern with a tool use agent with a reflection agent or with something additional."10:49
Who should watch
You are building an LLM application that needs tool calls, multi-step reasoning, or handoffs between specialized agents.
Your team is deciding whether a fixed workflow is enough or whether the model should choose the next action.
You need an evaluation approach for an agent whose cost, tool calls, and success rate vary across runs.
