LiveMCP-101 tests whether agents can coordinate several MCP tools across 101 multi-step tasks.
2
The benchmark compares an agent's final answer with a reference output and its tool-use trajectory, since live data can change between runs.
3
The strongest models still fail through wrong tool selection, malformed or incorrect parameters, and answers produced without calling the required tools.
Summary
This reading-group session discusses the LiveMCP-101 benchmark for testing agents that use the Model Context Protocol. The benchmark contains 101 tasks built from 41 MCP servers. Tasks span areas such as travel, software development, and office work, and require tool chains ranging from 2 to 15 calls. Sophia Skowronski explains how GPT-4.1 and an o3 reasoning model generated the queries, while human reviewers checked the execution plans. The evaluation compares a reference agent with test agents that must choose tools independently, while distractor tools add noise. Valdimar Eggertsson reviews results showing that newer reasoning models perform better, although even the best model completes only part of the benchmark perfectly. The discussion also questions the design: most tasks appear to run once, the judge is another language model, and the generated queries may favor models from the same family. The speakers treat the benchmark as a useful engineering test with limits, rather than a final measure of autonomous agents.
LiveMCP-101 tests multi-step tool coordination rather than isolated function calls
Valdimar Eggertsson introduces LiveMCP-101 as a 101-task benchmark for measuring MCP-enabled agents. The tasks are aimed at agents that receive a challenging natural-language request, decompose it into a plan, select tools, and return an answer. The benchmark evaluates foundation models and also supports experiments with prompts or tool-selection components. The queries are designed to be difficult and sometimes unusual. One example asks an agent to solve a riddle about an NBA team, find travel dates, search accommodation, calculate walking distances, and write the result to a file. This is closer to a code assistant or autonomous agent than to a chatbot answering a single question.
MCP gives agents a standard way to connect to external tools
The session describes the Model Context Protocol as a standard created by Anthropic for connecting agents to tools. Valdimar compares its intended role to HTTP or USB: an MCP server can expose tools and documentation that an agent can use across systems. The paper gives agents access to tools such as travel searches, coding utilities, maps, and other services. The agent receives the tool documentation as part of its context, then has to decide which tools to call and how to combine their results. The discussion also notes that the availability of MCP servers does not necessarily mean that many users are already relying on them.
The benchmark queries were generated and then revised by people
Sophia Skowronski explains that the benchmark was designed to move beyond simple synthetic function calls. Each task had to be solvable, have a verifiable end state, and fit an easy, medium, or hard difficulty level. The authors selected domains from 41 MCP servers, with travel and leisure, software development, and office work among the larger areas. GPT-4.1 selected domains, while an o3 reasoning model drafted the natural-language queries. The early drafts were often messy, so the authors used several rounds of manual revision. The final queries were intended to require multiple tools and to resemble requests made in changing real-world environments.
Reference execution plans make live-data evaluations possible
Because MCP servers can expose live data, a fixed final answer may change between runs. The authors therefore created ground-truth execution plans as well as reference outputs. The plans specify the intended tool order, parameters, and logical steps. Sophia says reviewers spent 120 PhD hours checking the tool choices, parameters, logic, and overall correctness. The reference agent, GPT-4.1, follows these validated plans to produce an output. A test agent receives only the natural-language query and a predefined pool of MCP tools, then has to select tools, write parameters, and decide when it is finished. The design evaluates both the result and the route used to reach it.
Tool distractors test whether agents can find the right tools
Each task includes the tools needed for its intended solution along with unrelated distractor tools. The available pool is built from roughly 15 MCP servers, with the exact tools varying by task. The test agent uses a ReAct-style loop: think, make a tool call, observe the result, and decide whether to continue. Each run has a limit of 30 iteration rounds. The speakers question whether this exactly matches production systems, since an application builder might not deliberately add unrelated servers. They still regard the setup as a useful stress test for tool discovery and selection, because an agent must ignore irrelevant options while finding the tools that match the request.
Valdimar reports that the models were tested on the 101 easy, medium, and hard tasks using task success and average result scores. The best reported task success rate was about 58%, while the 8-billion-parameter Llama model completed only one task perfectly. GPT-5 had the strongest average score at about 73%, compared with about 41% for GPT-4o. The speakers connect the improvement to model size and extended reasoning. Newer models also performed better than older or smaller models, although they used substantial token budgets and made many tool calls. Giving stronger models more iteration rounds helped, while performance for some other models eventually plateaued.
Agents fail through bad parameters, wrong tools, and confident answers without tool use
The error analysis groups failures into semantic and syntactic tool-call errors. A syntactic error has the wrong format, such as supplying a malformed date or the wrong data type. A semantic error has a valid format but an incorrect meaning, such as using the wrong year or an unsupported parameter value. The analysis also includes wrong tool selection, output parsing failures, unproductive thinking, and overconfident self-solving. Overconfident self-solving happens when an agent answers from its own model knowledge instead of calling a tool that contains the needed information. Larger models are less prone to some parameter errors, but tool selection and incorrect parameter meaning remain problems.
The evaluation design makes strong conclusions difficult
The speakers question several parts of the benchmark's experimental design. Most importantly, they understand that each task was run once per test agent, which makes it hard to estimate performance for nondeterministic models. They also note that the benchmark queries were generated with language models and judged with another language model, creating possible style and self-preference bias. Sophia points out that complex queries may have multiple valid tool trajectories, even though the benchmark compares against human-validated plans. The participants suggest repeated runs, paraphrased versions of the same query, more detail about the human evaluation, and judge models that are independent of the model family being tested.
"You cannot quite yet just ask an LLM with MCP tools to do anything you throw at it, especially not these challenging queries."Valdimar Eggertsson07:18
Who should watch
You are building an agent that must choose among several MCP tools and need a test set beyond single function calls.
You want to evaluate tool selection, parameter construction, and multi-step execution while live APIs can change.
You need to interpret agent benchmark scores carefully and want to understand the effects of one-shot runs and LLM-based judging.
