# Stop Shipping on Vibes: How to Build Real Evals for Coding Agents Jessica Wang, Braintrust | Coding Agents Conference 2026 | 29:09 Source: https://www.youtube.com/watch?v=VbX24V_JFQI Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/stop-shipping-on-vibes-how-to-build-real-evals-for-coding-agents Published: 2026-03-31 Tags: agents, coding-agents, evals, search ## TL;DR - Teams should replace prompt-based shipping decisions with evals that measure test cases, quality changes, regressions, cost, and performance across real scenarios. - A useful eval combines a dataset, a task, a scoring system, and repeated experiments, then feeds production observations back into the application. - In Jessica Wang's search experiment, agentic search scored at least as well as vector search and used fewer tokens in the reported comparisons, though she says the eval needs more trials and a better vector-search implementation. ## Summary Jessica Wang argues that teams often ship AI features because a PM tried a few prompts or an engineer declared the feature ready. She proposes using evals to make those decisions measurable. An eval needs test data, a defined task, a scoring method, and experiment runs that can be compared for regressions, quality changes, cost, and latency. Wang then walks through an unfinished comparison of agentic search and vector search for coding agents. She tested buggy code tasks from Microsoft's TypeScript Go repository and SWE-bench Verified Django examples. Agentic search performed as well as or better than vector search in the reported results, while vector search used more tokens and money. Wang is careful about the limits of the result. The vector implementation was basic, the trials were limited, and Claude Code's restrictions may have affected token usage. She recommends repeated trials, larger datasets, improved retrieval, and hybrid search. ## Key ideas ### Shipping AI features from a few prompts is shipping on vibes [00:57](https://www.youtube.com/watch?v=VbX24V_JFQI&t=57s) Wang has joined calls where teams said they shipped an AI feature because an engineer said it was ready or a product manager tried a few prompts and liked the result. She calls this a poor basis for a ship decision. A stronger decision could say that 200 test cases were run and 94% passed. It could also capture tradeoffs, such as a change improving tone while reducing accuracy by 5%. The point of an eval is to make those effects visible before and after a release, rather than relying on a small set of informal impressions. ### An eval combines data, a task, scoring, and experiments [03:17](https://www.youtube.com/watch?v=VbX24V_JFQI&t=197s) Wang breaks an eval into four parts. The dataset contains golden use cases, edge cases, and known failure modes. The task defines how the system should respond, including the prompt and selected model. The scoring system decides what counts as good or bad, using deterministic checks, an LLM judge, or human review. An experiment is one run of a particular dataset, task, and scoring configuration. Running different hypotheses and configurations creates comparable experiment runs, which can reveal regressions and improvements. ### Production logs can become the next version of the eval dataset [05:48](https://www.youtube.com/watch?v=VbX24V_JFQI&t=348s) Wang describes a loop that begins with live application logs. For a documentation chatbot, logs can contain the user's question, the model's response, and additional metadata. A team can sample 10 to 20% of production traffic and turn those examples into a dataset. It can then adjust the dataset, score, or prompt, such as asking for a more concise or more enthusiastic answer. The results inform code changes in the application, and the updated application generates more production data for future evaluation. ### Evals need several roles because the work includes judgment and analysis [07:41](https://www.youtube.com/watch?v=VbX24V_JFQI&t=461s) Wang calls evals a team sport. An AI engineer brings data into the platform and changes bugs or features. A product manager develops hypotheses and defines success criteria, although the same person may fill both roles in some teams. Subject matter experts label data and adjust prompts in fields such as medicine, law, and insurance. Data analysts help define scores and examine the results. This division matters because the technical setup alone cannot decide what a useful response means in a specialized domain. ### Agentic search follows code relationships while vector search retrieves nearby chunks [09:24](https://www.youtube.com/watch?v=VbX24V_JFQI&t=564s) Wang compares two ways for a coding agent to find relevant code. Vector search converts text or code into embeddings, stores them in a vector database such as Pinecone, and retrieves items near the query in semantic space. Agentic search gives the model command-line tools such as grep, find, ls, and cat. The model can search for a function, open its file, follow a referenced function, and continue through the codebase. Vector search provides proximity to relevant code, while agentic search can follow the connective tissue between files. ### The search comparison used buggy repositories and test outcomes as the score [12:23](https://www.youtube.com/watch?v=VbX24V_JFQI&t=743s) Wang built two datasets. The first used merged fixes from Microsoft's TypeScript Go repository. She checked out the parent commit so each task began in a known buggy state, then used Claude to create a bug description from the pull request diff. The second used 25 Django-related rows from SWE-bench Verified. The agent received the bug description and had to fix the issue. Passing the repository's relevant tests counted as success. Wang used two codebases because TypeScript Go is newer and more modular, while Django is a larger legacy codebase. ### A fair comparison required blocking Claude Code from using the wrong search method [14:34](https://www.youtube.com/watch?v=VbX24V_JFQI&t=874s) Claude Code already uses agentic search, so Wang could use it directly for that condition. The vector-search condition required a script and two restrictions. She used Claude Code's disallow-tools flag to block grep, find, and similar agentic tools. She also instructed the model to use vector search whenever it needed search. Without both controls, Claude Code kept switching back to agentic search, which would have invalidated the comparison. This setup took several days because the model strongly preferred its default search behavior. ### The reported results favor agentic search, but Wang does not treat them as final [18:40](https://www.youtube.com/watch?v=VbX24V_JFQI&t=1120s) On 25 Django tasks from SWE-bench Verified, vector search scored 60% and agentic search scored 68%. On the TypeScript Go tasks, both scored 70%. Wang says vector search used substantially more tokens and money in the comparison, though she doubts the exact cost shown because the reported total did not match her experience. Her trace review found vector search repeatedly retrieving incomplete code chunks. One run made 26 searches while guessing where the bug was. Agentic search could read files, follow imports, and trace calls more directly. ### A credible eval needs repeated trials, better retrieval, and broader data [22:34](https://www.youtube.com/watch?v=VbX24V_JFQI&t=1354s) Wang says she would not publish the comparison as a finished result. Language model runs are nondeterministic, so she would repeat each task and average the scores. She would improve the basic vector-search implementation with techniques such as chunk overlap, different text splitting, or retrieval models. She would also test a hybrid approach because companies combine vector and agentic search, then expand the dataset with more rows, languages, and codebases. She estimates repeated runs could vary the score by 10 to 15% under the same criteria. ## Notable quotes - "You are essentially making ship decisions based off of vibes, which is not good." (00:57) - "Evals are team sport." (07:41) - "Vector search gave the agent a lot of proximity to relevant code, but didn't give the connective tissue between the code for it to actually implement a fix." (21:29) - "I don't consider this eval close to being done at all." (22:34) ## Tools & references mentioned - Braintrust - OpenAI - Pinecone - Claude Code - Microsoft's TypeScript Go repository - SWE-bench Verified - Django - Claude Sonnet ## Who should watch - You ship AI features after informal prompt checks and need a concrete way to decide whether a change is better. - Your team is building coding agents and wants to compare retrieval approaches using real repository tasks and test outcomes. - You have production traces but have not yet turned user failures and successful interactions into an evaluation dataset. ## Related talks - [It's 2026, and We're Still Talking Evals](https://mlopstalks.com/talks/its-2026-and-were-still-talking-evals) (Maggie Konstanty, Prosus, 40:57) - [Evals Aren't Useful? Really?](https://mlopstalks.com/talks/evals-arent-useful-really) (Chiara Caratelli, Prosus Group, 25:25) - [The Science of Improving AI Agents](https://mlopstalks.com/talks/the-science-of-improving-ai-agents) (, 26:43) - [Iterating on Your AI Evals](https://mlopstalks.com/talks/iterating-on-your-ai-evals) (Mariana Prazeres, 13:47) - [Evaluating AI Agents: Why It Matters and How We Do It](https://mlopstalks.com/talks/evaluating-ai-agents-why-it-matters-and-how-we-do-it) (Annie Condon & Jeff Groom, Acre Security, 13:27)