# Before Building AI Agents Watch These Hard Earned Lessons

Nishikant Dhanuka, Prosus Group | MLOps Community | 52:37
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Z74Ok22wLeE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/before-building-ai-agents-watch-these-hard-earned-lessons
Published: 2025-09-05
Tags: agents, context-engineering, evals, search

## TL;DR
- Nishikant Dhanuka says context engineering matters more than choosing between already capable state-of-the-art models, because useful agents need live enterprise data, user history, and the user's current request.
- Shopping agents need hybrid search pipelines that combine keyword search, semantic search, query understanding, and LLM-based reranking for broad or ambiguous requests.
- Teams should start agent evals with business outcomes and real conversations, then use human labeling parties to improve LLM judges instead of beginning with complex tool-level analysis.

## Summary
Nishikant Dhanuka describes lessons from building productivity and e-commerce agents at Prosus Group. He argues that context engineering is usually more important than prompt tweaks or model selection. A useful agent needs live business data, promotions, opening hours, payment details, and user history. Search remains difficult, so shopping agents need a pipeline that combines keyword and semantic search with query understanding and reranking. Dhanuka has also found that a pure chatbot often fails in production because users do not adopt a new interface without immediate value. He prefers experiences that combine chat with dynamic UI components and narrow, contextual help. For evaluation, he recommends starting with business metrics and user intent, using synthetic queries before launch, and running regular labeling parties with business stakeholders. More detailed tool and trace analysis can follow when the simpler checks reveal a problem.

## Key ideas
### Context quality often matters more than model choice
[00:00](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=0s)
Dhanuka says teams spend too much time discussing system prompts, prompt tweaks, MCP tools, and which model is best. When two models are already state of the art, the larger difference comes from whether one has the right context. He compares this with data engineering and the idea of garbage in, garbage out. For a food agent, context includes live promotions, restaurant opening hours, payment-card support, and the user's situation. A model that can search for pizza but does not know that nearby restaurants are closed will not be useful in the real world.

### Enterprise context requires messy real-time data work
[05:25](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=325s)
The data needed by an agent is often scattered across real-time databases rather than stored in one clean source. A restaurant might run a sushi promotion only from noon to 3 p.m., which is difficult to represent in a conventional database. Dhanuka describes engineers connecting these sources and triggering data pipelines when a request arrives. If a user asks for sushi on promotion, the system must fetch the relevant information, add it to the prompt context, and then generate the response. He calls this unglamorous work a major part of building useful agents.

### User history provides a practical cold-start memory
[07:47](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=467s)
Dhanuka divides context into the system prompt, the user's current message, enterprise data, and user history. He says memory can create product stickiness because users may stay with a tool that knows their preferences and past activity. Teams do not need to begin with an elaborate long-term memory architecture. An existing shopping or food app already knows what users ordered and browsed, so that information can provide cold-start context when the assistant launches. Later conversations can add dynamic memory after users have used the product for some time.

### E-commerce search needs several stages
[11:20](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=680s)
Keyword search remains useful for direct requests such as burger, but it misses broad requests such as planning a romantic dinner or furnishing a house. Semantic search can connect related concepts, such as vegetarian pizza and pizza margherita, but it cannot fully resolve subjective terms such as romantic. Dhanuka describes a pipeline with query understanding before search, keyword and semantic retrieval, and an LLM reranking stage after retrieval. The system can use the user's profile and the original request to expand the query, select candidates, and present the most relevant options.

### A new interface needs immediate value
[18:25](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=1105s)
Dhanuka says internal testing can make a conversational shopping assistant look excellent, while an A/B test with real users shows poor conversion. Users already understand familiar interfaces, so they will not adopt a new one unless it solves a meaningful problem. He recommends showing the value within the first 30 seconds and guiding users with onboarding, examples, tooltips, and interactive elements. A plain chatbot can hide its capabilities, while a redesigned UI can create its own friction. The product has to make the reason for changing behavior obvious.

### Chat and dynamic UI work better together
[26:45](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=1605s)
Dhanuka's current view is that the best interface combines conversation with generated UI. The agent can choose among components such as item carousels, related products, or other widgets based on the user's request and earlier actions. This matters in shopping and food because people want to scroll, click, swipe, and use images while deciding. Traditional recommendation systems still matter when users click or browse. He also describes a proof of concept where an agent watches actions on the screen and responds to them, while acknowledging that users differ in how much observation they accept.

### Narrow contextual help can reduce interface friction
[31:00](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=1860s)
Instead of presenting one chatbot that claims to do everything, Dhanuka describes assistants that appear at the right moment for a small task. A floating control might offer to compare a headphone with a newer Apple model while the user is viewing that product. The comparison can use a simple LLM call connected to a tool, without requiring a full agent. He says these micro-tasks work better in experiments because users do not have to learn an entirely new interface or guess what the assistant can do.

### Start evals with business outcomes and real conversations
[36:36](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=2196s)
Dhanuka calls evals a system for checking whether an AI product is good enough before launch and after it reaches production. He recommends starting with practical signals such as whether the conversation satisfied the user's intent, moved toward an order, reached the cart, or produced another relevant funnel event. These deterministic signals should come before immediately using an LLM as a judge. Teams can create an initial dataset from manual queries, synthetic queries, or generated user personas. A simple first pass often reveals enough problems to fix before teams analyze every tool call and parameter.

### Human labeling improves automated judges
[47:16](https://www.youtube.com/watch?v=Z74Ok22wLeE&t=2836s)
Dhanuka describes a recurring labeling party where engineers, business stakeholders, and other team members review real conversations for about an hour and a half. They answer simple questions about user intent, business progress, and guardrails. The resulting human labels are compared with the LLM judge's labels. Differences show where the judge is too lenient or too strict, and those examples become few-shot context for a revised judging prompt. He recommends repeating the exercise regularly. Custom annotation tools can also show the products, restaurants, and other visual information needed to judge a shopping conversation.

## Notable quotes
- Nishikant Dhanuka: "What makes the difference between A and B is A is with context, B is without context." (03:20)
- Nishikant Dhanuka: "Search is about when you're searching for dishes or when you're searching for restaurants. But when you're searching for does McDonald's have promotion, does McDonald's accept payment card, does McDonald's open right now, that is context." (07:19)
- Nishikant Dhanuka: "The best interface is a mix of UI and chat." (26:45)
- Nishikant Dhanuka: "I believe that eval is the real moat of your product and not your system prompt." (36:45)
- Nishikant Dhanuka: "Don't run immediately to LLM as a judge. Look for those metrics whatever those metrics are." (42:37)

## Tools & references mentioned
- Prosus Group
- Tokan
- OLX
- ChatGPT
- Andrej Karpathy
- MCP
- Alexa
- TikTok
- Domino's
- Netflix
- Langfuse
- v0
- Vercel
- Label Studio

## Who should watch
- You are building a shopping, food-ordering, or productivity agent and need practical guidance on context, search, and interface design.
- Your agent works in demos but users do not adopt it or conversion is weak in production.
- You are designing an eval process and want to start with business signals and human review before building complex trace-level checks.

## Editor's note

Nishikant Dhanuka says useful agents depend on messy enterprise data being fetched and added to the context when a request arrives. ZenML lets teams write that workflow as Python pipeline steps and run it on different infrastructure through configuration. Each run records its steps, inputs, outputs, and code version, so each data artifact can be traced back to its source.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Everything Hard About Building AI Agents Today](https://mlopstalks.com/talks/everything-hard-about-building-ai-agents-today) (Shreya Shankar & Willem Pienaar, Cleric, 47:03)
- [Beyond the Gold Standard: Evaluating and Trusting Agents in the Wild](https://mlopstalks.com/talks/beyond-the-gold-standard-evaluating-and-trusting-agents-in-the-wild) (Sanjana Sharma, Prosus, 24:45)
- [Iterating on Your AI Evals](https://mlopstalks.com/talks/iterating-on-your-ai-evals) (Mariana Prazeres, 13:47)
- [Navigating the AI Frontier: The Power of Synthetic Data and Agent Evaluations in LLM Development](https://mlopstalks.com/talks/navigating-the-ai-frontier-the-power-of-synthetic-data-and-agent-evaluations-in) (Boris Selitser, Okareo, 57:22)
- [Advancing the Cost-Quality Frontier in Agentic AI](https://mlopstalks.com/talks/advancing-the-cost-quality-frontier-in-agentic-ai) (Krista Opsahl-Ong, Databricks, 12:45)