Nishikant Dhanuka says context engineering matters more than choosing between already capable state-of-the-art models, because useful agents need live enterprise data, user history, and the user's current request.
2
Shopping agents need hybrid search pipelines that combine keyword search, semantic search, query understanding, and LLM-based reranking for broad or ambiguous requests.
3
Teams should start agent evals with business outcomes and real conversations, then use human labeling parties to improve LLM judges instead of beginning with complex tool-level analysis.
Summary
Nishikant Dhanuka describes lessons from building productivity and e-commerce agents at Prosus Group. He argues that context engineering is usually more important than prompt tweaks or model selection. A useful agent needs live business data, promotions, opening hours, payment details, and user history. Search remains difficult, so shopping agents need a pipeline that combines keyword and semantic search with query understanding and reranking. Dhanuka has also found that a pure chatbot often fails in production because users do not adopt a new interface without immediate value. He prefers experiences that combine chat with dynamic UI components and narrow, contextual help. For evaluation, he recommends starting with business metrics and user intent, using synthetic queries before launch, and running regular labeling parties with business stakeholders. More detailed tool and trace analysis can follow when the simpler checks reveal a problem.
Context quality often matters more than model choice
Dhanuka says teams spend too much time discussing system prompts, prompt tweaks, MCP tools, and which model is best. When two models are already state of the art, the larger difference comes from whether one has the right context. He compares this with data engineering and the idea of garbage in, garbage out. For a food agent, context includes live promotions, restaurant opening hours, payment-card support, and the user's situation. A model that can search for pizza but does not know that nearby restaurants are closed will not be useful in the real world.
Enterprise context requires messy real-time data work
The data needed by an agent is often scattered across real-time databases rather than stored in one clean source. A restaurant might run a sushi promotion only from noon to 3 p.m., which is difficult to represent in a conventional database. Dhanuka describes engineers connecting these sources and triggering data pipelines when a request arrives. If a user asks for sushi on promotion, the system must fetch the relevant information, add it to the prompt context, and then generate the response. He calls this unglamorous work a major part of building useful agents.
User history provides a practical cold-start memory
Dhanuka divides context into the system prompt, the user's current message, enterprise data, and user history. He says memory can create product stickiness because users may stay with a tool that knows their preferences and past activity. Teams do not need to begin with an elaborate long-term memory architecture. An existing shopping or food app already knows what users ordered and browsed, so that information can provide cold-start context when the assistant launches. Later conversations can add dynamic memory after users have used the product for some time.
Keyword search remains useful for direct requests such as burger, but it misses broad requests such as planning a romantic dinner or furnishing a house. Semantic search can connect related concepts, such as vegetarian pizza and pizza margherita, but it cannot fully resolve subjective terms such as romantic. Dhanuka describes a pipeline with query understanding before search, keyword and semantic retrieval, and an LLM reranking stage after retrieval. The system can use the user's profile and the original request to expand the query, select candidates, and present the most relevant options.
Dhanuka says internal testing can make a conversational shopping assistant look excellent, while an A/B test with real users shows poor conversion. Users already understand familiar interfaces, so they will not adopt a new one unless it solves a meaningful problem. He recommends showing the value within the first 30 seconds and guiding users with onboarding, examples, tooltips, and interactive elements. A plain chatbot can hide its capabilities, while a redesigned UI can create its own friction. The product has to make the reason for changing behavior obvious.
Dhanuka's current view is that the best interface combines conversation with generated UI. The agent can choose among components such as item carousels, related products, or other widgets based on the user's request and earlier actions. This matters in shopping and food because people want to scroll, click, swipe, and use images while deciding. Traditional recommendation systems still matter when users click or browse. He also describes a proof of concept where an agent watches actions on the screen and responds to them, while acknowledging that users differ in how much observation they accept.
Narrow contextual help can reduce interface friction
Instead of presenting one chatbot that claims to do everything, Dhanuka describes assistants that appear at the right moment for a small task. A floating control might offer to compare a headphone with a newer Apple model while the user is viewing that product. The comparison can use a simple LLM call connected to a tool, without requiring a full agent. He says these micro-tasks work better in experiments because users do not have to learn an entirely new interface or guess what the assistant can do.
Start evals with business outcomes and real conversations
Dhanuka calls evals a system for checking whether an AI product is good enough before launch and after it reaches production. He recommends starting with practical signals such as whether the conversation satisfied the user's intent, moved toward an order, reached the cart, or produced another relevant funnel event. These deterministic signals should come before immediately using an LLM as a judge. Teams can create an initial dataset from manual queries, synthetic queries, or generated user personas. A simple first pass often reveals enough problems to fix before teams analyze every tool call and parameter.
Dhanuka describes a recurring labeling party where engineers, business stakeholders, and other team members review real conversations for about an hour and a half. They answer simple questions about user intent, business progress, and guardrails. The resulting human labels are compared with the LLM judge's labels. Differences show where the judge is too lenient or too strict, and those examples become few-shot context for a revised judging prompt. He recommends repeating the exercise regularly. Custom annotation tools can also show the products, restaurants, and other visual information needed to judge a shopping conversation.
"Search is about when you're searching for dishes or when you're searching for restaurants. But when you're searching for does McDonald's have promotion, does McDonald's accept payment card, does McDonald's open right now, that is context."Nishikant Dhanuka07:19
Who should watch
You are building a shopping, food-ordering, or productivity agent and need practical guidance on context, search, and interface design.
Your agent works in demos but users do not adopt it or conversion is weak in production.
You are designing an eval process and want to start with business signals and human review before building complex trace-level checks.
