Chinar Movsisyan argues that LLM products need production analytics based on user experience, alongside standard model evaluation.
2
Explicit feedback such as thumbs up and thumbs down is incomplete, so teams should also interpret implicit signals such as repeated queries, paraphrasing, and users leaving or reopening a chatbot.
3
Reliable AI products require product managers, domain experts, engineers, and customer-facing teams to share feedback and translate it into product and code requirements.
Summary
Chinar Movsisyan describes a gap between building an AI model and understanding how people experience the resulting product. Standard evaluation metrics such as accuracy, recall, mean average precision, and F1 score matter before deployment, but they do not explain how users actually use an LLM application in production. She compares this missing layer to the analytics available for ordinary web products through tools such as Google Analytics or Amplitude. For a chatbot, teams can combine explicit feedback with implicit signals, including repeated or paraphrased queries, abandoned conversations, reopened sessions, and clicks on retrieved documents. The aim is to diagnose root causes by connecting these signals with retrieval context, chunk size, prompts, and other available metadata. Chinar is honest that there is no universal metric combination yet. Her broader point is organizational: product managers and domain experts need tools they can use, and AI engineers need a clearer way to receive end-user requirements. LLM reliability depends on aligning these groups around the same product evidence.
Production constraints taught Chinar to connect models with real operating conditions
Chinar describes an early object-detection project for agricultural drones. The team annotated more than 10K high-resolution images, adapted YOLO 3, and then moved the model to a lighter deployment because a Raspberry Pi could not provide a GPU. Translating the model from Python 2 to C++ helped preserve accuracy and processing speed. The deployed system still produced false positives and false negatives when lighting and other conditions changed. She later worked on person and car detection for street surveillance, where the model automatically produced basic analytics from camera footage.
AI projects often fail at the boundary between domain experts and builders
Chinar says the same pattern appeared across agriculture, healthcare, surveillance, and fintech. Domain experts and end users may treat an AI output as if it were magic, while builders understand that the result has limits and needs explanation. During her PhD, she struggled to establish shared language with a professor in cardiovascular medicine. She says expectations from non-technical stakeholders can be further out of scope than the vocabulary problem. Demetrios Brinkmann adds that even technical people often expect LLMs to do more than they can. Chinar connects managing those expectations with getting AI products into production.
Chinar compares an AI product with a web application. Teams usually connect Google Analytics or Amplitude as soon as a web application is hosted, then check which features people use. She asks why AI products should be different. Standard evaluation still matters, including metrics such as accuracy, recall, mean average precision, and F1 score, but those metrics do not show how end users use a deployed product or what they expect from it. Her preferred direction is to begin with user experience and work backward toward the model, rather than stopping at model-to-user delivery.
User feedback includes behavior, not only thumbs up or thumbs down
For a retrieval-augmented chatbot, Chinar separates explicit and implicit feedback. Explicit feedback can be a thumbs-up, a thumbs-down, or a written complaint such as asking for customer transactions from Q4 2022 and receiving business transactions instead. Implicit signals include querying again, paraphrasing a question, leaving the chatbot, reopening it, and clicking a link to a retrieved document. Demetrios points out that a click or an abandoned session can have more than one interpretation. Chinar agrees that this is difficult and frames it as root-cause analysis rather than treating one action as a definitive label.
Root-cause analysis needs the surrounding execution data
Chinar says teams should analyze logs of queries and responses together with information about the retrieved context and the retrieval system. Useful metadata can include chunk size, the retrieved chunks, and retrieval hyperparameters such as temperature when available. She describes this as peeling back layers of an onion. The depth of the analysis depends on what data the product exposes. She says the team had not yet incorporated the embedding model, although access to that information could support further analysis. The goal is to turn many inputs into signals and then into a list of possible actions.
Agentic workflows multiply the difficulty of diagnosis
Chinar says chatbots are only one interface for LLM products. Other examples include conversational AI, intelligent document-processing dashboards, and agentic workflows. Diagnosing an agentic workflow is harder than diagnosing a standard retrieval-augmented application because the workflow combines several LLM-driven components, prompts, and other implementation choices. She describes it as an orchestration of separate root-cause analyses followed by an analysis of the orchestration itself. Teams need to solve the individual diagnostic problems before they can explain failures in the larger workflow.
An impact score can compare issues when no single metric is decisive
Chinar says Feedback Intelligence introduced an impact score calculated from the results of several metrics. It maps different issues onto one dimension so teams can compare their impact across customers and models. For example, recall may differ between two issues, but recall alone does not establish which issue deserves more attention. The impact score is intended to capture how much weight an issue has and how repetitive it is. Chinar says the team is still iterating on the score across areas such as fintech and legal AI, so she does not present it as a finished universal answer.
AI product work needs a shared place for non-engineering stakeholders
Chinar says product managers need to understand product usage, but developer-oriented evaluation tools do not fit their work. She imagines a combination of product analytics and evaluation features that lets different stakeholders inspect the same product evidence. She also describes a translation problem: product managers may gather end-user experience in tools such as Miro, then turn it into requirements for AI engineers. Those engineers may need to convert the requirements into data pipelines, retrieval changes, or code. Chinar's broader claim is that production AI is a multi-angle problem involving product, engineering, customer success, and domain expertise.
"Just the usage of chatbot, querying something, paraphrasing it, leaving the chatbot or closing it and then opening it again, these all signals can make a difference."Chinar Movsisyan22:45
Who should watch
You are shipping a chatbot or another LLM application and have model scores, but cannot explain why users abandon or repeat interactions.
Your product managers and domain experts rely on conversations or boards to report AI problems, while engineers lack structured feedback they can turn into changes.
You are designing observability for an agentic workflow and need to connect user behavior with prompts, retrieval context, and other execution metadata.
