Human-in-the-loop work for LLMs includes labeling, teaching models, correcting mistakes, and adding information to model knowledge.
2
Simple thumbs-up and thumbs-down feedback leaves much of the useful information that people could provide unused.
3
Many production systems still use LLMs as simple information retrievers, zero-shot models, or interfaces for other models rather than as continuous copilots.
Summary
Alberto Rizzoli argues that production teams have made limited progress on human-in-the-loop experiences for LLMs. The work includes more than labeling. People also need ways to teach models, correct poor answers, and add information that can influence future behavior. Rizzoli thinks thumbs-up and thumbs-down feedback is too crude for systems handling important information. He describes a gap between the promise of LLM copilots and how companies use them today. Most systems still ask a model for a short piece of information or use it to manipulate another model. In labeling, multimodal automation becomes difficult when expert users contribute unusual or out-of-distribution knowledge. Industry systems also tend to have discrete outcomes, limited tolerance for errors, and existing software interfaces that may not need an LLM. Rizzoli is direct about another problem: users often provide poor training feedback, while answers can be inappropriate because the information reached the wrong person rather than because it was factually wrong.
Human-in-the-loop work includes teaching and correcting models
Rizzoli uses human-in-the-loop to mean more than labeling data. It includes teaching an LLM, getting information into the model's knowledge, and giving people ways to fix mistakes. Those mistakes are not limited to hallucinations. A response can be poor, incomplete, or unsuitable for the task. He also includes multimodal work, where feedback spans language and vision. This matters because production systems need a record of what people changed and what the model should learn from that change. Rizzoli says progress in these experiences has been slower than he expected, despite the rapid adoption of LLMs in products.
Rizzoli criticizes the common pattern in which users respond to an LLM with only a thumbs-up or thumbs-down. He says this treats current systems like early machine-learning implementations, even when they are deployed in production and handle important information. A binary reaction does not explain what was wrong, what the answer should have contained, or which part of the response needs correction. The product challenge is to capture richer feedback without forcing every user to become a specialist labeler. Rizzoli presents this as an area with substantial unused potential.
LLM production use is still often simple and indirect
Rizzoli says most production use cases he sees remain relatively simple. In V7, LLMs are often used as a zero-shot model or as a way to manipulate another computer-vision model. Their multimodal behavior is still unreliable, so the model may act like a command interface: a user asks to label bees, the system finds the relevant model, runs it, and applies a basic transformation. He describes this as scratching the surface of what LLMs could do in production. The challenge is designing an experience in which the system actually learns from repeated use instead of just repeating isolated commands.
Expert users make automation difficult because they add unusual knowledge
V7's auto-label feature can take a small instruction, such as segmenting airplanes, and apply it to other images. Rizzoli says automation becomes harder when the human user is an expert or is working with data outside the model's distribution. If a model can fully automate the expert's work, the person may not need to label that data. If the person contributes genuinely new knowledge, the model cannot reliably automate the task by design. This creates a tension in human-in-the-loop product design: the cases where human judgment matters most are often the cases where automatic labeling works least well.
Industry workflows limit the value of general-purpose reasoning
Rizzoli says many industry systems have discrete outcomes and little room for error. In computer vision, some actions cannot be undone, such as a robot picking an apple or cutting a tree. Other systems already have narrow interfaces and defined actions. He uses Bloomberg Terminal as an example of software that already provides the controls needed for trading. In those situations, an LLM interface may add little value. A smaller fine-tuned machine-learning or deep-learning model can complete a defined action without the overhead of a general model and its broad vocabulary.
Users are poor teachers and answers can fail through information asymmetry
Rizzoli identifies two unresolved problems in production LLM systems. People who are allowed to retrain a model often provide incorrect information or feedback in a form that is unsuitable for reinforcement learning from human feedback. The system also may give a correct answer to the wrong person. In that case, the issue is not factual accuracy. It is that the information was shown to someone who should not receive it or who cannot use it. This makes user feedback and access to information part of the product design, rather than simple extensions of a chat interface.
Existing interfaces show that LLM interaction is still provisional
Rizzoli refers to Adept's website navigation, OpenAI's house-price interface, Sana, and Glean as examples of current approaches. He describes Adept's interaction as a way that works for now, rather than the only or final way to search for homes. He says OpenAI's interface is simple and functional, but not necessarily the best way to retrieve house prices. Sana receives praise for its visual feedback around LLM responses, and he also mentions Glean's enterprise search work. These examples show teams experimenting with feedback and action interfaces while the best patterns remain unsettled.
