Alex Strick van Linschoten's database shows that most production LLM projects are chatbots for customer service or conversations with company data.
2
LLM projects still depend on software engineering, DevOps, and MLOps practices, while agent systems work best when their tasks are narrowly constrained.
3
Published case studies rarely explain long-term return on investment, and many reported successes may be temporary as products and platforms change.
Summary
Alex Strick van Linschoten explains why he built ZenML's database of real-world LLM use cases. Inspired by Evidently, he collected scattered case studies, podcast episodes, and blog posts so readers could compare projects from small teams through large enterprises. Most examples fall into customer-service chatbots or tools that let people chat with company data. More unusual systems come from companies willing to accept technical risk. Alex says these projects still rely on familiar software engineering, DevOps, and MLOps practices. Agent systems are rarely given broad autonomy. The successful examples constrain them to specific tasks. The conversation also covers product interfaces, where buttons, voice, recommendations, and chat may work better together than chat alone. Evaluation often starts late, and LLM-as-judge systems have produced unreliable numeric scores. Alex is cautious about claims of lasting business value and expects many agent platforms to disappear or merge. He invites people to submit use cases and use the dataset on Hugging Face.
The database brings scattered LLM case studies into one place
Alex says Evidently's databases inspired ZenML's work. Real-world information was scattered across company blogs, conversations, podcast episodes, and isolated posts. His background as a historian made him comfortable collecting and organizing sources. The database covers the range from large enterprises to teams of five or six people starting something new. Alex used LLMs to create summaries for the collected posts, since doing that manually would have required far more time or a large budget. The work of gathering the material was substantial, although he says the overall project did not take as long as people might expect.
Most production use cases are familiar chatbot patterns
Alex groups the largest set of examples into chatbots. The common versions support customer service or let users chat with company data. Some are internal, some face customers, and some have an agentic element. A typical project begins with a leadership team seeing another company build a chatbot, or with an impressive Streamlit prototype created inside the company. Alex contrasts these repeated patterns with a smaller group of technically unusual systems. Those projects push into new territory and may work well for a specific company, although they are harder to generalize.
Public case studies underreport failures and operational mistakes
Alex says companies usually publish examples that make their teams look good. Even a failure may be presented as a success because the team detected and handled it. He praises Weights & Biases for sharing details about its internal support chatbot, including an evaluation mistake that cost several thousand dollars to redo. Alex would like more companies to publish systems that did not work and the reasons they failed. Without those accounts, readers see a biased sample of LLM projects and miss the operational decisions behind disappointing results.
Existing engineering and MLOps practices still determine reliability
Alex's broadest lesson is that established software engineering and DevOps principles remain necessary for LLM applications. Teams need to get those foundations right before the new model-driven features can work reliably. He prefers the term LLMOps because most of the work still concerns language models, while multimodal and image-generation systems remain less common in the database. He also connects LLMOps back to MLOps and, further down, to ordinary software engineering practice. The model may feel new, but the surrounding need for dependable systems is familiar.
Agents succeed when teams restrict them to a defined task
Alex has not seen evidence that broad, unbounded agents are working reliably. The successful examples constrain the system to a specific task and define its operating limits. Demetrios gives a marketing example: an agent can collect competitors' paid-search keywords and analyze which ones to bid on, but it cannot simply be told to do marketing for a company. Alex mentions Clara's customer-service deployment as a large reported success, while noting that the released details describe a narrow and controlled domain. Vague instructions make it harder to predict the result.
LLM interfaces may work better as product controls than as chat
The discussion moves from model capability to product design. Alex says many things presented as chat could instead use a button, voice input, or another interface while still using an LLM underneath. Demetrios describes OLX Magic, which combines search with recommendations rather than replacing the existing search experience. Alex says UX innovation is underrepresented in the database because technical teams tend to write the case studies. He also worries that prebuilt platforms such as Bedrock can push teams toward identical chatbot experiences and limit experiments with other ways for customers to interact.
Evaluation usually appears after a prototype has already been built
Alex says evaluation is often absent from the first proof of concept. It becomes a concern when a team prepares to expose the system to a wider internal or public audience. LLM-as-a-judge approaches have produced mixed results, especially when models are asked to return numeric scores. Some teams instead ask for qualitative feedback or use models to flag examples. Larger teams may collect data and iterate on the evaluation process, while smaller internal projects may stop with basic checks. Alex describes ZenML's Slack support bot as an experiment built partly to learn about usage and failure patterns.
Reported business value is difficult to judge over time
Alex found few case studies with concrete return-on-investment figures. Some reported more users, renewed interest, or improved journeys after adding retrieval or LLM-based search, but he is unsure whether those effects will last. He uses NotebookLM as an example of a fun and popular product whose long-term role is unclear. He expects many agent platforms to disappear, merge, or become parts of other products. Large claims can also reflect business pressures, such as preparing for an initial public offering, so the published result needs to be read alongside the company's situation.
The database is open to community additions and tool-based analysis
ZenML continues to maintain the database and accepts submitted use cases through a form at the top of the site. Alex also released the data as a Hugging Face dataset, so users do not need to scrape the website. His next desired feature is tool-based search, such as finding companies that use LlamaIndex, Pinecone, or Qdrant and comparing their use cases or failure patterns. Extracting tool names reliably is difficult, so he has not implemented it yet. He explicitly does not plan to add a chat-with-the-database feature, although users can build one from the released data.
"The places where people are managing to get this kind of to work is really, really constraining down on exactly the specific tasks or specifications for agents."Alex Strick van Linschoten14:19
Who should watch
You are deciding whether an LLM project should be a chatbot, a narrowly scoped agent, or a different product interface.
Your team has a prototype and needs practical guidance on evaluation, failure reporting, and the engineering work around the model.
You want to compare real production examples rather than rely on broad claims about agents, ROI, or the next dominant platform.
