LLMs work best when a product needs fluent, creative output and the cost of occasional inaccuracies is acceptable.
2
Production applications need evaluation, prompt versioning, feedback loops, and ways to handle refusals or uncertain answers.
3
Teams should begin with an accessible, capable model, then use their evaluations to decide whether a smaller or open-source model is sufficient.
Summary
This panel examines the practical decisions involved in building products with large language models. Natalia Burina frames model fit around fluency and accuracy. Creative writing can benefit from fluent output even when exactness is not essential, while decision-making applications need much stronger controls. The panel discusses hallucinations, prompt design, retrieval, citations, prompt chaining, model blending, and structured outputs. The speakers agree that evaluation remains less mature than deployment. Teams need benchmark questions, prompt versioning, automated and human feedback, and checks for bias, toxicity, and hallucinations. Cost depends on prompt length, output length, model size, and the number of chained calls. Smaller models, fine-tuning, semantic caching, streaming, and intermediate results can help. Model selection should start with the model a team can access and evaluate, then move to a cheaper or more specialised option when performance allows. The panel is candid that production work involves regular experimentation and imperfect safeguards.
Hallucinations should shape the use case rather than be treated as a problem that will disappear
Natalia Burina argues that hallucinations are a lasting property of current language models. They produce fluent answers that can be wrong, so trying to eliminate hallucinations entirely is a futile goal. She recommends using LLMs where their strengths matter, especially creativity and inspiration. Sahar Mor adds that hallucinations are especially difficult because the model can state incorrect information with confidence. The effect of a mistake depends on the application. A brainstorming partner has different requirements from a classification pipeline whose output triggers substantial downstream work.
Open-source and specialised models could move more AI work closer to users and private data
Sahar Mor argues that open-source language models can run at the edge, which suits privacy-sensitive products and cases that cannot rely on an internet connection. Developers can also fine-tune them for a particular purpose, with less compute and lower latency. The panel connects this with a broader move toward domain-specific models built around each organisation's proprietary data. George Mathew predicts that public training data will become harder to extend as models grow, increasing the value of private datasets and smaller models. The panel also acknowledges the counterargument that well-funded research organisations may retain an advantage through larger models, private data, and multimodal systems.
Model fit depends on the balance between fluency and accuracy
Natalia Burina proposes mapping a use case on two axes: fluency and accuracy. High fluency with low accuracy requirements suits writing poems, fiction, children's stories, and other creative tasks. High accuracy with less need for fluent language points toward precise information tasks. She adds a separate business decision framework. Teams should ask whether LLMs could make the business obsolete, increase revenue through uses such as customer support, create an advantage, or have little effect. These questions should come before major investment. Experimentation still makes sense when the business case is uncertain, but the team should understand what it is trying to learn.
Prompt applications become difficult when they move from a weekend prototype into production
Asmitha Rathis says tools such as LangChain make it easy to create a proof of concept quickly, while production introduces harder problems. For document question answering, prompts can include retrieved context and examples, and the desired output can be constrained to a structure such as JSON. Retrieval can supply examples relevant to the user's query instead of relying on a fixed set of examples. Chained prompts create latency, so product teams can stream partial output, show retrieved documents first, or provide intermediate messages while the final answer is generated. Sahar Mor recommends giving the model a predictable fallback, such as returning 'N/A' when an input is not a valid review, rather than forcing it to invent a classification.
The panel describes evaluation as the missing layer that would make teams more confident about changing prompts and deploying applications. Natalia Burina points to AI system cards, including OpenAI's GPT-4 system card, as a way to document risks and evaluation results. Sahar Mor recommends use-case-specific benchmarks and says a stronger model such as GPT-4 can judge outputs from a weaker model such as GPT-3.5 when the scoring criteria are explicit. Asmitha Rathis describes prompt versioning, benchmark questions, keyword checks, semantic similarity, and weighted evaluation scores. New failures should feed back into the benchmark set instead of remaining one-off incidents.
Hallucination controls trade extra cost and latency for more reliable answers
The panel lists several ways to reduce incorrect outputs. A prompt can require the model to decline or return a defined value when information is missing. Prompt chaining can ask one model to answer and then critique its response, although this adds latency and cost. Requiring citations gives users a way to inspect the source in a document. Sahar Mor also mentions LLM Blender, which combines models that perform well in different areas. George Mathew describes related work on visualising gaps in a model's latent space and retraining on missing data. These techniques do not remove the need to understand the consequences of errors in the particular product.
Prompt length, model size, and product design all affect the economics
Longer prompts with more examples can improve performance but increase input-token costs, and generated answers add output-token costs. Natalia Burina recommends time-boxing experiments, setting expectations with leadership and partner teams, and considering smaller models when their accuracy is sufficient. Asmitha Rathis suggests constraining output length and format, and using semantic caching when repeated questions do not need a new completion. Fine-tuning can reduce the amount of context needed on later calls, according to Sahar Mor. The panel also points out that model prices have been falling and that startups may need to prioritise customer value while the product is still being tested.
Model selection should begin with access and measured performance
Sahar Mor recommends starting with the most capable model that the team can access and evaluate. If GPT-4 works well, the team can test GPT-3.5 when lower cost or faster responses matter. Commercial access requirements and agreements affect which model can be tested first. Open-source selection should increasingly depend on a team's own evaluations rather than general leaderboards. The right model can differ by latency, cost, and accuracy requirements, and the panel expects better infrastructure to make it easier to compare and switch between models. Using more than one model for the same use case may also make sense when different requests have different constraints.
