LLM applications are difficult to evaluate because teams often start with prompts rather than datasets, and generative systems rarely have a clear objective function.
2
Public benchmarks help researchers choose models, but application teams need tests based on their users, domain, and outcomes.
3
Production evaluation should combine stable prompt sets, automated checks, human feedback, monitoring, and product-level outcome signals.
Summary
The panel explains why evaluating large language model applications differs from evaluating traditional machine learning systems. Josh Tobin says teams often begin with a desired behavior and a prompt rather than a training dataset, while tasks such as summarization lack an obvious objective function. The panel recommends keeping test prompts stable, measuring system behavior over time, and treating quality changes as an observability problem. Public benchmarks can help during research, but they do not show whether a model works for a particular product or user group. The panel discusses domain-specific evaluation, including medical and financial examples, alongside customer support metrics such as whether a user's problem was solved. The speakers also cover data curation, guardrails, human feedback, and the value of using production outcomes to improve applications. Tobin advises teams not to train their own language model before product-market fit when an API and prompt engineering can solve the problem faster.
LLM projects often begin with behavior and prompts instead of a fixed dataset
Josh Tobin contrasts traditional machine learning with LLM work. Earlier projects usually started by building a dataset and defining an objective function, which made holdout evaluation straightforward. With generative AI, teams often start by deciding what they want the system to do and writing a prompt to encourage that behavior. This leaves two hard questions: what data should the system be tested on, and what counts as a good answer when there is no clear objective function? Summarization is an example where two outputs may both be acceptable, yet one may still be better for the product.
Evaluation must cover the dimensions that matter for the product
Amrutha Gujjar says evaluation depends on the intended use. Accuracy can mean answering correctly, following instructions, remaining coherent, or avoiding hallucinations. Teams also need to check whether the model stays on topic, remembers context, and responds with acceptable speed and latency. Sohini Roy adds that qualities such as answer length, tone, and concision depend on the user's expectations. For systems that can call applications or handle sensitive information, the panel also considers safety, malicious code, privacy, toxicity, and bias.
LLM evaluation needs both human judgment and automated measurement
Tobin says the industry needs a practical mix of automated and human evaluation. As more non-technical stakeholders help build LLM applications, they can contribute domain knowledge and judge whether outputs are useful. Technical teams can consume those judgments in evaluation systems. The panel also points to domain specialists as an important source of feedback. A model may appear acceptable to a general reviewer while failing the expectations of people who understand a particular industry, workflow, or customer group.
Stable prompt sets turn changing model behavior into something teams can monitor
Sohini Roy recommends keeping a set of prompts and testing against them continuously. She notes that the same input can produce different answers from GPT-3.5 Turbo at different times, so a single test run is not enough. Teams can combine fixed inputs with sampling, ongoing health measurements, and alerts when quality falls below an acceptable level. This creates an observability workflow around model behavior instead of treating evaluation as a one-time experiment.
Public leaderboards are weak evidence for a product-specific application
Tobin separates research evaluation from application evaluation. Public or Elo-based benchmarks can help researchers compare models during early selection, but they do not measure the data or outcomes that a product's users care about. He describes a hierarchy in which real user outcomes are the most useful signal, proxy measures such as accuracy or model-based judging are easier but less direct, and public benchmarks are the easiest yet least informative for a specific application. Product teams should build evaluations around their own tasks.
Domain-specific applications need domain-specific data and human reviewers
The panel discusses Google's Med-PaLM as an example of evaluation designed for a medical purpose. Its tests used biomedical and scientific questions in different answer formats, with medical licensing questions as one metric. Human feedback came from clinicians assessing accuracy and non-clinicians from varied backgrounds assessing accessibility. BloombergGPT is mentioned as another example of a domain-specific benchmark for financial questions. The point is that the test data, success criteria, and reviewers should match the application's actual setting.
Product outcomes give customer support systems useful evaluation signals
Amrutha Gujjar identifies customer response and customer success applications as promising because teams can ask users whether a response solved their problem. They can also measure how many messages were needed to resolve an issue and whether users return. These signals are closer to product value than a generic language benchmark. Josh Tobin groups common applications into information retrieval, chat, and text generation, then argues that the surrounding product determines how difficult any of them will be.
Teams should avoid training a custom language model before proving the product
Tobin argues that most teams building NLP products should start with an available model, such as GPT-4, and use prompting or few-shot in-context learning before training their own model. He describes companies replacing long NLP projects with API calls and prompt engineering in weeks. His rule is to avoid training models until product-market fit has been established. After deployment, teams still need feedback loops that connect production outcomes to maintenance and improvement.
"The answer unfortunately for those of us in the ML side of the house is the reason why they ship those things so quickly is because the ML people on the team didn't build them."Josh Tobin11:54
Who should watch
You are building an LLM feature and need to decide what to measure beyond a public leaderboard.
Your model's outputs change over time, but you do not yet have a repeatable prompt set, alerts, or production monitoring.
You are considering fine-tuning or training a model before proving that an off-the-shelf model can support the product.
