LLM applications need deterministic assertions, embedding-based similarity measures, and evaluations performed by another LLM.
2
Teams should collect production traces, label useful examples, and turn those examples into a test dataset for later iterations.
3
Experiment tracking and LLM evaluation share the same workflow: record configuration, compare runs, inspect failures, and add automated checks to CI.
Summary
Gideon Mendels explains how testing changes when an application contains an LLM. Exact string assertions still help with hard rules, but they break when semantically equivalent answers use different wording. He recommends combining deterministic checks with embedding-distance metrics and LLM-as-a-judge evaluations. Teams can collect production questions, answers, and traces, then label examples manually or automatically and reuse them as a dataset for future prompt and pipeline changes. Mendels treats this as an experimental workflow. Developers change prompts, retrieval settings, embedding models, and other parameters, then compare results against the dataset. Opik records those configurations, traces, spans, and evaluation results, and its PyTest extension can put fuzzy checks into CI. He is direct about the limits: human labels remain valuable, alerts are difficult because of false positives, and large-scale evaluation is still an evolving problem. The conversation also covers collaboration, downstream product signals, and the boundary between Opik and existing A/B testing tools.
LLM applications need several kinds of evaluation metrics
Mendels groups LLM application metrics into three types. Deterministic assertions check hard rules, such as ensuring that an answer never contains a particular phrase. Embedding-distance metrics compare a generated answer with a preferred answer while allowing different wording. LLM-as-a-judge asks another model to score the original response, such as checking for bias or whether two answers convey the same meaning. He says the right choice depends on the task. Traditional metrics such as accuracy, F1, and perplexity still apply in the settings where labels or model-training objectives make them meaningful.
A pre-release gold dataset helps, but users will still produce unexpected questions after launch. Mendels describes how Opik traces production questions, responses, and chat sessions. Teams can manually label examples, such as responses that violate company policy, contain bias, or are simply wrong. They can also apply built-in LLM-as-a-judge metrics. Those labeled examples then become a reusable dataset for testing the next version. Mendels says this workflow separates teams that reach production with active users from projects that are released and then fail under real use.
Software engineers need a data-science approach to fuzzy behavior
String-based unit and integration tests are brittle for LLM applications. A provider can update its model and return a different string without making the result worse, which can break a build that expects exact text. Mendels says software engineers need to adopt the experimental, metric-driven habits common in data science. They should change prompts or pipeline settings, run the same examples, and inspect measured results. He also warns about overfitting: a prompt tuned on a small gold dataset can look perfect on those examples and fail in production.
LLM-as-a-judge is useful, expensive, and prompt-dependent
Mendels does not describe one standard method for LLM-as-a-judge evaluations. Teams write prompts that ask a model to detect properties such as bias or hallucination, and those prompts can behave differently across providers. He mentions OpenAI's playground support for this pattern. One cost-control approach is to use a cheaper model in the application and a more expensive model as the judge during testing. He presents the method as helpful, while acknowledging that it has disadvantages and still needs more work.
Opik is open source and has the same functionality as Comet's hosted version, according to Mendels. It integrates with major LLM providers and libraries, and its tracking decorator can instrument a function or class even when a team is not using one of those integrations. The recorded information can cover the full chain or pipeline. For a RAG application, that includes the question and answer, the vector database results, and the content inserted into the prompt. Teams can also record downstream product events, such as whether a user completed an activity after receiving an answer.
Experiment tracking and evaluation support the same iteration loop
Mendels compares LLM application development with traditional ML experimentation. In both cases, developers change a set of controllable parameters, run a dataset through the system, and compare the result with an acceptable metric. LLM teams control prompts, model parameters, retrieval and chunking settings, reranking, embedding models, and provider choices. Opik records an experiment configuration so another person can reproduce a result and continue the work. The interface supports dataset annotation, experiment comparison, text diffs, and filtering for examples with poor evaluation scores.
Mendels describes an Opik extension library for PyTest. A team can add an evaluation test to existing test code and set a threshold for the distance between a generated answer and the gold answer. This adds a check for regressions in LLM output alongside ordinary software tests. It can catch problems caused by changes elsewhere in the application, such as a broken vector database pipeline. Mendels says this gives teams more confidence when shipping and helps them move faster without relying only on manual checks.
Human labels remain valuable, while alerting is still difficult
Mendels says high-quality human-labeled data is valuable even when teams also use automated judges. He recommends labeling a small set of examples for a RAG question-and-answer application because the exercise can reveal prompt and product problems. Alerts are harder. A system that reports too many false positives creates alert fatigue, while hard rules can be effective for specific risks, such as preventing a chatbot from mentioning a competitor. Mendels says Opik did not yet have built-in alerts in this conversation, and that anomaly detection and large-scale observability remain areas where the product and industry are still developing.
"You control the prompt, you control some LLM parameters, if you're building a RAG you have all these hyperparameters around chunking and configuration of how the database does the reranking."Gideon Mendels31:31
Who should watch
You are building a RAG chatbot and need tests that tolerate different valid answers instead of checking exact strings.
Your team has production traces but no repeatable process for labeling failures and turning them into regression tests.
You need to connect LLM evaluations with experiment tracking, collaboration, or CI checks.
