LLM applications need objective evaluation throughout development, production monitoring, and model or prompt changes.
2
Teams should manage prompts with version history and give both technical and domain experts access to the experimentation process.
3
Starting with the largest capable model, then testing fine-tuning before adding complex chains or agents, makes performance easier to measure and can reduce cost and latency.
Summary
Raza Habib shares lessons from Humanloop's work with companies building large language model applications. He describes an LLM application as traditional software around a base model, a prompt template, and a data-selection strategy. Each part affects the result, so teams need evaluation that can show what changed and whether it helped. He recommends capturing user votes, user actions, and edits as feedback. Prompts also need version history, shared access, and a path from experimentation into production. Raza advises teams to begin with the strongest available model and push prompt engineering before adding complex chains or agents. Fine-tuning a smaller model can then reduce cost and latency while improving performance for a narrow task. He closes by explaining why LLM development needs software discipline adapted to faster iteration, non-deterministic outputs, subjective success criteria, and collaboration between engineers and domain experts.
An LLM application has three parts that all affect its result
Raza describes an LLM application as traditional software wrapped around an LLM block. The block contains a base model, a prompt template, and a strategy for selecting data to place into the prompt. The model might come from Anthropic, OpenAI, or a custom fine-tuned open-source model. The selected data might come from retrieval augmentation, conversation history, or a codebase, as with GitHub Copilot. Chains and agents connect these pieces, but the basic components remain the same. Raza says teams must choose the model, prompt, and data-selection strategy together because a weakness in any one of them can affect the application.
Objective evaluation is needed before teams can tell whether they are improving
Raza says teams often begin in the OpenAI playground by eyeballing a few examples. They may evaluate an entire retrieval-augmented system without separating retrieval quality from prompt or embedding quality. Later, they change retrieval, prompting, chains, or agents without knowing whether the result improved. Evaluation is often missing during prompt iteration, even though that is where changes happen quickly. It is also needed in production and after model or prompt changes, because regressions are easy to introduce. He gives GitHub Copilot as an example: GitHub tracks whether suggestions are accepted and whether they remain in the code over time, which provides a stronger signal of usefulness.
Production feedback should include votes, actions, and corrections
The strongest applications Raza has seen collect several types of feedback in production, alongside human feedback during development. Votes include simple controls such as thumbs up and thumbs down. Actions provide implicit signals about what users do after receiving a response. Corrections capture edits to generated content. For GitHub Copilot, edited text can show how the generated code was changed. Raza argues that these signals support monitoring, improvement, and development. Without a way to measure outcomes, teams can spend substantial effort changing prompts or retrieval systems without learning whether their work helped.
Prompt management needs shared history and access for domain experts
Raza says prompts are often treated less seriously than normal code. Teams may start in a playground, move experiments into a spreadsheet, or keep prompts in Google Docs. That makes it easy to lose the history of experiments and repeat work across teams. Keeping prompts only in Git solves versioning but can make it harder for non-technical domain experts to contribute. Raza recommends a system that records the full history from early playground experiments through quantitative evaluation, stores the history alongside model configuration, and remains accessible to technical and non-technical team members. Humanloop is one tool built around this problem, though he says teams can choose other systems.
Complex chains and agents should come after simpler approaches have been tested
Raza has seen customers move quickly to complicated chains or agents and later remove them. With several prompts in a chain, teams have more possible causes for each outcome, and changes in one place can affect later steps. That makes evaluation and maintenance harder. He recommends starting with the strongest available model, pushing prompt engineering as far as possible, and avoiding early optimization for cost, latency, or architectural complexity. Retrieval-augmented generation and ordinary chat are practical exceptions. More complicated chains or agents should come after simpler alternatives have been tested and found inadequate.
Fine-tuning a smaller model can fit a narrow task better than a general model
Raza says teams often underestimate fine-tuning because they assume smaller models need too much data or will not perform well. Humanloop has seen customers start with hundreds of annotated examples, while thousands can produce much stronger performance. A smaller model can reduce cost and latency and can be specialized for a task such as answering questions about legal documents or generating sales emails. He describes a customer that tried prompt engineering and retrieval augmentation for three weeks, then fine-tuned a smaller model with about three and a half thousand data points. The smaller model outperformed GPT-4 on that customer's task.
Feedback data can support repeated fine-tuning cycles
Raza describes a practical fine-tuning pattern. Teams generate data with an existing model, filter it using a success criterion, and fine-tune on the retained examples. The filter might use explicit user feedback or another LLM's score. Teams can repeat the cycle as they collect more feedback. He describes Find as a search engine for developers that gathers user feedback, then fine-tunes a custom open-source model focused on developer and code questions. Because the model is specialized, it does not need the broad capabilities of a general model. Raza says this approach can produce lower costs, lower latency, and better performance for a specific use case.
LLM development needs software discipline adapted to non-deterministic outputs
Raza does not argue for copying traditional software practices unchanged. LLM applications need prompt management because domain experts often work directly with engineers and because prompts and fine-tuned models change faster than ordinary software. They need evaluation that handles subjective outcomes and non-deterministic results, where there may be no single correct answer. They also need a CI/CD-like process that catches regressions while supporting fast iteration and feedback data. Traditional unit tests are harder to write for many LLM tasks, so teams may use LLMs to help evaluate outputs, while recognizing that this approach has limits.
"I would only go to agents or chains if you're in a situation where reasoning is really important and where you've explored these other alternatives and found them to be wanting first."15:04
Who should watch
You are building an LLM application and still judge prompt changes by looking at a few examples.
Your team needs engineers and domain experts to collaborate on prompts without losing experiment history.
You are considering agents or large models first and want to test simpler prompts or fine-tuning before accepting their cost and complexity.
