A strong LLM application combines a suitable base model, a carefully designed prompt and data selection strategy, and continuous evaluation.
2
Teams should use prompt engineering and retrieval to reach a useful first version, then fine-tune when private feedback or specialised data can create a smaller, cheaper, faster, more customised model.
3
Defensibility can come from private-data integrations, feedback-driven improvement, counter-positioning through privacy or custom models, and a product experience designed around model limitations.
Summary
Raza Habib explains how Humanloop sees teams build LLM applications from prototypes into production. He breaks an application into a base model, a prompt template, a strategy for selecting data, and an evaluation loop. GitHub Copilot illustrates the approach: it uses a code-specific model, retrieves relevant code from the developer's context, and measures whether suggestions remain in the codebase. Habib then covers recurring production problems, including prompt sensitivity, hallucinations, subjective evaluation, latency, and cost. His advice is to push prompt engineering and retrieval first, since they are fast ways to improve an initial system and add changing factual information. Fine-tuning becomes useful when production feedback or private data can produce better performance on a narrow task, lower latency and cost, or private deployment. For defensibility, he discusses switching costs from integrations, feedback data flywheels, counter-positioning, scale, brand, and user experience. He also cautions that solving a real user need should come before worrying about competitors.
An LLM application is a chain of reusable blocks with three core parts
Habib describes each LLM application block as a base model, a prompt template, and a data selection strategy. The base model may be GPT-3 or an open source model such as Llama. The prompt contains instructions and spaces for user input or retrieved data. The selection strategy decides what fills those spaces at run time. Applications can repeat these blocks or connect them through an agent. This framing gives teams separate decisions to improve instead of treating the whole application as one opaque system.
GitHub Copilot wins through choices around the model, context, and measurement
GitHub Copilot uses a 12-billion-parameter GPT model trained for code. Habib says the model is smaller than GPT-3 because suggestions need low latency, even though a larger model might generate better code. Copilot also examines the cursor location, recently touched files, and similar code elsewhere before constructing its prompt. Its evaluation goes beyond whether a suggestion is accepted. The system checks whether the code remains in the codebase after intervals such as a few seconds or minutes. This gives the team production feedback from a large developer base.
Prompt engineering needs fast experiments because small changes can alter results
Habib says prompt engineering remains an art. Small changes to a template can produce surprisingly large performance differences, so teams need a fast feedback loop and many experiments. He points to Chain of Thought prompting as an example. Asking a model to provide a reasoning trace can improve question-answering performance by many accuracy points compared with asking only for an answer. Other changes, such as output format or the role assigned in a prompt, can also matter. A defensible application therefore needs a practical way to test, compare, and revise prompts.
Retrieval adds changing factual information without putting that knowledge into fine-tuning
A common pattern for giving an LLM factual information is to split documents into pieces, embed those pieces, and make the embeddings available when the model generates an answer. Habib connects this to the retrieval strategy in an application such as GitHub Copilot. Different retrieval methods can have a large effect on performance, so retrieval needs its own experimentation rather than being treated as a fixed infrastructure step. He later distinguishes this use of retrieval from fine-tuning: prompt engineering and retrieval are better when the goal is to provide factual knowledge that changes over time.
Production feedback creates a loop from failures to prompts and fine-tuned models
Habib groups application feedback into actions, issues, and votes. Copilot observes whether generated code is accepted and retained, while ChatGPT uses thumbs-up and thumbs-down feedback with natural-language explanations. Teams can filter production data for failure cases, inspect those cases, form a hypothesis, and change the prompt or retrieval step. A second path uses successful examples for supervised fine-tuning. The model generates data in production, the team selects examples that worked, fine-tunes the model, and repeats the cycle. He also mentions Reinforcement Learning from Human Feedback, but says fewer companies use it because it is more complicated.
Fine-tuning is useful for specialised performance, latency, cost, and privacy
Habib recommends prompt engineering first because it is faster and avoids the work of hosting and fine-tuning models. Fine-tuning becomes valuable when a team has private data or feedback from a live application. A smaller fine-tuned model can approach the performance needed for a specific task, while reducing latency and cost. It can also support local or private deployment and a company's particular tone of voice. Fine-tuning is therefore a way to build an application around information and behaviour that competitors cannot access, although Habib advises against doing it prematurely.
Defensibility comes from software advantages plus LLM-specific data and model loops
Habib says LLM companies still rely on familiar software advantages such as scale, switching costs, network effects, and brand. He compares Jasper and Writer to show two different paths. Jasper scaled quickly with closed models from OpenAI and invested in marketing. Writer built custom fine-tuned models, using privacy, customer-specific tone of voice, and the ability to run on customers' machines to reach users who could not use Jasper's approach. GitHub Copilot adds a feedback flywheel, using usage data to improve its model for the task.
Private integrations, counter-positioning, feedback, and user experience can make products harder to replace
Habib identifies several practical sources of defensibility. Integrating with a company's private knowledge base, customer information, or code creates switching costs because replacing the system requires rebuilding those connections. Feedback capture can support repeated filtering and fine-tuning, producing a model that improves with use. Counter-positioning can offer privacy or deployment options that competitors built on other models cannot easily match. A product can also win through a distinct user experience. Copilot accepts that generated code is imperfect and makes suggestions in the user's code context, so the experience remains useful when the model is wrong.
"If you are able to get private customer information, a particular company's knowledge base, if you're doing customer service or code if you're indexing someone's code base or something like that, then that can be a big advantage."21:42
Who should watch
You are building an LLM product and need a practical way to separate model choice, prompting, retrieval, and evaluation.
Your first version works, but you need to decide whether more prompt iteration or fine-tuning is the right next step.
You are thinking about defensibility and want examples based on privacy, private data, feedback loops, and product design.
