LLM application development is moving from thin wrappers around models toward products with retrieval, fine-tuning, evaluation, and workflow-level monitoring.
2
Vector databases provide a practical form of memory by retrieving relevant embedded content and passing it to a language model as context.
3
Teams will need specialized tools for testing probabilistic workflows, controlling hallucinations, and adapting models to their own data.
Summary
Diego Oppenheimer surveys the early LLM application stack and compares it with earlier waves of software and machine learning tooling. He describes a first version of development built around model APIs, prompt experimentation, chaining, data integration, and retrieval. Vector databases help provide context to models that do not retain memory across interactions. A second version adds high-quality task-specific data, fine-tuning, distillation, and methods for reducing cost and latency. Production brings harder questions. LLMs are probabilistic, so ordinary software tests are not enough. Teams need to evaluate complete workflows, observe user interactions, compare models, and measure the effect of latency on the user experience. Diego also separates use cases where errors are easy to ignore from medical and financial applications where incorrect output needs strong safeguards. He predicts fast iteration in developer tooling and more models specialized for an organization's own data.
Foundation models provide broad capabilities that developers adapt to specific tasks
Diego distinguishes foundation models from large language models. Foundation models are broad, general-purpose systems trained on extremely large amounts of data, often with billions or more parameters. They can be adapted and fine-tuned for specific data. Large language models are the language-focused part of that broader category, with tasks such as classification, generalization, and summarization. He uses GPT-4, CLIP, and DALL-E as examples of foundation models, while LLaMA and GPT models fit the language-focused category.
The first LLM products are thin wrappers around model APIs
Diego says the current wave resembles earlier software waves, where new products initially wrapped lower-level capabilities. Developers are building convenience layers around foundation-model APIs, along with tools for prompt templates, orchestration, and multi-step reasoning. The first version of an LLM application can be assembled by connecting a model API to prompt experimentation, data integration, and, when needed, a vector database. He expects deeper applications to emerge as models and development methods improve.
Prompt experimentation and retrieval make general models more useful
Developers often need to try different prompts and chains of prompts to get a desired result. Diego describes LangChain and LlamaIndex as tools that help connect data sources, provide context, coordinate calls, and expose abstractions such as agents. Retrieval adds relevant content to the model input, which can improve accuracy, behavior, and cost. He compares a language model to the 'smartest goldfish' because it lacks persistent memory, and says vector databases can provide that memory by storing embeddings and finding similar documents efficiently.
Fine-tuning and distillation target accuracy, latency, and cost
Diego places fine-tuning in the second version of an LLM product. It depends on highly curated, task-specific datasets, which can improve accuracy and reduce inference latency. He also describes distilling a large model into a smaller model that keeps the needed accuracy while running faster and more cheaply. These techniques become relevant when a product has strict requirements around accuracy, response time, or the unit economics of inference.
LLM evaluation has to cover the whole probabilistic workflow
Traditional software developers expect the same input to produce the same output, but LLM applications are probabilistic. Diego says performance therefore has to be assessed through user interactions and the quality of generated content. Teams should test and run A/B experiments on the complete workflow rather than only on an individual model. They also need to compare models on specific tasks and measure how response speed affects the product experience.
Safety requirements change when errors are expensive
Diego separates high-affordability and low-affordability use cases. Suggestions in GitHub Copilot, email assistance, and image generation can tolerate errors because a person can ignore or check the result. Medical diagnosis and financial decisions have much less room for error. For these applications, he argues that teams need safeguards around safety, accuracy, and reliability. Tools that define rules, schemas, and heuristics for model outputs can help build trust.
Diego predicts rapid turnover in LLM developer tools. He compares the current period with earlier machine learning frameworks, where Caffe and TensorFlow were followed by PyTorch and other systems that gained adoption through faster iteration. He expects the same pattern around LLM libraries and application frameworks. The low barrier to starting with a model API should also produce many more first-version products.
Organization-specific models will depend on durable data and last-mile tooling
Diego predicts a future of highly contextualized open-source models trained or fine-tuned on an organization's data. He calls this idea 'GPTU' and says tooling will develop around adapting models to business and personal data. His core claim is that data is the most durable asset in these workflows, while value is generated in the last mile of specialization. The application stack will therefore need tools that help teams tune models to their own information.
"The core thing that's actually happened right is the capabilities got to that holy moment where we're looking at it really feels like magic."Diego Oppenheimer06:39
Who should watch
You are building an LLM feature and need to understand which tools fit prompt iteration, retrieval, fine-tuning, and evaluation.
Your application is moving beyond a demo and you need to think about latency, inference cost, user feedback, and full-workflow testing.
You work on a high-risk use case where plausible but incorrect model output requires rules and safeguards.
