LLMOps has to support proprietary and open-source language models because their infrastructure and operational needs differ.
2
A useful LLMOps stack covers prompt management, evaluation, retrieval-augmented generation, embedding updates, fine-tuning, inference, and observability.
3
Teams should choose stack components from their use cases while managing velocity, friction, and GPU-related costs.
Summary
Hien Luu shares lessons from exploring an LLMOps stack strategy for DoorDash. He starts with the practical challenges of deploying language-model applications, including inference, serving, cost, latency, and support for both proprietary and open-source models. He then describes the capabilities a stack may need for prompt engineering, evaluation, retrieval-augmented generation, embedding pipelines, and fine-tuning. Reference architectures from the community, LinkedIn, and Uber help him map the major areas, but he argues that companies should not build every component by default. The use cases should determine which parts come first and how deeply each area needs to be developed. Luu also points to model caching for high-QPS applications, especially when calls to external model vendors create cost and latency concerns. His final test for the stack is whether it lets teams build applications quickly with little friction while keeping GPU, fine-tuning, and inference costs under control.
Enterprise LLM adoption still lacks a settled operating model
Hien Luu opens with a quote comparing enterprise generative AI adoption to teenage sex: everyone talks about it, few people know how to do it, and people assume others are already doing it. He says this was an accurate description in 2023, although he expects the situation to improve as the machine-learning community gains experience with operating these systems at scale. His purpose is to share what he learned while exploring an LLMOps strategy for DoorDash.
Inference brings distinct cost, latency, and serving problems
Before choosing tools, Luu says teams need to understand what makes language-model applications different from earlier machine-learning systems. Inference and serving have attracted many startups because hosting models is difficult. Applications with high queries per second also need to control latency and cost. These concerns affect the design of the stack, rather than appearing only after an application reaches production.
The stack must handle proprietary and open-source models
Luu says organizations need to support both proprietary and open-source language models because their infrastructure needs differ. One model type may act as a backup for the other when reliability or performance becomes a problem, and some applications may use both. He expects enterprise teams to adopt more open-source models for use cases that need fine-tuning with domain-specific data. The stack therefore needs a practical path for fine-tuning those models efficiently.
LLMOps requires managed workflows around prompts, retrieval, and tuning
Luu identifies several areas that a production stack may need to make easier. Prompt engineering requires prompt management, versioning, release testing, and evaluation. Retrieval-augmented generation needs a vector database, embedding models, and pipelines that keep embeddings updated. Fine-tuning needs automation around templates, training, and evaluation. These capabilities turn recurring development work into repeatable workflows.
Caching vendor-model responses can reduce cost and latency
Luu points to a reference architecture with an LLM cache as an interesting component for high-QPS applications. When an application calls an external model vendor, caching can reduce both expense and response time. He says this is different from a typical key-value cache, so teams need to work out how such a cache should behave for language-model requests.
Reference architectures map the parts without prescribing a build plan
After showing a dense reference architecture, Luu groups its contents into broader pillars. He uses these pillars to think about which teams and capabilities an organization may need. Examples from LinkedIn and Uber provide useful views of the major areas, such as gateways, experimentation, responsible AI, training, tuning, inference, and observability. He treats these diagrams as maps of possible components, not as a required shopping list.
Use cases should decide what the team builds first
Luu says an organization does not need to build every component shown in a large LLM platform diagram. The applications the stack must support should determine which boxes are needed and which should be built first, second, and third. Teams can then decide how deeply to develop each area. This keeps the platform tied to actual product needs instead of copying another company's architecture.
Velocity, friction, and cost are the platform's main tests
Luu evaluates an LLMOps stack by asking whether it helps teams build generative AI applications quickly with minimal friction. That requires automation and useful tooling. It also requires close attention to cost because large language models need GPUs for fine-tuning and inference. Cost management belongs inside the stack's design rather than being treated as a separate concern.
"Implementing gen AI at scale in an enterprise is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims that they are doing it."01:17
Who should watch
You are deciding which parts of an LLM platform to build in-house and need a way to scope the work from actual application needs.
Your team is moving from model experiments toward production systems that need prompt versioning, retrieval pipelines, fine-tuning, inference, and observability.
You run high-QPS model applications and need to think about latency, vendor costs, GPU use, and caching together.
