LLM application development should start with a business problem, available data, and a measurable definition of success.
2
LLMOps adds work around prompts, unstructured data, vector databases, GPU infrastructure, model packaging, and evaluation.
3
RAG can provide current, domain-specific information without changing foundation-model weights, but its quality depends on how well the source documents are understood and parsed.
Summary
Yinxi Zhang explains how LLM application development extends familiar MLOps practices. She begins with business understanding, data analysis, prototyping, packaging, evaluation, CI/CD, and serving. The application pattern should grow with the problem: prompt engineering may be enough at first, while RAG, fine-tuning, or pre-training may be considered when data and results justify them. LLM systems add raw text, prompts, embeddings, vector databases, external APIs, and often GPU infrastructure. Zhang discusses the security risks of sending sensitive data to third-party model providers and the need for API governance. She also describes why evaluation remains difficult, including limited rubrics and high-quality evaluation data. Human feedback should be collected through the product, while LLM judges can provide scalable checks at extra cost. Her main practical advice is to understand the business use case and source documents deeply, start simply, and keep evaluating the system after deployment.
LLM development should begin with a real business problem
Zhang says teams should define the problem, check whether they have enough data, and establish quantified success criteria before experimenting with foundation models. The use case determines the model and system design. A chatbot may need a generative model, while personally identifiable information identification can be handled by a classification model. Teams also need to decide whether responses must be real time, then define expected throughput and latency. The surrounding user interface and integration with existing systems belong in the design from the start.
LLM systems work with more data formats than standard ML pipelines
Conventional ML pipelines often use tabular data and a model, while LLM applications may begin with PDFs, images, and other unstructured documents that must be parsed into text. Prompts become part of the application data, and RAG adds embeddings stored in a vector database. Zhang says these extra assets make LLMOps more difficult than regular MLOps. Teams must understand their documents, query examples, domain language, and document update frequency before choosing models or designing the pipeline.
Teams should move from prompts to RAG or fine-tuning only when needed
Zhang presents a progression from off-the-shelf models and prompt engineering to RAG, fine-tuning, and pre-training. Prompt engineering and RAG add information without changing foundation-model parameters. Fine-tuning updates model weights, while pre-training can involve training from scratch. She recommends moving to a more involved pattern when the team has sufficient data and the current result is not satisfactory. RAG can add proprietary or more recent information while avoiding the cost and complexity of changing the foundation model.
Prompts need explicit instructions, examples, and model-specific formatting
Zhang compares prompt design with giving detailed laundry instructions. A useful prompt can specify steps, provide examples, and tell the model to say it does not know when it lacks relevant information. She warns that prompts are model specific, so the same wording can produce different results across models. Teams should track prompt queries and responses as they would track conventional ML experiments, use prompt templates, and consider tools such as MLflow, LangChain, LlamaIndex, or DSPy to organize and automate the process.
RAG quality depends on retrieval and document preparation
In Zhang's description, a RAG system embeds the user query, searches a vector database for similar documents, adds retrieved content to the context, and sends the expanded prompt to the language model. Source documents need a continuing pipeline that creates and refreshes embeddings. The system therefore has to manage vector-database updates and package the retrieval components alongside the model. Zhang says better RAG results depend heavily on parsing documents well and creating context chunks that contain useful information.
Data security can determine whether teams use external APIs or host models
Third-party APIs can require sending data to external servers, which Zhang identifies as a serious concern for customers in healthcare and finance. Those customers may prefer to host models in a secure environment so they retain access to the model and avoid data leakage. She also recommends centralized API governance and payload encryption. This decision affects model packaging, deployment, and the amount of infrastructure the team must operate.
Evaluation remains difficult and should include human feedback
Zhang says LLM evaluation lacks broadly accepted rubrics and high-quality evaluation datasets. LLMs can act as judges, but the product should also let users provide feedback on results. For RAG, an evaluation can inspect the query, retrieved context, and final answer, including relevance and whether the answer is grounded in the context. Using an LLM judge costs more because it creates additional model calls. Teams must compare that cost with asking domain experts to perform the same work.
CI/CD and serving must cover the whole LLM pipeline
Zhang applies familiar CI/CD practices to LLM applications. Developers commit changes, run unit tests, deploy to staging, monitor behavior, run integration and end-to-end tests, and then deploy to production. Those tests can cover the path from data ingestion to the final API or user interface. Serving choices depend on latency needs. Batch serving can be distributed across multi-GPU clusters, while real-time serving requires packaging the model API call with its pre-processing and post-processing logic.
"It is really really important and critical to ground LLMs in real life problems before just jump into the world of LLMs and start experimenting different type of foundation models."09:14
Who should watch
You are building a RAG or other LLM application and need a practical path from experimentation to production.
Your team already uses MLOps and wants to understand what changes when prompts, unstructured documents, external APIs, and vector databases enter the pipeline.
You need to make decisions about model hosting, evaluation costs, CI/CD, or GPU-based serving for a business application.
