LLM applications create value when teams choose use cases that fit their industry rather than treating generative AI as a general solution.
2
Vector-enabled databases can provide current enterprise data and bridge the gap between language models and production applications.
3
Retrieval Augmented Generation grounds model responses in retrieved external knowledge, which can improve accuracy without retraining the entire model.
Summary
Hamsa Buvaraghan explains how teams can move from broad enthusiasm about generative AI to practical enterprise use cases. She covers applications such as document summarization, content discovery, product catalog improvements, marketing, healthcare research, and developer tools. She then breaks an LLM application into data processing and embedding, prompt construction and retrieval, and prompt execution. The data is chunked, embedded, and stored in a vector database before being retrieved as context for a model prompt. Buvaraghan is direct about model limits, including false information, source citation problems, bias, prompt hacking, and difficulty with maths. Her main recommendation is to ground applications in trusted organizational data. She explains how vector-enabled databases support semantic search over current, unstructured information, and how Retrieval Augmented Generation combines retrieved knowledge with a model's internal knowledge. The talk is a short practical introduction to choosing use cases and building more credible LLM applications.
LLM use cases should be chosen for their industry value
Buvaraghan says teams should first understand the available use cases, then identify which ones create the most value for their industry. She points to a McKinsey report on the economic potential of generative AI as a framework for assessing this. Examples include document summarization, content discovery for media companies, product catalog improvements for retailers, multi-channel marketing, healthcare research, and information discovery. Developers can also use tools built on LLMs to improve productivity.
Buvaraghan describes an application stack with data preprocessing and embedding, prompt construction and retrieval, and prompt execution or inference. Documents are split into chunks, passed through an embedding model, and stored in a vector database. When a user submits a query, the application combines a developer-written prompt template, few-shot examples, information from external APIs, and relevant documents. The resulting prompt is sent to a pretrained model, either through a proprietary API or an open-source or self-trained model.
LLMs have practical limits that production teams must address
Buvaraghan warns that LLMs are powerful but far from perfect. Problems include inaccurate source citations, bias, false information, difficulty with maths, and susceptibility to prompt hacking. She also mentions the cost of training and serving large models, saying that training a model such as GPT-3 could cost over $4 million. Understanding these limits helps teams use the technology more effectively and responsibly.
Enterprise databases can ground model responses in current data
Buvaraghan describes organizational databases as a source of truth for enterprise applications. Vector-enabled databases can store and search embeddings while continuing to provide current business data. They can supply production data to LLM prompts, improving the accuracy and relevance of generated answers. Semantic search also helps retrieve information from unstructured sources such as product descriptions, help desk tickets, and conversation history.
Retrieval Augmented Generation adds external knowledge to a model
Retrieval Augmented Generation, or RAG, improves generated responses by grounding a model in external knowledge. Buvaraghan says the technique originated in a research paper published by Meta. RAG can be adapted to knowledge-intensive tasks, and its knowledge can be changed or supplemented without retraining the entire model. This gives engineers more control over what information the system uses while avoiding the compute cost of retraining.
RAG combines retrieved context with the model's internal memory
Buvaraghan explains RAG through parametric and non-parametric memory. The external knowledge base provides non-parametric memory, while the model contains parametric memory. Retrieved documents cue the model toward correct responses. She says RAG can perform well when documents contain clues rather than stating the answer word for word, and it can sometimes produce a correct answer even when the exact answer is absent from the retrieved documents.
The basic RAG pattern retrieves relevant data from an external knowledge base and adds it to the prompt as context for the user's question. Buvaraghan says semantic search is part of this process because it considers the meaning of questions and source documents rather than matching only keywords. She gives an example of an internal question-answering system built around the RAG pattern that answers questions from private documents and adds references to relevant sources.
