# Retrieval Augmented Generation

Syed Asad, KiwiTech | MLOps Podcast | Episode 233 | 44:10
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=g4uRiyIydvM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/retrieval-augmented-generation
Published: 2024-05-17
Tags: evals, model-serving, observability, rag

## TL;DR
- Syed Asad found that direct embeddings were a poor fit for a 133 MB CSV with repeated entities and overlapping fields, while converting it to Parquet and querying it with a LlamaIndex agent worked quickly and correctly.
- Production failures came from infrastructure and application behavior, including parallel video processing, overloaded Celery workers, repeated model loading, and logs that did not give machine learning engineers enough detail.
- Syed Asad prefers tools that give his team control over data, models, and deployment, and KiwiTech's machine learning research lab tests tools against specific use cases before storing working examples for later projects.

## Summary
Syed Asad discusses the practical problems he has encountered while building and deploying RAG systems under tight client deadlines. A 133 MB agricultural CSV defeated several embedding models because repeated terms appeared in different contexts. Converting the file to Parquet reduced it to 9 MB, after which a LlamaIndex agent returned correct answers in one or two seconds. He also describes inference failures caused by memory use, parallel processing, overloaded Celery workers, repeated embedding-model loads, and weak operational logs. Syed compares AWS, Salad, vLLM, Ollama, and other inference options through the lens of cost, speed, observability, and control. He is cautious about Hugging Face models in production and about relying on LangChain or LlamaIndex without owning more of the framework. He also discusses ORPO and Unsloth for fine-tuning small language models, the cost and limits of RAG evaluation, and KiwiTech's internal research lab for testing tools before client work.

## Key ideas
### Repeated values made embeddings a poor fit for the CSV
[03:48](https://www.youtube.com/watch?v=g4uRiyIydvM&t=228s)
Syed Asad's first production problem involved a 133 MB, well-formatted CSV containing US agricultural data. A chat interface needed to answer questions about it. Directly creating embeddings failed across several models, including MixBred, Snowflake Arctic, sentence-transformers, and MiniLM, regardless of the machine or vector size he tried. He also tested PandasAI, but it repeatedly failed with a missing-module error tied to an open GitHub issue. Repeated terms created another problem: a query about sugar cane production in Mexico could match many separate rows and produce confusing context. Syed concluded that embeddings were a poor fit for this structured, repetitive data.

### Parquet and an agent solved the immediate structured-data problem
[09:16](https://www.youtube.com/watch?v=g4uRiyIydvM&t=556s)
With a Friday production deadline, Syed Asad moved away from the embedding pipeline. He converted the CSV to Parquet, reducing its size from 133 MB to 9 MB without what he described as much loss. He then used a normal LlamaIndex agent to query the file. The system returned answers in one or two seconds, and he said the results were correct. He still considers vector search useful for other data, especially PDFs, and said hybrid search can combine sparse and vector embeddings when that matches the data. He also recommended FastEmbed from Nan as a fast production option that reduced Docker container size.

### Inference failures often came from the surrounding system
[12:06](https://www.youtube.com/watch?v=g4uRiyIydvM&t=726s)
Syed described a multimodal RAG system whose embedding model initially used three to four GB on AWS, then dropped to about 400 MB while stopping its GPU use. Retrieval became slower even though the model itself worked on another machine. He traced the problem toward asynchronous processing: multiple videos were being searched in parallel even though they needed sequential processing, which increased memory use. AWS autoscaling did not resolve the issue. Kubernetes and Celery were part of the deployment, and Celery workers could become overloaded as pipelines burst. Increasing GPU and vCPU capacity helped keep production running, but it did not replace worker-load testing or better diagnosis.

### Observability must work for machine learning engineers
[16:05](https://www.youtube.com/watch?v=g4uRiyIydvM&t=965s)
Syed said AWS logs were useful for DevOps or job operations teams but did not provide enough information for a machine learning engineer. His team added time loggers to identify where processing time was being spent. Another unresolved issue was that the embedding model kept loading repeatedly, despite being loaded globally, which consumed memory and added delay. He also rejected Salad for production because its logging interface was a Jupyter notebook-style terminal that he could not easily hand to his DevOps team. The comparison showed that infrastructure can fail at the operational boundary, even when the model and application code appear to work.

### Inference choices trade speed, cost, control, and data locality
[17:29](https://www.youtube.com/watch?v=g4uRiyIydvM&t=1049s)
Syed tested Salad as an inference-as-a-service platform and reported a $58 bill for roughly one month on a small proof of concept. He described it as inexpensive and potentially based on distributed GPUs, but slower and difficult to monitor. He said OpenAI raised concerns about cost and about what happens to client data. Ollama was useful for local model experiments, but remote-connection errors made deployment on AWS difficult to debug under a time-bound workflow. vLLM increases inference speed and can support localization, while also giving more control over the models and deployment design. Syed preferred that added control when client data and customization mattered.

### Owning the framework can reduce dependency on fast-changing libraries
[21:35](https://www.youtube.com/watch?v=g4uRiyIydvM&t=1295s)
Syed said Hugging Face models were not a good fit for his production work because they consumed too much space, although he did not rule them out universally. He also said FastEmbed and Llama worked well in his tests. For production, he preferred building more of his own framework instead of depending entirely on LangChain or LlamaIndex. His reason was stability and control. He pointed to LangChain's frequent library changes as a source of broken pipelines. Ollama remained acceptable when there was time to investigate its errors, and for prototyping, but he would avoid it in an agile, deadline-driven production project.

### Small-model fine-tuning can replace some RAG complexity
[28:45](https://www.youtube.com/watch?v=g4uRiyIydvM&t=1725s)
Syed had been testing small language-model fine-tuning with ORPO and Unsloth. He explained that Odds Ratio Preference Optimization combines fine-tuning and preference treatment into one step. Unsloth provides ready-to-run notebooks and models quantized to four bits. A team can train a one or two billion parameter model on its own JSON or Hugging Face data and then build a small RAG system around it. Syed acknowledged that four-bit quantization can introduce lossy compression, but said that may be acceptable for the use case. His approach was to test a smaller model directly rather than automatically building a large retrieval system.

### Evaluation is technically possible but clients still resist paying for it
[29:45](https://www.youtube.com/watch?v=g4uRiyIydvM&t=1785s)
Syed tested DeepEval and found that its scoring required a context field alongside the question and expected output. He did not find that context score useful for his needs. He said AutoRAGAS worked better, although RAGAS increased costs. He also discussed CircleCI as a way to automate language-model evaluation in a CI/CD pipeline, citing an Andrew Ng course and a free DeepLearning.AI tutorial. He had not deployed it because client approval was still needed. In his experience, clients often preferred manual software testing and did not yet understand the value of language-model evaluation.

### KiwiTech maintains a research lab to turn tool tests into reusable code
[38:20](https://www.youtube.com/watch?v=g4uRiyIydvM&t=2300s)
KiwiTech created a machine learning research lab that tests tools and frameworks against defined use cases. The team compares options such as LLM evaluation frameworks, Prometheus, and Weights & Biases, ranks them, writes a quick proof of concept, and stores the result in GitHub. When a client arrives with similar requirements, engineers can reuse the tested code rather than starting from scratch. Syed called the goal 'research to production'. He admitted that stored examples can become a risk when tools introduce breaking changes, so the team must keep track of changes. The lab also reduces the time he personally spends evaluating every new platform.

## Notable quotes
- Syed Asad: "Vector embedding will not work in this case even if I try hard with the ETL process." (08:50)
- Syed Asad: "At the end you need to have your own framework." (22:00)
- Syed Asad: "The client or the end user who is actually paying for the project, they do not understand the value of evaluation in LLM." (32:32)
- Syed Asad: "Research to production is our goal." (39:25)

## Tools & references mentioned
- AWS
- MixBred
- Snowflake Arctic
- sentence-transformers
- MiniLM
- PandasAI
- GitHub
- MixPeek
- Ethan
- MongoDB Vector Search
- Parquet
- LlamaIndex
- FastEmbed
- Nan
- Kubernetes
- Celery
- Salad
- Ollama
- Llama.cpp
- vLLM
- OpenAI
- Hugging Face
- LangChain
- Unsloth
- ORPO
- DeepEval
- AutoRAGAS
- RAGAS
- CircleCI
- Andrew Ng
- Prometheus
- Weights & Biases

## Who should watch
- You are building a RAG system over CSVs, tables, or other structured data and need to decide whether embeddings are appropriate.
- Your inference service works in development but fails through memory growth, worker overload, model reloads, or inadequate logs in production.
- You are comparing hosted inference platforms and need practical trade-offs around cost, speed, observability, data control, and deployment effort.

## Editor's note

Syed Asad describes production failures caused by overloaded Celery workers, parallel video processing, and repeated model loading. ZenML records each pipeline run's steps, inputs, outputs, and code version, so the team has a trace of what ran when diagnosing workflow behavior across infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Supercharging Your RAG System: Techniques and Challenges](https://mlopstalks.com/talks/supercharging-your-rag-system-techniques-and-challenges) (Tengyu Ma, Voyage AI and Stanford University, 40:20)
- [A Survey of Production RAG Pain Points and Solutions](https://mlopstalks.com/talks/a-survey-of-production-rag-pain-points-and-solutions) (Jerry Liu, LlamaIndex, 30:00)
- [Building RAG-based LLM Applications for Production](https://mlopstalks.com/talks/building-rag-based-llm-applications-for-production) (Philipp Moritz & Yifei Feng, Anyscale, 30:23)
- [RAG Has Been Oversimplified](https://mlopstalks.com/talks/rag-has-been-oversimplified) (Yujian Tang, Zilliz, 48:56)
- [False Starts and Dead Ends: Building a Retrieval Augmented Generation System](https://mlopstalks.com/talks/false-starts-and-dead-ends-building-a-retrieval-augmented-generation-system) (Wes Ladd, Train GRC, 11:44)