Harcharan Kabbay argues that RAG applications should be designed as resilient microservices, with protection against failures in the LLM, vector store, and orchestration layers.
2
Harcharan Kabbay recommends separating code, configuration, and secrets so teams can promote the same image across development, testing, and production without rebuilding it for every parameter change.
3
Harcharan Kabbay says observability must cover each stage of an ML or RAG workflow, since monitoring only the final answer can hide retrieval problems, data drift, latency issues, and resource limits.
Summary
Harcharan Kabbay explains how to move GenAI applications from experiments into managed services. He treats a RAG system as several connected components, including retrieval, vector storage, an orchestration layer, and an LLM, so each part needs testing and protection from single points of failure. Kubernetes, Argo CD, containers, configuration maps, secrets management, and Terraform help teams deploy the same application through multiple environments. He also argues that observability must cover data, model performance, retrieval quality, final responses, latency, logs, and resource use. Kabbay is direct about the limits of local experimentation, manual provisioning, excessive logging, and dependence on one subject matter expert. His practical answer is to standardize repository structures, pull request checks, health checks, documentation, and incident response. The conversation also covers security, library vulnerabilities, licensing, and the possibility of applying chaos engineering to ML data and services.
Local LLM experiments need an operational path before they become team practice
Demetrios Brinkmann asks why Harcharan Kabbay is skeptical of local LLMs. Kabbay says local models are useful for experimentation, but teams need to think about how they will operationalize them and get value from them. He is concerned when every person runs a model independently with little control over access and deployment. His preferred direction is to expose the model through an API from the beginning, so local testing does not create habits that later become an Operations problem. Demetrios compares this with notebooks that work on a developer's machine and are then thrown over the fence to another team.
A RAG application should be treated as a set of failure-prone services
Kabbay describes RAG as a sequence of calls: a user query reaches an orchestration engine, the system searches a vector store, retrieved content is added to a prompt, and the prompt is sent to an LLM. Each hop can fail. He suggests using a pool of LLMs to handle rate-limit errors such as HTTP 429 responses, and clustering databases so one unavailable instance does not stop the application. In Kubernetes, the orchestration layer can run as a deployment with multiple replicas. This architecture should be considered before the CI/CD details, because deployment automation cannot compensate for a single point of failure.
CI/CD should promote the same container while configuration changes outside the image
Kabbay recommends containerizing application code and tracking library versions through image versions. A CI job can listen for GitHub changes, build an image after a merge, and store the artifact in an image repository rather than GitHub. Argo CD can watch the repository and synchronize changes to Kubernetes, with rollback available when needed. The same image can move through development, test, staging, and production. Non-confidential settings such as URLs and retrieval parameters belong in Kubernetes ConfigMaps. Secrets such as API keys belong in secret-management systems and can be synchronized into Kubernetes. Changing a retrieval setting from 10 results to 15 should not require rebuilding the image.
Kubernetes patterns should match the application instead of being applied uniformly
Demetrios asks how the deployment approach differs between an LLM and RAG service and a conventional recommender model. Kabbay says even traditional models need a different operational shape once they are served as APIs. KServe provides preprocessing, model serving, and postprocessing patterns, and Ray Serve is another option he plans to explore. A model does not always need to be baked into a Docker image. Kubernetes knowledge helps teams choose the right deployment pattern instead of applying the same rule everywhere. Kabbay also says no single person can operationalize an ML system alone. Data scientists, ML engineers, and Operations teams each have work to do.
Terraform turns repeated cloud setup into reviewed, versioned configuration
Kabbay says teams should avoid creating cloud resources manually. A new resource may require a naming convention, resource group, tags, firewall rules, and other settings. Repeating that process for storage, OpenAI, search, and database resources quickly becomes difficult to control. Terraform lets the team define those rules in versioned code. Kabbay describes a modular approach where resources have separate modules and inventory files, such as a CSV listing resource groups. Adding more resources then means changing the inventory and opening a pull request, while the shared configuration remains governed by code. He says this makes responsibility and review clearer.
Templates move operational knowledge from individuals into the team's process
Kabbay is a strong advocate for templates. Pull request templates can remind authors to check naming standards, remove confidential information, and test changes before submission. Reviewers still verify the parts they own, so approval becomes shared responsibility rather than a click-through exercise. Kabbay extends the same idea to repository templates. A new application can start from a skeleton containing the expected files, README instructions, logging, health checks, and other required patterns. When someone finds a missing part, they can update the template through a pull request. This reduces repeated omissions and helps people learn the application's operating model.
Teams should prevent subject matter experts from becoming blockers
Kabbay distinguishes between having a subject matter expert and making one person the only person who understands an application. He recommends documenting the design, writing useful README files and code comments, holding knowledge-sharing sessions, and mentoring others. A person who keeps all operational knowledge in their head becomes a dependency for every support request. Kabbay prefers process-oriented design, where automation is documented and others can inspect, operate, and improve it. Demetrios connects this to the people, process, and technology model of DevOps. Kabbay agrees that people remain part of the system, but the implementation should not depend on one person's memory.
RAG observability must measure retrieval and intermediate steps, not only final answers
Kabbay says a model that replaces a manual business process needs an agreed performance target and ongoing measurement. For conventional ML, teams should track training and runtime metrics, data drift, processed rows, and model behavior. For RAG, checking only the final LLM response gives an incomplete view. Teams also need to assess whether retrieved results are relevant, how they are ranked, and whether the augmented input supports the answer. Kabbay suggests exposing an endpoint for search results so retrieval can be evaluated separately from the final response. He also recommends measuring inference latency, handling batch and real-time workloads differently, and watching resource use through observability dashboards.
Security, licensing, health checks, and incident response belong in the delivery path
Kabbay says security is often missed when teams create resources manually or treat a new model as a simple package installation. Each component in a RAG workflow needs its own quality checks, including a replacement vector database or retrieval engine. Teams should scan libraries for vulnerabilities and check licenses before using them commercially. Applications also need an incident response plan for unavailable services, unexpected LLM behavior, and questions that designers did not anticipate. Kabbay recommends health checks for applications and their dependencies, including databases. Templates can include these checks from the start. He accepts that failures cannot be removed entirely, but teams can reduce risk and detect problems earlier.
