# LLM on Kubernetes

Shrinand Javadekar, Outerbounds & Manjot Pahwa, Lightspeed India & Rahul Parundekar, A.I. Hero & Patrick Barker | Conference in Production 2023 | 36:24

Source: https://www.youtube.com/watch?v=0e5q4zCBtBs
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/llm-on-kubernetes
Published: 2023-07-28
Tags: deployment, gpus, inference, platform-teams

## TL;DR
- Kubernetes gives teams a cloud-agnostic way to train and deploy models while keeping data inside their own VPC.
- Kubernetes handles the distributed parts of LLM systems well, but GPU cost, model size, startup time, scheduling, and inference still create hard problems.
- Data scientists need a layer that hides Kubernetes operations, while platform teams still need to manage repeatable deployment, feedback, and infrastructure workflows.

## Summary
The panel discusses where Kubernetes fits in LLM work, from foundation-model training and fine-tuning to inference and applications that combine models with vector databases and web services. The speakers describe Kubernetes as useful for workload isolation, orchestration, portability, and coordinating several services. They also describe its limits. Large model containers can take a long time to pull, GPU nodes are expensive and difficult to obtain, autoscaling is hard, and inference depends on hardware-specific libraries that containers do not fully hide. The panel distinguishes training and fine-tuning from inference, with Kubernetes considered more mature for the former. The practical advice is to optimize for deployment velocity, reserve GPU capacity when necessary, and use higher-level tools when a team does not have Kubernetes expertise. Manjot Pahwa also argues that the market still has room for infrastructure products that hide operational complexity while addressing cost, latency, reliability, and GPU availability.

## Key ideas
### Kubernetes applies its existing orchestration model to machine learning workloads
[06:26](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=386s)
Shrinand Javadekar explains that Kubernetes was created around the ideas of scalable, reliable container workloads, drawing on Google's Borg. Its original focus was general infrastructure such as APIs and microservices, not machine learning. The same abstraction can still help ML teams because Kubernetes separates workloads from hardware and provides orchestration, reliability, and scaling. The panel treats this as a useful foundation rather than proof that Kubernetes already solves every ML problem.

### Keeping data inside a company VPC is a major reason to run models in-house
[08:34](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=514s)
Rahul Parundekar says enterprises are interested in training and deploying models inside their own environments because they do not want data to leave their VPC. Kubernetes can provide a cloud-agnostic platform for this work, including model training and deployment. That portability comes with trade-offs, but it gives teams a common operating layer instead of tying the whole system to one cloud provider or hosted model service.

### LLM workloads can look like distributed microservices even when they are model systems
[10:33](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=633s)
Patrick Barker argues that generative AI workloads are moving beyond the batch-heavy pattern associated with earlier machine learning systems. Different model partitions may need to communicate with each other, which requires containers to be scheduled and networked together. In that sense, Kubernetes' existing strengths around microservices can fit some LLM architectures. The panel also notes that batch jobs still matter, and tools such as Argo have helped fill gaps in Kubernetes' treatment of batch workloads.

### GPU economics and scheduling remain difficult even with Kubernetes
[11:47](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=707s)
The speakers describe GPU cost and availability as major constraints. Kubernetes can scale nodes up and down and can use GPU node pools or spot instances, but starting a node and loading model weights can take minutes. Scheduling also has to account for where data moves between nodes and pods. The panel suggests that teams may need to reserve GPU capacity in advance rather than rely on autoscaling alone. Patrick Barker also raises hot-swapping LoRA or QLoRA components as a possible way to avoid restarting an entire node for every fine-tuning task.

### Large model containers make startup, storage, and network problems visible
[17:37](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=1057s)
Rahul Parundekar describes the operational cost of foundation models and fine-tuning. Foundation-model data can be measured in petabytes, while model containers can be tens of gigabytes. He gives the example of a roughly 15 GB container taking much longer to pull on some nodes than others. Repeatedly downloading such images increases startup time and cost, and a pod may fail if the node lacks enough disk space. These details make Kubernetes operation substantially harder than writing the training code itself.

### A good data scientist experience hides Kubernetes operations
[19:57](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=1197s)
Shrinand Javadekar says the best Kubernetes experience for a data scientist is not having to deal with Kubernetes directly. The panel describes a division between platform or deployment teams and data science teams, and Rahul Parundekar mentions products that let a user write Python or PyTorch code and submit it with a single command. That abstraction can be a useful starting point, but production systems still need to handle inference, user feedback, data movement, and repeatable deployment.

### Kubernetes is well suited to coordinating an LLM application's supporting services
[23:05](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=1385s)
The panel distinguishes model training from application architectures that combine a model with a vector database, a web app, and other services. Manjot Pahwa says those systems fit Kubernetes' service-oriented design because the components need to run together and communicate. Kubernetes provides one place to orchestrate them and one coherent API for the application. The speakers still reject the idea that Kubernetes is a universal answer, especially for smaller teams with limited budgets.

### Inference has different problems from training and is less mature on Kubernetes
[24:58](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=1498s)
The panel considers Kubernetes more battle-tested for training and fine-tuning than for inference. Inference may involve real-time services, vector databases, and hardware-specific libraries or drivers that work with acceleration features. Containers aim to hide hardware details, but LLM inference still depends on them. The speakers recommend using open-source solutions that abstract some of this complexity while the ecosystem continues to develop.

### The infrastructure market still has room for products that hide operational complexity
[28:59](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=1739s)
Manjot Pahwa says there is no settled best practice for hosting LLM applications in production because the field is changing quickly. She sees room for platforms that hide Kubernetes' moving parts while addressing GPU shortages, cost, reliability, and latency. Patrick Barker adds that the right choice depends on company size. For some startups, running Kubernetes alongside expensive LLM workloads may cost more than using a hosted model or an OpenAI API.

## Notable quotes
- Shrinand Javadekar: "The best experience for data scientists to deal with Kubernetes is not dealing with Kubernetes." (19:57)
- Manjot Pahwa: "Kubernetes is not the Silver Bullet." (23:48)
- Rahul Parundekar: "The real challenge right now is GPUs are super rare to get hands-on." (15:16)
- Manjot Pahwa: "I think you're better off picking up some of these open source solutions present that at least abstract some of that complexity away." (32:44)

## Tools & references mentioned
- Kubernetes
- OpenAI
- ChatGPT
- DALL-E
- Borg
- Argo
- DeepSpeed
- LoRA
- QLoRA
- H100
- A100
- PyTorch
- MetaFlow
- Replicate
- Anthropic
- OpenAI API
- Cloudflare Workers

## Who should watch
- You are deciding whether to run LLM training or inference on your own Kubernetes platform and need a candid account of GPU, container, and scheduling problems.
- Your data must stay inside a VPC, but your team needs a portable platform for training and serving models.
- You are building an ML platform and want to understand which Kubernetes details should be hidden from data scientists.

## Editor's note

Shrinand Javadekar says the best Kubernetes experience for a data scientist is not having to deal with Kubernetes directly. ZenML lets teams write ML workflows as Python steps and choose the orchestrator and infrastructure through configuration, so the same pipeline code can run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Challenges in Providing LLMs as a Service](https://mlopstalks.com/talks/challenges-in-providing-llms-as-a-service) (Hemant Jain, Cohere AI, 11:43)
- [Challenges and Opportunities in Building Data Science Solutions with LLMs](https://mlopstalks.com/talks/challenges-and-opportunities-in-building-data-science-solutions-with-llms) (Pascal Brokmeier, McKinsey and Company & Daniel Herde & Viktoriia Oliinyk, QuantumBlack, AI by McKinsey, 37:19)
- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)
- [Efficiently Scaling and Deploying LLMs](https://mlopstalks.com/talks/efficiently-scaling-and-deploying-llms) (Hanlin Tang, MosaicML, 25:14)
- [GPU For Machine Learning](https://mlopstalks.com/talks/gpu-for-machine-learning) (Ronen Dar & Gijsbert Janssen van Doorn, Run:ai, 1:03:33)