Kubernetes gives teams a cloud-agnostic way to train and deploy models while keeping data inside their own VPC.
2
Kubernetes handles the distributed parts of LLM systems well, but GPU cost, model size, startup time, scheduling, and inference still create hard problems.
3
Data scientists need a layer that hides Kubernetes operations, while platform teams still need to manage repeatable deployment, feedback, and infrastructure workflows.
Summary
The panel discusses where Kubernetes fits in LLM work, from foundation-model training and fine-tuning to inference and applications that combine models with vector databases and web services. The speakers describe Kubernetes as useful for workload isolation, orchestration, portability, and coordinating several services. They also describe its limits. Large model containers can take a long time to pull, GPU nodes are expensive and difficult to obtain, autoscaling is hard, and inference depends on hardware-specific libraries that containers do not fully hide. The panel distinguishes training and fine-tuning from inference, with Kubernetes considered more mature for the former. The practical advice is to optimize for deployment velocity, reserve GPU capacity when necessary, and use higher-level tools when a team does not have Kubernetes expertise. Manjot Pahwa also argues that the market still has room for infrastructure products that hide operational complexity while addressing cost, latency, reliability, and GPU availability.
Kubernetes applies its existing orchestration model to machine learning workloads
Shrinand Javadekar explains that Kubernetes was created around the ideas of scalable, reliable container workloads, drawing on Google's Borg. Its original focus was general infrastructure such as APIs and microservices, not machine learning. The same abstraction can still help ML teams because Kubernetes separates workloads from hardware and provides orchestration, reliability, and scaling. The panel treats this as a useful foundation rather than proof that Kubernetes already solves every ML problem.
Keeping data inside a company VPC is a major reason to run models in-house
Rahul Parundekar says enterprises are interested in training and deploying models inside their own environments because they do not want data to leave their VPC. Kubernetes can provide a cloud-agnostic platform for this work, including model training and deployment. That portability comes with trade-offs, but it gives teams a common operating layer instead of tying the whole system to one cloud provider or hosted model service.
LLM workloads can look like distributed microservices even when they are model systems
Patrick Barker argues that generative AI workloads are moving beyond the batch-heavy pattern associated with earlier machine learning systems. Different model partitions may need to communicate with each other, which requires containers to be scheduled and networked together. In that sense, Kubernetes' existing strengths around microservices can fit some LLM architectures. The panel also notes that batch jobs still matter, and tools such as Argo have helped fill gaps in Kubernetes' treatment of batch workloads.
GPU economics and scheduling remain difficult even with Kubernetes
The speakers describe GPU cost and availability as major constraints. Kubernetes can scale nodes up and down and can use GPU node pools or spot instances, but starting a node and loading model weights can take minutes. Scheduling also has to account for where data moves between nodes and pods. The panel suggests that teams may need to reserve GPU capacity in advance rather than rely on autoscaling alone. Patrick Barker also raises hot-swapping LoRA or QLoRA components as a possible way to avoid restarting an entire node for every fine-tuning task.
Large model containers make startup, storage, and network problems visible
Rahul Parundekar describes the operational cost of foundation models and fine-tuning. Foundation-model data can be measured in petabytes, while model containers can be tens of gigabytes. He gives the example of a roughly 15 GB container taking much longer to pull on some nodes than others. Repeatedly downloading such images increases startup time and cost, and a pod may fail if the node lacks enough disk space. These details make Kubernetes operation substantially harder than writing the training code itself.
A good data scientist experience hides Kubernetes operations
Shrinand Javadekar says the best Kubernetes experience for a data scientist is not having to deal with Kubernetes directly. The panel describes a division between platform or deployment teams and data science teams, and Rahul Parundekar mentions products that let a user write Python or PyTorch code and submit it with a single command. That abstraction can be a useful starting point, but production systems still need to handle inference, user feedback, data movement, and repeatable deployment.
Kubernetes is well suited to coordinating an LLM application's supporting services
The panel distinguishes model training from application architectures that combine a model with a vector database, a web app, and other services. Manjot Pahwa says those systems fit Kubernetes' service-oriented design because the components need to run together and communicate. Kubernetes provides one place to orchestrate them and one coherent API for the application. The speakers still reject the idea that Kubernetes is a universal answer, especially for smaller teams with limited budgets.
Inference has different problems from training and is less mature on Kubernetes
The panel considers Kubernetes more battle-tested for training and fine-tuning than for inference. Inference may involve real-time services, vector databases, and hardware-specific libraries or drivers that work with acceleration features. Containers aim to hide hardware details, but LLM inference still depends on them. The speakers recommend using open-source solutions that abstract some of this complexity while the ecosystem continues to develop.
The infrastructure market still has room for products that hide operational complexity
Manjot Pahwa says there is no settled best practice for hosting LLM applications in production because the field is changing quickly. She sees room for platforms that hide Kubernetes' moving parts while addressing GPU shortages, cost, reliability, and latency. Patrick Barker adds that the right choice depends on company size. For some startups, running Kubernetes alongside expensive LLM workloads may cost more than using a hosted model or an OpenAI API.
"I think you're better off picking up some of these open source solutions present that at least abstract some of that complexity away."Manjot Pahwa32:44
Who should watch
You are deciding whether to run LLM training or inference on your own Kubernetes platform and need a candid account of GPU, container, and scheduling problems.
Your data must stay inside a VPC, but your team needs a portable platform for training and serving models.
You are building an ML platform and want to understand which Kubernetes details should be hidden from data scientists.
