# The GPU Uptime Battle

Andy Pernsteiner, VAST Data | MLOps Podcast | Episode 346 | 1:33:46
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=IaFDFIIioKs
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-gpu-uptime-battle
Published: 2025-11-11
Tags: governance, gpus, platform-teams, reliability

## TL;DR
- A demo that works on a laptop hides the parallelism, scaling, failure handling, and support work required in production.
- Large GPU environments measure outages in GPU minutes, so storage and software updates must recover without forcing the whole platform offline.
- Platform teams need to understand why users are asking for a service, while application teams need to understand the infrastructure constraints behind their requests.

## Summary
Andy Pernsteiner explains why AI systems become difficult when they move beyond a laptop or a few servers. More data brings more variety, and larger workloads require careful sharding, memory movement, routing, cost control, and failure handling. In large GPU farms, a short outage becomes millions of GPU minutes, while power problems and hardware failures are increasingly common. Andy describes VAST Data's approach of separating logic from state so systems can scale in different directions, recover from data-center outages, and receive software updates without disruptive maintenance windows. The conversation also covers the human work around infrastructure. Platform engineers need to learn what application teams and researchers are trying to achieve, while developers need to gather requirements and think about production conditions early. Demetrios Brinkmann connects this to agent UX, prompt compression, model routing, privacy, and the risk of forcing users to interact with many separate agents. Andy argues that reliable tools, strong data controls, and attention to the end user still matter more than an impressive demo.

## Key ideas
### A laptop demo hides the cost of production
[00:00](https://www.youtube.com/watch?v=IaFDFIIioKs&t=0s)
Andy says a common path starts with a laptop and ends with a desktop machine under someone's desk before the creator asks infrastructure teams for help. The creator sees a project that cost a few hours and perhaps a couple hundred dollars, while the platform team sees a deployment that could require $10,000 of budget. Larger data sets add variety as well as volume. Teams must think about sharding across databases, storage, and GPUs, along with how much memory moves between machines. A job with 10,000 GPUs still gets poor value if its work bottlenecks on one GPU.

### GPU downtime becomes an operating cost
[02:10](https://www.youtube.com/watch?v=IaFDFIIioKs&t=130s)
Andy describes a customer that measures failures in GPU minutes. A five- or ten-minute interruption multiplied across a large GPU farm creates a large, direct cost, whether it happens during training or preprocessing. The price of the hardware makes each interruption more painful. At scale, teams also need to consider model routing, user concurrency, latency, and cloud spending. A policy that sends difficult requests from a cheap model to more expensive models can create unpredictable costs once usage grows.

### Production work includes the boring failure cases
[13:37](https://www.youtube.com/watch?v=IaFDFIIioKs&t=817s)
Andy says developers often focus on what they want to build and work around obstacles until they reach a demo. Production users still face the obstacles that were skipped. Platform teams must ask what happens when a server or network link fails, when a user has a slow connection, or when timeouts are reached. He connects this work to chaos engineering and describes testers who deliberately do unexpected things. AI can help document bugs, but teams still need to understand whether a failed result came from a bad prompt or from a task that the system cannot perform.

### Requirements gathering should happen before the demo
[08:52](https://www.youtube.com/watch?v=IaFDFIIioKs&t=532s)
Andy wants engineers to spend less time showing a finished feature and more time asking what users actually need. Demetrios recommends The Mom Test, which encourages questions about real pain rather than leading questions such as whether a proposed feature would be useful. Andy says a strong signal is that someone has already searched for a solution, joined a community, or tried to solve the problem themselves. He also argues that showing vulnerability can make it easier for users to give honest criticism instead of politely approving a feature they will not use.

### Data is rarely clean in a lasting sense
[22:38](https://www.youtube.com/watch?v=IaFDFIIioKs&t=1358s)
Andy says entire industries have formed around cleaning data, but cleaning always means choosing a format for a particular person or use case. That format may not suit the next person. Organizations are also being encouraged to retain more data because its future value is uncertain, even when they do not know how to store or structure it. A bank with decades of transactional history may have valuable information, but extracting it is difficult, especially under regulatory constraints. Andy rejects the idea that an organization can simply place a large archive somewhere and quickly build a useful service on top of it.

### Platform teams should provide services instead of exposing plumbing
[30:18](https://www.youtube.com/watch?v=IaFDFIIioKs&t=1818s)
Andy says platform engineers should spend less time manually provisioning low-level infrastructure and more time understanding their customers. A Kubernetes cluster, networking components, or storage system should be managed through repeatable playbooks so the team can offer a usable service. He describes this as a cultural issue as well as a technology issue. Data scientists and infrastructure engineers should move closer together, ask why a request exists, and find a solution that avoids repeated migrations or support calls.

### Availability means serving the customer's real workflow
[47:03](https://www.youtube.com/watch?v=IaFDFIIioKs&t=2823s)
Andy says VAST Data measures success partly through customer uptime, including failures caused by a changed behavior after an upgrade. Returning a 200 response is insufficient if the customer cannot use what follows. His team asks platform customers who is requesting a service and why they need it. That context helps them build something that works for the downstream users as well. Andy's team also acts as a final customer-like test before releases, following the documentation from the beginning and trying to find confusing instructions or user experience problems.

### Separating state from logic supports recovery and rolling updates
[52:57](https://www.youtube.com/watch?v=IaFDFIIioKs&t=3177s)
Andy explains that many file systems use buffers, journals, and file-system checks after a power failure. On a large system, replay and checking can keep everything offline for hours. VAST Data separates the state layer from the logic layer, which allows software updates to happen in a rolling fashion without taking the full system offline. The same design lets customers scale performance without automatically adding storage, or add capacity without paying for more performance. Andy says the company spent a long period designing the architecture before writing code and waited before deploying it to alpha customers.

### Agent systems still need reliable tools and data controls
[24:24](https://www.youtube.com/watch?v=IaFDFIIioKs&t=1464s)
Demetrios says the community is building a poor user experience by making every product an agent instead of a tool. He prefers the idea of one agent that can act across services, while also worrying about trust and sensitive information entering a shared context window. Andy agrees that reliable, consistent tools still have value even if they sound less impressive than agents. He adds that a small retrieval system on a laptop can become difficult to scale because users may not want its creator to see their data. Access controls, lineage, governance, authorization, and labels on derived vectors need to apply at the underlying data layer.

## Notable quotes
- Andy Pernsteiner: "If you have 10,000 GPUs at your disposal, but everything you're running ends up bottlenecking on one, then what's the point?" (05:59)
- Andy Pernsteiner: "The best thing that we can do is make sure that they don't have to wake up at midnight." (45:13)
- Andy Pernsteiner: "We can't defeat the speed of light, but we can cheat a little bit." (1:20:36)
- Demetrios Brinkmann: "Everybody wants to be an agent and nobody wants to be a tool." (1:24:24)
- Andy Pernsteiner: "The most important part of a computing environment or a computing network is the user." (09:32)

## Tools & references mentioned
- VAST Data
- The Mom Test
- Rob Fitzpatrick
- Hadoop
- Databricks
- Spark
- Common Crawl
- Nvidia
- Melanox
- CUDA
- Dynamo
- SourceCraft
- Ben Young
- frugal GPT
- Stanford
- OpenAI
- MCP

## Who should watch
- You are moving an AI prototype from a laptop into a shared platform and need to understand where the hidden cost and operational work will appear.
- Your team runs GPU-heavy training or inference and needs practical ways to think about outages, checkpointing, power failures, and rolling updates.
- You are building a platform for researchers or application teams and want better conversations about requirements, support boundaries, privacy, and user experience.

## Editor's note

Andy Pernsteiner says platform engineers should spend less time manually provisioning infrastructure and more time understanding what their customers need. ZenML lets teams define ML workflows as Python pipelines and choose the orchestrator and other infrastructure through a stack configuration. The same pipeline code can then run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud service.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Building Out GPU Clouds](https://mlopstalks.com/talks/building-out-gpu-clouds) (Mohan Atreya, Rafay Systems, 47:58)
- [Accelerating Growth Through Optimizing GPU Usage](https://mlopstalks.com/talks/accelerating-growth-through-optimizing-gpu-usage) (Sahil Khanna, Adobe, 23:53)
- [Efficient GPU infrastructure at LinkedIn](https://mlopstalks.com/talks/efficient-gpu-infrastructure-at-linkedin) (Animesh Singh, LinkedIn, 59:14)
- [GPU For Machine Learning](https://mlopstalks.com/talks/gpu-for-machine-learning) (Ronen Dar & Gijsbert Janssen van Doorn, Run:ai, 1:03:33)
- [Building Data Centers for GPU Clouds](https://mlopstalks.com/talks/building-data-centers-for-gpu-clouds) (Craig Tavares, Buzz HPC, 46:00)