# GPU For Machine Learning

Ronen Dar & Gijsbert Janssen van Doorn, Run:ai | MLOps Coffee Sessions | Episode 99 | 1:03:33
Hosted by Vishnu Rachakonda

Source: https://www.youtube.com/watch?v=clU3Z18eyeY
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/gpu-for-machine-learning
Published: 2022-05-20
Tags: gpus, model-serving, orchestration, synthetic-data

## TL;DR
- Run:ai adds a software layer above GPUs and Kubernetes so organizations can share GPU capacity across teams and workloads.
- Gijsbert Janssen van Doorn describes customers with data science teams, separate business units, and large model-serving fleets that need better GPU utilization.
- Ronen Dar argues that GPU scheduling is still static and exclusive compared with CPU resource management, leaving room for fractional allocation and workload-aware scheduling.

## Summary
Ronen Dar and Gijsbert Janssen van Doorn explain why Run:ai was built around GPU resource management for machine learning. Ronen describes a gap between compute-intensive AI workloads and the software available to manage GPUs. Gijsbert discusses teams that need to share expensive GPUs, centralize infrastructure across business units, and move many models into production. Ronen compares current GPU allocation with CPU and Kubernetes scheduling, where applications can share resources more flexibly. He describes Run:ai's Kubernetes scheduler, guaranteed quotas, fairness, and fractional GPUs. The conversation also covers inference, including the problem of running thousands of infrequently used models without dedicating one GPU to each model. The speakers discuss hosting multiple models on one GPU, model orchestration, and the limits of scaling GPU nodes to zero because startup can take several minutes. Ronen expects specialized hardware to become more common and GPUs to become first-class resources in cloud-native systems.

## Key ideas
### Run:ai was created because AI workloads introduced a missing layer in the software stack
[05:28](https://www.youtube.com/watch?v=clU3Z18eyeY&t=328s)
Ronen Dar says Run:ai started in early 2018 after the founders saw a gap between AI workloads and the software used to run them. Training and deploying models require compute-intensive workloads, while GPUs became central to breakthroughs in deep learning, natural language processing, and computer vision. Existing infrastructure had been designed around microservices and commodity CPUs. Run:ai built what Ronen calls an "operating system for AI", a software layer above GPUs that is closely tied to Kubernetes and cloud-native technologies. Its purpose is to make GPU access simpler and allow multiple workloads to share GPU capacity.

### GPU sharing is needed by both individual data science teams and large organizations
[09:44](https://www.youtube.com/watch?v=clU3Z18eyeY&t=584s)
Gijsbert Janssen van Doorn describes several customer situations. A data science team may have limited, expensive GPUs that need to be shared efficiently without forcing researchers to understand Kubernetes infrastructure. Larger organizations may have separate AI teams and business units that bought their own hardware, creating what Gijsbert calls "shadow AI". Run:ai helps centralize those resources and provide them to teams through the infrastructure organization. Another group has already trained models and needs to deploy many of them in production. Across these cases, the product aims to let data scientists focus on notebooks, models, and training while infrastructure teams manage policies and resources.

### GPU scheduling remains static and exclusive compared with CPU scheduling
[14:17](https://www.youtube.com/watch?v=clU3Z18eyeY&t=857s)
Ronen compares GPU management with the software built around CPUs over decades, including Linux resource management and Kubernetes container orchestration. In his description, GPU allocation is still static and exclusive. When a container receives a GPU, other applications cannot use it, even if the first application leaves the GPU idle. He compares this with a laptop where browser CPU cores would be reserved exclusively for the browser. Run:ai adds software intended to manage GPU resources more flexibly, alongside CPUs, memory, storage, networking, and other accelerators in a cluster.

### Run:ai uses a Kubernetes scheduler designed for flexible AI workloads
[20:36](https://www.youtube.com/watch?v=clU3Z18eyeY&t=1236s)
Ronen says the team studied high-performance computing schedulers and YARN and Spark workloads before building its Kubernetes scheduler. AI training jobs can run for a long time and may need very different amounts of compute. A researcher may need one GPU for a notebook, then twenty GPUs for a larger training job or many GPUs for hyperparameter optimization. Hard limits such as giving each person two or four GPUs would prevent that flexibility. Run:ai introduced guaranteed quota and fairness concepts, then built a scheduler that plugs into Kubernetes and runs with the default scheduler. The goal is to share clusters without imposing fixed GPU limits on every user.

### Fractional GPUs let small workloads use less than a whole accelerator
[31:25](https://www.youtube.com/watch?v=clU3Z18eyeY&t=1885s)
Ronen identifies fractional GPU support as another difficult feature. Kubernetes traditionally lets a workload request an integer number of GPUs, such as one, two, or four. It does not naturally let a user request half or a quarter of a GPU in the same way CPU resources can be divided. Run:ai enables containers to use fractions of a GPU. This fits notebook workloads that do not need a full accelerator and inference workloads that need much less than one full GPU. The feature changes how teams can right-size workloads and place more of them on the hardware they already have.

### Inference creates a model placement problem when thousands of models are only used occasionally
[40:50](https://www.youtube.com/watch?v=clU3Z18eyeY&t=2450s)
Ronen describes a customer with more than 3,000 models that need to run on GPUs. Assigning one GPU to each model would require 3,000 GPUs, which would be expensive and difficult to obtain from a cloud provider. Scaling GPU capacity to zero is also unsuitable for latency-sensitive applications because starting a GPU node can take six or seven minutes, including the node, drivers, and serving setup. One option is to host multiple models on a single server with systems such as TensorFlow Serving, TorchServe, or NVIDIA Triton. That reduces the one-model-per-GPU assumption, while creating new work around model scaling, priorities, memory collisions, and starvation.

### The product tries to hide infrastructure complexity while keeping administrators in control
[32:58](https://www.youtube.com/watch?v=clU3Z18eyeY&t=1978s)
Ronen says the team worked to wrap advanced infrastructure technology in simple features for researchers, engineers, administrators, and DevOps teams. Gijsbert describes the intended user experience as hands-off for data scientists. They submit jobs, while policies, priorities, queues, and fairness determine how resources are assigned. Users can consume available compute without being restricted to fixed quotas, while administrators still manage budgets and cluster behavior. The speakers acknowledge that unrestricted access can become expensive. Ronen gives the example of an A100 server costing about 32 dollars per hour on Amazon and says Run:ai provides controls to keep usage from going too far above a budget.

### Ronen expects specialized accelerators to become normal in cloud-native infrastructure
[54:01](https://www.youtube.com/watch?v=clU3Z18eyeY&t=3241s)
Ronen says the need for compute will continue to grow as teams process more data, train larger models, and solve more complex problems. He expects specialized hardware to become more important alongside CPUs, including GPUs, TPUs, and hardware developed by cloud providers and Intel. The challenge is making those accelerators work together and managing them efficiently. He believes GPUs will become first-class resources in Kubernetes, which currently gives that status mainly to CPU and memory. He also expects other widely used cloud applications to receive specialized hardware. In his view, the software layers for this infrastructure are still at an early stage.

## Notable quotes
- Ronen Dar: "We've built what we call an operating system for AI." (07:31)
- Gijsbert Janssen van Doorn: "We hate to see idle GPUs because that's a very expensive GPU, an idle GPU." (10:53)
- Ronen Dar: "When an application starts to run on a GPU, that GPU is allocated to that container only, to that application, no matter if the application is using the GPU or not." (17:39)
- Ronen Dar: "You don't want to limit your users to static quotas." (29:29)
- Ronen Dar: "We wanted to bring as much freedom and as much simplicity to the researchers, to the users." (36:29)

## Tools & references mentioned
- Run:ai
- Kubernetes
- Linux
- YARN
- Hadoop
- Spark
- TensorFlow Serving
- TorchServe
- NVIDIA Triton
- OpenAI
- Amazon

## Who should watch
- Your machine learning teams share expensive GPUs and researchers are waiting for access or leaving capacity idle.
- You run Kubernetes for AI workloads and need scheduling based on fairness, flexible GPU amounts, or fractional allocation.
- Your inference platform has many models with uneven traffic and a full GPU per model is too costly.

## Related talks

- [The Role of Resource Management in MLOps](https://mlopstalks.com/talks/the-role-of-resource-management-in-mlops) (Ronen Dar & Gijsbert Janssen van Doorn, Run:AI, 53:38)
- [Accelerating Growth Through Optimizing GPU Usage](https://mlopstalks.com/talks/accelerating-growth-through-optimizing-gpu-usage) (Sahil Khanna, Adobe, 23:53)
- [Building Out GPU Clouds](https://mlopstalks.com/talks/building-out-gpu-clouds) (Mohan Atreya, Rafay Systems, 47:58)
- [Efficient GPU infrastructure at LinkedIn](https://mlopstalks.com/talks/efficient-gpu-infrastructure-at-linkedin) (Animesh Singh, LinkedIn, 59:14)
- [Building Data Centers for GPU Clouds](https://mlopstalks.com/talks/building-data-centers-for-gpu-clouds) (Craig Tavares, Buzz HPC, 46:00)