# The Role of Resource Management in MLOps

Ronen Dar & Gijsbert Janssen van Doorn, Run:AI | MLOps Meetup | Episode 99 | 53:38
Hosted by Ben Epstein

Source: https://www.youtube.com/watch?v=NYfEeVyOgi8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-role-of-resource-management-in-mlops
Published: 2022-05-09
Tags: gpus, inference, orchestration

## TL;DR
- Resource management is the infrastructure foundation beneath the visible parts of an MLOps stack, including data processing, model training, and production inference.
- A scheduler can pool GPUs, enforce quotas, preempt lower-priority work, and share idle capacity so data scientists get resources without managing machines manually.
- Run:AI extends its Kubernetes-based resource management from training workloads to inference, including fractional GPU use and model-server sizing.

## Summary
Ronen Dar and Gijsbert Janssen van Doorn argue that resource management is an overlooked part of MLOps. Teams often focus on models, data, monitoring, and tools while the GPUs and other compute resources underneath remain static, fragmented, or difficult to access. They describe Run:AI's approach, which adds an AI-focused scheduling layer around Kubernetes. The scheduler pools GPUs, assigns guaranteed quotas, allows workloads to use spare capacity, and can preempt lower-priority jobs when a user needs their reserved resources. This lets teams share infrastructure without manually imaging machines or rescheduling workloads. The speakers also discuss containerized environments, which let researchers use different package versions on shared GPUs. They describe the same resource-management concerns in inference, where organizations may need to keep many models available while using GPU capacity efficiently. Their answer to when this matters ranges from a single researcher running parallel experiments to large organizations with many teams and scattered GPU systems.

## Key ideas
### Compute is a strategic resource for machine learning work
[05:47](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=347s)
Ronen Dar says GPUs enabled recent deep-learning breakthroughs and that more powerful compute lets teams train more models, process more data, and address larger or more difficult problems. Run:AI began after seeing a gap between researchers who needed substantial compute and infrastructure that could provide it. The company built a software layer dedicated to AI workloads, with resource management and workload orchestration for researchers and data scientists. The aim is to give people access to the compute they need without making them deal with the underlying infrastructure.

### Resource management sits beneath the visible MLOps tools
[08:43](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=523s)
Gijsbert Janssen van Doorn uses an MLOps iceberg to explain the problem. Teams commonly discuss data verification, analysis, monitoring, feature stores, and ML code, while the infrastructure underneath receives less attention. He says that infrastructure is the foundation on which the rest of the MLOps stack runs. Compute also affects how quickly teams can process data, train larger models, and iterate. If resources are hard to obtain, they become a bottleneck. He argues that resource management can affect the pipeline from data processing through training and production.

### Static machines make model development slow to scale
[12:42](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=762s)
Without resource management, a data scientist has a machine with a GPU and can use that machine for development and training. When more capacity is needed, they need another machine, along with its tools and environment. Gijsbert describes this setup as static and difficult to scale. Manual scaling can require imaging machines and reinstalling tools such as Python and TensorFlow. A resource-management system can abstract those steps, make capacity available when needed, and let researchers spend more time building and training models.

### Inference creates a different resource-management problem
[16:04](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=964s)
Ronen distinguishes inference from model building and training. An ML engineer or MLOps engineer has to deploy models on CPUs or GPUs, and production often involves multiple models or model pipelines rather than one model. He gives the example of a customer that needed to deploy 3,000 models, with all of them available on GPUs. That requirement makes efficient use of GPU capacity part of the deployment problem. Later, he says Run:AI is adding inference capabilities such as fractional GPU allocation and right-sizing model servers.

### The scheduler combines quotas with shared spare capacity
[19:55](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=1195s)
Run:AI Atlas includes a scheduler built around Kubernetes. The scheduler adds multiple queues, priorities, and quota policies. A guaranteed quota means that a user with a quota of two GPUs can use those two GPUs when needed, although the workload may wait if all resources are occupied. The over-quota system lets another user use spare capacity when reserved resources are idle. This gives teams access to more capacity without changing the underlying hardware allocation.

### Automatic preemption returns reserved resources quickly
[25:46](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=1546s)
Gijsbert's example uses two data scientists sharing a four-GPU pool. Each has a guaranteed quota of two GPUs. When one person goes on holiday, the other person's queued jobs can use the unused GPUs. When the first person returns and starts work, the scheduler preempts one of the other jobs so the reserved GPU becomes available. Gijsbert says this happens in seconds. The same principle applies to shorter idle periods, such as lunch breaks, so GPUs do not remain unused while other workloads are waiting.

### Kubernetes provides the base, while Run:AI adds AI workload scheduling
[21:08](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=1268s)
Run:AI is built around Kubernetes because many AI tools run as containerized workloads on it. The speakers say standard Kubernetes was built primarily for microservices on CPUs, so AI workloads create different scheduling needs. Run:AI's scheduler adds AI-oriented queues, quotas, priorities, fairness, and preemption while plugging into Kubernetes and working with its default scheduler. Ronen says the design drew ideas from high-performance computing schedulers and YARN, which was built for Spark workloads.

### Containers separate researcher environments on shared GPUs
[40:30](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=2430s)
Ronen explains that containers make dependency management easier. Platform engineers can define the required packages and software libraries in an environment, then containerize it. Researchers can connect to that environment through tools such as Jupyter Notebook, PyCharm, or VS Code. Two researchers can use different package versions while running workloads on the same GPU, because their environments remain separate. Those environments can also be replicated when more compute becomes available.

### The value starts with one researcher and grows with shared infrastructure
[44:05](https://www.youtube.com/watch?v=NYfEeVyOgi8&t=2645s)
Ronen says even one data scientist can benefit from a queue when running parallel experiments or hyperparameter optimization on a single GPU system. The value grows when a team shares resources, and it grows again when an organization brings scattered GPU systems into a central pool. Gijsbert describes this as a way for IT to manage GPU infrastructure centrally and make it available to more AI teams. Run:AI's resource management covers CPUs, memory, and GPUs, with its strongest focus on GPU workloads.

## Notable quotes
- Ronen Dar: "Compute is strategic as you have more compute, more powerful compute, you'll be able to run more workloads, train more models, do better AI." (05:47)
- Gijsbert Janssen van Doorn: "The resources or the infrastructure where all of that above actually is running on is kind of hidden somewhere, it's not top of mind." (10:17)
- Gijsbert Janssen van Doorn: "We shouldn't care about the underlying infrastructure. We'll care about it, we'll make sure that everything is being utilized and it's available, but we're trying to abstract a lot of the complexities." (30:30)
- Ronen Dar: "Even when you are a single data scientist and you're going out of just a single notebook, even then you need some kind of a queuing system." (44:34)
- Gijsbert Janssen van Doorn: "We can help organizations build that experience for their data scientists and ML engineers." (51:18)

## Tools & references mentioned
- Run:AI
- Run:AI Atlas
- Kubernetes
- Jupyter Notebook
- PyCharm
- VS Code
- Kubeflow
- Airflow
- MLflow
- Weights & Biases
- YARN
- Spark

## Who should watch
- You run GPU-backed training jobs and spend time waiting for, provisioning, or manually rescheduling machines.
- Your team has several data scientists sharing a limited pool of GPUs and needs quotas, priorities, or automatic use of idle capacity.
- You manage production inference with many models and want to understand why GPU allocation and model-server sizing become operational concerns.

## Related talks

- [GPU For Machine Learning](https://mlopstalks.com/talks/gpu-for-machine-learning) (Ronen Dar & Gijsbert Janssen van Doorn, Run:ai, 1:03:33)
- [Kubernetes, AI Gateways, and the Future of MLOps](https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops) (Alexa Griffith, Bloomberg, 50:12)
- [Considerations and Optimizations for Deploying Open Source LLMs at Your Company](https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company) (Oscar Rovira, Mystic AI, 11:31)
- [Packaging MLOps Tech Neatly for Engineers and Non-engineers](https://mlopstalks.com/talks/packaging-mlops-tech-neatly-for-engineers-and-non-engineers) (Jukka Remes, Haaga-Helia University of Applied Sciences, 8wave AI, 55:31)
- [Scaling AI in Production](https://mlopstalks.com/talks/scaling-ai-in-production) (Srivatsan Srinivasan, AIEngineering, 51:56)