Run:ai adds a software layer above GPUs and Kubernetes so organizations can share GPU capacity across teams and workloads.
2
Gijsbert Janssen van Doorn describes customers with data science teams, separate business units, and large model-serving fleets that need better GPU utilization.
3
Ronen Dar argues that GPU scheduling is still static and exclusive compared with CPU resource management, leaving room for fractional allocation and workload-aware scheduling.
Summary
Ronen Dar and Gijsbert Janssen van Doorn explain why Run:ai was built around GPU resource management for machine learning. Ronen describes a gap between compute-intensive AI workloads and the software available to manage GPUs. Gijsbert discusses teams that need to share expensive GPUs, centralize infrastructure across business units, and move many models into production. Ronen compares current GPU allocation with CPU and Kubernetes scheduling, where applications can share resources more flexibly. He describes Run:ai's Kubernetes scheduler, guaranteed quotas, fairness, and fractional GPUs. The conversation also covers inference, including the problem of running thousands of infrequently used models without dedicating one GPU to each model. The speakers discuss hosting multiple models on one GPU, model orchestration, and the limits of scaling GPU nodes to zero because startup can take several minutes. Ronen expects specialized hardware to become more common and GPUs to become first-class resources in cloud-native systems.
Run:ai was created because AI workloads introduced a missing layer in the software stack
Ronen Dar says Run:ai started in early 2018 after the founders saw a gap between AI workloads and the software used to run them. Training and deploying models require compute-intensive workloads, while GPUs became central to breakthroughs in deep learning, natural language processing, and computer vision. Existing infrastructure had been designed around microservices and commodity CPUs. Run:ai built what Ronen calls an "operating system for AI", a software layer above GPUs that is closely tied to Kubernetes and cloud-native technologies. Its purpose is to make GPU access simpler and allow multiple workloads to share GPU capacity.
GPU sharing is needed by both individual data science teams and large organizations
Gijsbert Janssen van Doorn describes several customer situations. A data science team may have limited, expensive GPUs that need to be shared efficiently without forcing researchers to understand Kubernetes infrastructure. Larger organizations may have separate AI teams and business units that bought their own hardware, creating what Gijsbert calls "shadow AI". Run:ai helps centralize those resources and provide them to teams through the infrastructure organization. Another group has already trained models and needs to deploy many of them in production. Across these cases, the product aims to let data scientists focus on notebooks, models, and training while infrastructure teams manage policies and resources.
GPU scheduling remains static and exclusive compared with CPU scheduling
Ronen compares GPU management with the software built around CPUs over decades, including Linux resource management and Kubernetes container orchestration. In his description, GPU allocation is still static and exclusive. When a container receives a GPU, other applications cannot use it, even if the first application leaves the GPU idle. He compares this with a laptop where browser CPU cores would be reserved exclusively for the browser. Run:ai adds software intended to manage GPU resources more flexibly, alongside CPUs, memory, storage, networking, and other accelerators in a cluster.
Run:ai uses a Kubernetes scheduler designed for flexible AI workloads
Ronen says the team studied high-performance computing schedulers and YARN and Spark workloads before building its Kubernetes scheduler. AI training jobs can run for a long time and may need very different amounts of compute. A researcher may need one GPU for a notebook, then twenty GPUs for a larger training job or many GPUs for hyperparameter optimization. Hard limits such as giving each person two or four GPUs would prevent that flexibility. Run:ai introduced guaranteed quota and fairness concepts, then built a scheduler that plugs into Kubernetes and runs with the default scheduler. The goal is to share clusters without imposing fixed GPU limits on every user.
Fractional GPUs let small workloads use less than a whole accelerator
Ronen identifies fractional GPU support as another difficult feature. Kubernetes traditionally lets a workload request an integer number of GPUs, such as one, two, or four. It does not naturally let a user request half or a quarter of a GPU in the same way CPU resources can be divided. Run:ai enables containers to use fractions of a GPU. This fits notebook workloads that do not need a full accelerator and inference workloads that need much less than one full GPU. The feature changes how teams can right-size workloads and place more of them on the hardware they already have.
Inference creates a model placement problem when thousands of models are only used occasionally
Ronen describes a customer with more than 3,000 models that need to run on GPUs. Assigning one GPU to each model would require 3,000 GPUs, which would be expensive and difficult to obtain from a cloud provider. Scaling GPU capacity to zero is also unsuitable for latency-sensitive applications because starting a GPU node can take six or seven minutes, including the node, drivers, and serving setup. One option is to host multiple models on a single server with systems such as TensorFlow Serving, TorchServe, or NVIDIA Triton. That reduces the one-model-per-GPU assumption, while creating new work around model scaling, priorities, memory collisions, and starvation.
The product tries to hide infrastructure complexity while keeping administrators in control
Ronen says the team worked to wrap advanced infrastructure technology in simple features for researchers, engineers, administrators, and DevOps teams. Gijsbert describes the intended user experience as hands-off for data scientists. They submit jobs, while policies, priorities, queues, and fairness determine how resources are assigned. Users can consume available compute without being restricted to fixed quotas, while administrators still manage budgets and cluster behavior. The speakers acknowledge that unrestricted access can become expensive. Ronen gives the example of an A100 server costing about 32 dollars per hour on Amazon and says Run:ai provides controls to keep usage from going too far above a budget.
Ronen expects specialized accelerators to become normal in cloud-native infrastructure
Ronen says the need for compute will continue to grow as teams process more data, train larger models, and solve more complex problems. He expects specialized hardware to become more important alongside CPUs, including GPUs, TPUs, and hardware developed by cloud providers and Intel. The challenge is making those accelerators work together and managing them efficiently. He believes GPUs will become first-class resources in Kubernetes, which currently gives that status mainly to CPU and memory. He also expects other widely used cloud applications to receive specialized hardware. In his view, the software layers for this infrastructure are still at an early stage.
"When an application starts to run on a GPU, that GPU is allocated to that container only, to that application, no matter if the application is using the GPU or not."Ronen Dar17:39
Who should watch
Your machine learning teams share expensive GPUs and researchers are waiting for access or leaving capacity idle.
You run Kubernetes for AI workloads and need scheduling based on fairness, flexible GPU amounts, or fractional allocation.
Your inference platform has many models with uneven traffic and a full GPU per model is too costly.
