# Composable Memory for GPU Optimization

Bernie Wu, MemVerge | MLOps Podcast | Episode 270 | 55:19
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ccaDEFoKwko
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/composable-memory-for-gpu-optimization
Published: 2024-10-22
Tags: deployment, gpus, inference, orchestration

## TL;DR
- Bernie Wu argues that many AI systems described as GPU-bound are actually limited by memory capacity, bandwidth, or latency.
- MemVerge is developing composable memory pools that can provide temporary capacity when workloads approach out-of-memory conditions, then reclaim it elsewhere.
- Checkpointing into memory before asynchronously writing to a file system could reduce interruption time for large training jobs and avoid making checkpoint management a major part of the training cycle.

## Summary
Bernie Wu argues that memory has become a central constraint in LLM infrastructure. GPU purchases are often sized around the memory needed by a model, leaving GPUs underused while workloads remain memory-bound. He describes composable CXL memory as a separate pool that can be allocated across compute nodes, used as a cache, and reclaimed as demand changes. MemVerge is also working on memory-level checkpointing, including a path where large checkpoints are written quickly to memory and then drained asynchronously to storage. The conversation covers distributed inference, KV-cache growth with longer prompts, LoRA adapters, heterogeneous pipelines, and the need for schedulers to account for memory alongside GPUs and service-level objectives. Bernie expects Kubernetes integrations to evolve through operators, dynamic resource allocation, and new standards. He is candid that several scheduler projects are still competing while the AI stack changes quickly.

## Key ideas
### AI workloads often leave GPUs underused because memory sets the purchase size
[02:03](https://www.youtube.com/watch?v=ccaDEFoKwko&t=123s)
Bernie Wu says production delivery is where companies get a return on AI investment, and Transformer workloads move large amounts of data. He describes a rough relationship of one floating-point operation per byte, or about ten bytes of memory movement for each floating-point operation. Teams often buy GPUs by matching model size to available HBM. The result can be underused GPUs, with some large training workloads operating around the 30% utilization level. Bernie says the systems are still memory-bound, so adding more GPU memory or a separate memory tier could raise utilization without simply buying more GPUs.

### An elastic memory pool could prevent spills, swaps, and out-of-memory termination
[05:25](https://www.youtube.com/watch?v=ccaDEFoKwko&t=325s)
When Kubernetes or an AI workload runs out of memory, it may spill to disk, slow sharply, or get killed when swapping is unavailable. Bernie describes a separate pool that provides surge capacity to a compute instance when memory pressure is detected. The system can give the workload a temporary amount of memory, then reclaim it when conditions change. He connects this approach to composable infrastructure, where GPUs and memory are pooled and assigned as workloads fluctuate. Demetrios Brinkmann points out that short spikes can kill jobs even when average GPU utilization is low, which is the failure this design tries to absorb.

### Memory-level checkpointing could reduce the cost of protecting long-running jobs
[07:07](https://www.youtube.com/watch?v=ccaDEFoKwko&t=427s)
Frameworks such as PyTorch already save model state so training can resume after a crash or roll back after an overshoot. Bernie says that approach does not cover every failure mode, especially memory failures and failures at very large scale. MemVerge is working on checkpointing machine and memory state as well as framework state. Large checkpoints can become a file-system bottleneck. Bernie describes using a memory pool as a fast cache, writing the checkpoint there and moving it to the file system asynchronously. He says this can make the transfer up to 10x faster and notes that checkpoint work can account for 25% to 30% of a training lifecycle.

### Distributed inference creates a second memory problem around KV caches
[13:06](https://www.youtube.com/watch?v=ccaDEFoKwko&t=786s)
Bernie explains that large models such as Llama 405B do not fit on many individual GPU nodes, so inference must be distributed. The prompt stage is more CPU-intensive, while decoding is strongly memory-bound. Longer prompts also make KV-cache requirements grow quadratically with prompt length. Those caches may no longer fit in one device, creating a distributed GPU and distributed memory problem. Memory becomes fragmented as different prompts create different-sized KV sets. Bernie says systems need mechanisms to allocate, migrate, compact, and schedule memory across GPUs and nodes.

### CXL turns memory into a pooled resource outside the processor's traditional bus
[18:38](https://www.youtube.com/watch?v=ccaDEFoKwko&t=1118s)
Bernie describes the CXL Consortium, formed in 2019, as part of a move away from traditional parallel memory buses. CXL allows memory modules to connect through a PCIe bus, expanding memory within a compute node. CXL-enabled PCIe switches can extend that model across nodes. Bernie defines the approach as a way to create external memory pools that compute resources can pull from and share. He imagines the pool holding chat histories or other reusable context so multiple users or agents can access it. MemVerge has prototyped a shared repository with LlamaIndex, though Bernie calls the demonstration a toy demo.

### Schedulers will need to allocate memory according to workload goals
[29:08](https://www.youtube.com/watch?v=ccaDEFoKwko&t=1748s)
Bernie says scheduling has an organizational and economic dimension because scarce resources must be assigned among jobs. A batch job can be interrupted more easily than an interactive session with a service-level agreement. Schedulers also need to balance GPU and memory utilization against session throughput, total throughput, and tail latency such as P99 latency. He says Kubernetes provides useful infrastructure but does not go far enough for accelerator and memory scheduling. Several Kubernetes-related scheduling projects are being developed, yet the right design is still unsettled because models and AI workflows are changing quickly.

### Kubernetes needs dynamic resource management below the node level
[44:48](https://www.youtube.com/watch?v=ccaDEFoKwko&t=2688s)
Bernie says current Kubernetes resource allocation is largely static after a node is configured. A more elastic system would react when available memory or GPUs increase or decrease. MemVerge is working with hardware partners that build memory pools and is exploring Kubernetes hooks or operators. Bernie says the industry may eventually need a standard that brings memory into Kubernetes as a first-class resource, alongside efforts such as Dynamic Resource Allocation. The intended behavior includes detecting memory pressure before a pod is killed, injecting additional memory, and adjusting allocation to workloads whose memory use follows a predictable pattern.

### The memory problem extends beyond AI infrastructure
[50:49](https://www.youtube.com/watch?v=ccaDEFoKwko&t=3049s)
Bernie expects composable memory to help high-performance computing, enterprise databases, and other applications that are evicted or killed under memory pressure. He says AI is currently absorbing most infrastructure attention, but those other workloads will remain part of data centers. He also discusses cost tiers. DRAM is faster but more expensive per bit than SSD, so future systems may combine memory and storage through caching. He mentions memory-semantic SSDs and newer memory technologies as attempts to narrow the large performance and cost gap between DRAM and SSD.

## Notable quotes
- Bernie Wu: "I believe the numbers around one flop per byte, or in other words, there's 10 bytes of memory movement or loading into registers for every floating point operation on average." (03:03)
- Bernie Wu: "The infrastructure needs to get more elastic and composable." (06:31)
- Bernie Wu: "It's not just scheduling GPU resources, but I think also we're going to need to figure out how to schedule memory resources." (17:37)
- Bernie Wu: "I think really when I listen to a lot of these talks, at the end of the day they're really memory-bound problems." (37:09)
- Bernie Wu: "I think over time we'll need to work with areas, there's a standard for the storage layer, and there has to be an equivalent standard for how do we bring memory as a separate first-class citizen into a Kubernetes environment." (49:17)

## Tools & references mentioned
- MemVerge
- MLOps Community
- Demetrios Brinkmann
- UC Berkeley
- Kubernetes
- PyTorch
- Llama 405B
- LlamaIndex
- CXL Consortium
- PCIe
- Ultra Ethernet Consortium
- Unified Accelerator Link
- NVIDIA
- InfiniBand
- RoCE
- Intel Optane
- DRAM
- SSD
- Dynamic Resource Allocation
- Alibaba

## Who should watch
- You are sizing GPU clusters for LLM training or inference and suspect that memory capacity or bandwidth is limiting utilization.
- Your Kubernetes workloads sometimes hit out-of-memory failures, and you want to understand how pooled memory and dynamic allocation could change that.
- You manage large checkpoints, KV caches, or multi-node inference pipelines and need practical context on CXL, memory tiers, and scheduler design.

## Related talks

- [Accelerating Growth Through Optimizing GPU Usage](https://mlopstalks.com/talks/accelerating-growth-through-optimizing-gpu-usage) (Sahil Khanna, Adobe, 23:53)
- [Efficient GPU infrastructure at LinkedIn](https://mlopstalks.com/talks/efficient-gpu-infrastructure-at-linkedin) (Animesh Singh, LinkedIn, 59:14)
- [Building Out GPU Clouds](https://mlopstalks.com/talks/building-out-gpu-clouds) (Mohan Atreya, Rafay Systems, 47:58)
- [Building Data Centers for GPU Clouds](https://mlopstalks.com/talks/building-data-centers-for-gpu-clouds) (Craig Tavares, Buzz HPC, 46:00)
- [Handling Multi-Terabyte LLM Checkpoints](https://mlopstalks.com/talks/handling-multi-terabyte-llm-checkpoints) (Simon Karasik, Nebius AI, 55:37)