Kashish Mittal argues that data I/O, rather than model architecture, is usually the constraint that prevents distributed training from scaling.
2
Caching remote data was not enough because Petastorm still had to convert PyArrow data to NumPy before it could feed the GPU.
3
Petastorm reached 85% GPU utilization after caching the transformed output, and training time fell from a day to one or two hours on the same resources.
Summary
Kashish Mittal describes GPU starvation as a data pipeline problem. Models can sit idle because training reads too much data, moves duplicated features between CPU and GPU hosts, or spends time converting data into a format the GPU can use. At Uber, models running on A100 chips reached only 15% to 20% utilization. Loading the data directly into RAM raised utilization to 85%, which showed that the model itself was not the problem. Tracing Petastorm revealed two bottlenecks: slow reads from remote Parquet files and an expensive PyArrow-to-NumPy conversion. Local caching fixed the first issue, but the team had to cache the transformed NumPy output to fix the second. Kashish also explains the serving trade-off between latency and GPU efficiency, plus a reproducibility problem caused by highly parallel, nondeterministic queues. The conversation ends with practical advice about warming serving models and diagnosing slow Claude Code workloads through their network and initialization paths.
Kashish says that when teams try to scale a model, the model architecture is rarely the main problem. Data I/O is usually the constraint. Reading a small slice from a Parquet table can still mean reading everything and filtering it on the CPU, which wastes time while the GPU waits. He also connects the problem to engineering velocity. Teams cannot assume they can keep adding GPUs, so poor feeding of the accelerator slows both resource use and model iteration. His proposed boundary is that ML engineers should specify the data they need, while infrastructure should handle how that data reaches the GPU.
For models that need particular slices of data, Kashish recommends rewriting the data on disk so those slices can be read in batches. Point lookups are expensive when the underlying files are organized for a different access pattern. He gives recommendation models as another example. A query may be paired with thousands of items, and sending the query features repeatedly creates a large, duplicated payload between the CPU host and GPU host. The team instead sends one query with all the items, then packs and unpacks the representation near the GPU. This reduces the data transferred before the forward pass.
At Uber, models on A100 GPUs were reaching only 15% to 20% utilization. The team removed one variable by loading a data slice into RAM and running training without reads or writes during the test. Utilization rose to 85%, which showed that the model could use the GPU efficiently when data was immediately available. Increasing CPU threads, parallelism, and queue storage did not change the original result. That pushed the investigation toward profiling the data pipeline instead of changing the model.
Petastorm had separate producer and consumer bottlenecks
Kashish describes Petastorm as a producer-consumer pipeline. The producer reads data from a remote file system and places it in a queue. The consumer slices batches, converts them to tensors, and passes them to the GPU. Tracing showed that the queue was often empty because the producer could not read Parquet data fast enough. The team first tried caching the data locally on the GPU host so later epochs would avoid remote calls. The queue then stayed full, but utilization still did not improve, which showed that filling the queue was not the whole problem.
Caching transformed output fixed the hidden bottleneck
The missing detail was the data format. Reads from Parquet produced PyArrow data, while the GPU path needed NumPy arrays or dense tensors. When the team loaded everything into RAM for the diagnostic test, it was already in NumPy form. In the normal path, Petastorm converted PyArrow to NumPy on the fly, and that transformation consumed the remaining GPU headroom. Kashish says the fix was to cache the transformed output rather than only the raw data. That change raised utilization to 85% and reduced training time from a day to an hour or two using the same resources.
Kashish distinguishes offline training from online serving. Serving has to balance how quickly an inference result arrives against how efficiently the GPU is used. If there is not enough data for a full batch, the system can run a padded batch with fake data and keep latency low, but it wastes GPU capacity. Waiting for enough requests to form a fuller batch improves efficiency while adding latency. He says ads are highly latency-sensitive, and GPU inference can still cost less than CPU inference because GPUs handle larger batches with fewer instances, even when some capacity is wasted.
Large serving batches create another data problem. A batch of 1,000 rows with 200 features needs a large matrix before the model can run. The client can assemble and send that payload, which increases serialization and network cost, or the GPU endpoint can fetch features from systems such as Cassandra or Redis, which adds its own delay. Either side has to pay for gathering the data. Real-time features and current user embeddings are harder to cache than fixed offline data, so the GPU may remain idle while the full input matrix is assembled.
Increasing Petastorm parallelism improved utilization, but it also changed the order in which examples entered batches. Kashish says label skew then caused different runs on the same data to produce different model quality. The Michelangelo team replaced the single worker-driven queue arrangement with per-worker queues. They pre-allocated or predetermined the work, kept workers from starving, and had the worker queues fill the master queue in a deterministic way. This preserved the efficiency gains while making repeated runs produce consistent results. Adding more threads eventually stopped helping because the CPU imposed its own upper limit.
