GPU clouds must manage unreliable, heterogeneous hardware while giving users an experience closer to a public cloud.
2
Inventory management is the foundation for allocating GPU capacity, building tenant networks, and handling changing data center infrastructure.
3
GPU providers need to offer a range of services, from bare metal and notebooks to token-based inference and automated fine-tuning.
Summary
Mohan Atreya explains why GPU infrastructure is harder to provide than ordinary cloud compute. Customers often need a specific GPU, but availability, pricing, long commitments, hardware failures, networking, thermals, storage, and upgrades make access difficult. New GPU clouds are trying to serve enterprises, universities, and other customers that cannot or do not want to build this infrastructure themselves. Their operational problem starts with basic inventory: many providers still track servers, GPUs, switches, memory, and policies in spreadsheets. From there, they need automated provisioning, tenant isolation, elastic networking, storage access, and user-facing services such as notebooks and data science environments. The conversation also covers token clouds, which let providers monetize spare or older GPUs through inference APIs. Mohan expects users to choose among bare metal, managed environments, tokens, and higher-level workflows such as zero-code fine-tuning. The practical argument is that GPU providers need to support this full range because their customers have different technical skills, budgets, data constraints, and workloads.
GPU access is difficult because experimentation needs flexibility before commitment
Mohan says customers often need an exact GPU type, yet the hardware may be unavailable or too expensive. AI and machine learning work is experimental, so teams may need to try a setup before they know whether it will work. Long-term commitments and expensive reserved pricing make that harder. General-purpose compute does not usually create the same problem. Customers can end up stuck when a large cloud provider lacks the required capacity or only supports a standard offering. The uncertainty is especially painful when a team has a limited window to run experiments and may learn that its original hypothesis was wrong.
New GPU clouds carry the cost of hardware before they have customers
The newer GPU clouds have to find real estate, data center power, and GPUs before they can offer a service. The hardware is power-hungry, and idle capacity continues to cost money. Mohan says many providers have hardware and data center skills but need help turning a collection of GPUs into an AWS-like customer experience. He contrasts newer providers with CoreWeave, whose public filings showed that a large share of revenue came from Microsoft. A smaller provider may instead sell to enterprises or universities. One example is a university that wants to launch an AI and machine learning lab and needs notebooks, Ray, and Kubeflow backed by GPUs.
GPU failures affect training runs because large systems have many connected parts
Mohan describes multi-GPU systems as complex connected environments rather than ordinary computers. He cites a Meta Llama paper with a reported failure rate of about 30 percent. A failure during a training run can force a team to abandon work that has already run for weeks unless the provider can replace capacity underneath the workload. The problem includes GPU failures, switches, inter-node connectivity, thermals, SSDs, memory, and updates. A service needs to detect failures, reroute connections, and preserve the training job. The user still expects a working service-level agreement even though the infrastructure underneath is changing.
GPU clouds need a live inventory before they can provision capacity reliably
Mohan says some GPU providers do not have a precise view of their own infrastructure. Inventory includes the server holding a GPU, its memory, network connections, MAC address, firewall policies, and the teams managing each part. Some providers track these details in Google Sheets or Excel, which helps explain why provisioning a negotiated block of GPUs can take weeks. A live source of truth can allocate capacity to tenants as infrastructure changes. It also needs to create a tenant access network, connect high-speed GPU traffic and storage, and expand or shrink allocations when customers change their requirements.
Terraform can automate requests but cannot decide from current GPU inventory
Mohan says Terraform is one interface that users can use for automation, but Terraform does not understand the provider's current inventory. It does not know how many switches exist or which kinds of automation each underlying technology supports. A GPU cloud may have equipment from many network, storage, and infrastructure vendors. The provider therefore needs a unified interface that translates a higher-level request, such as asking for 100 GPUs, into the different network and storage operations required underneath. The same variety that helps providers control costs also creates the integration problem.
GPU users need managed environments because bare metal alone leaves too much work
A data scientist cannot always be given a bare-metal server and left to assemble the rest of the environment. Mohan compares the choices to pizza. One user may want raw ingredients and full control, while another wants a ready-made meal. In infrastructure terms, that means some users want bare metal, while others want a notebook or a more managed service. Public clouds make this experience simple by letting someone click a button to get a Jupyter notebook, GPUs, and pipeline tools. GPU clouds and enterprise platforms need to offer comparable choices, especially when regulation or data location prevents a team from moving everything to a public cloud.
Token clouds give providers a way to monetize spare and aging GPUs
Mohan describes a possible shift toward token-based GPU clouds for generative AI workloads. Instead of renting a dedicated inference endpoint, a customer can request an API endpoint and pay for the tokens consumed. The provider can use spare GPU capacity for serverless inference rather than leaving it idle. This also creates a use for older hardware as demand moves to newer GPU generations. Mohan says GPU systems may have a 12 to 18 month period of high interest, so token services can extend the revenue life of equipment. The customer buys access to model output, while the provider sells a different service on the same underlying GPUs.
Higher-level services can let non-developers fine-tune models without managing infrastructure
Mohan connects token clouds to a broader group of users he calls citizen scientists. A finance team might have years of proprietary documents but no developer who understands GPUs, quantization, or model settings. A higher-level service could let the user provide data, choose a model, click fine-tune, and receive an endpoint or chatbot for testing. He calls this zero-code fine-tuning. The user does not need to know whether the workflow used GPUs or tokens. Mohan is careful about the limits: fine-tuning may produce a worse result if done badly, and its long-term role is still unsettled. Providers need to offer workflows alongside lower-level infrastructure.
