AI infrastructure is becoming a physical engineering problem because power, cooling, building strength, manufacturing capacity, and skilled workers limit how quickly it can grow.
2
The short upgrade cycles of GPU hardware create waste because chip makers do not offer the same return, reuse, and financing systems that exist for products such as cars and iPhones.
3
Data centers could become more useful to their communities by supplying high-grade waste heat to municipal heating systems, while engineering leaders can reduce resource use through green software practices.
Summary
Erica Hughberg and Zachary Smith discuss what happens when AI demand meets the physical world. They define alternative clouds as the growing group of infrastructure services outside the hyperscalers, then move through the constraints behind AI expansion. GPU systems need far more power and cooling than older computers, racks can weigh thousands of pounds, and some buildings cannot support their floor loading. Manufacturing is concentrated among a small number of suppliers, while the industry lacks enough material scientists, electricians, and other experienced workers. Smith also criticizes the short hardware replacement cycle and the lack of effective recycling and reuse systems for data center equipment. He sees a possible benefit in using high-grade data center heat for municipal heating networks. He expects more cloud providers and new abstractions to widen access to infrastructure, and predicts that Apple could become a major inference provider through on-device AI. For engineering leaders, he recommends measuring energy and resource impact as part of software development.
Alternative clouds cover the infrastructure services outside the hyperscalers
Smith describes the cloud market as a range of services rather than only a few hyperscalers. An alternative cloud can be a database cloud such as Supabase, a front-end service such as Vercel, or a storage provider such as Wasabi or Backblaze. His definition includes infrastructure-associated services that operate as a service and can be consumed globally. He also places neoclouds, which focus on specific infrastructure, within this wider group. The same idea applies inside large companies when internal private clouds provide shared infrastructure to many teams. Hughberg connects this to the raised expectations created by services where developers can move from GCP to another provider and expect the same quick experience.
Inference turns AI growth into a networking and compute problem
Smith became interested in AI infrastructure after considering what inference requests mean for networks and computers. As requests become larger, encryption, decryption, and other processing consume more resources. Training is only one part of the system. Models also need to run inference for users, and that demand affects networking, memory, power, and data center design. Hughberg compares the old goal of loading a website in under 500 milliseconds with the current pressure to return an inference request within a similar time. Smith says the consequences extend beyond GPUs and TPUs because every part of the infrastructure has to move and process more data.
AI hardware is constrained by power, cooling, and the limits of semiconductor scaling
Smith says the easy gains from smaller transistors are running out. Manufacturers are adding and stacking more transistors, which increases power use and heat. A computer that once drew a few hundred watts can now draw thousands, and dense systems cannot simply be spread across a rack because the networking links between them need to stay close together. He gives the example of data center racks moving from roughly 4.5 to 5 kilowatts at Equinix to designs reaching 100 or 120 kilowatts. Those systems are difficult to power and cool, and their weight can reach 6,000 to 8,000 pounds per rack. Many buildings cannot support that floor loading.
Short hardware cycles leave the industry without a circular reuse system
Smith argues that chip makers earn money by selling new hardware, so they have little incentive to make equipment last longer or take it back. Data center equipment is hidden from its users, which makes the physical waste easier to ignore. He contrasts this with Apple, which he says can take apart old iPhones and recycle most of their materials into new products. He also points to Cisco's effort to recover and recycle old switches, partly to control the gray market for used equipment. Nvidia's upgrade cycle is much less mature: Smith says its current goal is an 18-month cycle in which the new generation makes the old one seem irrelevant, without a clear mechanism for returning or reusing the old hardware.
Concentrated manufacturing and missing skills can block infrastructure growth
The supply chain for AI hardware depends on specialized companies, with only a small number able to make advanced memory, fabrication processes, and other components. Smith describes this as a market with one or two suppliers in many parts of the chain and little spare capacity. New entrants would need expensive equipment and workers who already know how to build these systems. He compares this with the nuclear industry, where a long gap in plant construction reduced the number of people studying and teaching the required skills. Data centers face a similar labor problem in areas such as concrete, plumbing, electrical work, and material science. Smith argues that apprenticeship and long-term training programs are needed because critical physical systems cannot be staffed like ordinary software projects.
Data center heat could become a useful service to nearby communities
Liquid cooling can produce hotter heat than the air systems used in older data centers. Smith says that higher-grade heat could be used for homes or municipal heating systems instead of being discarded. He describes Scandinavian municipal heating loops as an example, where water moves through buildings and transfers heat into each property. A data center could make a long-term commitment to supply heat to a city, allowing the city to avoid building some new boilers or natural gas capacity. This would require agreements that guarantee heat over many years. Smith thinks the arrangement could change how communities view data centers, since a local facility would provide a visible service rather than only consume energy while sending its computing output elsewhere.
More cloud providers and simpler interfaces could widen access to infrastructure
Smith is encouraged by the growth of alternative cloud providers after years of infrastructure consolidation. He believes a more diverse set of companies and business models creates more choices and more room for experimentation. Anthropic is one example he gives of a provider that could offer a simpler abstraction than a traditional cloud. He says cloud-native infrastructure remains difficult, with Kubernetes requiring skills that only a small share of people have. New interfaces could let more people deploy useful systems without understanding every underlying component. He also expects multicloud to become a wider mix of specialized services, private infrastructure, and regional or sovereign clouds, which creates a need for open standards that let those systems work together.
On-device AI could make Apple a major inference provider
Smith calls Apple his dark-horse prediction. He thinks Apple could become the world's largest inference service provider within three years if on-device AI changes where computation happens. Some requests could run on an iPhone, Mac, or another local computer, while only tasks needing more power would move to a remote system. Apple has little inference market share today, in Smith's view, but a strong local computing experience could change that. The result would be a different balance between cloud and consumer hardware. Users might rely less on token budgets and remote services, although the electricity used by local devices would still matter.
Engineering leaders should measure resource use as part of software development
Smith recommends the Green Software Foundation, part of the Linux Foundation, as a starting point for engineering leaders. He says teams should understand the energy and resource impact of their software, then choose resources that fit the work rather than automatically scaling out in the cloud. The decision should account for more than the bill. He suggests incorporating these measurements and practices into the software development life cycle. His advice connects infrastructure choices with the physical systems behind them, including power, cooling, and hardware, instead of treating compute as an unlimited service.
"We can't even support the floor loading of those. They're running to 6 to 8,000 pounds of weight per rack."Zachary Smith00:00
Who should watch
You are choosing between hyperscalers, alternative clouds, private infrastructure, and specialized services, and need to understand the physical constraints behind those choices.
Your AI plans depend on GPUs, dense racks, new data center capacity, or faster inference, and you need to account for power, cooling, buildings, supply chains, and labor.
You lead an engineering team that wants to measure the energy and resource cost of software instead of treating cloud capacity as unlimited.
