Machine learning repeatedly outgrows the systems built for it, from small research experiments to global maps, real-time services, and large language models.
2
Most companies should use an existing machine learning platform unless machine learning is their core business, because building and maintaining flexible infrastructure requires a large specialist team.
3
Large language models create serious cost, deployment, governance, and coordination problems, while open-source models and human-machine workflows may make them more practical over time.
Summary
Waleed Kadous describes scalability as a problem that has followed machine learning throughout his career. He recalls distributing PhD experiments across 17 machines, scaling mapping algorithms from small areas to the United States at Google, and handling real-time data and hundreds of models at Uber. He argues that most companies should buy or adopt existing infrastructure rather than build a full ML platform, unless machine learning is their core business. Systems need to let developers move from local experiments to distributed training and serving without abandoning normal Python libraries. The discussion then turns to large language models. Waleed points to their high API costs, large memory requirements, multi-GPU coordination, human feedback needs, and governance issues. He expects high-value and batch use cases to appear first, with open-source models helping companies fine-tune and control their systems. He also explains how Ray and Anyscale address distributed workloads.
Machine learning has repeatedly moved from small experiments into much larger systems
Waleed traces the same scaling problem across different stages of his career. During his PhD, hyperparameter experiments would have taken six months, so he connected 17 machines and distributed the work across them. At Google, a mapping algorithm that worked on a small area had to process the United States. At Uber, machine learning handled location quality, imagery, accident detection, and ETA prediction. The systems used around 700 city-specific ETA models, while Uber processed two to three million drivers across more than 15 million trips per day.
Most companies should adopt existing infrastructure instead of building a full ML platform
Waleed says building a world-class ML platform is expensive because it needs separate data science, data engineering, and ML infrastructure roles. He gives Uber as an example, with hundreds of people across those groups. Unless machine learning is the company's core business, he thinks most teams should start with an off-the-shelf system. The difficult choice is finding one that combines flexibility with scale. A platform that only runs a model on a larger machine eventually reaches limits, especially for serving workloads that cannot justify keeping an expensive machine running continuously.
Infrastructure should let developers move from local code to distributed workloads
Waleed recommends systems that let developers move quickly from development to scalable serving while continuing to use familiar Python libraries. He contrasts flexible programming interfaces with fixed-function pipelines that assume every workflow has the same monitoring and training stages. Semi-supervised learning and closed-loop systems become difficult when the platform cannot support unusual control flow. He points to PyTorch's FSDP and DDP as examples of libraries moving toward scalability, driven partly by large language models that require distributed infrastructure from the start.
Large language models make cost and multi-GPU coordination practical constraints
Waleed warns that large language models are expensive to use and difficult to run yourself. He cites a price of up to 12 cents per 1,000 tokens for a fine-tuned DaVinci model, with 1,000 tokens being about 700 words. A 175-billion-parameter model requires about 350 gigabytes, while an A100 has 80 gigabytes, so serving it requires at least four GPUs in the example he gives. Training or serving across many GPUs also requires software to coordinate the machines. Fine-tuning can reduce model size in some cases, but Waleed says many of those techniques remain between research and practical deployment.
Early LLM applications will favor high-value or batch workloads
Waleed expects prices to fall through better infrastructure, model compression, and related research, but he says useful techniques often take two to three years to become reliable. He compares the likely adoption pattern with speech recognition, where medical dictation and legal work provided valuable early markets. Batch workloads may arrive sooner because they allow teams to trade latency for throughput. He also says applications do not need to use a language model wholesale. Embeddings can support document search, and systems such as LangChain can chain several processing stages together.
Open-source models matter because companies need control over data and fine-tuning
Waleed expects strong pressure toward open-source language models because companies may not want to send internal customer-service data to an external provider. He names BLOOM and OPT-175B as examples, and mentions DeepSpeed as infrastructure for training models. He also distinguishes ChatGPT from a basic supervised learning system because reinforcement learning from human feedback is part of its behavior. Smaller companies may struggle to reproduce that process because it needs large amounts of human review. Open-source datasets and tooling could make this more accessible, but Waleed expects the work to take time.
Ray combines a Python interface, cluster scheduling, and scalable ML libraries
Waleed describes Ray as a distributed computing system with a modern annotation-based Python interface. A scheduler distributes Ray workloads across a cluster, and an ecosystem of libraries supports reinforcement learning, hyperparameter optimization, and deep learning at larger scales. He gives several examples: creating 10,000 product models as parallel tasks, running NLP over 50 million documents, and backtesting over 50 years of data. Anyscale provides a hosted version with workspaces, security, cost management, and other features intended for companies that do not have their own ML infrastructure team.
More data usually improves accuracy, while human input offers another expensive route
Waleed is skeptical of the claim that big data is disappearing. He says that, over the last 30 years, adding more data has often produced better models, especially when higher accuracy has economic value. Teams can instead add human intelligence through active learning, semi-supervised learning, labeling functions, and machine teaching, but those methods also cost money. He describes the direction he wants as a 'cyborg model', where people and machines work together. He remains interested in neurosymbolic systems that combine learned models with explicit rules, but does not expect that problem to be solved within five years.
"It's going to be a scalability problem again ever again, you know, it's just kind of like this recurring loop of like we get more computational power and then we use it to do even crazier stuff which needs more computational power."Waleed Kadous12:41
Who should watch
You are choosing between buying ML infrastructure and building it internally, and need a realistic view of the people and maintenance work involved.
Your models fit on a laptop or one virtual machine today, but you expect more customers, more data, or distributed training soon.
You are building on language-model APIs and need to think through inference cost, GPU coordination, human feedback, data control, or batch processing.
