Hanlin Tang argues that many companies will use small, specialized language models alongside external APIs, rather than relying on one general-purpose model.
2
Private training data gives companies control over data provenance, model behavior, model ownership, and deployment costs.
3
Open-source tooling has made training and deploying smaller language models more accessible, with fully sharded data parallelism and automated recovery reducing infrastructure work.
Summary
Hanlin Tang argues that the future of large language models will include many small, specialized models owned by individual companies. Companies may still use external APIs, but private models can protect sensitive data, give teams control over training data and model weights, and reduce inference costs for focused tasks. He points to biomedical, financial, legal, genomics, and healthcare use cases where domain-specific data can make smaller models useful. Tang also challenges the idea that training these models is always prohibitively expensive or technically difficult. He describes models in the one to seven billion parameter range as practical for many business applications, and discusses tools such as PyTorch fully sharded data parallel and MosaicML Composer. Automated handling of memory errors, node failures, checkpoint recovery, and data streaming can remove much of the operational burden. His recommendation is to start with a smaller model, validate its business value, then scale up if needed.
The model ecosystem is moving toward many specialized models
Hanlin Tang says the early assumption was that every use case would depend on one giant AGI model controlled by a small number of companies. He sees the ecosystem moving toward many smaller models for specific tasks, owned by many companies. Businesses will still use external APIs for some applications, while training their own models when private data or specialized behavior matters. This would put language-model capabilities inside individual companies rather than concentrating them entirely with API providers.
Private training gives companies control over data and model behavior
Tang gives data privacy as one reason to train internally, since companies may not want to send private data or core intellectual property to an external service. He also focuses on data provenance. A financial model may need to exclude sources such as Reddit's WallStreetBets channel, while a healthcare model may need carefully selected medical data. Controlling the training data can reduce the need for downstream modifications used to align model outputs. Companies may also want their own weights for portability, introspection, and explainability.
Smaller domain models can be cheaper to run and still perform well
Tang argues that many applications do not need a 175-billion-parameter model. A focused task may benefit more from a smaller model trained on relevant data. He cites BioMedLM, trained mainly on PubMed by Stanford University with a team of two or three machine-learning engineers. The three-billion-parameter model reached state-of-the-art accuracy at release time on the MedQA USMLE benchmark and performed similarly to the much larger Galactica model on the example he discusses. He also mentions applications such as loan-document classification, summarization, code generation, genomics, and electronic health records.
Continued pretraining can combine public and proprietary data
Tang describes Bloomberg's 50-billion-parameter model as an example of continued pretraining. Bloomberg combined web data with internal Bloomberg data instead of treating model development as only a choice between training from scratch and fine-tuning. Tang says the resulting model outperformed existing open-source models on a broad set of financial tasks. The example supports his argument that proprietary business data can improve performance on company-specific work.
Training costs are lower when the model matches the use case
Tang addresses the belief that useful language models always cost millions of dollars to train. He says a GPT-3-quality model can be trained for about half a million dollars with current tooling, while a model costing around thirty thousand dollars may already be sufficient for some business applications. He presents one- to seven-billion-parameter models as a practical starting point. The economic case depends on choosing a model size that fits the task instead of automatically pursuing the largest available model.
Fully sharded data parallelism makes large-model training fit across GPUs
Tang explains that large models cannot fit into the memory of one GPU. PyTorch's fully sharded data parallel approach splits the model and optimizer across GPUs, then fetches the needed parts during training. This saves memory without requiring more exotic parallelism strategies. Tang describes combining it with the MosaicML Composer library and its optimization work. The approach is intended to make distributed training more flexible and easier to use.
Training systems need to recover from ordinary hardware failures
At large scale, Tang says nodes can fail every few days, and training logs from a Meta OPT run show the operational cost of those failures. A restart can leave a cluster idle for about 50 minutes while the model reloads and the data loader catches up. He also mentions lost spikes, out-of-memory errors, accidental cluster deletion, and the difficulty of monitoring many nodes. Systems can now detect bad nodes, restart runs, resume from checkpoints, and adjust memory use dynamically, reducing the need for constant manual supervision.
Teams can scale from familiar models into larger deployments
Tang describes a gradual path for enterprises that have already trained or deployed models such as BERT. They can use available tooling to cross the multi-node barrier, train a smaller model, deploy it, and check whether it produces business value. If it does, they can move toward larger models or more accurate systems. The full training stack now includes experiment tracking, data streaming, distributed training frameworks, deep-learning libraries, deployment, orchestration, and device tooling.
A practical starting point is a private model with a narrow purpose
Tang closes by encouraging teams not to be intimidated by large language models. He says training seven-billion-parameter models, and smaller models, on private data is now cost-effective and straightforward enough to attempt. Existing experience with BERT or computer-vision models can transfer to this work because open-source models and training tools are available. His expectation is that models dedicated to areas such as law, medicine, and individual businesses will continue to appear as this tooling spreads.
"The world doesn't actually look like one AGI model. It looks like many small specialized models that are used to solve very specific use cases owned by many companies."00:46
Who should watch
You are deciding whether a private language model is justified by data privacy, regulatory, or model-ownership requirements.
Your team has deployed smaller machine-learning models and wants a practical path toward distributed LLM training.
You need to understand the infrastructure work behind multi-GPU training, checkpoint recovery, memory management, and production deployment.
