Businesses can often replace a slow, expensive foundational-model API with a smaller model specialized for their own task.
2
Quantization, pruning, knowledge distillation, and pseudo-labeling reduce model size or improve the speed and cost of serving it.
3
At Neeva, combining pseudo-labeling, decoder pruning, and faster Transformers reduced web summarization from 8 seconds to 300 milliseconds for 10 documents on an A10.
Summary
Daniel Campos argues that many businesses do not need a large foundational model to perform every task. A smaller model trained for a narrow job can be cheaper, easier to scale, and less exposed to API outages and variable latency. He explains four ways to make that model practical: quantization, pruning, knowledge distillation, and pseudo-labeling. The most detailed example comes from Neeva's real-time web summarization system. GPT-3 and GPT-3.5 took about two seconds per response and had roughly a 10 percent failure rate. A T5 large model on an A10 reduced the time for 10 documents to eight seconds. After generating pseudo-labels, pruning the decoder, and using faster Transformers, the team reached 300 milliseconds for a batch of 10. This made offline summarization of the full index affordable. Campos also explains why A10 instances were easier to obtain and much cheaper than A100s for this workload.
A specialized smaller model can replace a broad foundation-model API for a narrow business task
Campos says businesses usually do not need their model to do everything that a foundational API can do. They often need a few specialized capabilities. A smaller model can approximate the behavior of a larger model while being cheaper to serve and easier to scale. He compares the large model to a movie star and the compressed model to a stunt double. The smaller model may have different characteristics, but it can be better suited to the workload. This also gives the business more control when an external API becomes slow or unavailable.
Quantization trades numerical precision for faster inference
Quantization changes a model from a higher-precision representation such as FP32 or FP16 to a lower-precision format such as INT8 or INT4. Campos describes post-training quantization, where the model is converted after training, and quantization-aware training, where the forward pass uses lower precision while the backward pass uses full precision. Lower precision can introduce rounding errors and cause cascading failures, so quantization-aware training lets the model learn to tolerate them. He says quantization generally produces a 2x to 4x speedup, although the result is often closer to 2x.
Pruning removes parts of a network that a task does not need
Pruning removes weights, activations, or larger network components. Unstructured pruning removes individual weights or activations, while structured pruning removes logical units such as layers or channels. Structured pruning usually gives more direct speedups because entire layers can be removed from a language or vision model. Unstructured pruning needs specialized software to turn the reduced model into faster runtime behavior. Campos mentions NVIDIA software and Neural Magic's DeepSparse inference engine as examples of tools for this type of optimization.
Knowledge distillation trains a small student to imitate a large teacher
Distillation uses a large, capable teacher model to supervise a smaller student model. The student is the model intended for deployment, while the teacher may be too expensive to ship. Instead of training only on the original labeled dataset, the student learns to emulate the teacher's behavior. Campos says this can improve the student by significant margins. The tradeoff is that training becomes slower because the teacher must remain in memory for a forward pass. The basic approach is relatively simple to implement.
Pseudo-labeling expands a small labeled dataset with outputs from an expensive model
Campos starts with roughly 100 to 10,000 labeled examples and trains a high-quality but expensive model such as FLAN-T5 11B or Falcon 40B. He then uses that model offline to label an unlabeled collection of about 10,000 to 1 million items. The resulting pseudo-labeled dataset can be much larger than the original data. A smaller model, such as T5-base, T5-small, or a BERT model, is trained on those generated labels and deployed in production. The extra examples teach the smaller model to approximate the larger model and can improve its quality.
Neeva combined several compression methods to make web summarization fast enough for offline processing
At Neeva, GPT-3 and GPT-3.5 took about two seconds per response for web summaries and had about a 10 percent failure rate. A T5-large model running on an A10 reduced the time for 10 documents to eight seconds. The team then combined pseudo-labeling with pruning and used faster Transformers, pruning only the decoder portion of the network. The result was 300 milliseconds for a batch of 10 on an A10, which Campos describes as nearly a 20x speedup. This let Neeva summarize its entire index offline for about $10,000 to $20,000, compared with an estimated $12 million to $13 million using a GPT model.
A10 instances fit this workload better than scarce A100 capacity
Campos explains that A100s are often purchased in groups of eight, and a resilient service may need at least two such groups. A10s can be obtained individually, including through spot pricing, so the team could scale out across many instances and recover when one disappeared. He says A10s were about three times slower than an A100 for the relevant workload but 10 to 15 times cheaper and broadly available. With NVIDIA's faster Transformers libraries, T5 models could achieve speedups of up to 20x.
The same compression ideas apply beyond language models
Campos says pruning and quantization have an especially long history in computer vision, where researchers have worked on channel and filter pruning for deep networks. Speech models often resemble language sequence-to-sequence models, so similar methods apply there. He says a sequence-to-sequence model can sometimes keep a deep encoder and use a shallow decoder without much performance loss. Since the decoder runs repeatedly during generation, reducing it can produce a large speed improvement.
"We could summarize our entire index for about ten or twenty thousand dollars versus if we had run summaries in our entire index with a GPT model it would have been on the order about 12 or 13 million dollars."09:15
Who should watch
You are deciding whether to self-host a model instead of accepting slow or unreliable API calls.
Your team has a small labeled dataset and needs a practical way to train a cheaper model for one task.
You need to reduce inference cost and want an example that combines pseudo-labeling, pruning, model selection, and GPU economics.
