Moving a model in-house can reduce latency, avoid API outages and rate limits, and give a team control over hardware and serving.
2
Teams can reduce cost by choosing a smaller model, pruning and distilling it, selecting suitable hardware, and reducing unnecessary LLM calls.
3
LLM monitoring should focus on bad production experiences, including user feedback, long inputs, tail latency, and evaluation methods that avoid comparison bias.
Summary
The panel discusses the cost and performance decisions involved in putting large language models into production. Daniel Campos describes Neeva's move from foundation-model APIs to smaller in-house models as usage grew, especially for processing a web index. He reports large latency gains from local serving, model compression, better kernels, and hardware selection. Luis Ceze argues that teams should match the model and hardware to their actual latency and throughput needs rather than defaulting to expensive accelerators. Mario Kostelac recommends starting with the best available model to test feasibility, then reducing calls, using simple classifiers, and moving to cheaper models when the use case is understood. Jared Zoneraich discusses production monitoring through user behavior, human labeling, and LLM-based critics. The speakers also describe practical latency tactics such as streaming output, controlling response length, batching carefully, and investigating outliers caused by unusually long inputs.
Large-scale workloads can make API use financially impossible
Daniel Campos says foundation-model APIs were useful for testing whether a product and ranking signal worked, but cost and scale became bottlenecks. Real-time summaries may be feasible through an API, while summarizing billions of documents in a web index is not. Neeva therefore moved toward a smaller system that could run in-house. The decision depends on how many users need access, the required turnaround time, and the volume of data. Campos also says local models gave Neeva much faster responses and avoided dependence on an external provider's availability.
A smaller model and suitable hardware can change the cost calculation
Daniel Campos describes a practical target of fitting a model on one A100, one A10, or a CPU. He contrasts expensive multi-GPU deployments with cheaper A10 spot instances, then says some smaller query encoders can run directly on CPU alongside retrieval. Luis Ceze agrees that many teams do not need high-end silicon. His recommendation is to choose a model for the task, optimize it, and then select hardware that meets the latency and throughput requirement. OctoML is presented as a way to automate model optimization and hardware selection.
Pruning and distillation can make a large model fit smaller hardware
Daniel Campos names structured pruning and knowledge distillation as two effective techniques. He says a 11-billion-parameter Flan-T5 model did not fit on an A10 until at least 60 percent of its weights were removed. Distillation can recover some of the lost quality by training a smaller student model from a larger teacher. Campos recalls seeing 30x, 40x, and 50x improvements on text classification tests when distillation was pushed far. The trade-off is that moving this work in-house slows iteration from minutes with prompts to days or weeks for data, compression, and deployment changes.
Teams should reduce calls before they build a model platform
Mario Kostelac says Intercom first tests whether a capability is possible with the strongest available model, then tries to find the cheapest model that produces acceptable results. One basic cost control is simply to call an LLM less often. A simple classifier can sit before an expensive model and decide whether the request needs a complex answer at all. Jared Zoneraich adds that teams should first understand which prompts and system components are driving spend. Ceze also points out that an open-source model may be far cheaper when it can handle a specialized task.
The right deployment choice depends on scale and staffing
Daniel Campos describes a case where a company could have spent roughly $500,000 to $600,000 per year on an NLP team and infrastructure for a business classifier. Its actual workload was about 10,000 examples, so using a general model and tuning prompts was cheaper than hiring a specialist team. The panel contrasts this with search and customer-support workloads, where high request volume changes the calculation. Campos says Neeva had about four people working directly on bringing models in-house, with about ten people working with models overall.
Serving and batching choices can cut latency without changing the model
Campos gives a concrete serving example. A T5-large model initially took about eight seconds for a batch of ten on an A10. Switching from native PyTorch serving to Nvidia's Faster Transformers reduced that to about 1.8 seconds. Asymmetric pruning then reduced the model from 24 encoder and 24 decoder layers to 24 encoder layers and four decoder layers, bringing the batch latency to roughly 300 milliseconds. The panel also discusses kernel and compiler choice, batch size, and model-specific hardware optimization. Large batches can waste work when one long output determines the duration for every item.
User experience can hide some latency, while APIs leave fewer options
Mario Kostelac says teams using an external API have limited control. They can use shared capacity when available, avoid calls when a quick classifier can answer, shorten the requested output, and stream tokens to the user. Luis Ceze says showing that the system is typing can improve perceived responsiveness without changing actual latency. The speakers warn that streaming can complicate safety checks. They also note that API latency varies over time, so a service that matters to users should measure those spikes instead of treating latency as a fixed property.
Production evaluation should start with failures and bad user experiences
Jared Zoneraich describes three evaluation approaches: direct user signals such as thumbs up or thumbs down, human labeling through services such as Mechanical Turk, and synthetic evaluation by another LLM. He says Prompt Layer's focus is to find what is failing in production, such as rude chatbot responses, instead of only asking which prompt has the highest score. Daniel Campos says GPT-4 was close to human judges on several evaluation dimensions, but warns about positional bias when comparing two outputs. Both speakers recommend examining outliers, including long documents that create tail-latency spikes, because the fix is often simple, such as truncating an oversized input.
"The first step is just knowing what's going on and kind of just building version one and kind of see just understanding how the cost works."Daniel Campos14:56
Who should watch
You are deciding whether to keep using hosted LLM APIs or move a workload to smaller models that your team runs itself.
Your inference bill or response time is rising, and you need concrete options involving pruning, distillation, batching, hardware, and serving libraries.
You are building evaluation for an LLM application and need to connect user feedback and production failures to model and prompt changes.
