# LLMs vs LMs in Production

Denys Linkov, Voiceflow | LLMs in Production 2023 | 24:44

Source: https://www.youtube.com/watch?v=y1HqPNwBp0U
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/llms-vs-lms-in-production
Published: 2023-08-21
Tags: build-vs-buy, latency, model-serving, testing

## TL;DR
- Voiceflow uses large language models for flexible, cross-domain generation, while smaller custom models handle fast, specific production tasks.
- Voiceflow chose API providers for large language model inference because self-hosting changes quickly and is outside its core business, while it hosts some language models itself.
- A custom NLU model beat GPT-4 on accuracy and cost in one test, with GPT-4 costing a thousand times more for inference.

## Summary
Denys Linkov compares Voiceflow's use of large language models with its smaller, task-specific language models. Voiceflow added large language models for assistant creation, runtime generation, prompt chaining, and knowledge-base responses. The company uses OpenAI and Anthropic APIs because self-hosting large models would require changing GPU infrastructure and is not central to its business. This creates problems with JSON formatting, prompt testing, cost, latency, and provider outages. Linkov describes an internal testing framework that records failed responses and turns them into regression tests. For smaller models, Voiceflow owns more of the training and serving stack because these models power core functions such as intent and entity detection. A Pub/Sub architecture worked for longer-running jobs but produced high p99 latency for real-time NLU inference, so the team moved queuing to Redis. The talk argues that model choice should follow the product requirement: flexible generation may justify an API, while a fast, narrow task may justify a custom model.

## Key ideas
### Voiceflow separates creation-time and runtime generative features
[01:33](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=93s)
Voiceflow added large language model features in two places. During creation, platform users can generate bot data while building an assistant. At runtime, an assistant can use a generate step or prompt chain to produce an additional response for the end user. The company also added an AI playground for trying different models and a knowledge-base feature that turns uploaded data into summarized answers. Linkov explains these examples to show that the model requirements differ depending on who uses the output and when it must be produced.

### A large language model is useful across many tasks and domains
[03:37](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=217s)
Linkov uses a practical definition rather than a parameter-count threshold. He asks whether BERT and T5 should count as large language models, then defines one as a general-purpose model that supports many tasks such as summarization and generation. That matters for Voiceflow because its platform supports different industries, including automotive, retail, and banking. The useful property is broad generation and understanding across domains, rather than simply having a particular architecture or number of parameters.

### Voiceflow uses APIs because large-model infrastructure changes too quickly
[05:29](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=329s)
Large models can require multiple GPUs at full precision, although quantization can allow some models to run on a single A100. Voiceflow decided not to build a fleet of this infrastructure because the hardware and model-serving techniques were changing quickly and self-hosting was not its core business. Its ML Gateway connects platform services to model endpoints, applies prompt validation, rate limiting, and usage tracking, and passes generative requests to OpenAI. Claude is connected through the same general approach.

### Generation introduces formatting failures that ordinary classifiers do not have
[07:56](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=476s)
Voiceflow's earlier encoder-based models returned classifications or embeddings. Large language models introduced a different failure mode because the service expected generated JSON and the models did not always produce clean output. The team used prompt engineering, regular expressions, and handwritten formatting rules. It recorded parsing errors, ran failed prompts and responses through additional prompts, and built a test suite for backtesting changes. Fine-tuning improved formatting in some experiments, but smaller fine-tuned models lost answer quality compared with larger models, so Voiceflow did not use that approach for the affected task.

### Production prompt design has direct cost and latency consequences
[12:51](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=771s)
Few-shot prompting can avoid fine-tuning, but adding examples increases prompt size and cost. Linkov says a 2,000-token prompt can become expensive for GPT-4, especially at higher production volume. Latency also varied much more for ChatGPT and its API than for Voiceflow's internal models, with unstable p99 values. In Voiceflow's tests, Azure ChatGPT was almost three times faster and had lower standard deviation than the standard service, although it cost more. Provider changes make it harder for a platform to control the customer experience.

### Voiceflow chooses hosting separately for each model
[14:49](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=889s)
The company does not use one hosting strategy for every model. It runs its own ML platform for fine-tuning, hosting, and inference, and it lets customers train some models in real time. It also uses OpenAI where a managed API makes more sense. Linkov presents model hosting as a business trade-off involving infrastructure ownership, training data, operational effort, and product value. A large model may fit a managed service better than an in-house deployment that requires the team to operate its own GPUs.

### A real-time NLU model forced a redesign of the serving platform
[20:34](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=1234s)
Voiceflow's NLU model detects intents and extracts entities, such as matching a request for a medium cheese pizza to an order-pizza intent and its attributes. An earlier Pub/Sub architecture worked for longer-running utterance recommendation jobs, with a 150-millisecond p50 SLA in each direction. It produced high p99 latency when used for real-time NLU inference, even though the model itself responded in 16 to 18 milliseconds. Voiceflow moved the queue to Redis and placed it closer to the application layer, which brought the system within its p50 and p99 targets.

### A narrow custom model can beat a general model on production economics
[22:43](https://www.youtube.com/watch?v=y1HqPNwBp0U&t=1363s)
Linkov compares the custom NLU model with GPT-4 on one test. He says the custom model had better accuracy and lower cost, while GPT-4 had higher latency and cost a thousand times more for inference. The test involved 3,000 inferences. His conclusion is specific to the task: GPT-4 is easy to use and capable, but a narrow model can make more sense when the product needs predictable, fast intent and entity detection at production volume.

## Notable quotes
- "You have to know what you're building and what problem you're solving before you just go deploy models left, right and center." (01:15)
- "We haven't invested into that LLM infrastructure because it's not core to our business." (06:53)
- "You don't actually have to choose the same hosting solution for all your models. You just need to make sure it makes business sense." (18:08)
- "Our NLU model still outperforms GPT-4 in this one test that we did both on cost and on accuracy." (22:43)

## Tools & references mentioned
- Voiceflow
- OpenAI
- Anthropic
- Claude
- GPT-3
- GPT-3.5
- GPT-4
- ChatGPT
- BERT
- T5
- Falcon
- Llama
- MMLU
- Azure ChatGPT
- Helicone
- Google Pub/Sub
- Redis

## Who should watch
- You are deciding whether to call a hosted large language model or deploy a smaller model inside your own platform.
- Your generation system returns malformed JSON, has changing latency, or needs a regression-testing process for prompts.
- You are evaluating whether a general model is worth its cost for a narrow, high-volume classification or NLU task.

## Related talks

- [Large Language Models in Production Round-table Conversation](https://mlopstalks.com/talks/large-language-models-in-production-round-table-conversation) (Diego Oppenheimer, Factory HQ & David Hershey, Unusual Ventures & Hannes Hapke, Digits & James Richards, Bountiful & Rebecca Qian, Facebook AI Research, 57:21)
- [Challenges in Providing LLMs as a Service](https://mlopstalks.com/talks/challenges-in-providing-llms-as-a-service) (Hemant Jain, Cohere AI, 11:43)
- [Efficiently Scaling and Deploying LLMs](https://mlopstalks.com/talks/efficiently-scaling-and-deploying-llms) (Hanlin Tang, MosaicML, 25:14)
- [Making LLM Inference Affordable](https://mlopstalks.com/talks/making-llm-inference-affordable) (Daniel Campos, Snowflake, 32:07)
- [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale, 34:57)