Voiceflow uses large language models for flexible, cross-domain generation, while smaller custom models handle fast, specific production tasks.
2
Voiceflow chose API providers for large language model inference because self-hosting changes quickly and is outside its core business, while it hosts some language models itself.
3
A custom NLU model beat GPT-4 on accuracy and cost in one test, with GPT-4 costing a thousand times more for inference.
Summary
Denys Linkov compares Voiceflow's use of large language models with its smaller, task-specific language models. Voiceflow added large language models for assistant creation, runtime generation, prompt chaining, and knowledge-base responses. The company uses OpenAI and Anthropic APIs because self-hosting large models would require changing GPU infrastructure and is not central to its business. This creates problems with JSON formatting, prompt testing, cost, latency, and provider outages. Linkov describes an internal testing framework that records failed responses and turns them into regression tests. For smaller models, Voiceflow owns more of the training and serving stack because these models power core functions such as intent and entity detection. A Pub/Sub architecture worked for longer-running jobs but produced high p99 latency for real-time NLU inference, so the team moved queuing to Redis. The talk argues that model choice should follow the product requirement: flexible generation may justify an API, while a fast, narrow task may justify a custom model.
Voiceflow separates creation-time and runtime generative features
Voiceflow added large language model features in two places. During creation, platform users can generate bot data while building an assistant. At runtime, an assistant can use a generate step or prompt chain to produce an additional response for the end user. The company also added an AI playground for trying different models and a knowledge-base feature that turns uploaded data into summarized answers. Linkov explains these examples to show that the model requirements differ depending on who uses the output and when it must be produced.
A large language model is useful across many tasks and domains
Linkov uses a practical definition rather than a parameter-count threshold. He asks whether BERT and T5 should count as large language models, then defines one as a general-purpose model that supports many tasks such as summarization and generation. That matters for Voiceflow because its platform supports different industries, including automotive, retail, and banking. The useful property is broad generation and understanding across domains, rather than simply having a particular architecture or number of parameters.
Voiceflow uses APIs because large-model infrastructure changes too quickly
Large models can require multiple GPUs at full precision, although quantization can allow some models to run on a single A100. Voiceflow decided not to build a fleet of this infrastructure because the hardware and model-serving techniques were changing quickly and self-hosting was not its core business. Its ML Gateway connects platform services to model endpoints, applies prompt validation, rate limiting, and usage tracking, and passes generative requests to OpenAI. Claude is connected through the same general approach.
Generation introduces formatting failures that ordinary classifiers do not have
Voiceflow's earlier encoder-based models returned classifications or embeddings. Large language models introduced a different failure mode because the service expected generated JSON and the models did not always produce clean output. The team used prompt engineering, regular expressions, and handwritten formatting rules. It recorded parsing errors, ran failed prompts and responses through additional prompts, and built a test suite for backtesting changes. Fine-tuning improved formatting in some experiments, but smaller fine-tuned models lost answer quality compared with larger models, so Voiceflow did not use that approach for the affected task.
Production prompt design has direct cost and latency consequences
Few-shot prompting can avoid fine-tuning, but adding examples increases prompt size and cost. Linkov says a 2,000-token prompt can become expensive for GPT-4, especially at higher production volume. Latency also varied much more for ChatGPT and its API than for Voiceflow's internal models, with unstable p99 values. In Voiceflow's tests, Azure ChatGPT was almost three times faster and had lower standard deviation than the standard service, although it cost more. Provider changes make it harder for a platform to control the customer experience.
Voiceflow chooses hosting separately for each model
The company does not use one hosting strategy for every model. It runs its own ML platform for fine-tuning, hosting, and inference, and it lets customers train some models in real time. It also uses OpenAI where a managed API makes more sense. Linkov presents model hosting as a business trade-off involving infrastructure ownership, training data, operational effort, and product value. A large model may fit a managed service better than an in-house deployment that requires the team to operate its own GPUs.
A real-time NLU model forced a redesign of the serving platform
Voiceflow's NLU model detects intents and extracts entities, such as matching a request for a medium cheese pizza to an order-pizza intent and its attributes. An earlier Pub/Sub architecture worked for longer-running utterance recommendation jobs, with a 150-millisecond p50 SLA in each direction. It produced high p99 latency when used for real-time NLU inference, even though the model itself responded in 16 to 18 milliseconds. Voiceflow moved the queue to Redis and placed it closer to the application layer, which brought the system within its p50 and p99 targets.
A narrow custom model can beat a general model on production economics
Linkov compares the custom NLU model with GPT-4 on one test. He says the custom model had better accuracy and lower cost, while GPT-4 had higher latency and cost a thousand times more for inference. The test involved 3,000 inferences. His conclusion is specific to the task: GPT-4 is easy to use and capable, but a narrow model can make more sense when the product needs predictable, fast intent and entity detection at production volume.
