Voice systems need separate latency and throughput measurements because improving one can hurt the other.
2
Connection pooling can reduce TTS latency by avoiding repeated connection handshakes and cleanup.
3
Whisper and Orpheus can be sped up with WebSockets, C++ execution, dynamic batching, compilation, quantization, and removal of initial silence.
Summary
Madison Kanna and her Baseten colleague explain how to run speech-to-text and text-to-speech models with lower latency and higher concurrency. They separate latency measures such as time to first token, end-to-end response time, and token generation speed from throughput measures such as concurrent sessions per GPU, requests per second, queue depth, and wait time. The talk covers connection pooling, model engines, KV cache routing, and caching model weights in containers. For Whisper, the speakers recommend WebSockets for streaming transcriptions, a C++ backend for real multithreading, and dynamic batching. For Orpheus, they discuss compiling both model components, quantizing weights and the KV cache to FP8, and removing roughly 600 milliseconds of initial silence from the base model. In the questions, they describe the infrastructure burden of running these systems on AWS or Azure and explain when an inference provider can absorb that work.
Voice performance has separate latency and throughput measures
The speakers divide performance into latency and throughput. Latency includes time to first token, end-to-end response time, and token generation speed. Throughput includes concurrent sessions per GPU, requests per second, queue depth, and wait time. They stress that the two goals can conflict. Increasing batch size can improve throughput while making an individual request slower, so production systems need to choose the trade-off deliberately.
Connection pooling is presented as one of the easiest improvements. The speakers say they have seen TTS latency fall from above 500 milliseconds to below 300 or even 250 milliseconds after applying it. Instead of opening and closing a connection for every request, an asynchronous client keeps a session and reuses established connections. This avoids repeated handshakes and cleanup, which become more expensive as traffic and concurrency increase.
A model engine supplies batching and quantization features
The speakers recommend using an inference engine for transformer models, with TensorRT-LLM as their preferred choice. They describe it as a framework that supports features such as in-flight batching, quantization, and speculative decoding. These features apply to the Whisper and Orpheus components discussed later. The point is to use an engine that can coordinate model execution instead of relying only on the basic model implementation.
Cache routing and weight caching reduce scale-up delays
KV cache routing sends a request to a replica that already contains the relevant cache, which can keep latency low as more replicas are added. The speakers also discuss copying cache data when a new replica starts, while noting that this prefill takes time. Weight caching addresses cold starts by putting model weights inside the container rather than downloading them separately from Hugging Face. This reduces the time required to wake a sleeping model.
Whisper benefits from streaming, C++, and dynamic batching
For Whisper, Baseten implemented a WebSockets version that can push partial transcriptions while someone is speaking. The speakers report less than 50 milliseconds for each generated audio chunk. They also moved from a Python implementation to a C++ framework to avoid Python's Global Interpreter Lock and gain true multithreading, with an 18 percent speed increase. Dynamic batching sends requests continuously instead of waiting for a full batch, reducing the time before Whisper begins responding.
Orpheus can be optimized across both of its model components
The speakers describe Orpheus as having a Llama model and a SNAC codec model. They compile both models for the specific GPU kernels in use. They also quantize the models and their KV cache to FP8, which reduces GPU memory use and allows more requests to run concurrently. This can improve throughput and reduce queueing. The approach depends on the model architecture and the execution engine available.
The base Orpheus model has about 600 milliseconds of silence at the beginning as a result of its training. The speakers recommend cutting that silence out. They distinguish time to first audio byte from time to first text byte and say that removing the silence improves end-to-end latency and makes the interaction feel smoother.
Infrastructure optimization is harder than model optimization
In the question period, the speakers say model-side changes are easier to adopt through tools such as inference engines, while infrastructure and networking require more specialized knowledge. Running the system directly on AWS or Azure can involve a load balancer, EC2, SageMaker, and other services. Simply adding a larger GPU or more replicas may work, but cost limits that approach, especially for startups and hobbyists. An inference provider can handle more of the scaling work.
