LLM inference cost and latency depend heavily on the ratio between a GPU's compute capacity and memory bandwidth.
2
For a 7B model, memory limits make practical decoding heavily memory-bandwidth bound, so reducing model and KV-cache size improves serving efficiency.
3
Open-source serving works well for small models, but Python overhead and incomplete batching optimisations still leave room for faster inference code.
Summary
Timothée Lacroix explains how to reason about serving an open-source language model by looking at throughput, per-token latency, and cost together. During decoding, each step requires computation proportional to the batch size while the model weights must be loaded from memory. The relationship between GPU FLOPs and memory bandwidth determines the batch size at which the hardware stops being memory-bound and becomes compute-bound. For a 7B model, GPU memory usually limits the batch size long before that point. Lacroix covers grouped-query attention, quantization, PagedAttention, sliding-window attention, continuous batching, CUDA graphs, and custom kernels as ways to improve serving. He argues that long prefill requests should be chunked so they do not delay decoding for other users. His quick benchmarks compare models and GPUs, and he recommends starting with the cheapest available hardware, then testing larger options against the actual workload. He also discusses open-source models, moderation systems, and multilingual training data in the questions.
Throughput, latency, and cost describe different parts of the serving problem
Lacroix uses queries per second for throughput, seconds per token for latency, and lower cost as the third objective. Throughput matters for batch jobs and for supporting more users. Per-token latency controls how responsive an application feels. He gives 250 words per minute as a useful reading-speed reference, saying users should not get bored when token latency stays below that level. The best operating point depends on whether the service is optimising user experience, total requests handled, or the price of serving them.
The compute-to-memory ratio determines the useful batch size
For a model with P parameters, one decoding step needs roughly two times P times the batch size in floating-point operations. The full model also has to be loaded from GPU memory for each step. Compute demand grows with batch size, while the model's memory movement stays roughly constant until very large batches. Their curves meet at a hardware-dependent batch size, which Lacroix calls B star. Below it, the GPU is memory-bound and its compute capacity is wasted. Above it, the GPU is compute-bound and latency rises. B star is therefore the point where latency is still low without leaving compute unused.
A 7B model usually cannot reach the hardware's ideal batch size
Using a 7B model with FP16 weights, Lacroix estimates about 14 GB for the model itself. The KV cache adds roughly 2 GB for every batch element when the maximum sequence length is 4K. An A10 with 24 GB can therefore support a maximum batch size of about five, while an A100 with 80 GB supports about 33. Both are far below the example ideal batch size of 400. This leaves practical 7B decoding severely memory-bandwidth bound. The size of both the weights and KV cache directly affects how many sequences can fit and how efficiently the GPU can run.
Attention and quantization reduce the memory pressure
Grouped-query attention associates one key-value pair with several queries. Mistral uses four queries per key-value pair, keeping the amount of computation similar while reducing KV-cache memory to one quarter of the standard arrangement. Quantization reduces model storage as well. INT8 halves the model size and INT4 divides it by four. Lacroix says the theoretical computation gain from INT4 is around 2x, though model shapes made about 1.5x more realistic in his experience. In the memory-bound regime, halving the model's load time can make decoding twice as fast. He reports little precision loss with INT8 and says some INT4 quality loss can be recovered with QLoRA for suitable uses.
PagedAttention prevents unused context space from filling GPU memory
A conventional KV cache reserves a rectangular region based on the maximum batch size and maximum sequence length. Short requests then occupy rows sized for the longest possible request, wasting memory. PagedAttention divides the available GPU memory into blocks, allocating only the blocks needed by each prompt and adding more as the sequence grows. A sequence does not need contiguous blocks. When decoding ends, its blocks can be released immediately. Lacroix says the Berkeley authors reported a 20x throughput increase over the standard implementation, and he considers that result plausible.
Sliding-window attention gives the cache a fixed size
Mistral trained its model to use only the past K tokens in the attention cache. Once a sequence exceeds the sliding-window length, new tokens overwrite old cache entries in a rotating buffer. This keeps the cache size fixed while still allowing a context length larger than the window. Lacroix says the implementation is straightforward because position information is encoded through positional embeddings, so the physical position in the rotating cache does not matter.
Prefill should be chunked so long prompts do not block decoding
The prefill phase processes many prompt tokens together, while decoding usually processes one new token per sequence. Lacroix says a 4K-token prompt can raise every other user's latency if the serving system processes it in one large operation. Chunking prefill into smaller groups would keep the workload closer to the useful operating regime. It would also make it easier to combine prompt processing with ongoing decoding, giving the scheduler finer control over GPU resources.
Serving code still leaves performance on the table
At these model sizes, Python overhead can be significant. Lacroix points to CUDA graphs as one way to remove that overhead and mentions the XFormers repository as an example. He also mentions NVIDIA's TensorRT-LLM as a system that traces inference and applies pattern matching, plus custom fused kernels that reduce memory traffic by doing operations while data is already loaded. FasterTransformer avoids some overhead but is harder to deploy. His summary is that hardware, device memory, model design, and serving code all shape the throughput-latency curve.
Hardware selection should start with cheap, direct benchmarks
Lacroix's quick comparisons show that changing the model's grouped-query attention can improve throughput, while moving from an A10 to an A100 can shift the cost and performance trade-off substantially. He recommends trying hardware in increasing order of cost and availability. A short benchmark on each candidate gives a workload-specific answer instead of relying on a general ranking. For a user who only wants to chat with a model, he says a MacBook may be cheaper and simpler. For roughly a million requests per day, he gives an A10 as a useful point at which dedicated GPU serving can make sense.
"My strategy since it's so easy to deploy everywhere is just to start with the cheapest and increase if I don't have the throughput or the speed I want."24:07
Who should watch
You are choosing GPUs or serving software for an open-source 7B model and need to compare latency, throughput, and cost on the same workload.
Your service has long prompts, variable sequence lengths, or many simultaneous users, and you want to understand why KV-cache allocation and batching affect performance.
You are building inference infrastructure and want practical ideas for reducing memory use and Python overhead before buying more hardware.
