Matt Squire argues that teams should benchmark latency, throughput, request rate, software versions, environments, and data before optimizing an LLM application.
2
Vaibhav Gupta shows how BAML parses less constrained model output into reliable structured data, which can reduce token use while improving accuracy across models.
3
Tom Sabo combines information extraction with LLMs to filter large collections of text before summarization, then uses extracted terms and source statements to inspect the results.
Summary
This session contains three talks about making LLM systems more useful in production. Matt Squire presents a self-hosted documentation assistant using RAG, guardrails, Mistral 7B Instruct, and a GPU-backed model server. He explains how Locust benchmarking exposed bottlenecks, how vLLM improved inference, and how Ray Serve supported horizontal scaling, autoscaling, and GPU sharing. Vaibhav Gupta demonstrates BAML, a programming language and toolchain for structured outputs. BAML uses local parsing and generated algorithms to turn compact, sometimes invalid-looking model output into application data, while supporting tests, hot reload, multimodal inputs, and several programming languages. Tom Sabo describes a complementary approach based on information extraction. In a public-comment analysis example, SAS rules filtered relevant statements before an LLM summarized them. Visualizations then linked claims back to terms, organizations, and source statements. The session is practical and candid about tradeoffs, especially for self-hosted systems and smaller models.
Scaling means improving user experience under concurrency while controlling GPU cost
Matt Squire defines scaling around the experience of the person asking a question. Users want a fast response, while the system must support hundreds or thousands of concurrent sessions. The business also wants to minimize the number of GPUs needed to provide that response time. His example includes a Python orchestration service, a vector database for RAG, guardrails that check whether questions are on topic, and a model server running Mistral 7B Instruct on AWS. He stresses that the system has several components to scale, including guardrails and vector computation, rather than only one language model.
The team used Locust to simulate different traffic patterns before changing the system. Their scenarios covered development testing, an estimated typical day with 20 concurrent users, a 50 percent increase, a larger spike, and the point where the system failed. The initial system failed at five users, which made the breaking point useful to study. They measured total latency, output tokens per second, and successfully served requests per unit of time. They also recorded the test, software release, environment, Git commit, and dataset version so later experiments could be compared with the original baseline.
vLLM improved inference without requiring more GPUs
The initial Hugging Face pipeline with FastAPI had high latency that grew quickly as users were added. Matt explains that vLLM addresses GPU-memory bottlenecks in transformer inference with its paged-attention approach. In the team's tests, replacing the original pipeline improved latency and slowed its growth as concurrency increased. The change also improved throughput without increasing the number of GPUs. Matt presents this as one result from one project, rather than a universal performance promise, and notes that the exact numbers may change as the software evolves.
Ray Serve combines horizontal scaling, autoscaling, and GPU sharing
After improving single-server inference, the team used Ray Serve to deploy across multiple servers. Ray Serve provided an abstraction for distributing models and could adjust the number of servers through autoscaling. It also allowed different workloads to share GPUs, including the model server, guardrails, and vector operations. Matt describes the combination of Ray Serve and vLLM as a way to join horizontal scaling with the single-server inference improvements. He also warns that self-hosting a production LLM creates operational problems that teams must measure and solve for their own workload.
Structured outputs make model results easier to store, inspect, and reject
Vaibhav Gupta compares unstructured and structured extraction from a resume. Structured output produces data that can be saved to a database and makes links more reliable. In a prompt-injection example, an unstructured system responds to an unrelated chess request, while the structured system returns nothing. His citation example also distinguishes valid links from hallucinated ones and connects a valid citation to the relevant passage in a Wikipedia article. He uses these examples to show that structured output affects how an application can validate and present model results, rather than only changing their visual format.
BAML uses code, tests, and local parsing to make structured generation easier
BAML lets developers describe functions and data models with code instead of relying only on plain-English instructions or JSON schema. Its development workflow shows the full prompt, inputs such as images or audio, and test cases through hot reload. BAML then runs algorithms locally to process model output into the desired data type. Gupta demonstrates output without quotation marks and reports a reduction from 108 tokens to 83 in that example. He says the approach can work with existing models, avoids modifying the model, and can be used from TypeScript, Ruby, Python, Java, Go, and Rust.
Information extraction can narrow the data sent to an LLM
Tom Sabo uses traditional text analytics to identify entities, recommendations, terminology, and categories before calling an LLM. In public-comment analysis, thousands of letters can contain many separate statements and opinions. Rules can select statements about a particular aspect of a regulation, such as recommendations or negative responses. The LLM then summarizes a smaller and more focused collection. Sabo describes this as similar to a small RAG process assembled from many relevant statements, rather than a few retrieved documents. The same pattern can apply to customer complaints, call-center requests, and other large text collections.
Traceability lets reviewers inspect an LLM summary against source statements
Sabo's SAS visualization connects a generated summary to extracted terms, organizations, and individual source statements. In the EPA carbon-capture example, the system summarized negative recommendations about a proposed regulation. Reviewers could inspect terms such as safety, carbon waste, carbon dioxide, and unproven carbon capture, then see which organizations had raised related concerns. They could also open the underlying statements. This gives reviewers a way to examine the evidence behind a summary and focus attention on particular groups or claims instead of treating the LLM response as a final answer.
"When you can do things like engineer a prompt on one side and simultaneously get the right data against that prompt on the other side, it can give much more focused answers."Tom Sabo46:43
Who should watch
You are self-hosting an LLM application and need a way to measure latency, concurrency limits, throughput, and GPU use before making infrastructure changes.
Your application depends on reliable JSON-like data from models, especially when smaller models are attractive for cost or speed.
You are summarizing large collections of public comments, complaints, or other text and need to filter inputs and trace generated claims back to source material.
