Open LLMs improved quickly because better training data and instruction tuning produced strong results with relatively small models.
2
The main difficulty in turning an LLM into a product is the surrounding software, data, infrastructure, monitoring, governance, and cost management.
3
Teams need infrastructure that can accommodate rapidly changing models and tools instead of locking them into one opinionated stack.
Summary
Joseph Gonzalez describes two perspectives on LLMs: Berkeley's work on model and systems research, and Aqueduct's focus on what software teams need today. He explains how Llama, Alpaca, ShareGPT data, and the Vicuna project helped open models improve quickly. The Chatbot Arena then provided a way to compare models through human preferences and pairwise battles. Gonzalez argues that innovation will increasingly concern how models are used inside applications, including tools such as Gorilla for launching cloud resources from natural-language instructions. Vikram Sreekanti then describes the missing software around an LLM application. Teams need model access, retrieval, prompt chains, ordinary Python code, cloud deployment, data controls, telemetry, and cost management. Their advice is to start with a model that is easy to try, use retrieval rather than stuffing prompts with data, and choose infrastructure that can keep changing as the ecosystem develops.
Research and production are moving closer together
Joseph Gonzalez says Berkeley works on frontier problems in model architectures, evaluation, inference, training, and memory use, while Aqueduct studies what teams can actually build now. He says these paths are colliding for the first time because research advances are affecting products within weeks rather than years. Aqueduct's goal is to help every software team build LLM-powered applications. The talk combines the research view with the practical problems that appear when teams try to deploy these systems.
Better data made small open models much more capable
Gonzalez traces the open-model progress from Llama to Alpaca and Vicuna. Llama used better training data and training methods, although it was weak at following instructions. Alpaca fine-tuned Llama with self-instruction data to make it behave more like a conversational assistant. The Vicuna team used about 70,000 shared ChatGPT conversations from ShareGPT, around 800 megabytes of high-quality, multi-turn examples. Gonzalez says the first Vicuna training run was done by a few students in about a week and cost a few hundred dollars.
Model evaluation needs open-ended and human-based tests
The team built a GPT-4 evaluation method because standard NLP benchmarks did not capture instruction following well. GPT-4 judged answers on criteria such as engagement, insightfulness, and factuality. They later created Chatbot Arena, where people ask randomly paired models questions and choose between the responses. The results are scored with the Elo rating system. Gonzalez also describes limitations: some models abstain on unusual questions, while humans may prefer any answer, so a richer benchmark is needed.
Application design will matter more than training another model
Gonzalez says the field may be shifting from building models to using them in applications. His group built Gorilla to help researchers obtain cloud resources through natural-language instructions, such as launching virtual machines with particular GPUs in a specified region. The model reads documentation and uses it to invoke cloud services. He presents this as an example of an LLM making an existing workflow easier rather than as a new general-purpose model.
Start with an accessible model and retrieve the right data
Sreekanti advises teams to begin with the largest model they can easily use, which will often be a hosted model. Sensitive industries may later need another arrangement, but a hosted API can help prove that an application works. He warns against relying on a clever prompt with no data and against putting all available data into every prompt. Vector databases can retrieve relevant information at the right time and provide a more focused context.
An LLM application still needs ordinary software around the model
Sreekanti says models cannot handle every part of an application. Teams still need code to retrieve data from APIs, clean and featurize it, call the model, validate the result, and connect the workflow to the rest of the product. Tools such as LangChain and LlamaIndex help teams chain prompts, compare models, and experiment. The model is one component inside a larger application, not the entire application.
After a prototype works, teams must connect it to cloud infrastructure and the existing product. They need controls around data access so that information is not exposed accidentally, along with governance for the systems that feed the application. They also need records of inputs and outputs for debugging and complaints. Sreekanti adds budgeting and cost management because hosted APIs and self-managed open models can produce large bills quickly.
Fast-changing tools require infrastructure that stays flexible
Sreekanti says innovation is arriving at every layer, including models, databases, and Python libraries. Teams are adopting new research outputs quickly, sometimes only days after release. He expects teams to choose infrastructure that helps them keep up with these changes. In his view, infrastructure cannot be built around supporting one preferred tool while rejecting others, because the surrounding ecosystem will continue to change.
Retrieval can help with latency and context limits
In the question session, Gonzalez mentions memory-bandwidth techniques, batching, hardware utilization, and quantization as ways to improve serving performance. For long context, he is cautious about simply increasing the context window. He recommends retrieval methods that select the information needed for a request, because too much context can distract the model. He also expects LLM tooling to spread beyond Python, with implementations for languages such as Go, Rust, JavaScript, and TypeScript.
"At the end of the day, to use these models, just like with any machine learning model, you're going to want to take some code that retrieves some data from an API, cleans it, featurizes it, passes it into the model, takes the result, validates it."Vikram Sreekanti17:09
Who should watch
You are building an LLM prototype and need to understand what has to exist around the model before it can become a product.
Your team is deciding whether to use hosted models, open models, retrieval, or prompt-chaining tools.
You are responsible for cloud deployment, data access, debugging, or costs for an LLM application.
