Open source models give teams control over model weights, infrastructure, data handling, updates, and fallback options.
2
Model choice should be based on how well a model works in the complete application, rather than on small differences in public benchmark scores.
3
Production inference needs optimized serving, quantization, batching, caching, autoscaling, logging, and observability around the model.
Summary
Philip Kiely explains why teams may move from hosted proprietary model endpoints to open source models that they can run and control themselves. He defines an open source model for this talk as one whose weights can be obtained and run on a local or cloud GPU. That control can support privacy, security policies, compliance, model versioning, provider independence, and specialized model choices. Kiely then surveys language, image, audio, and multimodal models. He argues that evaluation must use the application's own results and quality requirements, rather than relying only on leaderboards. The production section covers model customization, GPU selection, optimized serving engines, quantization, batching, caching, autoscaling, logging, and observability. His examples include TensorRT, vLLM, TGI, TensorRT-LLM, Stable Diffusion XL, and Mixtral. The talk is practical, though the final Q&A makes clear that some infrastructure details depend on the specific model and hardware.
Open source means running weights on infrastructure you control
For this talk, Philip Kiely defines an open source model as one whose weights can be downloaded from places such as Hugging Face or GitHub and run on a GPU owned by the team or rented from the cloud. He contrasts this with GPT-4, where the provider controls the model endpoint. The definition lets him focus on operational control rather than debate the exact meaning of open source. A team can run the model end to end, decide where inputs and outputs go, and choose how the model is exposed to its application.
Model weights provide variety, independence, and control over changes
Kiely says the large number of available models gives teams both variety and specialization. They can choose a model built for a particular use case or combine several models in a retrieval-augmented generation or multimodal pipeline. Running the weights also removes dependence on one provider's endpoint uptime. Teams can create fallbacks, choose an infrastructure provider, and decide when to update a model. If they keep the weights unchanged, the model does not change underneath the application. He also connects this control to privacy, security policies, and SOC 2 and HIPAA compliance at Baseten.
The model must fit the application, not just the leaderboard
Kiely compares language models by parameter size, since models in similar weight classes can have different GPU and performance requirements. He also distinguishes foundation models from fine-tunes, which can perform differently on particular evaluations. Public benchmarks are useful signals, but he says they are not the whole truth. A model that scores slightly higher on an evaluation may still be a worse choice for a real product. Teams should test the complete system and judge both its results and its quality for the intended use case.
Open source models cover distinct language, image, audio, and multimodal needs
The examples span several model categories. For language, Kiely names Mistral, Llama, Gemma, Qwen, Phi, and StableLM families. For images, he discusses Stable Diffusion XL, its Turbo version, ControlNet, and Playground V2. Whisper supports audio transcription, while Piper handles text to speech across voices and languages. Qwen-VL and LLaVA provide general visual language capabilities. Kiely says open source multimodal models are still behind GPT-4V in general performance, while specialized systems for document question answering, OCR, and structured extraction can already be very useful.
Production inference requires a deployment around the model
A production model needs more than weights and a GPU. Kiely defines model serving as running inference, sending inputs to a model, and receiving outputs. A deployment provides the dedicated resources needed to run it, while an endpoint gives the application an API to call. Teams can set Python and system requirements, choose a GPU based on speed and cost, define request and response formats, and add application-specific steps such as parsing inputs or saving outputs to a database. This is where an open source model becomes part of a controlled product system.
Serving engines and quantization trade quality, speed, and cost
Kiely starts optimization with the hardware, then recommends using a serving engine suited to the model. He discusses TensorRT from NVIDIA, along with vLLM and TGI, and says the right engine can improve inference results on the same hardware. He reports a 40 percent latency improvement for Stable Diffusion XL with TensorRT and describes good results with Mixtral on TensorRT-LLM. Quantization provides another tradeoff. A model commonly runs at FP16, but careful validation can allow INT8 or INT4, with possible speed or cost improvements.
Batching and autoscaling keep high-traffic inference affordable
Batching lets a system trade some latency for higher throughput, which helps serve large audiences. Caching can reduce cold-start times. Kiely says autoscaling should handle traffic spikes by adding capacity, then scale to zero when the endpoint is unused so the team does not pay for idle GPUs. Logging and observability sit above these mechanisms. The operational work is substantial because teams must manage resources, serving behavior, traffic, and system visibility alongside the model itself.
Dedicated deployments reduce reliance on security promises
In the Q&A, Kiely contrasts trusting an endpoint provider's security promises with running a dedicated deployment of an open source model through a provider such as Baseten. His point is that a dedicated deployment gives the team a way to inspect and control where the model runs instead of relying only on a provider's stated policy. He also mentions that models can leak data unintentionally, even when a company is following its stated practices. For healthcare use cases, he points to retrieval-augmented generation and specialized work such as OCR and document parsing.
"What really matters when evaluating large language models is if it's going to work for your use case."09:37
Who should watch
You are deciding whether a hosted model endpoint gives your product enough control over privacy, model versions, uptime, and infrastructure.
Your team has selected an open source model and needs to turn it into an API with sensible GPU, serving, batching, and scaling choices.
You evaluate language or multimodal models and want a reminder to test them in the complete application instead of relying only on public leaderboards.
