# Running open models A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 3h of video. Page: https://mlopstalks.com/packs/running-open-models The model runs on your GPU, but concurrent requests exhaust its memory. A quiet endpoint takes too long to wake up. A faster model does little for users because requests wait elsewhere in the service. Start with the control that downloadable weights give you and the deployment work that comes with it. Then examine accelerator communication, memory pressure and the serving techniques that change what fits on the hardware. Application and service accounts show how narrow workloads and related model variants alter deployment choices. Autoscaling brings the discussion back to uneven real traffic. Finish with a team that deliberately hosts different models in different ways. These 2023 and 2024 examples explain operating tradeoffs; their prices, hardware comparisons and product capabilities belong to that period. ## This pack is for you if - An open model works locally but cannot serve the concurrency your application needs. - You are choosing GPUs and a serving engine without a clear explanation of the bottleneck. - You need control over model versions but are unsure how much serving infrastructure to operate. ## The talks, in order ### 1. Graduating from Proprietary to Open Source Models in Production Philip Kiely, Baseten | 23:16 | AI in Production 2024 Video: https://www.youtube.com/watch?v=I_PnkrzLihk Summary: https://mlopstalks.com/talks/graduating-from-proprietary-to-open-source-models-in-production.md Why first: Kiely frames downloadable weights as control over deployment, data flow and model updates. He also lists the service machinery that an API used to hide. Start with that exchange: the reason to run an open model should be clear before choosing a GPU or an inference engine. ### 2. Considerations and Optimizations for Deploying Open Source LLMs at Your Company Oscar Rovira, Mystic AI | 11:31 | MLOps Community Video: https://www.youtube.com/watch?v=TkJLx46yVJQ Summary: https://mlopstalks.com/talks/considerations-and-optimizations-for-deploying-open-source-llms-at-your-company.md Why second: Rovira turns the deployment into concrete requirements: model-specific environments, enough GPU memory, streaming and monitoring. His comparison of infrastructure approaches makes the maintenance burden visible. This establishes what your service must supply before optimizing the computation inside it. ### 3. Challenges in Providing LLMs as a Service Hemant Jain, Cohere AI | 11:43 | LLMs in Production 2023 Video: https://www.youtube.com/watch?v=R4obCHN2QHc Summary: https://mlopstalks.com/talks/challenges-in-providing-llms-as-a-service.md Why here: Splitting a large model across accelerators introduces communication costs as well as capacity. Jain connects that constraint to batching, quantization and horizontal scaling. A model that technically fits is only a starting point; the next explanation shows why the remaining memory can determine useful concurrency. ### 4. Exploring the Latency/Throughput & Cost Space for LLM Inference Timothée Lacroix, Mistral | 30:25 | MLOps Community Video: https://www.youtube.com/watch?v=mYRqvB1_gRk Summary: https://mlopstalks.com/talks/exploring-the-latency-throughput-cost-space-for-llm-inference.md Why here: Lacroix explains the memory used by weights and the attention cache, then examines allocation and scheduling. His 2023 discussion of paged attention and chunked prompt processing connects memory waste to stalled decoding. Use these mechanisms to form a bottleneck hypothesis, rather than treating a larger accelerator as the only response. ### 5. Making LLM Inference Affordable Daniel Campos, Snowflake | 32:07 | LLMs in Production 2023 Video: https://www.youtube.com/watch?v=I7zMJJlHdxQ Summary: https://mlopstalks.com/talks/making-llm-inference-affordable.md Why here: Campos supplies an application-level example of changing the workload instead of only enlarging the hardware. His web-summarization account combines a smaller model and compression with a deployment sized around available instances. The historical result is useful because the serving design follows one concrete task, not a general model ranking. ### 6. Efficient Serving of LLMs for Experimentation and Production with Fireworks.ai Dmytro Dzhulgakov, Fireworks.ai | 11:43 | MLOps Community Video: https://www.youtube.com/watch?v=O-UUy0ZiaBI Summary: https://mlopstalks.com/talks/efficient-serving-of-llms-for-experimentation-and-production-with-fireworks-ai.md Why here: Many quiet variants create a different problem from one busy model. Dzhulgakov describes batching requests for adapters that share a base model onto common hardware. This is the serving consequence of customization, rather than a lesson in training adapters: separate allocations can waste the capacity you just worked to recover. ### 7. Scalable Evaluation and Serving of Open Source LLMs Waleed Kadous, Anyscale | 34:57 | LLMs in Production 2023 Video: https://www.youtube.com/watch?v=xQRdOkVCAUY Summary: https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms.md Why here: Kadous brings the model fleet into contact with spiky traffic. His service needs per-model prompt configuration, warm replicas and startup behavior that can respond before a burst is over. The case complicates a simple scale-to-zero promise and shows why autoscaling settings belong in the deployment design. ### 8. LLMs vs LMs in Production Denys Linkov, Voiceflow | 24:44 | LLMs in Production 2023 Video: https://www.youtube.com/watch?v=y1HqPNwBp0U Summary: https://mlopstalks.com/talks/llms-vs-lms-in-production.md Why last: Linkov describes choosing hosting separately for different models and finding latency in the queue rather than the model. That is a useful final check on all the preceding optimization: owning the weights does not require owning every endpoint, and faster inference is wasted if the surrounding request path still misses its target.