# Serving and deployment

204 sessions · follows the tags model-serving, deployment
Page: https://mlopstalks.com/threads/serving-and-deployment

The mechanics of getting a model to answer requests, from Flask apps and Kubernetes to inference servers and hosted model APIs.

## 2020

32 sessions.

- [Building an ML Platform at SurveyMonkey](https://mlopstalks.com/talks/building-an-ml-platform-at-surveymonkey) (Shubhi Jain, SurveyMonkey). Pushed back: Shubhi Jain said SurveyMonkey's inference architecture did not run on Kubernetes. [34:48](https://www.youtube.com/watch?v=oq1g4s2dUHE&t=2088s)
- [TrueLayer's MLOps Pipeline](https://mlopstalks.com/talks/truelayers-mlops-pipeline) (Alex Spanos, TrueLayer). Pushed back: Alex Spanos favors simple interpretable algorithms in regulated financial services, rather than moving to more complex models when interpretability is reduced. [19:45](https://www.youtube.com/watch?v=cUxK28ocZcw&t=1185s)
- [10 Years Deploying ML in the Enterprise: The Inside Scoop!](https://mlopstalks.com/talks/10-years-deploying-ml-in-the-enterprise-the-inside-scoop) (Charles Martin, MLOps Community). Pushed back: Charles Martin disputes the idea that running a machine learning model inside a database such as SQL Server is a simple solution. [27:20](https://www.youtube.com/watch?v=WmZL1AQPbOw&t=1640s)
- [Machine Learning at Scale in Mercado Libre](https://mlopstalks.com/talks/machine-learning-at-scale-in-mercado-libre) (Carlos de la Torre, Mercado Libre). Pushed back: Carlos de la Torre argued that Fury Data Apps should not automate model deployment because responsibility for production deployment must remain explicit. [45:30](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=2730s)

28 more from 2020 on this thread: https://mlopstalks.com/threads/serving-and-deployment/2020

## 2021

44 sessions.

- [Lessons Learned From Hosting the ML Engineered Podcast](https://mlopstalks.com/talks/lessons-learned-from-hosting-the-ml-engineered-podcast) (Charlie You, Workday). Pushed back: Charlie You argues that the hardest or most differentiating part of production machine learning is often not the algorithm but data, deployment and maintenance. [58:57](https://www.youtube.com/watch?v=J1WpAJRt3rg&t=3537s)
- [MLOps Engineering Labs Recap, Part 1](https://mlopstalks.com/talks/mlops-engineering-labs-recap-part-1) (John Savage, Overstock & Alexey Naiden & Varuna Jayasiri & Michel Vasconcelos, Bank of Nordeste). Pushed back: Michel Vasconcelos argues that MLflow is a strong experimentation tool but is not sufficient by itself for large-scale, containerized deployments. [34:54](https://www.youtube.com/watch?v=KdWoavM-JRI&t=2094s)
- [Operationalizing Machine Learning at a Large Financial Institution](https://mlopstalks.com/talks/operationalizing-machine-learning-at-a-large-financial-institution) (Daniel Stahl, Regions Bank). Pushed back: Daniel Stahl disputes the assumption that models should be deployed as microservices by default, arguing that batch processing was simpler and sufficient for Regions' current needs. [28:08](https://www.youtube.com/watch?v=vrvagiFVzI4&t=1688s)
- [Law of Diminishing Returns for Running AI Proof-of-Concepts](https://mlopstalks.com/talks/law-of-diminishing-returns-for-running-ai-proof-of-concepts) (Oguzhan Gencoglu, Top Data Science). Pushed back: Oguzhan Gencoglu says the most crucial role skill is problem translation, not model training or deployment, which may be controversial. [45:06](https://www.youtube.com/watch?v=j09xbtudJgs&t=2706s)

40 more from 2021 on this thread: https://mlopstalks.com/threads/serving-and-deployment/2021

## 2022

37 sessions.

- [Federated Learning: Machine Learning on the Edge](https://mlopstalks.com/talks/federated-learning-machine-learning-on-the-edge) (Varun Kumar Khare, Nimble Edge). Pushed back: The speaker disputed the idea that federated learning is not production-ready by describing existing large-scale deployments in products. [40:02](https://www.youtube.com/watch?v=IWxBKGPHOBQ&t=2402s)
- [Platform Thinking: A Lemonade Case Study](https://mlopstalks.com/talks/platform-thinking-a-lemonade-case-study) (Orr Shilon, Lemonade). Pushed back: Automatic model testing was not considered sufficient to make the team comfortable deploying a model, so people still had to take responsibility for testing. [41:17](https://www.youtube.com/watch?v=KemCHs7Xbrs&t=2477s)
- [MLOps Critiques](https://mlopstalks.com/talks/mlops-critiques) (Matthijs Brouns, Xccelerated.io). Pushed back: Matthijs Brouns rejects the idea that model deployment is simply an already solved problem, arguing that important gaps remain in MLOps. [17:08](https://www.youtube.com/watch?v=SS2_jQN3sG0&t=1028s)
- [MLflow vs Kubeflow 2022](https://mlopstalks.com/talks/mlflow-vs-kubeflow-2022) (Byron Allen, Contino). Pushed back: George Pearse argues that Kubeflow's complexity makes it less suitable for small shops, while Byron Allen says Kubeflow has a valid role through managed services and in larger organizations. [45:30](https://www.youtube.com/watch?v=9YcLBSqZNzE&t=2730s)

33 more from 2022 on this thread: https://mlopstalks.com/threads/serving-and-deployment/2022

## 2023

33 sessions.

- [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale). Pushed back: The displayed model cost and performance estimates do not fully account for batching, which can change results substantially. [10:19](https://www.youtube.com/watch?v=xQRdOkVCAUY&t=619s)
- [Build and Customize LLMs in Less than 10 Lines of YAML](https://mlopstalks.com/talks/build-and-customize-llms-in-less-than-10-lines-of-yaml) (Travis Addair, Predibase). Pushed back: Travis argued that fine-tuning a smaller model can match or outperform a much larger model for a sufficiently bounded task at lower latency and cost. [17:17](https://www.youtube.com/watch?v=De6RY2GN-e4&t=1037s)
- [LLM on Kubernetes](https://mlopstalks.com/talks/llm-on-kubernetes) (Shrinand Javadekar, Outerbounds & Manjot Pahwa, Lightspeed India & Rahul Parundekar, AI Hero & Patrick Barker). Pushed back: Rahul Parundekar argues that companies should prioritize a repeatable deployment platform and model iteration over over-optimizing Kubernetes autoscaling while GPUs are scarce. [16:42](https://www.youtube.com/watch?v=0e5q4zCBtBs&t=1002s)
- [Using Large Language Models at AngelList](https://mlopstalks.com/talks/using-large-language-models-at-angellist) (Thibaut Labarre, AngelList). Pushed back: The idea that off-the-shelf machine-learning services were sufficient for AngelList was rejected because they were too expensive, could not scale as needed, and lacked the required extraction capabilities. [12:57](https://www.youtube.com/watch?v=qhGaS1SGkKI&t=777s)

29 more from 2023 on this thread: https://mlopstalks.com/threads/serving-and-deployment/2023

## 2024

32 sessions.

- [Graduating from Proprietary to Open Source Models in Production](https://mlopstalks.com/talks/graduating-from-proprietary-to-open-source-models-in-production) (Philip Kiely, Baseten). Pushed back: Philip Kiely does not give a definite position on whether companies will serve models with alternative hardware such as Groq. [22:02](https://www.youtube.com/watch?v=I_PnkrzLihk&t=1322s)
- [Productionizing Health Insurance Appeal Generation](https://mlopstalks.com/talks/productionizing-health-insurance-appeal-generation) (Holden Karau, Netflix). Pushed back: Holden Karau rejects using the latest container version as a good deployment practice and says the version should be pinned. [11:50](https://www.youtube.com/watch?v=lfoZPp6tLm4&t=710s)
- [Streamlining Model Deployment](https://mlopstalks.com/talks/streamlining-model-deployment) (Daniel Lenton, Unify). Pushed back: The discussion disputes the assumption that selecting the same model across endpoint providers necessarily produces the same output quality. [17:35](https://www.youtube.com/watch?v=jodNnvBFYws&t=1055s)
- [Productionizing AI: How to Think From the End](https://mlopstalks.com/talks/productionizing-ai-how-to-think-from-the-end) (Annie Condon). Pushed back: The idea that deploying LLMs has a quick and dirty approach was challenged because production deployment is not actually simple. [10:17](https://www.youtube.com/watch?v=77OOqXd_97A&t=617s)

28 more from 2024 on this thread: https://mlopstalks.com/threads/serving-and-deployment/2024

## 2025

19 sessions.

- [Kubernetes, AI Gateways, and the Future of MLOps](https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops) (Alexa Griffith, Bloomberg). Pushed back: Demetrios Brinkmann suggested that KServe is built on KNative and helps provision model resources, while Alexa Griffith clarified that KNative can run serverless services generally and KServe is specifically focused on AI and ML models. [19:39](https://www.youtube.com/watch?v=18aHn9E5Sb8&t=1179s)
- [MLOps with Databricks](https://mlopstalks.com/talks/mlops-with-databricks) (Maria Vechtomova, Ahold Delhaize | Marvelous MLOps). Pushed back: Maria Vechtomova disagrees with the idea that Databricks should be used for every model-serving situation. [26:50](https://www.youtube.com/watch?v=Oa6qZPlOv3c&t=1610s)
- [The Rise of Sovereign AI and Global AI Innovation in a World of US Protectionism](https://mlopstalks.com/talks/the-rise-of-sovereign-ai-and-global-ai-innovation-in-a-world-of-us-protectionism) (Frank Meehan, Frontier One AI). Pushed back: Frank Meehan responds that anonymized research data can be shared, but identifiable medical data used for patient care and internal health-system inference requires local protection. [19:02](https://www.youtube.com/watch?v=6-y49WMbR_M&t=1142s)
- [Integration of AI into Traditional Systems](https://mlopstalks.com/talks/integration-of-ai-into-traditional-systems) (Hakan Tek). Pushed back: Hakan Tek rejects the idea that using a third-party service can provide complete certainty about data security. [14:30](https://www.youtube.com/watch?v=0WuUP7vsf2U&t=870s)

15 more from 2025 on this thread: https://mlopstalks.com/threads/serving-and-deployment/2025

## 2026

7 sessions.

- [From Notebooks to Production FASTER](https://mlopstalks.com/talks/from-notebooks-to-production-faster) (Shahd Alghrsi, Virgin Media). Claim: The team serves about 200 data scientists and develops platform tools with their feedback. [2:01](https://www.youtube.com/watch?v=4JNvJwnXmcA&t=121s)
- [Speed and Scale: How Today's AI Datacenters Are Operating Through Hypergrowth](https://mlopstalks.com/talks/speed-and-scale-how-todays-ai-datacenters-are-operating-through-hypergrowth) (Kris Beevers, NetBox Labs). Claim: NetBox is the system of record for infrastructure, modeling everything from space, power, and cooling through racks, GPU servers, switching, cabling, IP addresses, configurations, and automation. [1:39](https://www.youtube.com/watch?v=KF15E_F4viw&t=99s)
- [Fast & Asynchronous: Drift Your AI, Not Your GPU Bill](https://mlopstalks.com/talks/fast-asynchronous-drift-your-ai-not-your-gpu-bill) (Artem Yushkovskiy, Delivery Hero). Claim: Data scientists use plain Python functions that take dictionaries and return dictionaries, while platform teams manage the deployment and autoscaling configuration. [12:01](https://www.youtube.com/watch?v=q9wq1znpa4w&t=721s)
- [Why AI Agents Shouldn't Replace Your Fraud Models](https://mlopstalks.com/talks/why-ai-agents-shouldnt-replace-your-fraud-models) (Varant Zanoyan, Zipline AI). Claim: High-stakes systems need auditability, high query throughput, and low latency, so they are better suited to rules engines and machine learning models than to full agentic decisioning. [6:49](https://www.youtube.com/watch?v=HaWk8kAD8ZU&t=409s)

3 more from 2026 on this thread: https://mlopstalks.com/threads/serving-and-deployment/2026