# Model serving in production

A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 6h 23m of video.
Page: https://mlopstalks.com/packs/model-serving-in-production

A model that predicts quickly on your laptop can still leave users waiting. Requests need features, workers need loaded models, queues fill, and media must move through decoding and encoding before anyone sees the result. Start by writing down the response and traffic requirements, then choose how much prediction work can happen ahead of time. Follow that decision into request-based, queued and scheduled execution, with separate attention to scaling each stage and measuring the complete user response. The final pair asks whether inference belongs on the device at all, and how to validate that choice on real hardware. These talks compare operating designs and their constraints; the tools demonstrated in their historical examples are not a requirement to copy the same stack.

## This pack is for you if

- Your prediction function is fast, but users still wait too long for the result.
- You need to decide whether to precompute predictions, run a service, or process events through a queue.
- A model works on a server, but its device deployment has different latency and resource limits.

## The talks, in order

### 1. Don't Listen Unless You Are Going to Do ML in Production

Kyle Morris, banana.dev | 51:30 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=eDFCZNZnN-Q
Summary: https://mlopstalks.com/talks/dont-listen-unless-you-are-going-to-do-ml-in-production.md

Why first: Morris asks for numbers before infrastructure: response time, traffic spikes, model load time and reliability. His 2022 hosting account shows why a notebook exposed as an endpoint stops being enough when users and updates arrive. Those requirements give you a way to choose among the serving patterns that follow.

### 2. Common ML Serving Architectures Explained

Rebecca Taylor, Lidl e-commerce | 17:33 | MLOps Community
Video: https://www.youtube.com/watch?v=Y8wvwgHBI6Y
Summary: https://mlopstalks.com/talks/common-ml-serving-architectures-explained.md

Why second: Taylor starts with stored predictions and adds live computation only where the request needs it. Her mixed design serves existing users from batch results and uses a lighter model when history is missing. A serving requirement need not force every caller through the same expensive path.

### 3. System Design for Recommendations and Search

Eugene Yan, Amazon | 58:23 | MLOps Meetup
Video: https://www.youtube.com/watch?v=lh9CNRDqKBk
Summary: https://mlopstalks.com/talks/system-design-for-recommendations-and-search.md

Why here: Yan divides a live recommendation request into fast retrieval and more precise ranking, supported by artifacts prepared offline. This develops Taylor's mixed design into a concrete computation plan. Removing invalid candidates before ranking also avoids spending the request's budget on results you cannot show.

### 4. Declarative MLOps: Streamlining Model Serving on Kubernetes

Rahul Parundekar, A.I. Hero, Inc. | 58:58 | MLOps Meetup
Video: https://www.youtube.com/watch?v=qGC7eNHAL-k
Summary: https://mlopstalks.com/talks/declarative-mlops-streamlining-model-serving-on-kubernetes.md

Why here: Parundekar turns the request pattern into a runnable service. His 2023 Kubernetes demonstration uses one model container with different commands for HTTP, queue and batch work. The important distinction is whether a caller waits, an event starts background work, or a schedule starts a large prediction job.

### 5. Real-Time Event Processing for AI/ML with Numaflow

Sri Harsha Yayi, Intuit | 22:38 | DE4AI 2024
Video: https://www.youtube.com/watch?v=-EAexlXbN1I
Summary: https://mlopstalks.com/talks/real-time-event-processing-for-ai-ml-with-numaflow.md

Why here: Queued inference needs a scaling signal that describes the queue. Yayi's Numaflow example uses pending events and processing rate at each stage, rather than CPU and memory alone. Separating inference logic from sources and sinks makes that pressure visible without placing every messaging concern inside the model function.

### 6. Real-time Model Inference in a Video Streaming Environment

Brannon Dorsey, Runway | 58:02 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=TNO6rYwP3yg
Summary: https://mlopstalks.com/talks/real-time-model-inference-in-a-video-streaming-environment.md

Why here: Runway's 2022 video editor makes the full response path impossible to ignore: fetch, decode, infer, encode and stream before the playhead catches up. Dorsey's workers checkpoint progress so another worker can continue after interruption. Queue throughput matters only if the complete pipeline keeps the editing experience usable.

### 7. Different Ways of Serving ML Models

Byron Allen | 1:04:46 | MLOps Coffee Sessions
Video: https://www.youtube.com/watch?v=dyFjJIkrbnI
Summary: https://mlopstalks.com/talks/different-ways-of-serving-ml-models.md

Why here: Moving the model onto a device can remove the network call altogether. This discussion weighs that benefit against coupling model updates and resource use to the application. After several server-side designs, it is a useful check that the server is actually the right place for the prediction.

### 8. Efficient Deployment of Models at the Edge

Krishna Sridhar, Qualcomm | 51:34 | MLOps Podcast
Video: https://www.youtube.com/watch?v=sFqm7GTeulg
Summary: https://mlopstalks.com/talks/efficient-deployment-of-models-at-the-edge.md

Why last: Sridhar makes the device choice measurable. His 2025 examples distinguish CPU, GPU and neural-processor trade-offs, then profile models on physical target devices. Finish here because an architectural label such as edge or cloud cannot tell you whether the actual model fits the hardware and response requirements you started with.