A model that predicts quickly on your laptop can still leave users waiting. Requests need features, workers need loaded models, queues fill, and media must move through decoding and encoding before anyone sees the result. Start by writing down the response and traffic requirements, then choose how much prediction work can happen ahead of time. Follow that decision into request-based, queued and scheduled execution, with separate attention to scaling each stage and measuring the complete user response. The final pair asks whether inference belongs on the device at all, and how to validate that choice on real hardware. These talks compare operating designs and their constraints; the tools demonstrated in their historical examples are not a requirement to copy the same stack.
Why first: Morris asks for numbers before infrastructure: response time, traffic spikes, model load time and reliability. His 2022 hosting account shows why a notebook exposed as an endpoint stops being enough when users and updates arrive. Those requirements give you a way to choose among the serving patterns that follow.
Why second: Taylor starts with stored predictions and adds live computation only where the request needs it. Her mixed design serves existing users from batch results and uses a lighter model when history is missing. A serving requirement need not force every caller through the same expensive path.
Why here: Yan divides a live recommendation request into fast retrieval and more precise ranking, supported by artifacts prepared offline. This develops Taylor's mixed design into a concrete computation plan. Removing invalid candidates before ranking also avoids spending the request's budget on results you cannot show.
Why here: Parundekar turns the request pattern into a runnable service. His 2023 Kubernetes demonstration uses one model container with different commands for HTTP, queue and batch work. The important distinction is whether a caller waits, an event starts background work, or a schedule starts a large prediction job.
Why here: Queued inference needs a scaling signal that describes the queue. Yayi's Numaflow example uses pending events and processing rate at each stage, rather than CPU and memory alone. Separating inference logic from sources and sinks makes that pressure visible without placing every messaging concern inside the model function.
Why here: Runway's 2022 video editor makes the full response path impossible to ignore: fetch, decode, infer, encode and stream before the playhead catches up. Dorsey's workers checkpoint progress so another worker can continue after interruption. Queue throughput matters only if the complete pipeline keeps the editing experience usable.
Why here: Moving the model onto a device can remove the network call altogether. This discussion weighs that benefit against coupling model updates and resource use to the application. After several server-side designs, it is a useful check that the server is actually the right place for the prediction.
Why last: Sridhar makes the device choice measurable. His 2025 examples distinguish CPU, GPU and neural-processor trade-offs, then profile models on physical target devices. Finish here because an architectural label such as edge or cloud cannot tell you whether the actual model fits the hardware and response requirements you started with.