# Different Ways of Serving ML Models

Byron Allen | MLOps Coffee Sessions | Episode 2 | 1:04:46
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=dyFjJIkrbnI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/different-ways-of-serving-ml-models
Published: 2020-07-04
Tags: deployment, edge, model-serving

## TL;DR
- Offline scoring works when the possible inputs are known in advance and predictions can be stored in a database.
- Embedding a model in an application or device avoids network latency, but it couples model deployment to the application and creates security and scaling concerns.
- Microservice serving allows models and applications to be deployed, scaled, monitored, and rolled back independently, though it requires more infrastructure and engineering work.

## Summary
Demetrios Brinkmann, David Aponte, and Byron Allen discuss three patterns from Bugra Akyildiz's blog post: offline prediction materialized in a database, an embedded model inside an application or device, and model serving through an independent microservice. They explain that offline scoring fits bounded input spaces, such as ranking genes for known diseases or sending marketing recommendations on a schedule. It is simple and fast to query, but it becomes impractical when users can submit unpredictable inputs. Embedded models make sense on edge devices where a network call would add unacceptable latency, although the model becomes harder to protect, monitor, update, and scale. The group prefers microservices for many production applications because they separate model changes from application changes and can scale horizontally. They also caution that microservices are not automatically the right answer. Teams need suitable infrastructure, engineering skills, and a use case that justifies the added complexity.

## Key ideas
### Offline scoring fits bounded input spaces
[08:21](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=501s)
David Aponte explains that the first pattern materializes predictions offline and stores them in a database such as Postgres. This works when the inputs needed for prediction are known in advance and come from a finite set. He gives the example of ranking genes for known diseases. The team can run batch inference over many diseases, store the results, and make them available after the job finishes. Byron Allen adds that marketing systems often use this pattern because predictions can be combined with warehouse data and used to produce scheduled recommendations. It does not fit applications such as search, where users can enter unpredictable queries at runtime.

### Database-backed predictions are simple, fast, and easy to replace
[17:56](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=1076s)
The group lists several practical benefits of materialized predictions. The approach needs a database rather than special serving infrastructure. Since predictions are already computed, serving them can be as simple as querying a table. Replacing a bad set of predictions is also relatively direct because a new batch can be generated and stored. The prediction data needs associated metadata, including the model and data versions, if the team wants to compare results later. New variables are harder to introduce because the system must generate and store the additional predictions. The possible search space can also become too large when the number of queries and users grows.

### Data warehouses may absorb more model computation
[21:46](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=1306s)
Byron Allen points out that the first pattern usually extracts data from a database before running the model somewhere else. He mentions newer data warehouse capabilities that may reduce that separation. Snowflake can call an API through a user-defined function, while BigQuery ML can build and host a model inside BigQuery and expose predictions through queries. The group does not claim that these options cover every MLOps need. They discuss the possibility that more model computation could move into data warehouses as those capabilities mature.

### Embedded models avoid network latency
[24:00](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=1440s)
The second pattern puts the model inside the main application or on an edge device. David Aponte and Byron Allen connect this approach to cameras, phones, and driverless cars, where a cloud connection may be unavailable or too slow. Avoiding a network call can reduce latency and remove dependencies such as message brokers or Redis that might be needed to make network-based communication more reliable. This makes the pattern useful for computer vision and other applications where the response has to happen locally.

### Embedded serving exposes the model to deployment and security constraints
[24:53](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=1493s)
An embedded model shares the application's technology stack and infrastructure, so the same team can release the model and application together. That can reduce operational overhead. The cost is tight coupling. A model bug, dependency change, or model upgrade cannot be handled independently from the application. Multiple models also consume the application's memory and resources, which makes the system less flexible. Byron Allen raises the need to protect proprietary models stored on devices, including possible encryption or obfuscation. The team also says embedded systems need a way to report model performance back to the rest of the platform.

### Microservice serving separates model and application changes
[37:03](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=2223s)
The third pattern runs the model as an independent microservice. Demetrios Brinkmann and the guests describe this as the most flexible option for deployment, while warning that it brings more development and maintenance work. A model can live in its own repository, be tagged, released, rolled back, or replaced without deploying the main application. This separation also lets the application focus on orchestration while the model service focuses on producing predictions. Microservices can support both batch and online prediction, depending on how the surrounding system is built.

### Microservices require a team and infrastructure that can operate them
[41:05](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=2465s)
The group discusses several ways to build model-serving microservices, including custom containers and Kubernetes-based infrastructure, open-source tools, enterprise products, and cloud vendor services. Each option has trade-offs, and an organization's existing architecture may rule out some tools. A fully custom system needs dedicated infrastructure expertise. The speakers expect machine learning infrastructure to become a more distinct specialization because data scientists, application developers, ML engineers, and site reliability engineers have different responsibilities. They also describe the coordination and cultural problems that arise when these groups work in silos.

### Horizontal scaling and canary releases make microservices useful for online traffic
[51:51](https://www.youtube.com/watch?v=dyFjJIkrbnI&t=3111s)
Microservices can scale horizontally when an online prediction endpoint receives more traffic. With container and cluster infrastructure, the system can add resources during spikes and reduce them when demand falls. The speakers connect this capability to Kubernetes and explain that it matters when traffic is difficult to predict. They also discuss canary models, where a new model receives a configurable share of live traffic before it is promoted. This can support model comparison, concept-drift responses, and automated promotion, although the team does not present a single standard trigger for retraining or promotion.

## Notable quotes
- David Aponte: "The first way, the first option he described as materializing or computing predictions offline." (08:38)
- David Aponte: "You can't prepare predictions for all of those options if you did, you would have to have a huge database." (09:35)
- David Aponte: "The question is, do you want your model on those devices? Do you feel happy with that? Is it safe? Is it secure?" (25:32)
- David Aponte: "I don't think that this fits everyone." (1:00:39)
- David Aponte: "There's no perfect way. It's really what works best for you and your company, for your use case." (1:01:36)

## Tools & references mentioned
- Bugra Akyildiz
- Kubernetes
- Kubeflow
- MLflow
- Postgres
- Snowflake
- BigQuery ML
- Docker
- Redis
- Martin Fowler
- Seldon
- Algorithmia
- Amazon
- Docker Swarm
- GitLab CI

## Who should watch
- You are deciding whether to batch predictions into a warehouse, embed a model in an application, or expose it as a service.
- Your team is considering microservice serving and needs to weigh independent deployments and horizontal scaling against infrastructure and maintenance work.
- You are building an edge application where network latency, connectivity, model security, and local monitoring affect the design.

## Related talks

- [Common ML Serving Architectures Explained](https://mlopstalks.com/talks/common-ml-serving-architectures-explained) (Rebecca Taylor, Lidl e-commerce, 17:33)
- [Scaling ML Capabilities in Large Organizations](https://mlopstalks.com/talks/scaling-ml-capabilities-in-large-organizations) (Bertjan Broeksema & Axel Goblet, BigData Republic, 1:02:47)
- [Databricks Model Serving V2](https://mlopstalks.com/talks/databricks-model-serving-v2) (Rafael Pierre, Databricks, 43:17)
- [Scalable Evaluation and Serving of Open Source LLMs](https://mlopstalks.com/talks/scalable-evaluation-and-serving-of-open-source-llms) (Waleed Kadous, Anyscale, 34:57)
- [Streamlining Model Deployment](https://mlopstalks.com/talks/streamlining-model-deployment) (Daniel Lenton, Unify, 21:39)