# Serving Models with Kubeflow

 | MLOps Coffee Sessions | Episode 1 | 49:57
Hosted by David Aponte

Source: https://www.youtube.com/watch?v=NNXoZ53gHyE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/serving-models-with-kubeflow
Published: 2020-06-15
Tags: deployment, model-serving, platform-teams

## TL;DR
- David Aponte defines model serving as making a trained model's predictions available to users through an application interface.
- Kubeflow provides an abstraction over serving frameworks and can expose custom models through serverless services.
- The hard part of Kubeflow serving is often the surrounding Kubernetes and container infrastructure, rather than the prediction code itself.

## Summary
David Aponte and Demetrios Brinkmann use their first MLOps Coffee Session to explain model serving through Kubeflow. David defines serving as making predictions from a trained model available to users when they need them. He describes Kubeflow as a layer over frameworks such as TensorFlow Serving and Seldon, then explains how Kubernetes, containers, Istio, and Knative fit around it. The conversation covers serverless behavior, API endpoints, real-time prediction, offline predictions, scaling, and the infrastructure needed before Kubeflow becomes easy to use. David then walks through a scikit-learn iris model, wraps it in a custom KFServing class, loads a serialized model, parses JSON requests, returns predictions, and runs the service locally. He also explains where preprocessing, postprocessing, Docker images, and model registry artifacts fit. The discussion is honest about debugging, production setup, and cases where an offline database lookup may be simpler than a live endpoint.

## Key ideas
### Serving means making model predictions available to an application's users
[02:49](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=169s)
David Aponte defines serving as taking a trained machine learning model and making its predictions available to the users of an application. The prediction needs to arrive when the user needs it and in a form the application can use. He distinguishes this from merely training or storing a model. Demetrios Brinkmann connects the idea to an endpoint that an application can call, while David explains that the end user may only see the resulting recommendation in the interface. An online recommendation system could send live user information to an endpoint, receive a prediction, and show the result through the application.

### Kubeflow sits above several model-serving frameworks
[03:46](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=226s)
David explains that Kubeflow is an abstraction over serving frameworks such as TensorFlow Serving, Seldon, XGBoost, and ONNX-related serving. A Kubeflow job can be optimized for a particular framework, such as TensorFlow models, while the broader layer can also support Python pickled objects and custom models. The format used to submit a model differs between frameworks, so the abstraction does not erase every implementation detail. David says he uses Kubeflow to serve a custom model at his company, which gives him a way to expose models without being limited to one standard framework.

### Serverless model services start on demand and scale with traffic
[07:28](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=448s)
David describes a container as a box containing the code and software needed to run an application. He describes serverless as an event-driven setup where the service is not running all the time. A request to the API can start the service, and the service can scale up when traffic increases and scale down when traffic falls. This can reduce the cost of an endpoint that needs real-time responses but is only called occasionally. David also acknowledges tradeoffs. Serverless gives the user less control over the server, and some workloads may need a service running continuously.

### Knative and Istio provide general infrastructure around Kubeflow serving
[08:48](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=528s)
David places Kubeflow serving inside a wider Kubernetes stack. Kubernetes orchestrates containers, Istio provides a service mesh for communication between services, and Knative adds custom resources for serverless containers. Knative is not specific to machine learning, so it can deploy other serverless applications and web services as well. Kubeflow serving uses these layers to expose models as services. Demetrios asks what Knative serves when the workload is not a model, and David clarifies that it can serve any suitable application or container.

### Real-time endpoints are useful when the input is only known at request time
[11:01](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=661s)
David gives online or real-time prediction as a common model-serving case. The application cannot prepare the prediction in advance because the required input arrives when the user is interacting with it. He uses recommendation systems as an example, where a service can receive current user information and return a result. A live endpoint is less necessary when predictions can be generated offline, stored in a database, and retrieved later. For teams that do not need real-time responses, David says an offline workflow and database lookup may require less work than a live microservice.

### The infrastructure around Kubeflow can be harder than the serving class
[13:01](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=781s)
David says Kubeflow serving is easy to use once the infrastructure is already installed, but getting that infrastructure in place can be difficult. He points to Kubernetes as the fundamental tool because training jobs, batch inference, and model services run on Kubernetes resources. Engineers need to understand the Kubernetes API, use kubectl to inspect resources and logs, and build Docker images that package applications without unnecessary weight. APIs, REST, serverless concepts, and service architecture also become relevant. A small configuration or installation problem can turn into hours of debugging when someone is following a tutorial without the surrounding platform.

### A custom KFServing class needs model loading and prediction logic
[23:51](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=1431s)
David's demo uses the iris dataset from scikit-learn and a random forest model. The training script loads the data, performs a train-test split, trains the model, and serializes it as a pickle artifact. The serving code inherits from the KFServing model class. It overrides a load method to load the trained artifact and a predict method to read the request object, extract the instances, pass them to the model, and return predictions as a dictionary that can be encoded as JSON. David says a model stored in an S3 bucket or model registry could be downloaded during loading instead of being kept locally.

### Preprocessing, postprocessing, and containers extend the basic serving pattern
[29:55](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=1795s)
The minimal serving example needs a load method and a predict method. David explains that a preprocessing method can transform incoming data before prediction, such as converting an image into the expected format or scaling features. A postprocessing method can change the output into a more useful form, such as returning a class name instead of a numeric label. He also shows a Dockerfile that installs requirements, copies the repository, and starts the KFServing process. The same image can support different entry points, including training and serving, which lets a team reuse a base image rather than rebuild every dependency from scratch.

### Production serving adds deployment configuration beyond the local demo
[36:51](https://www.youtube.com/watch?v=NNXoZ53gHyE&t=2211s)
David's local example starts a service on port 8080 and accepts a JSON request through a REST API. A production deployment would provide a model ID, a container image path, and the location of the trained artifact, then use the serving infrastructure to create an endpoint. The service could scale with incoming traffic. David is clear that the local demo does not include every production concern, especially the Kubernetes and Kubeflow setup needed around it. Demetrios summarizes that having Kubeflow running is a large part of the work before data scientists can benefit from the serving layer.

## Notable quotes
- David Aponte: "It means it's the process of taking some sort of trained machine learning model and making its predictions available to its users." (02:49)
- David Aponte: "Serverless is best way I like to think about is that it's not on all the time, it's going to be turned on by some event." (07:28)
- David Aponte: "The fundamental tool is Kubernetes." (17:28)
- David Aponte: "What you do is you're going to inherit from this class, build on top of that, and overwrite certain methods that KFServing has told me to overwrite." (30:31)
- Demetrios Brinkmann: "The infrastructure part of Kubeflow is probably 90% of the battle, it seems like." (44:09)

## Tools & references mentioned
- MLOps Community
- Kubeflow
- KFServing
- TensorFlow Serving
- Seldon
- XGBoost
- ONNX
- Kubernetes
- Knative
- Istio
- Docker
- scikit-learn
- Python
- Tornado
- S3
- Katacoda
- Martin Fowler
- Vicki Boykis
- Thoughtworks
- Amazon
- Netflix
- GitHub

## Who should watch
- You are deciding between online endpoints, serverless services, and offline predictions for a model application.
- You need a practical introduction to the Kubernetes and Docker concepts that sit underneath Kubeflow serving.
- You want to see the shape of a custom scikit-learn serving class before dealing with a full production deployment.

## Related talks

- [Databricks Model Serving V2](https://mlopstalks.com/talks/databricks-model-serving-v2) (Rafael Pierre, Databricks, 43:17)
- [Why and When to Use Kubeflow for MLOps](https://mlopstalks.com/talks/why-and-when-to-use-kubeflow-for-mlops) (Ryan Russon, Maven Wave Partners, 58:57)
- [Building ML Blocks with Kubeflow Orchestration with Feature Store](https://mlopstalks.com/talks/building-ml-blocks-with-kubeflow-orchestration-with-feature-store) (Aniruddha Choudhury, Publicis Sapient, 1:26:03)
- [Optimizing Your ML Workflow with Kubeflow 1.0](https://mlopstalks.com/talks/optimizing-your-ml-workflow-with-kubeflow-1-0) (Josh Bottum, Arrikto, 1:03:41)
- [Bring Your On-Prem ML Use Cases to Production on Google Cloud using Kubeflow](https://mlopstalks.com/talks/bring-your-on-prem-ml-use-cases-to-production-on-google-cloud-using-kubeflow) (Chanchal Chatterjee, Google, 20:28)