# Operationalize Open Source Models with SAS Open Model Manager

Ivan Nardini, SAS | MLOps Meetup | Episode 39 | 56:53
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=SNRsTYmb19U
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/operationalize-open-source-models-with-sas-open-model-manager
Published: 2020-10-27
Tags: deployment, governance, model-serving, monitoring

## TL;DR
- ModelOps gives teams capabilities for governing models in production, automating training, and managing deployment, monitoring, and retraining.
- The model itself is only a small part of a machine learning system, while dependencies, scalability, automation, security, and business decisions create much of the operational work.
- SAS Open Model Manager versions models, compares champion and challenger models, deploys them to targets such as containers, and monitors their performance after deployment.

## Summary
Ivan Nardini explains why open source models are easy to build but harder to run as business products. Business users care about the value visible in a dashboard, analytics teams focus on model pipelines and accuracy, and IT teams focus on architecture, compute, dependencies, and security. These groups often have different definitions of what a model is and what production means. Nardini argues that the model is only a small part of the full machine learning system. The rest includes data preparation, deployment, reproducibility, monitoring, governance, and linking predictions to business decisions. He then demonstrates SAS Open Model Manager with a credit scoring example. The workflow reads data from MongoDB, trains and compares models, registers metadata and artifacts, validates them through Jenkins, deploys a model as a Docker container, and monitors scored data for drift and performance. He also explains how champion and challenger models can support continued retraining when production data changes.

## Key ideas
### ModelOps manages models after development and into production
[04:59](https://www.youtube.com/watch?v=SNRsTYmb19U&t=299s)
Nardini describes ModelOps as a set of capabilities for managing models in production. That includes governance, automated training, deployment, monitoring, and retraining. He treats ModelOps as the main topic because customers are increasingly aware that building a model is only one part of the work. The goal is to move analytics models from development into production and keep managing them there. ModelOps also includes the decision part of the process, where a model's output is connected to a business action.

### Different groups define a useful model in different ways
[07:12](https://www.youtube.com/watch?v=SNRsTYmb19U&t=432s)
Nardini says the business sees a model as productive when its results can be consumed in a dashboard and the business uplift can be seen. Heads of analytics tend to connect ModelOps with their analytics teams, machine learning pipelines, and tuning parameters. IT teams focus on the architecture needed to put models into production, including machines and GPUs. These perspectives create different needs in the same conversation. Nardini's practical advice is to ask, "What is a model?" before asking customers many detailed questions.

### The model is a small part of the full machine learning system
[11:38](https://www.youtube.com/watch?v=SNRsTYmb19U&t=698s)
Nardini refers to Google's discussion of hidden technical debt in machine learning systems and says the model itself accounts for only a small part of the system. The larger system includes data, dependencies, deployment processes, monitoring, and the surrounding application. He has seen this complexity directly in customer discussions with business, IT, and analytics teams. A model can have good accuracy and still be difficult to turn into a productive company asset.

### Business problems often come from communication, time, and governance
[12:52](https://www.youtube.com/watch?v=SNRsTYmb19U&t=772s)
From the business perspective, Nardini describes a data product with a dashboard that cannot be used after three months of work. He connects this situation to weak communication and collaboration, lost business value from the delay, and a lack of governance that makes it hard to understand why the process took so long. The issue is not limited to the model's quality. The organization also needs a process that lets people understand what was built, how it moved through development, and why the resulting product is or is not usable.

### IT systems need reproducibility, scalability, automation, and security
[14:12](https://www.youtube.com/watch?v=SNRsTYmb19U&t=852s)
IT teams may build architectures with many tools for putting models into production, yet those architectures still create hidden problems. A changed library or dependency can break reproducibility. An environment designed for one model may not meet the computational needs of another model, which creates a scalability problem. Automation can break somewhere across a system made of many pieces, making failures difficult to locate. Nardini also agrees that security is a major concern, especially under regulatory constraints in Europe.

### Analytics teams face a long path from an open source model to production
[17:52](https://www.youtube.com/watch?v=SNRsTYmb19U&t=1072s)
Analytics teams may know how to build open source models, but production requires more than model training. Nardini says the path to deployment needs skills, code, and coordination across the pieces of the system. Once a model is deployed, teams must monitor it because analytical models behave differently from other software. They also need to connect model results to business decisions. In his view, these operational difficulties explain why ModelOps is needed.

### Customers vary in technology use while sharing the same operational gaps
[20:09](https://www.youtube.com/watch?v=SNRsTYmb19U&t=1209s)
Nardini describes customers using Java platforms for Python models, Hadoop and Spark data lakes, containers such as Docker, platforms such as OpenShift and Cloud Foundry, and cloud services from Google, Azure, and Amazon. He also mentions MLflow and Kubeflow. Their ModelOps maturity differs, but he sees common gaps: manual iterative training and deployment, few models in production, limited automation, little model monitoring, and no automated way to attach business decisions to model results. Some customers have built complex pipelines, while others are still experimenting with cloud services.

### SAS Open Model Manager connects model artifacts with validation and production monitoring
[30:55](https://www.youtube.com/watch?v=SNRsTYmb19U&t=1855s)
The demonstration uses a credit scoring application. Nardini reads data from MongoDB in a Jupyter notebook, prepares variables, trains several models, and compares them. The workflow creates requirements, score code, a pickle file, JSON metadata, and other model properties. A library registers and versions these artifacts in SAS Open Model Manager. A Jenkins pipeline runs code-quality checks, metadata validation, unit tests, and an integration test through a sidecar container before registering the model. The platform compares models, assigns champion and challenger roles, publishes a model as a Docker container, runs a remote test, and reads scored data to produce monitoring views.

### A small model inventory is still a reason to build an operational foundation
[53:18](https://www.youtube.com/watch?v=SNRsTYmb19U&t=3198s)
In response to Demetrios Brinkmann, Nardini says ModelOps can help a company with only one or two models. A small number of models means less process complexity, fewer users, and more time to establish the operating process. Starting early lets a team improve the process use case by use case and scale it more easily later. Waiting until there are many users and models makes the operational work harder because the organization has less experience with ModelOps.

## Notable quotes
- Ivan Nardini: "For me ModelOps is the main topic." (05:24)
- Ivan Nardini: "The model itself counts just for the five percent of our machine learning systems." (11:58)
- Ivan Nardini: "You don't need to ask thousand of questions, you need just one, what is a model?" (11:05)
- Ivan Nardini: "The road is really long to deployment, you need skills, you need a lot, you need codes." (18:18)
- Ivan Nardini: "If you don't do that now that you have just one or two models, when you have to do that?" (53:44)

## Tools & references mentioned
- SAS Open Model Manager
- SAS Model Manager
- MLOps
- ModelOps
- Google
- Docker
- OpenShift
- Cloud Foundry
- Hadoop
- Spark
- MLflow
- Kubeflow
- Jenkins
- GitLab
- MongoDB
- Jupyter
- Python
- scikit-learn
- XGBoost
- random forest
- logistic regression
- decision tree

## Who should watch
- You are building open source models and need to move them through validation, deployment, and monitoring.
- Your organization has different business, analytics, and IT expectations about what production means.
- You have a small model inventory and want to establish a repeatable operating process before the number of models grows.

## Editor's note

Ivan Nardini argues that a model is only a small part of the work required to make an open source model a production business asset. ZenML turns that operational workflow into Python pipelines whose runs record steps, inputs, outputs, and code versions. This gives teams a trace from a model or artifact back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Graduating from Proprietary to Open Source Models in Production](https://mlopstalks.com/talks/graduating-from-proprietary-to-open-source-models-in-production) (Philip Kiely, Baseten, 23:16)
- [Practical MLOps Part 2](https://mlopstalks.com/talks/practical-mlops-part-2) (Alfredo Deza, Author and Speaker, 1:01:38)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [The Motivation for MLOps](https://mlopstalks.com/talks/the-motivation-for-mlops) (Steven Fines, CoreLogic, 56:42)
- [Operationalize Machine Learning at Scale with MLOps](https://mlopstalks.com/talks/operationalize-machine-learning-at-scale-with-mlops) (Christopher Bergh, DataKitchen, 57:50)