# DevOps, Security, and Observability in ML

Luke Marsden, MLOps Consulting | MLOps Meetup | Episode 106 | 32:46
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ZZQ-LiETK4U
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/devops-security-and-observability-in-ml
Published: 2022-07-22
Tags: deployment, drift, monitoring, observability, security

## TL;DR
- MLOps teams can integrate many tools by recognizing shared patterns instead of trying to connect every tool to every other tool.
- Kubernetes and GitOps provide a common deployment and version-control layer for training systems, model serving, infrastructure, and model artifacts.
- ML monitoring needs to compare production data and predictions with what was observed during training, since latency and error rates cannot reveal data drift or model drift.

## Summary
Luke Marsden explains how to assemble an MLOps stack from a large and changing set of tools. He starts with the basic flow from data and code to a deployed model, then monitoring and iteration. Kubernetes can provide a common runtime where tools discover each other through cluster networking. GitOps puts cluster state, infrastructure definitions, and ideally model deployments under version control. For access control, Luke describes OIDC and Keycloak as a way to connect users and groups to tools such as Argo CD, JupyterHub, Grafana, and Kubeflow. He also recommends agreeing on an ontology so teams use consistent names for clusters, projects, models, datasets, registries, and other resources. Model observability requires monitoring input and output distributions alongside ordinary software metrics. At the end, he demos Helix, an early project that connects Kubernetes, Git, Argo CD, object storage, Jupyter, MLflow, and KServe to train, register, and deploy a model.

## Key ideas
### MLOps tools should be connected through recurring integration patterns
[01:15](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=75s)
Marsden presents the MLOps ecosystem as a crowded collection of tools whose integrations would be impossible to enumerate exhaustively. Connecting every tool to every other tool would create a combinatorial explosion. He proposes looking for patterns instead. The basic MLOps flow starts with data and code, produces a model, deploys it so an application can use it, and monitors the result. Teams follow this flow even when they work manually on a laptop or through SSH. The difference is whether they use an ad hoc stack or a deliberate one.

### Kubernetes gives separately deployed tools a shared runtime and network
[05:17](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=317s)
Kubernetes can make integrations easier because many MLOps tools run on it. When tools share a cluster, they can address each other through stable in-cluster DNS names such as a service and namespace. Marsden contrasts this with older enterprise deployments on physical servers, where the way one system reached another was less consistent. A cluster can host model-training jobs, production model servers, and supporting services. Training may use GPUs scheduled by Kubernetes, while model artifacts and training data move through object storage such as S3.

### GitOps puts infrastructure and deployment state under version control
[07:18](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=438s)
Marsden describes GitOps as keeping the state of production in version-controlled manifests. If a cluster were destroyed, the team should be able to recreate it from those files, apart from secrets that need separate handling. The approach extends infrastructure as code to Kubernetes operations. A pull request could propose scaling an API server or changing another operational setting. Flux and Argo CD are examples he names. Argo CD reconciles YAML, Helm charts, and other configuration in Git with the actual cluster, and can also support the deployment of model artifacts.

### OIDC and Keycloak can connect identity management across the stack
[12:37](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=757s)
Marsden focuses on OIDC as a common way to connect users and applications. An authorization server, sometimes called an identity provider, can federate authentication through Google, GitHub, LDAP, or another provider. It then issues short-lived tokens to users and applications. He recommends Keycloak for its provider federation and detailed role-based access control, while acknowledging that its documentation and interface can be difficult. Argo CD, JupyterHub, Grafana, Kubeflow, and Pachyderm can integrate with OIDC, allowing one identity system to manage access across the stack.

### Teams need a shared ontology for resources and their relationships
[18:36](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=1116s)
Marsden defines an ontology as the concepts and categories in a domain, along with their properties and relationships. In an MLOps project, the team might define a cluster provisioner, a cluster target, a project, a notebook server launcher, an experiment tracker, a model management server, a model, a Git server, a dataset, and a container registry. This naming exercise matters because tools and security systems use inconsistent language. Teams also need to decide what they mean by a model, such as a weights file or a container image containing that file, and how to tie it to the code version that produced it.

### A workable stack combines identity, notebooks, data, experiments, serving, and monitoring
[21:17](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=1277s)
Marsden sketches one possible stack rather than claiming there is one universal choice. Keycloak can manage teams, projects, and access. Cloud-hosted notebooks can provide GPU resources only when users need them. Pachyderm can track data versions and lineage, while alternatives include Airflow with DVC or Kubeflow Pipelines. MLflow can handle experiments and model management, with other products available for that role. KServe can run models in production. The stack needs glue code or configuration to connect these components.

### Model monitoring must compare production distributions with training data
[24:18](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=1458s)
Ordinary service monitoring can track latency and error rates, but those metrics may look normal while a model becomes inaccurate. Marsden gives the example of a self-driving car model trained on sunny images of road signs that struggles when it snows. Teams should monitor data drift, where production inputs differ from training inputs, and model drift, where the distribution of predictions changes. His approach records histograms during training and compares them with production histograms. KL divergence can be plotted in Grafana, and a KS test can be used for continuous inputs and outputs.

### Helix automates a path from a notebook model to a deployed service
[27:34](https://www.youtube.com/watch?v=ZZQ-LiETK4U&t=1654s)
Marsden demos an early project called Helix. An account receives an isolated Keycloak tenant, and the service can connect to an existing Kubernetes cluster or provision one through TestFaster. It connects a Git repository to Argo CD, an object-storage bucket, and optionally a container registry. In the demo, a model is trained in Jupyter and logged to MLflow. Choosing the model in MLflow creates a KServe inference service, writes its deployment definition to Git, points it at the model in S3, and provides a URL. A request to that URL returns the model's prediction.

## Notable quotes
- Luke Marsden: "What we can do is look at patterns in how you can integrate these tools together when you plug them into each other." (02:54)
- Luke Marsden: "If your entire cluster exploded tomorrow you should be able to recreate it from scratch modulo secrets." (07:38)
- Luke Marsden: "Observing models is different to observing software." (24:18)
- Luke Marsden: "We just trained a model in Jupyter, deployed it to KServe via MLflow." (31:59)

## Tools & references mentioned
- MLOps Community
- Kubernetes
- S3
- GitHub
- GitLab
- Argo CD
- Flux
- Terraform
- OIDC
- Keycloak
- OAuth
- LDAP
- JupyterHub
- Grafana
- Kubeflow
- Pachyderm
- MLflow
- Neptune
- Comet ML
- DVC
- Airflow
- Kubeflow Pipelines
- KServe
- Box Kite
- Helix
- TestFaster
- Docker Hub
- ECR
- Grafana
- PromQL
- KL divergence
- KS test
- GitOps
- Weaveworks
- Red Hat
- Laszlo
- Dan Jeffries

## Who should watch
- You are choosing how to combine training, experiment tracking, model serving, identity, and infrastructure tools into one operating setup.
- Your team has Kubernetes services but lacks a clear GitOps approach for recreating clusters and managing model deployments.
- Production predictions look wrong even though service latency and error rates remain normal, and you need a practical drift-monitoring approach.

## Editor's note

Luke Marsden says connecting every tool to every other tool would create a combinatorial explosion, so teams should build around recurring integration patterns. ZenML lets teams write workflows as Python pipelines and choose the orchestrator, artifact store, and other infrastructure through configuration. The same pipeline code can then run on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud service.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47)
- [Engineering MLOps](https://mlopstalks.com/talks/engineering-mlops) (Emmanuel Raj, TietoEvry, 51:55)
- [Creating MLOps Standards](https://mlopstalks.com/talks/creating-mlops-standards) (Alex Chung, Social Good Technologies & Srivathsan Canchi, Intuit, 47:45)
- [CI/CD & Continuous Training in ML](https://mlopstalks.com/talks/ci-cd-continuous-training-in-ml) (David Hershey, Determined AI, 1:00:53)
- [Practical MLOps Part 2](https://mlopstalks.com/talks/practical-mlops-part-2) (Alfredo Deza, Author and Speaker, 1:01:38)