# SGT Model Deployment Working Group May 19, 2021 Meeting

 | MLOps Community | 52:00
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ABSgSlyRO9Y
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/sgt-model-deployment-working-group-may-19-2021-meeting
Published: 2021-05-20
Tags: enterprise, model-serving, platform-teams, testing

## TL;DR
- The group wants MLOps systems to support mix-and-match infrastructure and best-of-breed tools across cloud, on-premise, and open-source environments.
- The working group is concentrating on the V2 inference protocol, including adoption, metadata, client SDKs, and possible compliance testing.
- Participants see a need to hide protocol complexity from data scientists while preserving direct, flexible access for teams with legacy systems or high-performance requirements.

## Summary
The meeting introduces the Social Good Tech model deployment working group and its focus on practical standards for enterprise MLOps. The discussion starts with a view that enterprises will increasingly mix cloud services, open-source tools, and internal systems instead of choosing one end-to-end platform. Participants then review progress on the V2 inference protocol, including MLServer support, adoption in Seldon Core and KFServing, SageMaker integration, and Triton's HTTP and gRPC interfaces. They discuss how metadata, client SDKs, adapters, and compliance tests could make the protocol easier to adopt. Data scientists need simpler ways to deploy and invoke models, while production teams still need support for legacy request formats and direct, high-performance access. The meeting closes with possible next steps around a neutral repository, a compliance suite, and broader language bindings.

## Key ideas
### Enterprise MLOps is moving toward mix-and-match systems
[00:00](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=0s)
The speaker expects enterprises to combine best-of-breed tools across cloud, on-premise, and open-source environments. A CIO survey covering 100 people in Europe and the United States found that about half wanted to mix tools while using cloud compute as a commodity, while the other half preferred an end-to-end ML platform from a major cloud provider. The group sees a common SDK and shared contracts as a way to give users a consistent experience across independently chosen components.

### The working group is intended to reduce MLOps fragmentation
[00:58](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=58s)
The working group has three aims: spread MLOps practices and standards, reduce boilerplate, and provide education from online courses through university programs. It also wants to accelerate tooling, with a strong open-source focus and occasional support for vendors building broader enterprise tool suites. The speaker describes the MLOps lifecycle as eight connected areas, from data collection and processing through deployment and monitoring, with multiple teams and personas involved.

### Deployment and monitoring are grouped together for now
[08:02](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=482s)
A participant questions whether deployment and monitoring should be one area because monitoring includes outliers, observability, application-level concerns, and explainability. The response is that this is an easy area to split later, while combining it keeps the initial structure simpler. Tools such as Seldon Alibi, SageMaker, Fiddler, and Arize are cited as evidence that monitoring is developing quickly. The speaker expects deployment and monitoring to separate as interest grows.

### The V2 inference protocol is being adopted through several implementation paths
[15:26](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=926s)
The Seldon team describes three lines of work. MLServer provides an inference runtime built around the V2 protocol and is used with scikit-learn and XGBoost models in Seldon Core and KFServing. The teams are adding framework support and improving stability. Seldon Core and KFServing also expose the protocol through packaged servers, including Triton and servers brought in through MLServer. A third effort, the early-stage Tempo SDK, aims to simplify model deployment and inference for data scientists.

### A simple client layer could reduce the burden on data scientists
[17:37](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=1057s)
The participants argue that many data scientists should not need deep Kubernetes or serving-platform knowledge to deploy and use a model. Tempo explores a client that lets a user describe a model and its metadata, select a runtime, and interact with the resulting deployment. The design separates model artifacts from deployed model instances. The group questions how much of this could become a general standard instead of remaining tied to one vendor's implementation.

### The protocol must work with existing enterprise request formats
[27:41](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=1661s)
Production systems often depend on legacy payload structures that cannot be changed easily. KFServing's transformer is discussed as an adapter that accepts an organization's request format, runs pre- and post-processing, and converts the result to the V2 protocol. The group sees a possible project in documenting this pattern so enterprises can adopt a shared protocol without rewriting every upstream system.

### Client libraries matter as much as the wire protocol
[42:03](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=2523s)
Participants say many users do not want to interact with the protocol directly. They want client libraries in the languages used by their teams, including Python, C++, Java, C#, Go, and Rust. Triton already provides clients and can use gRPC-generated bindings, but the group notes that generated bindings may not provide the language-specific asynchronous behavior or HTTP support that users expect. A client SDK could make the standard easier to use without removing direct access for advanced teams.

### Compliance testing could give the standard a neutral foundation
[47:26](https://www.youtube.com/watch?v=ABSgSlyRO9Y&t=2846s)
The group observes that implementations claim V2 compatibility but there is no formal compliance suite. A shared test suite could check whether vendors and internal platforms implement the same protocol behavior. The proposed next steps include creating a neutral Social Good Tech repository, using automated tests, and keeping the project independent of any single implementation. The organizers also plan to contact teams working on TorchServe and TensorFlow.

## Notable quotes
- Demetrios Brinkmann: "This whole era I think for ML is going to be based on the idea of mix and match and best in breed." (00:32)
- Demetrios Brinkmann: "I'm not shooting for everything all at once. It's really going step by step and it's really helping accelerate people who are already interested and growing that momentum." (20:07)
- "Most of them don't even want to really interact with it." (25:25)
- Demetrios Brinkmann: "We do expect implementations to be tied to our protocols and our compliance suite though." (50:59)

## Tools & references mentioned
- Social Good Tech
- MLOps Community
- Seldon
- Seldon Core
- Seldon Alibi
- KFServing
- MLServer
- Triton
- SageMaker
- Fiddler
- Arize
- Tempo
- PyTorch
- XGBoost
- TensorFlow
- TorchServe
- FastAPI
- Kubernetes
- gRPC
- HTTP
- AI Infrastructure Alliance
- xkcd

## Who should watch
- You are comparing a managed model-serving platform with open-source runtimes and need a common interface between them.
- Your production models use legacy request formats, and you want to understand adapter patterns for adopting an inference standard.
- You are designing model-serving SDKs, protocol tests, or language bindings for enterprise users.

## Editor's note

The working group argues that enterprises will combine cloud services, open-source tools, and internal systems instead of choosing one end-to-end platform. ZenML lets teams write pipelines once and run the same code on a laptop, Kubernetes, Airflow, Kubeflow, or a cloud provider's services by changing the configured stack. That fits the group's call for mix-and-match infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [SGT Model Deployment Working Group June 22, 2021 Meeting](https://mlopstalks.com/talks/sgt-model-deployment-working-group-june-22-2021-meeting) (, 40:55)
- [SGT Model Deployment Working Group](https://mlopstalks.com/talks/sgt-model-deployment-working-group) (Srivathasan Canchi, Intuit, 46:51)
- [Creating MLOps Standards](https://mlopstalks.com/talks/creating-mlops-standards) (Alex Chung, Social Good Technologies & Srivathsan Canchi, Intuit, 47:45)
- [Iceberg, MCP, and MLOps: Bridging the Gaps for Enterprise](https://mlopstalks.com/talks/iceberg-mcp-and-mlops-bridging-the-gaps-for-enterprise) (Caleb Baechtold, Snowflake & Hamza Tahir, ZenML & Simba Khadder, Featureform, 1:02:19)
- [Kubernetes, AI Gateways, and the Future of MLOps](https://mlopstalks.com/talks/kubernetes-ai-gateways-and-the-future-of-mlops) (Alexa Griffith, Bloomberg, 50:12)