The group wants MLOps systems to support mix-and-match infrastructure and best-of-breed tools across cloud, on-premise, and open-source environments.
2
The working group is concentrating on the V2 inference protocol, including adoption, metadata, client SDKs, and possible compliance testing.
3
Participants see a need to hide protocol complexity from data scientists while preserving direct, flexible access for teams with legacy systems or high-performance requirements.
Summary
The meeting introduces the Social Good Tech model deployment working group and its focus on practical standards for enterprise MLOps. The discussion starts with a view that enterprises will increasingly mix cloud services, open-source tools, and internal systems instead of choosing one end-to-end platform. Participants then review progress on the V2 inference protocol, including MLServer support, adoption in Seldon Core and KFServing, SageMaker integration, and Triton's HTTP and gRPC interfaces. They discuss how metadata, client SDKs, adapters, and compliance tests could make the protocol easier to adopt. Data scientists need simpler ways to deploy and invoke models, while production teams still need support for legacy request formats and direct, high-performance access. The meeting closes with possible next steps around a neutral repository, a compliance suite, and broader language bindings.
Enterprise MLOps is moving toward mix-and-match systems
The speaker expects enterprises to combine best-of-breed tools across cloud, on-premise, and open-source environments. A CIO survey covering 100 people in Europe and the United States found that about half wanted to mix tools while using cloud compute as a commodity, while the other half preferred an end-to-end ML platform from a major cloud provider. The group sees a common SDK and shared contracts as a way to give users a consistent experience across independently chosen components.
The working group is intended to reduce MLOps fragmentation
The working group has three aims: spread MLOps practices and standards, reduce boilerplate, and provide education from online courses through university programs. It also wants to accelerate tooling, with a strong open-source focus and occasional support for vendors building broader enterprise tool suites. The speaker describes the MLOps lifecycle as eight connected areas, from data collection and processing through deployment and monitoring, with multiple teams and personas involved.
Deployment and monitoring are grouped together for now
A participant questions whether deployment and monitoring should be one area because monitoring includes outliers, observability, application-level concerns, and explainability. The response is that this is an easy area to split later, while combining it keeps the initial structure simpler. Tools such as Seldon Alibi, SageMaker, Fiddler, and Arize are cited as evidence that monitoring is developing quickly. The speaker expects deployment and monitoring to separate as interest grows.
The V2 inference protocol is being adopted through several implementation paths
The Seldon team describes three lines of work. MLServer provides an inference runtime built around the V2 protocol and is used with scikit-learn and XGBoost models in Seldon Core and KFServing. The teams are adding framework support and improving stability. Seldon Core and KFServing also expose the protocol through packaged servers, including Triton and servers brought in through MLServer. A third effort, the early-stage Tempo SDK, aims to simplify model deployment and inference for data scientists.
A simple client layer could reduce the burden on data scientists
The participants argue that many data scientists should not need deep Kubernetes or serving-platform knowledge to deploy and use a model. Tempo explores a client that lets a user describe a model and its metadata, select a runtime, and interact with the resulting deployment. The design separates model artifacts from deployed model instances. The group questions how much of this could become a general standard instead of remaining tied to one vendor's implementation.
The protocol must work with existing enterprise request formats
Production systems often depend on legacy payload structures that cannot be changed easily. KFServing's transformer is discussed as an adapter that accepts an organization's request format, runs pre- and post-processing, and converts the result to the V2 protocol. The group sees a possible project in documenting this pattern so enterprises can adopt a shared protocol without rewriting every upstream system.
Client libraries matter as much as the wire protocol
Participants say many users do not want to interact with the protocol directly. They want client libraries in the languages used by their teams, including Python, C++, Java, C#, Go, and Rust. Triton already provides clients and can use gRPC-generated bindings, but the group notes that generated bindings may not provide the language-specific asynchronous behavior or HTTP support that users expect. A client SDK could make the standard easier to use without removing direct access for advanced teams.
Compliance testing could give the standard a neutral foundation
The group observes that implementations claim V2 compatibility but there is no formal compliance suite. A shared test suite could check whether vendors and internal platforms implement the same protocol behavior. The proposed next steps include creating a neutral Social Good Tech repository, using automated tests, and keeping the project independent of any single implementation. The organizers also plan to contact teams working on TorchServe and TensorFlow.
"I'm not shooting for everything all at once. It's really going step by step and it's really helping accelerate people who are already interested and growing that momentum."Demetrios Brinkmann20:07
Who should watch
You are comparing a managed model-serving platform with open-source runtimes and need a common interface between them.
Your production models use legacy request formats, and you want to understand adapter patterns for adopting an inference standard.
You are designing model-serving SDKs, protocol tests, or language bindings for enterprise users.
