# The Shipyard: Lessons Learned While Building an ML Platform

Joseph Haaga, Interos | MLOps Coffee Sessions | Episode 91 | 40:00

Source: https://www.youtube.com/watch?v=_n-GthGQw7o
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/the-shipyard-lessons-learned-while-building-an-ml-platform
Published: 2022-04-07
Tags: build-vs-buy, developer-experience, governance, platform-teams

## TL;DR
- Interos built its ML platform around a specific NLP workload because its existing models and mixed framework requirements did not fit a single off-the-shelf system.
- The platform team uses shared API and project standards, with Snitch checking model code, metadata, documentation, and container configuration automatically.
- GitOps gives Interos a versioned audit trail for model promotion, while self-service tools keep ML engineers from waiting on the platform team.

## Summary
Joseph Haaga explains how Interos built an internal ML platform for models that read supply-chain news, extract events and relationships, and resolve entities into a large knowledge graph. The team started with about 15 NLP models running as Kubernetes deployments, then evaluated Kubeflow and KServe to address GPU use and scaling. Haaga describes the standards that let model teams work while the platform was still being built, and Snitch, an internal tool that checks those standards automatically. Boarding Pass provisions repositories, registries, storage paths, and service metadata for a new model project. GitLab environment repositories then track promotion through development, integration, staging, and production. Haaga is direct about the trade-offs. The team accepts purpose-built interfaces when they help current users move faster, even when future formats may require more work. His consistent design test is whether the platform gives engineers self-service access instead of making them wait for platform support.

## Key ideas
### Interos uses machine learning to turn supply-chain news into structured risk data
[05:33](https://www.youtube.com/watch?v=_n-GthGQw7o&t=333s)
Interos maps customers' extended supply chains through a relationship knowledge graph and monitors news about the companies in those networks. Joseph Haaga says machine learning is needed to process the volume of articles and events because the company cannot hire enough people to review everything manually. The main workloads are NLP for event and relationship extraction, plus entity resolution. One example is resolving an entity mention to a specific record in a canonical data store rather than only classifying it as an organization. Forecasting and route selection were on the timeline or roadmap, while the active platform work centered on these language and resolution models.

### The platform started with existing production constraints rather than a blank slate
[07:31](https://www.youtube.com/watch?v=_n-GthGQw7o&t=451s)
Haaga frames build-versus-buy decisions around architectural drivers, including how quickly the system must go online, cost, and what the next proof of concept needs to do. Interos already had about 15 NLP models running as Kubernetes deployments. Those services were wrapped in FastAPI, but the team was unsure whether GPU nodes and CUDA were being used correctly, and scaling for demand required extra work. They evaluated Kubeflow and KServe as purpose-built options. That evaluation became the starting point for the ML platform team, although Haaga describes administering and installing Kubeflow as difficult.

### Shared interfaces let model teams keep working while the platform develops
[11:12](https://www.youtube.com/watch?v=_n-GthGQw7o&t=672s)
The information extraction team could not stop training and testing models while one platform engineer built the new system. Haaga created mutually agreed standards so both teams could work in parallel. A model project had to include a serve.py module with the expected server subclass, load method, and predict method. Teams could implement the internals however they wanted as long as the interface was present. This decoupled model development from platform development and avoided making the platform team a bottleneck for roughly eight ML engineers working on separate projects.

### Snitch turns platform standards into automated checks
[12:32](https://www.youtube.com/watch?v=_n-GthGQw7o&t=752s)
Interos built Snitch over a weekend to enforce the model standards without relying on manual review. It parses the Python abstract syntax tree to check the expected server structure and methods. It also checks that the README links to model weights and training data in S3, that a Dockerfile exists, and that the image contains metadata such as the model version and the pipeline URL that built it. Haaga describes the name as a playful reference to a tool that reports what is missing. The intended result is a descriptive error that helps a developer fix a failed deployment.

### Platform rules work only when the platform team treats model engineers as users
[14:57](https://www.youtube.com/watch?v=_n-GthGQw7o&t=897s)
Haaga says enforcement depends on trust between adjacent teams that report to the same manager. When a rule causes friction, the platform team has to examine its own design. A rule may no longer fit, or its error message may not tell the user what to do. He connects this to developer experience: even when the user made a mistake, the platform should make the correction clear and pleasant. The team changed its approach when model and pipeline engineers asked for a request and response format designed for news articles. They accepted the tighter coupling because it would help the team deliver value faster, while agreeing to address future data formats later.

### Boarding Pass and GitOps connect each model's code, artifacts, and deployments
[24:11](https://www.youtube.com/watch?v=_n-GthGQw7o&t=1451s)
Boarding Pass provisions the GitLab repository, ECR container registry, S3 paths for model weights and datasets, and service metadata in Jira Service Management for a new model project. The model promotion pipeline then updates GitLab environment manifest repositories for development, integration, staging, and production. Each repository contains Argo CD applications and Helm charts. Promoting a model opens a merge request that changes the image tag and the S3 path for the newer model weights. This gives the team a commit history and an audit trail for changes in every environment.

### Self-service is the platform team's main design test
[29:02](https://www.youtube.com/watch?v=_n-GthGQw7o&t=1742s)
Haaga says engineers often become blocked when they need another team to open a ticket or perform an operational task. His team does not want ML engineers to become Kubernetes administrators, but it does want them to deploy without waiting for platform support. That means the platform must hide operational complexity while still giving users the actions they need. The same principle explains Interos's preference for building around its own constraints. Existing models used different frameworks and composite architectures, including PyTorch, Hugging Face Transformers, and spaCy, so a rigid product with a small set of supported base images would have forced the model teams to change their work.

## Notable quotes
- "You can't solve every problem up front or else you'll never build anything." (08:28)
- "I don't want the ML engineers here to ever feel like that they don't necessarily need to become certified Kubernetes admins to deploy something to Kubernetes." (29:23)
- "We want people to be able to do everything they need to do without waiting on us or us being a blocker to them." (29:49)
- "For now it's simply there's too much velocity we would gain by implementing this change." (20:24)

## Tools & references mentioned
- Interos
- Kubeflow
- KServe
- Kubernetes
- FastAPI
- CUDA
- Snitch
- Boarding Pass
- GitLab
- ECR
- S3
- Jira Service Management
- Argo CD
- Helm
- PyTorch
- Hugging Face Transformers
- spaCy
- Kafka
- Wikidata
- Amazon

## Who should watch
- You are deciding whether an ML platform should be built internally or assembled from open-source and vendor components.
- Your model teams need to keep shipping while the platform team is still defining interfaces, deployment workflows, and standards.
- You want practical examples of self-service deployment, automated compliance checks, and GitOps-based model promotion.

## Editor's note

Joseph Haaga says Interos used shared interfaces so model teams could keep training while the platform was being built, without making the platform team a bottleneck. ZenML lets teams write ML workflows as Python steps and run the same pipeline code on different infrastructure through configuration. That gives platform teams a way to separate workflow code from infrastructure choices.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Building an ML Platform: Insights, Community, and Advocacy](https://mlopstalks.com/talks/building-an-ml-platform-insights-community-and-advocacy) (Stephen Batifol, Wolt, 45:49)
- [How to Leverage ML Tooling Ecosystem](https://mlopstalks.com/talks/how-to-leverage-ml-tooling-ecosystem) (Mariya Davydova, Neu.ro, 55:57)
- [Challenges Operationalizing ML (And Some Solutions)](https://mlopstalks.com/talks/challenges-operationalizing-ml-and-some-solutions) (Nathan Ryan Frank, WW Grainger, 52:28)
- [Building ML/Data Platform on Top of Kubernetes](https://mlopstalks.com/talks/building-ml-data-platform-on-top-of-kubernetes) (Julien Bisconti, 48:13)
- [Platform Thinking: A Lemonade Case Study](https://mlopstalks.com/talks/platform-thinking-a-lemonade-case-study) (Orr Shilon, Lemonade, 51:42)