# Scaling MLOps for Computer Vision

David Espejo, Union & Fabio Grätz, Recogni & Arno Hollosi, Blackshark.ai | MLOps Meetup | Episode 4 | 58:54
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=X4HwStAt15U
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/scaling-mlops-for-computer-vision
Published: 2023-12-06
Tags: data-quality, evals, orchestration, platform-teams

## TL;DR
- Flyte applies software engineering practices to machine learning by versioning artifacts, enforcing typed interfaces, and managing infrastructure for repeatable workflows.
- Recogni uses Flyte to automate model conversion benchmarks, provision multi-GPU experiments, cache unchanged work, and resume training on cheaper preemptible machines.
- Blackshark.ai learned that planet-scale computer vision requires careful data reuse, manual quality checks alongside metrics, cost measurement beyond compute, simple workflows, and tools for domain experts.

## Summary
The session presents three views of MLOps for computer vision. David Espejo introduces Flyte as a platform that connects model developers with operations through typed task interfaces, versioned artifacts, containerized execution, and infrastructure declared in code. Fabio Grätz describes how Recogni uses Flyte in its internal developer platform. A commit can trigger reproducible conversion benchmarks across many networks, with caching, multi-node training, and checkpointing for preemptible machines. Arno Hollosi explains how Blackshark.ai runs models over satellite imagery at planetary scale. His lessons concern changing customer and sensor requirements, expensive labels, image alignment, model reuse, metrics that can hide local failures, storage and I/O costs, and the effort spent on the whole development process. He also describes point-and-click tools that let geospatial specialists run workflows and prototype models without managing the infrastructure themselves.

## Key ideas
### MLOps teams need a shared delivery process with model developers
[02:01](https://www.youtube.com/watch?v=X4HwStAt15U&t=121s)
David Espejo describes a recurring conflict between data science teams and operations teams. Model developers may focus on experiments and accuracy, while operations engineers care about versioning, code quality, documentation, and whether users can actually use the result. He connects this to software delivery principles from Continuous Delivery: automate the process, keep artifacts in version control, catch defects early, and treat a completed product as released to users. He also argues that everyone involved should own the delivery process. The platform should support repeatable development, early error detection, versioning, and flexible infrastructure.

### Flyte enforces a contract between model code and infrastructure
[10:41](https://www.youtube.com/watch?v=X4HwStAt15U&t=641s)
Flyte was created at Lyft as an ML-aware continuous delivery platform and was donated to the Linux Foundation in 2021. David explains that model developers define expected inputs and outputs with type hints, along with the infrastructure their tasks need. Flyte checks types early, versions pipeline artifacts, and orchestrates requested platform or external services. Each task runs in a container, workflows compose tasks, and launch plans provide standard inputs, schedules, and shared execution parameters. Projects and domains group work across stages and support multi-tenancy.

### Recogni turns experiments into automated benchmark workflows
[23:01](https://www.youtube.com/watch?v=X4HwStAt15U&t=1381s)
Fabio Grätz explains that Recogni develops low-power, low-latency inference systems for autonomous driving. Its engineers build perception models and tools that compress and mathematically convert trained networks for deployment. Instead of asking engineers to create cloud machines, install environments, and run one-off notebooks, Recogni lets a commit message trigger benchmark runs. CI builds a Docker image, runs tests, starts Flyte workflows across a range of networks, evaluates conversions, and writes an aggregate report into the pull request. Reviewers can follow a link to inspect the workflow execution and decide whether the proposed library change should be merged.

### Python task declarations reduce infrastructure work for ML engineers
[29:26](https://www.youtube.com/watch?v=X4HwStAt15U&t=1766s)
Recogni's engineers declare experiment resources in a Flyte task decorator, such as GPU count, CPU count, and memory. Flyte works with Kubernetes to provision the requested resources, so engineers do not need to create virtual machines or edit build pipelines in YAML. Fabio also describes Flyte's caching. If a change affects only a later task, earlier unchanged tasks can return cached results, which speeds up iteration and avoids recomputing work. Flyte also handles multi-node, multi-GPU execution through plugins for PyTorch and TensorFlow, allowing the same logic to run locally and then at larger scale in a cluster.

### Checkpointing makes preemptible compute practical
[34:08](https://www.youtube.com/watch?v=X4HwStAt15U&t=2048s)
Recogni marks suitable tasks as interruptible, and its platform schedules them on preemptible Google Cloud nodes. Training code writes checkpoints through Flyte's intra-task checkpointing mechanism. If the cloud provider removes a machine, Flyte retries the task and continues from the latest checkpoint. On the final retry, it switches to a non-preemptible machine so the task can finish. This lets the team use cheaper machines without making engineers manually recover interrupted training runs.

### Computer vision data changes require reuse and alignment
[37:42](https://www.youtube.com/watch?v=X4HwStAt15U&t=2262s)
Arno Hollosi says Blackshark.ai creates 3D digital twins from petabytes of satellite imagery. Customer requirements, satellite sensors, image processing, and elevation models change over time. A new sensor can have different optics and spectral sensitivity, while updated orthorectification can warp images differently. Labels may also be misaligned by more than 10 meters or shift when the satellite viewing angle changes. Blackshark.ai therefore invests in adapting labels and aligning imagery so models and expensive training data can be reused. Arno says teams should design processes, data, and models for reuse from the start.

### Metrics need manual inspection because local failures can hide
[43:04](https://www.youtube.com/watch?v=X4HwStAt15U&t=2584s)
Blackshark.ai tracks familiar measures such as training loss, accuracy, and F1, but Arno warns that these measures can hide bad regions inside an otherwise good result. He describes a model that scored well while some neurons failed to fire in parts of an image, producing a different inference mask. The team combines manual review with heuristics, AI-guided inspection, and randomized tests. The right balance depends on the domain. Arno recommends designing metrics carefully and investing in manual quality assurance tools instead of treating aggregate scores as sufficient.

### Scale creates storage, process, integration, and usability costs
[47:06](https://www.youtube.com/watch?v=X4HwStAt15U&t=2826s)
Arno says infrastructure costs include storage and I/O, not only compute. Large intermediate results need retention policies, and billions of writes can justify merging workflow tasks to avoid storing intermediate data. Teams should also measure the human effort from an idea through deployment and operation, then profile that process to find bottlenecks. Using Flyte across ML and non-ML workflows reduces the number of systems engineers must learn and allows tasks to be reused. For domain experts, Blackshark.ai built point-and-click interfaces for running workflows and a rapid prototyping tool that provides immediate model feedback from drawn examples.

## Notable quotes
- David Espejo: "We are not just developing the best model out there, we are developing an AI product that should get into the hands of users quickly and keeping high quality." (08:48)
- Fabio Grätz: "We want them to be able to quickly implement the experimental change and then have the infrastructure for that managed automatically." (25:01)
- Arno Hollosi: "Your metrics are not an end to themselves, but they actually stand in for some quality which you would like to measure." (44:57)
- Arno Hollosi: "You should assign your processes for change, because not doing so is very costly." (39:39)
- Arno Hollosi: "Your domain experts are not your developers, so you probably need to build tools for your experts as well." (54:30)

## Tools & references mentioned
- Flyte
- Union
- Lyft
- Linux Foundation
- Continuous Delivery
- Google Cloud Architecture Center
- Kubernetes
- PyTorch
- TensorFlow
- Google Cloud
- GitHub
- Blackshark.ai
- Microsoft Flight Simulator
- OpenStreetMap

## Who should watch
- You are building an internal ML platform and need model teams to run reproducible experiments without managing cloud infrastructure.
- Your computer vision pipeline uses large datasets, expensive labels, or changing sensors, and aggregate metrics do not reveal every failure.
- You are deciding how to run multi-GPU training, benchmark model transformations, or reduce the cost of interrupted cloud jobs.

## Editor's note

Fabio Grätz explains that Recogni replaced one-off notebooks with commit-triggered benchmark workflows that build, test, run across networks, and report results in pull requests. ZenML records each run's steps, inputs, outputs, and code version, so benchmark artifacts can be traced to the code and data that produced them. Unchanged steps are cached instead of recomputed.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [MLOps EngineeringLabs](https://mlopstalks.com/talks/mlops-engineeringlabs) (Niels Bantilan & Haytham Abuelfutuh, Union.Ai, 1:15:40)
- [MLOps vs ML Orchestration](https://mlopstalks.com/talks/mlops-vs-ml-orchestration) (Ketan Umare, Union.ai, 49:46)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [Packaging MLOps Tech Neatly for Engineers and Non-engineers](https://mlopstalks.com/talks/packaging-mlops-tech-neatly-for-engineers-and-non-engineers) (Jukka Remes, Haaga-Helia University of Applied Sciences, 8wave AI, 55:31)
- [MLOps at Volvo Cars](https://mlopstalks.com/talks/mlops-at-volvo-cars) (Leonard Aukea, Volvo Cars, 57:31)