The Engineering Labs were designed as a Kaggle-style format for machine learning infrastructure, with teams building products around required tools.
2
Team 1 built an automated pipeline that trained a PyTorch model with MLflow, registered the model, packaged it with TorchServe, and deployed it to Google Cloud Run.
3
The team found MLflow useful for experiment tracking and model registry work, while larger workflows needed additional tools and more careful dependency management.
Summary
Demetrios Brinkmann speaks with Team 1 from the first MLOps Engineering Labs. The open-ended project required MLflow and PyTorch, leaving the team to choose a problem and design the rest. Alexey Naiden describes how the group selected a news classification project from an MLflow example repository. John Savage explains how the team moved from source code and pull requests to automated training, MLflow tracking, model registration, Docker packaging, TorchServe, and Google Cloud Run deployment. GitHub Actions coordinated the workflow. Michel Vasconcelos created an architecture diagram that gave the team a shared language, then evolved as the design moved toward a more cloud-native setup. The group found MLflow strong for experiments and registry work, but less complete for large, multi-stage systems. They also spent substantial time tracing serving errors to mismatched Python versions. Each participant describes gaining practical experience with parts of the MLOps pipeline that had previously been unfamiliar.
The Engineering Labs let teams build machine learning infrastructure with very few constraints
Demetrios Brinkmann describes the Engineering Labs as a Kaggle-style format for MLOps. Teams form, choose a project, and create products around machine learning infrastructure. The first lab kept the requirements open, with only MLflow and PyTorch mandated. Four teams started, but only two finished. Team 1 completed its project before February 1 after working for several weeks. The format gave participants room to try tools and make their own architectural choices, although the lack of direction made choosing a project difficult for some teams.
The team chose a news classification project after deciding what was feasible
Alexey Naiden says the group began in a Slack room without assigned leadership or detailed instructions. John Savage informally led the early discussion by organizing a call. The team considered which models and MLflow and PyTorch integrations they could address within the available time. They chose a news classification example from the MLflow repository, where a model assigns news excerpts to categories such as business or entertainment. The existing example gave them a workable starting point that could become a complete product during the lab.
A shared architecture diagram helped four unfamiliar engineers work in the same way
John Savage says the team first discussed what it wanted to learn, with automation, Dockerization, and cloud deployment receiving most of the attention. After exploring the project and tools, they agreed on an architecture and several workflows, including training and deployment. Michel Vasconcelos then created a diagram that made the system concrete. John says the diagram helped the group see which parts needed changing and gave them a common language when problems appeared. Michel created it because software architecture was part of his work and because a visual design helped bridge language and background differences within the team.
The architecture evolved from separate servers toward a more cloud-native design
Michel Vasconcelos explains that the diagram was treated as a living design. The first version imagined separate servers for training, serving, and control. During the project, the team removed those elements and moved toward a more cloud-native architecture. The diagram also began from a traditional software delivery perspective, then changed through discussions with John Savage, Alexey Naiden, and Varuna Jayasiri. Those conversations helped Michel understand differences between delivering a conventional software product and building, training, and deploying a machine learning component.
The completed pipeline connected source changes to training, registration, serving, and deployment
John Savage describes a workflow that starts in the source repository after experimentation is complete. A reviewed pull request merged into the master branch triggers a full training run on Google Cloud. MLflow tracks metrics and parameters and stores model artifacts in its model registry. The pipeline then builds a Docker image, retrieves the registered model, registers it with TorchServe, and creates an image ready for inference. The image is deployed to a new Google Cloud Run endpoint, where acceptance tests run before a release sends it to the production endpoint.
GitHub Actions automated the infrastructure steps and shut down temporary compute
Alexey Naiden explains that GitHub Actions allowed repository events such as commits, releases, tags, merges, and pull requests to trigger workflows described in YAML. A commit to the master branch started the process. The action built and pushed images, started a Google Cloud virtual machine for MLflow training, ran the training, stopped the machine afterward, built the serving container, pushed it to the cloud registry, and deployed it to Google Cloud Run. The workflow issued commands against the team's Google Cloud account and cleaned up after itself.
MLflow worked well for experiments and registry tasks, but did not cover the whole system
Michel Vasconcelos says MLflow fit the individual experimentation activity well, especially when a data scientist works alone on models and experiments. He became less convinced when the project involved scaling, containerizing, and deploying several connected pipelines. John Savage agrees that MLflow can remain one component in a larger architecture, alongside tools such as Kubeflow or Airflow. The team left open whether a mixed setup is better than a more integrated platform. Alexey Naiden calls MLflow a useful building block rather than a tool that handles every part of the workflow.
Dependency mismatches created errors that were hard to diagnose
Alexey Naiden says deployment commands gave little information while images or models were being prepared, which made failures difficult to understand. The team spent two or three weeks investigating cryptic errors during model deserialization. John Savage eventually found that training and serving used different Python versions, including Python 3.7 and Python 3.8. Varuna Jayasiri adds that serialized models and package changes contributed to the problem. Alexey recommends a shared baseline for training and serving environments, with fixed versions for libraries and command-line tools because machine learning APIs and behaviors change quickly.
The lab gave participants a practical anchor for learning MLOps
John Savage says the team worked unusually well across Zoom, Slack, and GitHub, and that the project gave him something concrete to understand within the wider MLOps ecosystem. Varuna Jayasiri gained experience with Docker, cloud systems, and GitHub Actions that tutorials had not provided. Alexey Naiden learned how researchers and engineers use training, configuration, registry, and serving tools in a real project. Michel Vasconcelos added Terraform and Ansible code so others could recreate the Google Cloud infrastructure. The participants saw the repository as a checkpoint that could support later work on other models, monitoring, or feedback loops.
"It was really nice to see that in a month we started with one perspective, one basic line, and we evolved that to another that I think that's much more modern."Michel Vasconcelos21:11
Who should watch
You are designing a small training and serving pipeline and want to see how a team connected MLflow, PyTorch, GitHub Actions, and Google Cloud Run.
You are deciding whether MLflow can cover your whole workflow or should sit alongside an orchestrator and other infrastructure tools.
You want practical examples of how architecture diagrams, shared environments, and version pinning can reduce confusion in a mixed-experience engineering team.
