Meetup

Operationalize Machine Learning at Scale with MLOps

Christopher Bergh, DataKitchenEpisode 64 · 57:50 · May 2021 · 408 viewsHosted by Demetrios Brinkmann
Thumbnail for Operationalize Machine Learning at Scale with MLOps Watch on YouTube
TL;DR
  1. 1

    MLOps applies engineering principles to the whole system around a model, including source control, testing, infrastructure, automation, deployment, and monitoring.

  2. 2

    Teams should own the customer result instead of stopping at the model, data transformation, or visualization they personally built.

  3. 3

    Small releases, automated tests, and fast feedback help teams learn what customers need before they spend weeks building the wrong system.

Summary

Christopher Bergh argues that machine learning work needs a factory around it. The model is only one part of a chain that includes data providers, transformations, infrastructure, testing, deployment, monitoring, visualization, and governance. He recommends versioning source code, testing the whole system in development, monitoring it in production, scripting infrastructure, and automating work that would otherwise require clicking buttons. Teams should own the result delivered to the customer, rather than treating each component as someone else's responsibility. Bergh also argues for rapid feedback. A simple model tested with a customer can teach a team more than six weeks spent building a sophisticated model in isolation. He is skeptical that one cloud suite or one abstraction will replace varied toolchains, especially in companies with legacy systems. His practical starting point is to review operational problems regularly, fix one with an automated script, and write tests that run in development and production.

Key ideas
07:59

MLOps and DataOps use the same engineering principles as other complex systems

Bergh says the label matters less than the shared problem: a team is working on something technically complicated and needs a system that helps it succeed. He compares model and data work with software engineering, car factories, lean manufacturing, and the Toyota Production System. The shared system includes deployment, monitoring, testing, and environment management. He also says honesty matters because problems in one part of the chain can affect everyone else. The model or data transformation is the interesting output, but the factory around it must be built to produce that output reliably.

14:06

A model needs version control, testing, infrastructure, and monitoring

Bergh describes the model-building artifact as code and parameters. His first recommendation is to put that work in Git and version it. In production, teams should monitor the model and check whether it remains correct, since the model can go out of tune, input data can be wrong, or a data transformation can break. During development, teams should check accuracy and run regression and functional tests. Infrastructure should also be scriptable, testable, and deployable. Work should move from development to production through automation rather than manual button clicks.

16:16

Teams should own the delivered result instead of only their component

Bergh warns against the attitude of saying, "I did the data transformation" or "I did the model" and treating the work as finished. A team should focus on the value delivered to the customer and own the whole chain. That requires checking how a change affects other parts of the system. Demetrios Brinkmann connects this idea with involving data scientists in the wider process, while Bergh describes the cultural shift as moving from task focus to customer-value focus. He also says rapid feedback makes it possible to discover wrong assumptions sooner, before a team spends weeks on work that customers do not want.

22:06

Simple models and early customer feedback can beat extended isolated work

Bergh describes the temptation to spend six weeks building a polished model before showing it to anyone. The customer may then want only part of the result and request changes the team never considered. His alternative is to deliver smaller pieces quickly. A basic regression built in four hours and shown to a customer can be more useful than an elaborate ensemble model built in isolation. He learned this approach at NASA, where the team ran a simulation every week, brought in air traffic controllers, and used their feedback to adjust the algorithms. The cycle helped the team learn how wrong or right it was.

37:36

Teams need a shared technical view of the whole data value chain

Bergh says more meetings will not solve the coordination problem between data engineers, data scientists, machine learning engineers, visualization teams, governance, and IT. They need a shared abstraction that shows how their work relates. If a data engineer changes a source that feeds both a model and a dashboard, the engineer should be able to run tests and see those effects from their own environment. Bergh calls this pulling the pain forward. The goal is to detect the consequences of a local change before a later meeting or handoff reveals that something is broken.

40:18

Automation and testing can make contributors replaceable in a good way

Bergh says a well-functioning team should let a recent computer science graduate change a model and deploy it with a high probability that the system will work. That requires investment in the system around the work, as software engineering teams do when they aim to get a new hire's first bug fix into production quickly. He argues that making individual contributors replaceable is healthy because they can move on to new work instead of remaining tied to systems they built years earlier. Without this structure, people become responsible for old work and spend their time in handoffs, meetings, and reactive fixes.

44:30

Tests should run through the value chain in development and production

Bergh recommends treating tests as code artifacts that are automated. He prefers a functional style in which the same input data can pass through the system and produce a comparable result. Tests can include row counts, line counts, and statistical controls, although teams cannot test everything. He suggests spending at least 10 percent of data engineering or data science time writing automated tests. Some development tests can also run in production. The difference is that development has fixed data and changing code, while production has fixed code and changing data.

53:15

Legacy teams can start MLOps with a recurring problem list and automated fixes

For teams working in legacy systems, Bergh recommends writing down every operational problem, large or small. Once a month, the team should review the list, choose one problem to fix, and solve it with an automated script so it does not recur in the same way. He also recommends writing tests that run during development and in production. These steps do not require a special tool. Bergh says teams need to reserve about 15 percent of their time for operational work, rather than spending nearly all their effort processing incoming requests and reacting to failures.

"Source code, make sure you test in development, but test the whole thing, the whole system, and then in production monitor it and check it's right."Christopher Bergh14:53
Who should watch
  • You are responsible for getting models or analytics into production, but failures in upstream data, infrastructure, or downstream reporting keep becoming your problem.
  • Your team spends weeks building models before customers see them, then discovers that the requested result was different.
  • You work in an enterprise with legacy systems and want a practical way to introduce testing, automation, and operational ownership without replacing the whole toolchain.