# MLOps: A Leader's Perspective Stephen Galsworthy, Quby | MLOps Coffee Sessions | Episode 39 | 54:08 Hosted by Demetrios Brinkmann Source: https://www.youtube.com/watch?v=LoKMLW1v4EY Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/mlops-a-leaders-perspective Published: 2021-05-18 Tags: data-quality, platform-teams, product-strategy, team-adoption ## TL;DR - Stephen Galsworthy says MLOps at scale depends on technology, people, and organizational processes, with the latter two often becoming harder after the initial platform work. - Quby moved data scientists from a central innovation function into product teams, where they worked close to business problems alongside data engineers and product owners. - The strongest way to choose ML work is to stay close to end-user value, since many problems can be solved with simple models or without machine learning at all. ## Summary Stephen Galsworthy describes MLOps from the perspective of a data leader who moved from hands-on data science into executive work at Quby. He says production is only the start. Teams also need monitoring, organizational support, shared processes, and a clear connection to customer value. Quby began with a central data science function, then moved data scientists and data engineers into product teams as its services became part of the business. Galsworthy explains how the company used internal education to teach different groups what AI meant for their work, from precision and recall for product managers to practical notebooks for developers and strategic consequences for executives. He also discusses labeled data, platform investments, prioritization, and saying no. His account is candid about the cost of the first use case and the difficulty of changing established company cultures. The practical test for an ML initiative is whether it solves a real user problem and creates measurable product value. ## Key ideas ### Production is the beginning of the operating problem [05:28](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=328s) Galsworthy says that proving accuracy in a notebook and solving a business problem are only early steps. At Quby, putting models into production exposed the work that followed, including the technology, processes, and organizational support needed to run several models as part of the business. He recalls a period when getting a data science model into production was treated as a special event. His experience changed that view. A data leader with hands-on experience can move between executive discussions and technical details, while still giving team members room to grow. ### MLOps at scale combines technology, people, and processes [07:40](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=460s) Galsworthy divides machine learning at scale into three areas: technology, people, and organizational processes. He thinks technology is less of a barrier than it was several years earlier because cloud providers and large technology companies have made many components available off the shelf. Quby worked with AWS and Databricks rather than rebuilding every platform component. That allowed data scientists and engineers to spend more time on the company's specific problems. The harder work was making the organization operate as one system and deciding where skilled engineers should spend their time. ### Data science needs to move from a center of excellence into products [10:57](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=657s) Quby initially used a central data science and engineering function to create proof of concepts, demonstrate value, and secure further funding. Once those concepts became pilots and then production services, the organizational challenge changed. Quby moved toward agile product teams, placing one or two data scientists with a data engineer inside different product groups. This put the technical staff closer to business problems. Galsworthy says the shift required support from the CEO, COO, CTO, software developers, and product managers. Everyone had to understand that retraining could change model outputs and that product choices involved trade-offs such as precision and recall. ### Labeled data can matter more than access to algorithms [17:34](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=1054s) Galsworthy warns that data generated by one team may be consumed by another team without the producer knowing it. Communication about data ownership and use therefore matters. He puts special weight on labeled data, since organizations may have access to similar technologies and compete for similar talent. At Quby, a pilot with roughly 150 to 200 households used smart plugs on major appliances. The resulting labels helped the team identify inefficient appliance use and later move from traditional random forest approaches toward deep learning. The pilot was first intended for demand-side response, but that use case was too early for the domestic market in the Netherlands. ### Different groups need different forms of AI education [21:19](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=1279s) Quby created an internal data university called Data Science 101 and opened it to the whole company, including product, finance, administration, reception, and HR. Galsworthy describes three groups that needed more focused teaching. Product managers needed to understand precision, recall, and accuracy in the context of an AI product. Developers learned by running simple Python and SQL notebooks and training a basic model themselves. Executives needed to understand how AI could affect company strategy and how its behavior differed from ordinary software. The lessons were matched to each group's decisions rather than taught as one generic technical course. ### Platform investment is easier to defend when product value appears quickly [33:03](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=1983s) Galsworthy describes a later Quby project that used smart-meter data rather than data from Quby's own thermostat. The team could build the new pipelines with fewer constraints from the existing technology stack and connect them to existing data. Quby used Databricks, MLflow, and Delta Lake-related tooling. After a senior data scientist developed a proof of concept in a notebook, the team had a new technology running about three months after receiving approval to put it into production. Galsworthy says this speed came from having the platform, a coordinated team, and disciplined processes already in place. ### Agile processes can help creative data science teams work as a system [35:57](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=2157s) Galsworthy recalls resistance when Quby first tried to introduce stand-ups and other agile practices. Some data scientists felt that process would restrict their creativity. As machine learning became part of normal business operations, those rhythms became more useful. Product owners and agile coaches helped adjust the process and improve coordination across the organization. He compares process to traffic lights: an individual driver may move more slowly, while traffic across a city can move more smoothly. The point is to improve the flow of the whole organization rather than optimize one person's work. ### End-user value should determine which ML problems get attention [46:25](https://www.youtube.com/watch?v=LoKMLW1v4EY&t=2785s) Galsworthy says the best way to avoid solving low-value problems is to stay focused on the end user. For Quby, that meant helping homeowners control heating costs and reduce electricity use. He estimates that machine learning is not the answer most of the time, and says a simple data-driven method or a non-AI approach may be enough. Product management and user-experience teams help maintain this perspective. Teams also need a process for prioritizing scarce data science capacity. In practice, that means rejecting many plausible projects, stopping work when a model is good enough, and moving toward problems that can create more customer value. ## Notable quotes - Stephen Galsworthy: "In production is not the end." (05:29) - Demetrios Brinkmann: "The difficult parts aren't really the technology parts." (07:40) - Stephen Galsworthy: "If you're always looking at how can you add value to that end user, that helps to simplify things because a lot of the time the answer is not machine learning." (46:49) - Stephen Galsworthy: "What it really comes down to is saying no to things." (50:04) ## Tools & references mentioned - MLOps Community - Quby - AWS - Databricks - Kubernetes - MLflow - Delta Lake - Amazon - Azure - Doordash - Lemonade - Luigi Petruno - Amazon.com ## Who should watch - You lead a data or engineering group that has moved beyond proof of concept and now has to make ML part of ordinary product work. - Your company is deciding whether data science should stay centralized or move into product teams. - You need a practical way to explain platform costs, labeled data, AI education, and project prioritization to business leaders. ## Related talks - [MLOps as Tool to Shape Team and Culture](https://mlopstalks.com/talks/mlops-as-tool-to-shape-team-and-culture) (Ciro Greco, Coveo, 43:02) - [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47) - [MLOps - The Blind Men and the Elephant](https://mlopstalks.com/talks/mlops-the-blind-men-and-the-elephant) (Saurav Chakravorty, Brillo, 55:02) - [Organisational Challenges of MLOps](https://mlopstalks.com/talks/organisational-challenges-of-mlops) (Adam Sroka, Origami Energy, 54:29) - [I Am Once Again Asking "What is MLOps?"](https://mlopstalks.com/talks/i-am-once-again-asking-what-is-mlops) (Oleksandr Stasyk, Synthesia, 1:06:22)