# Harnessing MLOps in Finance

Michelle Marie Conway, Lloyds Banking Group | MLOps Podcast | Episode 174 | 1:05:16
Hosted by Stephen Batifol

Source: https://www.youtube.com/watch?v=nIEld_Q6L-0
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/harnessing-mlops-in-finance
Published: 2023-09-05
Tags: deployment, finance, security, testing

## TL;DR
- Michelle Marie Conway moved from statistics into data science and machine learning engineering by learning Python fundamentals and software engineering practices.
- Lloyds Banking Group is moving much of its on-premises data and model infrastructure to Google Cloud under strict security, access, validation, and cost controls.
- Michelle argues that early-career engineers need to study documentation and legacy code, while workplaces need allies who make room for women in technical teams.

## Summary
Michelle Marie Conway describes her move from mathematical statistics into data science and machine learning engineering at Lloyds Banking Group. She explains why Python fundamentals, modular code, version control, and readable repositories matter when statistical models become production systems. The conversation then turns to Lloyds' large migration from on-premises infrastructure to Google Cloud. The work includes moving data, rebuilding models, running old and new systems in parallel, and checking that both systems receive the same inputs and produce consistent outputs. Security controls are strict because the bank handles customers' financial information. Michelle also discusses model validation, pandemic-related data changes, snapshots, and cooperation between data scientists, data engineers, database administrators, and business stakeholders. She sees internal language models as a possible way to search protected documentation. On gender diversity, she recommends encouraging girls into STEM and asking colleagues to intervene when women are interrupted or overlooked in meetings.

## Key ideas
### Python fundamentals made the move from statistics into engineering possible
[06:43](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=403s)
Michelle Marie Conway studied mathematical science in Dublin and later completed a part-time master's in data science. Her early work focused on statistical and analytical tasks, including production models built with SAS. Learning Python exposed her to object-oriented programming, pipelines, and the need to build more of the system herself. She says the basic Python standard library was the turning point. Understanding loops, nested lists, nested dictionaries, modular code, and how to read a repository helped her work inside large production codebases rather than treating installed packages as a complete solution.

### Production code needs structure before a notebook becomes a tangled system
[10:32](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=632s)
Michelle describes sending a notebook to a machine learning engineer after putting data processing, versioning, and several models into one large file. The response was that the notebook needed tidying up. She now keeps notebooks for plots, data-frame outputs, and exploration, while putting reusable work into modular code with GitHub and version control. She follows the DRY principle, avoids hardcoding variables into functions, and keeps configuration in one place. She teaches these habits to junior data scientists in small groups and uses mob programming so people can help each other when they get stuck.

### Lloyds' cloud migration has to move data, models, and controls together
[14:30](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=870s)
Michelle explains that Lloyds Banking Group brings together around 16 brands and 70,000 colleagues, with much of its data infrastructure built on premises. The bank is moving its data to Google Cloud, with other providers also involved, and has publicly set aside three billion pounds for the wider work. The environments are restricted and scanned through security controls rather than being open cloud accounts. The migration is happening through releases because the bank has to protect live production models, move the data feeding them, and create usable environments without allowing uncontrolled access or cloud spending.

### Parallel model runs require identical inputs and careful reconciliation
[19:37](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=1177s)
During the migration, Lloyds will need to run on-premises and cloud versions of live models at the same time. Michelle says the teams must check that both systems receive the same data and compare the outputs. Small input differences could create different results, especially for models that continue to train and change over time. She expects the parallel period to last around two to three months once it is active, after which the legacy system can be retired. The work involves debugging upstream data handling as well as the model code itself.

### Security controls govern packages, data access, and cloud use
[21:17](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=1277s)
Michelle says Lloyds does not install Python packages directly from remote websites. Packages go through internal clearance and are stored in the bank's own repository, which reduces the risk of malicious code or typosquatted packages. Data access is monitored as well. Queries against the bank's Teradata warehouse can generate alerts, with extra scrutiny when someone retrieves customer records. Access is documented, reviewed, and removed when managers do not renew it. Michelle accepts that these controls add friction because they protect financial data and reduce the chance of harmful changes leaving the organization.

### Model validation should identify the failing component and the reason
[32:33](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=1953s)
Michelle gives an example of validation checks that test whether model features are statistically significant. If a p-value crosses the configured threshold, the pipeline stops instead of producing a forecast. The user interface identifies the specific component, model, and reason for the failure. She also describes how the pandemic changed customer spending and savings patterns, causing many model validations to fail. The team applied COVID exclusion periods because those observations did not describe normal behavior. The checks exposed the problem, and the data treatment had to be changed rather than ignored.

### Data scientists and data engineers need to debug failures together
[37:05](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=2225s)
Michelle rejects a finger-pointing relationship between upstream data teams and model teams. In the groups she has worked with, data scientists and data engineers operate together so they can understand both the data and the model. Database administrators can still change data outside those teams' control, which makes distribution checks and data snapshots useful. One model snapshots its data every month, allowing the team to keep operating with the previous version when a new extract causes trouble. This gives the team a way to inspect what changed and roll back the data used by the model.

### New engineers need to understand both current tools and old code
[42:00](https://www.youtube.com/watch?v=nIEld_Q6L-0&t=2520s)
Michelle says technology now changes faster than when she entered the industry, so engineers cannot stop learning after university or a master's degree. She advises newcomers to read the latest package documentation and also learn how older code works. A codebase may outlive the career of the person who wrote it, and older Python versions may lack methods available today. She teaches junior colleagues the standard library before moving to pandas and NumPy, because the documentation explains how packages are intended to work. She finds ChatGPT useful for writing and basic training material, but says it lacks the depth of developer documentation for technical work.

## Notable quotes
- Michelle Marie Conway: "You need to know how to apply it and with code new methods and tips and tricks are constantly coming out and new releases you need to roll with those." (42:40)
- Michelle Marie Conway: "Some people's code that they've written can last longer than their careers and they don't realize that." (43:36)
- Michelle Marie Conway: "It has to be what we need, it has to pass all the risk, it has to be protected of customer data." (26:51)
- Michelle Marie Conway: "If people are kind, courteous and helpful, it shouldn't matter what your gender is." (56:41)

## Tools & references mentioned
- Lloyds Banking Group
- Google Cloud
- SAS
- Python
- pandas
- NumPy
- statsmodels
- scikit-learn
- PyTorch
- GitHub
- GitHub Enterprise
- Nexus
- Hugging Face
- Teradata
- BigQuery
- Domino Data Lab
- ChatGPT
- Women in Data
- Financial Conduct Authority

## Who should watch
- You are moving statistical or analytical models into production and need practical advice on code structure, validation, and debugging.
- Your bank or other regulated company is moving from on-premises systems to cloud infrastructure while live models continue to run.
- You manage or mentor early-career data scientists and want concrete guidance on documentation, legacy code, collaboration, and inclusion.

## Editor's note

Michelle Marie Conway says Lloyds must run on-premises and cloud versions of live models together and compare their inputs and outputs. ZenML records each run's steps, inputs, outputs, and code version, so teams can trace a model or artifact back to the data and code that produced it. That record can support the reconciliation work Conway describes during migration.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [MLOps - The Blind Men and the Elephant](https://mlopstalks.com/talks/mlops-the-blind-men-and-the-elephant) (Saurav Chakravorty, Brillo, 55:02)
- [Organisational Challenges of MLOps](https://mlopstalks.com/talks/organisational-challenges-of-mlops) (Adam Sroka, Origami Energy, 54:29)
- [Operationalize Machine Learning at Scale with MLOps](https://mlopstalks.com/talks/operationalize-machine-learning-at-scale-with-mlops) (Christopher Bergh, DataKitchen, 57:50)
- [Enterprise Security and Governance MLOps](https://mlopstalks.com/talks/enterprise-security-and-governance-mlops) (Diego Oppenheimer, Algorithmia, 53:24)