Lloyds Banking Group moved a portfolio of around 10 production machine learning models from on-premises infrastructure to Google Cloud and completed the migration.
2
A lift-and-shift move reduced one model pipeline's runtime from five hours to 20 minutes because the cloud environment provided more computing capacity and fewer infrastructure restrictions.
3
The team recommends involving production and platform engineers early, choosing maintainable libraries, standardizing infrastructure where possible, and making technical changes in small releases.
Summary
Michelle Marie Conway and Andrew Baker discuss Lloyds Banking Group's move from on-premises infrastructure to Google Cloud. Their team maintained around 10 production models and had to coordinate with platform engineers, risk partners, business users, and other teams responsible for a wider portfolio. The migration exposed differences between models, unsupported or outdated packages, Python compatibility issues, connectivity problems, and gaps in the new platform's capabilities. One pipeline went from five hours to 20 minutes after a largely unchanged move to the cloud. Conway and Baker argue that production teams should influence design from the beginning because they must maintain models after launch. They favor maintainability, explainability, simplicity, and consistency over small gains in model accuracy. Their most practical advice is to migrate incrementally, avoid large bundles of changes, test the platform with difficult enough use cases, and keep up with ongoing package and infrastructure maintenance.
The production team has moved upstream from model handover to end-to-end delivery
Andrew Baker explains that the team originally took models built by central teams and made them work in production. That created repeated handover problems around monitoring, risk, and long-term ownership. The team now gets involved from the start of an idea, through development and stakeholder review, to deployment and ongoing improvement. Baker says a project is never truly finished because models can lose performance, stakeholder interest, or business value after launch. Their production experience also gives them a strong view of what good engineering looks like across the portfolio.
Long-term maintainability matters more than squeezing out the last gain in accuracy
Baker describes rejecting a library proposed by a consultancy because it had been built by a PhD student for a thesis and had not been maintained for roughly 18 months or two years. The package might have become incompatible with a future Python upgrade, leaving the team without a clear replacement. He says the team values the longevity of a solution over having every desired feature. In practice, robustness, explainability, simplicity, and user understanding repeatedly matter more than a narrow pursuit of accuracy, provided the model still performs acceptably for the business.
Cloud infrastructure removed limits that had made model runs depend on quiet hours
Michelle Marie Conway gives a direct comparison from the migration. One pipeline took five hours on the old on-premises setup and 20 minutes after moving to the cloud without changing the model's layout. The old infrastructure was heavily restricted, difficult to configure, and sometimes usable only in the evening when fewer people were working on it. Cloud capacity allowed the team to use more nodes and computing power. Conway also reports better connectivity and fewer delays. Baker connects this scalability to the bank's desire to use more data and support more machine learning use cases without infrastructure becoming the limiting factor.
The migration required the model team and platform team to build together
The move to Google Cloud passed through cloud services, analytics and AI platform engineers, and the MLOps team. Conway says the model team sometimes had to reject or reshape platform capabilities because generic infrastructure did not meet the needs of real users. The relationship became collaborative rather than a simple customer-and-provider arrangement. Conway and Baker also had to account for cloud risk controls and the bank's regulated financial-services setting. Some platform changes could be explained once across the portfolio, while models with different designs needed separate treatment. This made production knowledge useful during platform design, not only after the platform was delivered.
Consistency is easier to achieve in the infrastructure around models than in the models themselves
The portfolio had been built by different teams, so the models varied widely. Baker compares this to asking five people to write chapters for one book and expecting a coherent result. The team uses the migration as a chance to standardize engineering patterns, templates, and reusable approaches. Conway says business problems and data sources will still differ, but the infrastructure around them can be made more consistent. She gives the example of choosing a cloud database path that creates fewer schema changes instead of changing everything at once. The team also takes a materiality-based approach when it cannot immediately re-engineer every inherited model.
Small, separate releases make platform and dependency changes easier to diagnose
Conway warns against combining a product change, data-source change, infrastructure change, and library cleanup into one large release. Removing an old product can require removing related Python configuration and code, while changing a package can affect everything that depends on it. The team therefore tries to change one area at a time and release incrementally. That makes it possible to identify which change caused a failure. Baker extends the point to routine maintenance. If package upgrades are ignored because they do not create immediate business value, several dependencies can break together and force the team to spend much longer recovering.
The migration was a coordination problem as much as a technical one
Conway says the wider organization had around 16 or 18 live machine learning models to move, while her team owned around 10. Each migration needed platform engineers who understood the infrastructure and subject matter experts who used the models every day. Models also differed greatly in size and complexity, with some having 10,000 lines of code and others 40,000. Planning required squads, cross-team coordination, and several senior managers. Conway spent much of the migration embedded with central engineering teams so she could explain where each model touched the infrastructure and help engineers understand its pipeline.
Working inside the platform team made Conway better at challenging new designs
Conway describes spending about half her time with the platform engineering squads while still leading her own team. She had to learn software engineering practices quickly and participate in technical meetings where she initially felt out of her depth. Her communication skills helped her ask engineers for focused help, while her machine learning knowledge exposed gaps in platform designs. By the end, she understood the platform closely enough to assess generative AI proof of concepts and reject designs that would not work on the existing foundation. She also helped risk partners learn the platform through pair programming, which made their reviews more informed and demanding.
"If you do it incrementally at least you'll be able to be like, right, okay, we changed just the product type here, changed the data source there, or we changed a new infrastructure thing over here."Michelle Marie Conway33:46
Who should watch
You are moving production machine learning models from on-premises infrastructure to a cloud platform and need to plan the work across model owners, platform engineers, and risk teams.
Your team receives models after development and is trying to get involved earlier so that monitoring, maintainability, compliance, and handover do not become late-stage problems.
You are maintaining a portfolio with inconsistent code and dependencies and want practical advice on incremental migrations and platform feedback.
