# TrueLayer's MLOps Pipeline

Alex Spanos, TrueLayer | MLOps Meetup | Episode 7 | 56:17
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=cUxK28ocZcw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/truelayers-mlops-pipeline
Published: 2020-04-24
Tags: data-quality, model-serving, monitoring, training-pipelines

## TL;DR
- TrueLayer moved from transaction-classification rules toward a hybrid rules and machine learning system because rules covered only part of the transaction data and became difficult to maintain.
- The pipeline runs machine learning as a Python Flask microservice on Kubernetes and AWS, with DVC tracking data, code, models, and experiment provenance.
- Alex Spanos defines the machine learning engineer as an engineer who understands the full machine learning workflow and can take a model from experimentation into production.

## Summary
Alex Spanos describes how TrueLayer introduced machine learning into its Open Banking API products. The company classifies bank transactions, identifies merchants, and adds categories and other context that applications can use. It began with rules, such as matching transaction descriptions to merchants, but those rules became hard to extend and covered only part of the traffic. User feedback created a route to supervised learning, so TrueLayer adopted a hybrid system and considered moving further toward machine learning. The production service runs as a Python Flask microservice on Kubernetes and AWS. Prometheus and Grafana monitor API behavior alongside input and prediction distributions. DVC provides provenance for code, data, models, and results, while CircleCI and GitOps automate training and deployment with a manual approval step. Alex also discusses hiring. He wants general engineering ability, an understanding of the full machine learning workflow, and people who can learn quickly in a startup.

## Key ideas
### Open Banking gives applications access to bank data through dedicated APIs
[05:55](https://www.youtube.com/watch?v=cUxK28ocZcw&t=355s)
Alex explains that PSD2 required financial institutions to open systems through APIs. Users can consent to applications fetching their transaction data. Before this, applications often had to reverse-engineer online banking login flows through screen scraping, which could break whenever a bank changed its interface. TrueLayer acts as a bridge between applications and financial institutions. Its products cover both data access and payments, although Alex focuses on the data side. Applications can use TrueLayer's transaction data to build personal finance features and other products.

### Transaction classification began with simple rules
[10:41](https://www.youtube.com/watch?v=cUxK28ocZcw&t=641s)
TrueLayer first used straightforward rules to classify transactions. A rule could inspect whether a transaction was a debit card payment and whether its description contained a string such as 'Uber'. Similar logic could identify the merchant or entity. This worked while transaction descriptions stayed stable, but descriptions differed by provider. Alex says the rules could identify up to 60% of the transactions flowing through the system. Adding more rules would create a system that was prone to change and break.

### User feedback pushed the team toward supervised learning
[13:03](https://www.youtube.com/watch?v=cUxK28ocZcw&t=783s)
The team had to choose between continually adding rules and using feedback from clients whose transactions had been classified incorrectly. Human annotation offered a way to turn that feedback into training data. Commercial pressure to improve the product and the team's desire to make its methods more sophisticated led to the change. The move was gradual. TrueLayer started with a hybrid rules and machine learning approach, and Alex says the team was still considering whether to keep the hybrid design or move to a fully machine learning-based system.

### The production model runs as a Python microservice on Kubernetes
[16:26](https://www.youtube.com/watch?v=cUxK28ocZcw&t=986s)
TrueLayer runs Kubernetes on AWS. The existing rules-based classification engine was a C# microservice, but the team chose Python for the machine learning service. The service communicates through REST APIs and uses Flask. Alex presents this as one of the first Python microservices in production for the use case. The approach let the team add machine learning without abandoning its broader microservice architecture.

### Regulation makes reproducibility and interpretability requirements
[19:11](https://www.youtube.com/watch?v=cUxK28ocZcw&t=1151s)
For financial services use cases, TrueLayer needs to reproduce the workflow behind an estimate or decision. Alex gives the example of an income estimate that could contribute to a credit decision. The team must be able to identify which model, code, dataset, and version of the process produced a result, then replay it later. The regulatory setting also creates a bias toward simpler, more interpretable algorithms. Alex says the company is reluctant to choose more complex methods when a simpler option is available.

### Monitoring covers model inputs and predictions alongside API health
[25:11](https://www.youtube.com/watch?v=cUxK28ocZcw&t=1511s)
TrueLayer uses Prometheus across its microservice stack and Grafana for dashboards. For the machine learning service, it tracks standard API measures such as latency and error rate. It also monitors the distribution of inputs over time and the frequencies of predicted categories. Alex says the team looks at these dashboards every day. He would prefer to measure model accuracy directly rather than rely on proxies, possibly through an automated human-annotation process on anonymous, unlabeled data.

### Deployment is automated, while model promotion still has a manual gate
[33:34](https://www.youtube.com/watch?v=cUxK28ocZcw&t=2014s)
The team wanted machine learning services to follow the same deployment patterns as other TrueLayer services. CircleCI builds and deploys the service. The Python machine learning workflow is annotated with DVC, and a training-and-save flow runs as part of CircleCI. That flow starts a job on a DVC cluster, fetches data from an S3 bucket, runs the workflow, and exposes its accuracy and performance for review. A person then approves the next deployment step manually. Kubernetes and a GitOps continuous deployment workflow handle the later deployment stage. Retraining below an accuracy threshold was a future idea rather than something the team had implemented.

### A machine learning engineer needs engineering skills and machine learning judgment
[31:59](https://www.youtube.com/watch?v=cUxK28ocZcw&t=1919s)
Alex describes the role as an engineer who understands machine learning. The person should handle general programming and engineering work while understanding the end-to-end workflow from machine learning prototyping to production. He contrasts data scientists, who focus on experimentation and tolerate failed hypotheses, with machine learning engineers, who must deliver a working system. In a startup, Alex values generalists who can learn quickly, work with cloud systems and databases, and keep developing through practical projects, open source, blogging, or other community activity.

## Notable quotes
- Alex Spanos: "You can either go write the new rule, you know, with whatever chaos that might entail, or you can actually leverage that feedback from the client in a loop kind of way in order to incrementally improve your system." (15:31)
- Alex Spanos: "What I really want to know is, is my model making the right call, on aggregate or not." (27:54)
- Alex Spanos: "We were looking for someone with very kind of generalist kind of programming skills that are modern and applicable to today's world." (32:47)
- Alex Spanos: "Working in a vacuum in Jupyter notebooks is not an option, I think, anymore." (41:23)
- Alex Spanos: "I think machine learning engineers are more about getting something and deploying it and putting it out to the world." (42:41)

## Tools & references mentioned
- PSD2
- TrueLayer
- AWS
- Kubernetes
- C#
- Python
- REST API
- Flask
- Prometheus
- Prometheus Flask Exporter
- Grafana
- FastAPI
- CircleCI
- DVC
- Amazon S3
- GitOps
- Jupyter notebooks
- IBM
- Black Swan Data

## Who should watch
- You are building transaction or event classification and need to decide when rules have become too difficult to maintain.
- Your regulated machine learning service needs to replay past decisions and connect deployed results to code, data, and model versions.
- You are hiring a machine learning engineer and want a practical distinction between experimentation skills and production engineering skills.

## Editor's note

Alex Spanos says TrueLayer must be able to identify which model, code, dataset, and version of the process produced a result, then replay it later. ZenML records each run's steps, inputs, outputs, and code version, so artifacts and models can be traced to the data and code that produced them. Its pipelines can run on configured infrastructure without changing the pipeline code.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Operationalize Machine Learning at Scale with MLOps](https://mlopstalks.com/talks/operationalize-machine-learning-at-scale-with-mlops) (Christopher Bergh, DataKitchen, 57:50)
- [Doing MLOps](https://mlopstalks.com/talks/doing-mlops) (Noah Gift, Pragmatic AI Labs, 1:01:22)
- [Machine Learning Operations: What Is It and Why Do We Need It?](https://mlopstalks.com/talks/machine-learning-operations-what-is-it-and-why-do-we-need-it) (Niklas Kühl, IBM and Karlsruhe Institute of Technology (KIT), 58:47)
- [End to End MLOps Basics](https://mlopstalks.com/talks/end-to-end-mlops-basics) (Raviraja Ganta, Enterpret, 57:59)
- [The Motivation for MLOps](https://mlopstalks.com/talks/the-motivation-for-mlops) (Steven Fines, CoreLogic, 56:42)