Regions Bank treats model training, scoring, and monitoring as three reusable pipelines across its machine learning work.
2
Reproducibility comes from immutable source data, versioned code and model objects, production reruns, and approval-based CI/CD workflows.
3
The team uses batch processing because its feature pipelines can take from one to 24 hours, while leaving room for low-latency services later.
Summary
Daniel Stahl describes how Regions Bank built an on-premise machine learning platform for a regulated financial institution. The platform runs on a Hadoop data lake, Cloudera Data Science Workbench, Spark, Bitbucket, and Bamboo. Rather than forcing data scientists to learn every deployment detail, the team gives them a notebook-based workflow and a command-line tool called create data product that scaffolds projects and connects them to production pipelines. Stahl organizes the system around training, scoring, and monitoring pipelines. Reproducibility depends on immutable historical data, versioned code, versioned model objects, and rerunning the full training process in production. The team chose batch scoring because preparing features can take from one to 24 hours. Stahl also explains how model-validation experience shaped the use of approvals, monitoring, and other controls. He is candid that Spark is restrictive for some use cases and that Kubernetes may eventually support more flexible container-based deployments, but the immediate priority is solving real operational pain without adding unnecessary complexity.
Regions defines data products as end-to-end applications that use data or machine learning
Stahl says the data analytics team works on data products, which are complete applications that happen to use data or machine learning. Fraud models are embedded in the fraud workflow. Other examples include recommending additional products to commercial clients and identifying customers who may leave so relationship managers can intervene. Lending, stress testing, and economic capital remain with other groups when they rely on complex mathematics rather than machine learning. The team is also moving toward more personalized consumer experiences.
The platform uses familiar interfaces on top of an on-premise Hadoop environment
Regions runs an on-premise Hadoop data lake as the base for its analytics work. Data scientists use Cloudera Data Science Workbench, which provides a Jupyter notebook and IDE interface for connecting to the big-data cluster. Spark is used for data engineering and model training. Bitbucket provides Git version control, while Bamboo handles continuous integration and continuous delivery. Stahl chose the workbench because interacting directly with on-premise Spark can have a steep learning curve.
Training, scoring, and monitoring are the three model primitives
Stahl initially compared model training with software compilation, since both turn code into an executable artifact. He later adapted the design around three data science primitives. The training pipeline combines feature engineering and label engineering with data to produce a model object. The scoring pipeline takes feature engineering code and that model object to create predictions. The monitoring pipeline compares predictions with ground truth and checks whether the model continues to perform adequately. Stahl expects these three pipelines to remain useful across batch, streaming, and microservice architectures.
Reproducibility depends on preserving data, code, packages, and model objects
Regions keeps an immutable, append-only data source in HDFS that reaches back a decade across its source systems. Code is versioned in Bitbucket, and model objects are versioned in object storage. Every production run executes the full training pipeline again rather than copying a model created during notebook exploration. The team packages Python code as a versioned library and publishes it to an internal PyPI registry, so production can install the exact version it needs. This creates provenance across the code, data, pipeline, and resulting model.
The team gives data scientists production paths without removing governance
Regions created a command-line tool named create data product. It asks for basic project details, such as the project name and whether it uses Scala or Python, then scaffolds the project and its deployment paths. Data scientists can continue working in notebooks while the platform handles much of the CI/CD setup. Code moves through development, main, and feature branches using a Git Flow variant. Approvals are required before changes enter the relevant branches, and releases reach production through the existing pipeline.
Batch scoring is simpler when feature preparation takes hours
Stahl originally expected models to run as low-latency REST microservices on Kubernetes. He built Scala wrappers for Spark models serialized with MLeap so they could run without a Spark instance. A conversation with a data science manager changed the direction. Feature preparation could take from one to 24 hours because source data needed extensive manipulation. A real-time design would have required a low-latency store such as Redis or Cassandra to cache features. Since most use cases did not require immediate responses, Regions chose Spark-based batch processing for training, scoring, and monitoring.
Controls come from model-risk and audit experience
Stahl connects his work to the three lines of defense used in banking. Model validation and audit focus on processes, risks, and controls that reduce those risks. In the platform, branch approvals provide a second review before code moves forward. The monitoring pipeline is treated as a first-class component because it can detect statistical changes, missing values, or unexpected categories. For fraud, ground truth arrives within days, so accuracy can be checked quickly. For longer-horizon models, data scientists define the metrics and ranges that indicate a problem.
Automation should remove real pain before adding more infrastructure
Stahl advises teams to find a painful manual process and automate the next step toward the desired architecture. Regions once spent about five hours every two weeks on manual deployments, with many people involved because data and code both had to line up. Automating the process made releases easier to run and helped data scientists adopt the approach. For teams starting this work, he recommends putting Git and infrastructure as code at the center. Regions moved from checking out repositories during execution to packaging code as versioned libraries and installing the required version during deployment.
"The three primitives that I think are really important for all of our models regardless of algorithm, regardless of architecture is a training pipeline, a scoring pipeline, and a monitoring pipeline."Daniel Stahl10:52
Who should watch
You work in a regulated company and need model controls, approvals, and reproducibility to fit into the delivery process.
Your data science team relies on notebooks and manual releases, and you want a practical path toward automated production workflows.
You are deciding between batch scoring and microservices while your feature pipelines still depend on slow or complicated source-data preparation.
