Loblaw Digital built an ML and data platform to help more than 60 data science users move projects from exploratory analysis to production.
2
Moving a recommendation system from an on-premises Hadoop stack to GCP reduced its target response time from one second to less than 100 milliseconds and made recommendations available as a service.
3
Reusable templates, example projects, feedback from data scientists, and managed GCP services helped the platform team settle on a practical tooling stack.
Summary
Mefta Sadat describes how Loblaw Digital built an ML and data platform for production systems in search, recommendations, inventory, labor forecasting, and other grocery use cases. The team first built its data platform around streaming pipelines and BigQuery, then moved legacy ML projects from an on-premises Hadoop environment to GCP. A recommendation system became a Kubernetes microservice backed by Bigtable, with Cloud Composer handling scheduled pipelines, retries, and alerts. The team later evaluated open-source tools and Vertex AI, using pilot projects and data scientist feedback to shape its platform. Sadat is direct about the costs of cloud adoption, including permissions work, tool learning curves, logging mistakes, and autoscaling that can increase spending quickly. He also explains how policy tags, DLP, and Google Cloud permissions helped manage data access. For recommendation systems, he sees post-deployment feedback, web analytics, and model monitoring as areas that still need better integration.
The platform team owns the path from analysis to production
Loblaw Digital's ML and data platform team supports data scientists through the full lifecycle, from exploratory data analysis and model building to validation and production deployment. The team creates tooling and example projects, then pairs with data scientists on individual systems. Engineers help with system design and discuss which tools fit a particular problem. The goal is to make the process quicker and easier rather than leaving each data science group to build its own deployment approach. The platform group has around six to ten people, while more than 60 people across several data science teams use or interact with the platform.
Loblaw Digital built the data platform before expanding its ML platform
The data platform came first because ML projects depend on reliable access to business data. Point-of-sale and inventory data arrive through streaming systems, while other sources arrive as feeds. BigQuery is the unified data warehouse, and Airflow, Kafka, and Pub/Sub support orchestration and data movement. The platform also exposes streaming topics for other teams to consume. As Loblaw Digital moved from its data-center Hadoop environment to the cloud, the team first established this warehouse and then modernized legacy recommendation and other ML projects on top of it.
Cloud migration changed how the team managed compute
In the old data-center setup, a large Spark cluster stayed available, so application authors did not have to think much about resource efficiency or cost. In the cloud, the team could create Kubernetes or Dataproc Spark clusters when needed and shut them down afterward. Sadat says this required more attention to on-demand resource allocation and to tools that were easy to move to GCP. Examples included moving from HBase to Bigtable and from Spark to Dataproc. The platform gained business support after early projects showed measurable effects, such as labor forecasting, and the time to put another system into production fell from about six months to two or three months.
The recommendation system became a low-latency service
The legacy recommendation system used MapReduce and HBase to produce batch recommendations in the data center. During the cloud migration, Loblaw Digital wanted to improve both performance and reuse. The redesigned system used Bigtable for online recommendation storage and ran as a Kubernetes microservice. Other backend teams could call the recommendation service directly instead of reading a database, so recommendations became available to more applications. Sadat gives a target of reducing response time from about one second to less than 100 milliseconds. Cloud Composer scheduled the pipeline, supported retries, and sent alerts when runs failed. Service monitoring tracked latency and throughput.
Reusable platform components came from early operational pain
The first cloud projects exposed problems with permissions, Cloud Composer, and the learning curve for data scientists. Applications needed carefully scoped access to databases and storage, and some tools were difficult to use. The team responded by shifting toward reusable libraries, template projects, and documented examples. Seldon Core became the model serving tool instead of a custom microservice architecture. Once the team had a fixed set of tools and working examples, data scientists could start from an existing project rather than repeat the same deployment work.
Tool choices were tested through pilot projects and user feedback
Loblaw Digital ran proof-of-concept projects with tools including Feast, MLflow, Seldon Core, and Vertex AI. After Google announced Vertex AI, the team compared its managed capabilities with what they had built using open-source tools. They then asked data scientists to run one of their own projects on the platform. Feedback from these pilots shaped the platform's design. One issue with the open-source version of MLflow was the lack of role-based access control, which meant users could have the same access to experiments and could edit other people's work. Vertex AI and Google Cloud's existing permission model fit the team's need for finer-grained access.
Cloud costs require limits and project-level monitoring
Sadat warns that cloud resources can create unexpected costs when teams do not monitor them carefully. In one example, a long message produced by a readiness check created excessive logging between Kubernetes and BigQuery, which increased the bill. Autoscaling can also grow quickly. A Dataflow job for image classification could create many workers if the team does not set limits. Loblaw Digital uses billing alerts at the project level so the organization can see cost changes. Resource allocation and logging volume became part of the platform's operating discipline.
Recommendation systems need feedback after deployment
Sadat wants recommendation capabilities to become more reusable, so teams can call recommendation services without rebuilding the whole system for each use case. He also says prediction quality is only part of the problem. Teams need to know how recommendations perform in production, including engagement and the effect of new models. Loblaw Digital uses Seldon for feedback loops and model-serving patterns, along with Adobe Analytics and Snowplow data from consumer applications. Those data feed dashboards and alerts for possible model or data drift. Sadat sees a remaining gap between web analytics and ML monitoring.
"Before it would take like six months to put another system into production, but now it could be like two to three months just because we have everything in place."Mefta Sadat14:18
Who should watch
You are building an internal ML platform and need to support several data science teams with different business use cases.
Your organization is moving ML workloads from an on-premises data center to GCP and wants practical warnings about permissions, autoscaling, and cloud costs.
You are deploying recommendation systems and need to connect model serving with production feedback, web analytics, and monitoring.
