Nick Masca argues that teams should start with a small end-to-end model before adding features, data, and system complexity.
2
Marks and Spencer is reducing deployment friction by measuring manual steps, deployment speed, recovery time, and team ownership.
3
His 2015 production incident exposed how undocumented data transformations, generated code, and handoffs can make a model behave differently in production.
Summary
Nick Masca discusses the organisational and technical problems he has encountered while putting machine learning into production. At Marks and Spencer, he is helping bring technology in-house and build foundations that let data scientists own more of the path to production. The team starts with small deliveries while working toward a longer-term plan for automation and faster iteration. Nick describes a 2015 model that performed well in development but failed to deliver its expected impact in production. The causes included mismatched data transformations, undocumented platform behavior, generated code, and handoffs between several teams. He recommends starting with a simple model and a small amount of data, then using results to find system problems before scaling. He also explains how KPIs can turn MLOps goals into concrete work, including reducing productionisation steps and recovery time. Organisational change remains gradual. Cross-functional teams, written standards, and clearer ownership are replacing older waterfall processes.
At Marks and Spencer, Nick is working within a three-year plan while starting with small deliveries. His first assignment had a model that had spent about nine months in research and had accumulated complexity and technical debt. It needed to reach production roughly two weeks after he joined. The deadline was frantic, but getting a working system in front of customers early produced useful impact. Nick says that gaining trust and demonstrating value early can free a team to tackle more ambitious work later. His team has since paid down technical debt and built foundations for faster iteration.
MLOps KPIs need to describe the work teams must change
Nick uses measures that connect strategy to day-to-day action. His team tracks the time from committing code to running it in production, the number of manual productionisation steps, and mean time to recover from failures. The initial process had about eight manual steps and had been reduced by roughly half, with a longer-term goal of one review step. The operations team had previously deployed only two days each week and had moved to daily deployments. Nick also discusses traffic-lighting changes, so low-risk deployments need little review while higher-risk changes receive more scrutiny.
Handoffs can make recovery slower even after the fix is ready
Data scientists at Marks and Spencer still depend on an operations team for production deployment. Nick gives an example where a problem was found at 3 a.m. on a Saturday, diagnosed by 9:30 a.m., and fixed by 10 a.m., yet the team could not deploy because another group was unavailable until Monday. That dependency increased recovery time from hours to days. He wants data scientists to own more of the end-to-end service while retaining appropriate review for risky changes. The problem is partly technical, but much of it comes from team structure and permissions.
The 2015 model failed because development and production handled data differently
Nick's main war story concerns a near-real-time algorithm deployed around 2015. The model used more than a hundred features and several thousand parameters, and its business case looked strong. Small production traffic did not produce the expected impact. The development data came from clean analytical databases where transformations and formatting changes had already been applied. Production did not reproduce those transformations. The target platform also applied additional, undocumented transformations before receiving the data. Changes such as converting a missing value from zero to minus one could alter predictions for continuous variables. Generated code and bugs added further problems.
Small models make production problems easier to localise
Nick's strongest lesson is to get a simple system working end to end before trying to improve it. A model with hundreds of features made debugging difficult because features were correlated and data could be wrong at several stages. Some online journey fields were not populated as expected, while audited data still looked fine. A model with one or two features, or a small handful at a time, would have exposed problems much faster. The team could then have fixed system issues before scaling. Nick also says that a small feature set can often capture most of a model's performance, making extra complexity hard to justify early.
Large data pipelines add risk when they have little time margin
Nick describes another algorithm that processed a table with about a trillion records through many transformations and feature engineering steps. The pipeline took 10 to 12 hours, while the available overnight window was only about 12 or 13 hours. A failure left almost no time to recover before the morning deadline. The team found that using 1% of the data would have caused less than a 1% drop in predictive performance. They also found that most features had marginal impact and that a small number of features could have delivered much of the accuracy. The larger pipeline added time, effort, and failure risk.
Deployment friction often comes from the process around the model
Nick describes a deployment path that moved a model through three environments. Each change required a form to be completed, read by another team, and implemented in separate pieces. Forms were sometimes filled in or interpreted incorrectly, causing failures through human error. The team has since gained permission to deploy to pre-production, which removes one step. It has also introduced basic CI/CD, written standards for data platform inputs, and a traffic-light approach to deployment risk. These changes let teams do more work themselves and ask other groups for a quick review instead of scheduling work months in advance.
Organisational change happens through gradual team redesign
Marks and Spencer has traditionally used siloed functional teams and many handoffs. Nick is helping introduce cross-functional, mission-based squads with engineers, data scientists, data engineers, and product people. The change still involves permissions and political negotiations because functional groups retain ownership of some systems. He describes the process as an evolution, not a revolution. Clearer ownership matters because months of work can otherwise end with another team refusing to sign off. Nick says that seeing the impact of their work can motivate people more than working for months before handing a project to another group.
"The most important learning for me that's really been very influential throughout my career and ever since is just the importance of starting small, actually getting something to working end to end before trying to improve it."28:17
Who should watch
You are introducing MLOps practices inside a large company with established teams, permissions, and handoffs.
Your model works in development but behaves differently in production, and you need examples of where to inspect the data and process.
Your team is tempted to start with a large dataset, many features, or a complex model before proving a small end-to-end path.
