Machine learning projects often fail before modeling starts because the required training data has not been collected.
2
Teams make more progress when they build a useful system with a small pilot, then make it available for other teams through self-service.
3
Model monitoring must continue after deployment, and explainability helps teams debug models, meet regulatory needs, and build confidence in predictions.
Summary
The discussion follows an ML project from its initial idea through deployment and monitoring. Lex Beattie says teams often choose a model before they have the infrastructure or training data needed to build it. Michael Munn adds that teams need to define the business goal and evaluation metrics early, since a technically impressive model may have no useful outcome. Mike Moran describes how Skyscanner moved away from isolated work by building a hotel-focused pilot, extracting reusable systems, and helping other teams use them. The panel recommends starting with a heuristic or simpler model before adding complexity. After deployment, teams need to watch evaluation metrics, data drift, and changes in the problem. Explainability is discussed as a way to debug models, understand feature effects, support fairness work, and answer questions from regulators. The speakers also describe explainability as part of the wider ML workflow, although no single tool covers the full process.
Training data must be planned before choosing the model
Lex Beattie says one of the most common roadblocks appears before modeling. Teams decide what kind of ML they want to use, then discover that their existing infrastructure cannot collect the data needed for training. Some teams reach the project with no training data at all. Michael Munn sees the same pattern in customer engagements. A use case can be clearly defined while the required data does not exist. In that situation, the first task is to start collecting the exact data needed for the intended model. Lex also says that broad ideas need to become specific predictions with clear metrics and a stated reason for doing the work.
Shared systems are more useful than isolated ML heroics
Mike Moran describes Skyscanner's earlier ML work as an isolated effort owned by one team. The model could be deployed, but the approach depended on the people who built it and did not spread through the company. Later, people who had complained about the company's ML practices were brought together to improve them. They built and tested systems through a hotel project, where the problem was useful and manageable compared with flights. The group then extracted systems for model serving and a feature store, documented them, and focused on self-service. The result was other teams using the systems rather than one central group doing every project itself.
A business metric should decide when a model is good enough
Michael Munn recommends setting benchmarks as early as possible. Without a definition of success, an engineering team can keep iterating on a model without knowing when to move it to the next stage. Lex Beattie recommends building a heuristic or another simple solution before using ML. That baseline helps the team understand the problem and shows how much progress is possible without a complex model. If the simple approach reaches the required metric, it may be sufficient for now. A more complicated model is justified when the baseline does not meet the target. The speakers also warn that teams can waste time optimizing model complexity when other business improvements would matter more.
Post-production monitoring needs its own engineering work
Michael Munn says getting a model into production is only part of the work. Teams also need to know how long the model remains fit for purpose and when it should be revisited. Lex Beattie separates the decision to use ML from the later decision to refresh a model. In both cases, the team needs a baseline metric. After deployment, it should check whether the model continues to meet the evaluation target, monitor data drift, and watch whether the problem itself has changed. Michael describes this post-production work as potentially as large as, or larger than, the original effort to build and deploy the model.
Simple models can build trust before teams add complexity
Mike Moran explains that Skyscanner started with a simple regression approach for a ranking problem. The team had an ambitious idea of what a high-performing ranking system might look like, but the simpler model was easier to understand. Product owners could ask why one result ranked above another, and the team could point to three factors behind a specific outcome. That explanation helped build confidence. Mike says that starting immediately with a more complicated model such as XGBoost would have made those conversations harder and could have caused the project to fail earlier. The simple model also created evidence for whether more complexity was needed.
Explainability methods are difficult to use without context
Lex Beattie distinguishes between white-box models such as linear regression and black-box models such as deep neural networks. As models become more complex, the methods used to explain them can become difficult to understand themselves. She says SHAP is widely used in Google's tools, SageMaker, and its Python package, but users may not understand how the method works or what its visualizations mean. Michael Munn adds that methods such as integrated gradients and SHAP require choices about parameters, and those choices carry trade-offs. Teams need guidance that connects the dataset and problem type to suitable methods, along with the limits of each approach.
Explainability has different audiences and can mean traceability
Mike Moran says the required explanation depends on who is asking. A regulator may require an exact account of where a price came from, rather than a general description of which features influenced a prediction. In that setting, traceability matters alongside model explainability. Skyscanner only began using neural networks after simpler approaches had been tested and found insufficient, because more complex techniques carry more risk in a regulated context. Lex Beattie says explainability can also help developers debug models, check whether features matter, inspect neural network behavior, and assess fairness. The same model may need different forms of explanation for developers, product owners, and regulators.
ML products need product and engineering conversations together
Michael Munn describes a long engagement where the technical work was only one part of the challenge. The product team was used to planning ordinary software features in blocks of time, while ML work required discussion about what the model could actually do and which parts of the product should use it. Product stakeholders sometimes asked for behavior that ML could not provide directly. The team had to coordinate with UX, product, and engineering while defining the model's role. Michael describes this as an organizational lesson rather than a technical failure. The model's place in the wider product needs to be agreed as the work develops.
"If we'd gone for the fully ambitious like XGBoost or whatever immediately, then that would have been a lot harder to explain and we probably would have failed earlier because it would have looked like dangerous magic."Mike Moran19:28
Who should watch
You are starting an ML project and need to check whether the data, business goal, and evaluation metric are ready before building a model.
Your team has models in production but has not decided how to detect drift, declining performance, or the need for retraining.
You need to explain predictions to product teams, developers, customers, or regulators and want to understand why simple models may be a sensible starting point.
