# Learning from 150 Successful ML-enabled Products at Booking.com Pablo Estevez, Booking.com | MLOps Coffee Sessions | Episode 46 | 56:32 Source: https://www.youtube.com/watch?v=7MEcm3zINDw Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/learning-from-150-successful-ml-enabled-products-at-booking-com Published: 2021-07-13 Tags: model-serving, monitoring, product-strategy ## TL;DR - Booking.com treats machine learning as a tool for meeting business and customer goals, so model metrics such as precision and recall are not the final objective. - The teams connect technical work to business outcomes through hypotheses, broad solution options, and experimentation with randomized controlled trials. - Booking.com builds shared infrastructure around simple interfaces, while allowing teams to use different modeling approaches and deployment methods. ## Summary Pablo Estevez explains how Booking.com studied about 150 successful customer-facing machine learning applications and the larger set of attempts that did not work. The company organizes work around product and customer needs, with machine learning scientists working alongside designers, developers, product managers, copywriters, lawyers, and commercial teams. Pablo argues that model quality is only a health check. Teams need a hypothesis connecting a technical change to a business outcome, followed by an experiment that can measure the effect. Booking.com's experimentation platform makes randomized controlled trials easy to run and gives teams statistical reports for deciding whether to stop, continue, or change direction. The conversation also covers monitoring, delayed feedback, response distributions, latency, model serving, and platform design. Booking.com supports diverse modeling approaches through shared deployment interfaces, including simple lookup tables. Pablo says the next steps are online learning and grouping related solutions to reduce the maintenance burden of running many models. ## Key ideas ### Machine learning is one tool inside a product business [04:01](https://www.youtube.com/watch?v=7MEcm3zINDw&t=241s) Pablo says Booking.com does not make or sell machine learning. It uses machine learning to support business goals, customers, and partners. Product teams can include designers, developers, a product manager, and a machine learning scientist. Their interaction changes the model itself. A copywriter may show that the model's output does not make sense to users. A designer may need more specific answers or ten alternatives instead of one. A developer may reject features that take too long to fetch. In fraud or cybersecurity, machine learning teams still work with lawyers and other specialists. Pablo's point is that the team composition follows the product problem. ### The paper came from reviewing successful and failed exploration [08:48](https://www.youtube.com/watch?v=7MEcm3zINDw&t=528s) Pablo describes the paper as a retrospective on several years of experimentation across Booking.com's front end. Teams started from user research and product hypotheses, then explored many possible applications of machine learning. The paper focuses on 150 successful applications, but Pablo says there were also a couple of hundred attempts that did not work. Those failures provided information too. The polished paper makes the process look organized, while the actual work was more organic. The authors looked back across this body of work to find patterns that could help practitioners get value from machine learning more effectively. ### A business hypothesis connects model work to an outcome [14:11](https://www.youtube.com/watch?v=7MEcm3zINDw&t=851s) Pablo says practitioners must align with the company's business and customer objectives before choosing a model metric. Precision or recall is not the final goal when the company is trying to help people find an accommodation or improve value for partners. The team needs to connect the technical action to that outcome through a testable hypothesis. For example, a ranking change might be based on the idea that better recommendations help users find the right option faster. The team can then test whether the ranking change affects the intended behavior. Pablo also stresses that ranking is only one possible intervention. Copy and design may address the same user decision. ### Experimentation is part of the product development culture [18:42](https://www.youtube.com/watch?v=7MEcm3zINDw&t=1122s) Booking.com trains employees in experimentation and basic statistical concepts across roles. Pablo says people routinely ask whether an idea was tested, because the company has made testing easy and has a culture of using it. A product, design, or copy change can begin as a hypothesis, move into an implementation, and enter an experiment. The result informs whether the team stops, continues, or iterates. Tests can also reveal secondary effects. A change may fail on its primary metric while reducing errors or customer-service contacts, which can suggest another direction. This approach applies beyond machine learning. ### Offline metrics are health checks, while experiments measure business impact [20:49](https://www.youtube.com/watch?v=7MEcm3zINDw&t=1249s) Pablo explains that a better offline score does not automatically tell a team how much value users or the business will receive. A ten percent improvement in an error metric for a recommendation system may have an unclear effect on user decisions. Booking.com uses experiments to connect model changes to online behavior. Teams can compare models where their predictions differ, separate prediction quality from latency, and test whether a more complex model is worth its cost. Monitoring then uses proxies such as feature drift, concept drift, response distributions, and request latency. These signals do not prove business success, but they can show that the live system has moved away from the conditions used in the original test. ### Shared infrastructure supports different models through simple interfaces [42:16](https://www.youtube.com/watch?v=7MEcm3zINDw&t=2536s) Booking.com moved from localized deployment for individual teams toward common infrastructure for serving and monitoring models. Pablo gives lookup tables as an early example. A model can precompute its possible outputs and store them in a table, allowing teams to use different modeling methods and programming languages while the serving layer queries a stable interface. The platform also supports linear models, TensorFlow runtimes, and Python script runners, although Pablo says script runners receive more cautious treatment in production because they have fewer safeguards. The platform prioritizes the number and variety of use cases it can support, rather than forcing every team into one modeling framework. ### Online learning and maintainability extend the lessons from the paper [52:16](https://www.youtube.com/watch?v=7MEcm3zINDw&t=3136s) Pablo says two areas have developed since the paper. Online learning and continuous retraining can maintain a connection between model behavior and the business value the model is meant to produce. The second is maintainability. Running 150 successful solutions in parallel creates a significant operational burden. Booking.com is trying to recognize when several business problems are instances of a common problem and package them into shared solutions. This can improve monitoring, scalability, and maintenance while creating a stronger reason to invest in the shared system. Pablo connects this effort to the platform's broader goal of supporting many use cases without making the deployment layer unstable. ## Notable quotes - "Machine learning is one of the tools we have to achieve our business goals and to support our partners and support our customers." (04:01) - "Your end goal cannot be precision or recall or anything like that." (14:26) - "The concept, the overall concept, is actually a scientific concept which is hypothesis testing." (15:15) - "We try to provide users tools that allow them to be diverse on how they approach some problems." (43:57) - "It is not so nice to maintain 150 things in parallel." (52:47) ## Tools & references mentioned - Booking.com - Cassandra - TensorFlow - Python - Java - Kubernetes ## Who should watch - You are building customer-facing ML systems and need a practical way to connect model changes to product outcomes. - Your team has offline metrics and monitoring in place, but cannot explain how they relate to user behavior or business value. - You are deciding whether to build shared ML infrastructure and want to support different modeling approaches without making production operations unmanageable. ## Related talks - [MLOps at GetYourGuide](https://mlopstalks.com/talks/mlops-at-getyourguide) (Jean Machado, Meghana Satish, Olivia Houghton & Theodore Meynard, GetYourGuide, 1:03:53) - [ProductizeML: Assisting Your Team to Better Build ML Products](https://mlopstalks.com/talks/productizeml-assisting-your-team-to-better-build-ml-products) (Adrià Romero, ProductizeML, 50:53) - [Machine Learning Education at Uber](https://mlopstalks.com/talks/machine-learning-education-at-uber) (Melissa Barr & Michael Mui, Uber, 57:43) - [A Journey in Scaling ML](https://mlopstalks.com/talks/a-journey-in-scaling-ml) (Gabriel Straub, Ocado Technology, 52:41) - [MLOps at Stripe](https://mlopstalks.com/talks/mlops-at-stripe) (Emmanuel Ameisen, Stripe, 44:36)