Lemonade brings ML engineers into the data science lifecycle early, then builds platform interfaces that let data scientists train, deploy, and monitor models themselves.
2
The platform team used automation and internal building blocks to support more than 20 data scientists with two ML engineers for several months.
3
Lemonade keeps monitoring and model testing partly manual because fully automated approaches created noisy alerts and did not give data scientists enough confidence.
Summary
Orr Shilon describes how Lemonade built an internal machine learning platform around early engineering involvement, automation, and clear ownership. Data scientists can explore data in Snowflake, reuse features from an internal feature store, translate notebook code into the company framework, train models through the Cooper Slack bot, deploy services, and configure monitoring. The platform supports more than 20 data scientists, and for several months two ML engineers supported the group. Orr is direct about the limits of automation. Automatic monitoring produced many low-value alerts, so data scientists now configure monitors for the features and models they own. Model-side testing is still manual. The hardest platform problem is designing interfaces that work for people with very different levels of engineering experience. Lemonade also uses business points in time, such as a quote, policy purchase, or claim, to create training data that matches when predictions are made.
Lemonade uses machine learning to improve both customer interactions and insurance operations
Orr describes Lemonade as a full-stack insurance carrier offering homeowners and renters, pet health, life, and car insurance in the United States and parts of Europe. Machine learning supports product improvements, such as predicting the intent behind a customer service chatbot request before it reaches a representative. It also supports business goals such as predicting customer lifetime value. These examples give the platform work a direct connection to the insurance product and to business decisions.
The platform moves a model from warehouse data to production through a shared framework
Researchers can explore raw data or prepared dimension tables in Snowflake, with hundreds of existing features available. Orr says the feature store had about 1,500 features at the time of the conversation. After modeling in a notebook, data scientists translate their training and inference code into Lemonade's internal framework. That framework lets other people train a model, supports periodic training with cloud resources, and uses the same Cooper Slack bot to deploy an online inference service. Developers at Lemonade then integrate with that service.
Lemonade recycles features while keeping feature and model monitoring separate
Lemonade has dozens of production models, each using dozens of features, so features are reused across models. The team monitors features independently from the models that use them. Checks include null values and whether a feature is equal at training and inference time. Model monitoring covers data drift, concept drift, and performance drift. Data scientists manually configure alerts for each model and feature because the team wants high precision, even when that means lower recall.
Cooper makes routine machine learning operations available through Slack
The Cooper Slack bot is part of Lemonade's broader automation culture. It can bring up or shut down a SageMaker notebook, remind people when notebooks remain active, train models with cloud resources, manage periodic training, deploy models, and take services down. Orr says the bot is easy to connect to the internal platform because Lemonade already uses Slack commands for many operational tasks. The same ownership model sends alerts to the data scientists responsible for a model.
Automation is tied to business efficiency at Lemonade
Orr says Lemonade tracks measures such as customers per employee and IFP per employee. That makes automation a company-wide business priority rather than an engineering-only effort. The platform team's leverage came from long-term investment in reusable infrastructure and automation. Orr gives a concrete example: two ML engineers supported more than 20 data scientists for several months, while the team had four people at the time of the conversation.
The hardest platform decision is designing interfaces for different kinds of data scientists
Lemonade's platform must support data scientists who have only delivered notebooks as well as people with many years of software engineering experience and strong preferences about frameworks. Orr says the main challenge is choosing and maintaining interfaces that work across this range. Lemonade moved from an open first version of its model framework to a more restricted second version that fit about 80 percent of use cases. It later returned to a model that supports both paths, with more responsibility placed on users who need custom feature queries or other choices.
Clear ownership keeps data scientists responsible for production models
Orr says Lemonade does not use a model where data scientists throw work over a wall to engineers. Data scientists write the code that runs in production and receive alerts for their models, including application errors, drift, and null features. Engineering and data science have distinct ownership, even though some areas remain gray. This arrangement also affects hiring because it attracts data scientists who want end-to-end responsibility for their work.
Business points in time make training data match prediction events
A business point in time is the event when a company usually makes a prediction. At Lemonade, examples include creating a quote, purchasing a policy, or making a claim. The platform creates training data around these events, so researchers can ask how the data looked at the moment the prediction would have been made. Orr says this approach improves training and inference consistency, makes testing easier, and reduces mistakes and repeated engineering work. He also says its usefulness depends on the product.
"People don't need that at our company. They want to know how data looked when someone purchased a policy, and that's all they want to know in most cases."48:43
Who should watch
You are building an internal ML platform and need to support data scientists with very different levels of software engineering experience.
Your team is deciding which ML workflows to automate and where people should still configure monitors or tests themselves.
You want a concrete example of how early platform investment can let a small ML engineering team support a much larger data science group.
