Etsy calibrates click-through-rate predictions because downsampling negative examples makes raw model scores unlike real-world probabilities.
2
Calibration matters when scores feed thresholds, other models, or business decisions, while ranking-only systems can often use uncalibrated scores.
3
Etsy evaluates calibration with calibration score, normalized cross entropy, reliability diagrams, offline ranking metrics, and online A/B tests.
Summary
Erica Greene and Seoyoon Park explain how Etsy added calibration to its machine learning workflow for ad ranking. Their click-through-rate model trained on downsampled data, which helped avoid a trivial model that predicted almost every example as negative, but made its output scores unlike true click probabilities. Calibration adds a post-processing layer trained on data with the real label distribution. The team used Platt scaling with TensorFlow and considered isotonic regression. Calibration was useful when scores were used for thresholds, as inputs to other models, or in a nonlinear ranking formula. The implementation exposed changes in ad ordering, seller charges, and displayed listings, so the team added more offline and online evaluation. They also describe a TensorFlow model-loading bug that took months to resolve, plus two production incidents involving expensive feature transformations and an overloaded ad-serving path. The conversation is candid about the engineering work around model quality.
Seoyoon Park moved from backend engineering into machine learning through hands-on work
Seoyoon Park started as a software engineer at Viacom, then worked on big-data analytics at Signifier before joining Etsy. He did not have a machine learning background when he joined a small Etsy team that touched presentation, backend, and ML work. As the team grew, he focused more on ML systems and learned through courses, experienced teammates, and direct work on production problems. Erica Greene says people moving into ML need to opt in and want to learn. Etsy supported that transition with Coursera lessons, one-on-ones, and practical assignments. Seoyoon says the combination of team support and applying course material to his real job helped him progress quickly.
Etsy's ML team gained room to work on data quality and evaluation as it grew
When Etsy's team had four engineers, the same people handled page presentation, backend issues, and ML work. Revenue pressure and urgent product problems caused frequent context switching. More headcount allowed the team to focus on ML-specific concerns such as data quality and evaluation metrics. The team also moved its ML work toward Google Cloud Platform and TensorFlow, which increased model experimentation speed. Erica describes this as a shift toward an ML engineering team, even though that term was not widely used at Etsy at the time. She asked engineers whether they wanted to learn the area, then paired formal study with hands-on work.
A move from feature-heavy models to neural networks showed that operations work could dominate model work
Erica describes Etsy's earlier models as having very large, highly engineered feature inputs, including hashed and crossed features. The team moved to neural networks and used more raw features, text embeddings, and a relatively small fully connected network. The first major win depended heavily on data flow jobs, cloud training, and pipeline wiring. Seoyoon handled much of that work. Erica estimates that the effort was roughly 80 or 90 percent DevOps and data engineering and 10 or 20 percent modeling. That experience convinced the team to invest more in neural-network-based models and the surrounding ML workflow.
Calibration converts scores from a sampled training distribution into probability-like outputs
Etsy's click-through-rate model predicts a rare event, since users see many listings and click only a small number. Training directly on such sparse data can produce a useless model that predicts zero almost everywhere, so Etsy downsampled negative examples and trained on a more balanced set. That works for ranking because only the relative order matters. It changes the meaning of the scores, though. Calibration is a later process that maps the model's output distribution toward the true probability distribution. Erica stresses that calibration is a name for this post-processing goal, rather than one specific algorithm.
Calibration matters when predictions drive decisions or become inputs to other models
Seoyoon explains that calibration is unnecessary when a model is used only to rank search results and the ordering stays the same. It matters when a score is compared with a threshold, used in a healthcare or content-moderation decision, or passed into another model. Etsy also used model scores inside a feature set, and switching from a tree-based model to a deep neural network changed the score distribution enough to affect the next model. Neural networks are not guaranteed to produce stable score distributions, even when the architecture and training data change only slightly. Erica and Seoyoon cite the paper "On Calibration of Modern Neural Networks," which discusses why modern neural networks can become overconfident.
Etsy implemented calibration as a second training step on top of a frozen model
Seoyoon describes calibration as training another layer after the base model has been trained. The base model weights are frozen, and the calibration layer learns from data that preserves the true probability distribution. Etsy used Platt scaling in TensorFlow by reloading the old model, adding a final layer, and training that layer on data without negative-class downsampling before the sigmoid output. Isotonic regression was another option. It groups predictions into buckets and assigns monotonic output values, so the item ordering does not change. The calibrated data was added to Etsy's ETL pipelines, while the difficult work came from understanding how the changed score distributions affected the final ad-ranking formula.
The team evaluated both the calibrated model and the downstream ad-ranking system
Erica says Etsy first checked AUC because the calibration function was monotonic and should preserve ranked ordering. They also measured calibration directly by comparing the rate of true clicks with the model's predicted rate. Seoyoon adds normalized cross entropy, which measures how close the output distribution is to the evaluation distribution, and reliability diagrams, which show overconfidence or underconfidence across probability bins. These metrics describe the calibrated model, but the final ad-ranking function combines multiple scores nonlinearly. Etsy therefore added offline precision and recall measures based on historical clicks and purchases, then used an A/B testing framework to compare engagement on different traffic groups.
Production model work exposed both a TensorFlow bug and expensive feature transformations
Etsy spent about four months investigating segmentation faults in TensorFlow 2.3 when saving and reloading large models on GPUs. The same issue did not occur on CPUs, and debugging was complicated by the interaction between TensorFlow, hardware, cloud infrastructure, and supporting libraries. Google eventually provided a patch. Erica also describes a separate model that produced strong offline and online results but was too expensive to serve after the team added inefficient feature transformations. The team had to isolate the problem by repeatedly removing groups of transformations. Her practical lesson is to vectorize feature transformations and treat flexible Python-based transformations as a possible source of severe serving costs.
"Calibration is not one particular algorithm, not one particular technique, it's just a name for anything that converts the distribution of scores from a model to the true distribution."Erica Greene20:53
Who should watch
You are building a ranking model whose scores later feed thresholds, features, pricing, or another model.
Your team downsampled labels and needs to understand when its model outputs can be treated as probabilities.
You maintain TensorFlow models in production and want practical examples of serving-cost and model-loading failures.
