# Better Use Cases for Text Embeddings Vincent Warmerdam, Explosion | MLOps Coffee Sessions | Episode 83 | 48:20 Source: https://www.youtube.com/watch?v=4bbLiMXmFLI Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/better-use-cases-for-text-embeddings Published: 2022-02-28 Tags: data-quality, debugging, evals ## TL;DR - Vincent Warmerdam argues that understanding the system and its data should come before choosing models or metrics. - Simple heuristics can provide useful baselines for fraud detection and text classification, especially when the relevant patterns are easy to explain. - Comparing a complex model with a simpler rule-based model can expose bad labels and reveal new patterns to investigate. ## Summary Vincent Warmerdam talks about how machine learning systems fail, why teams should understand their data before selecting models, and how simpler methods can be easier to test and maintain. He uses fraud detection to show that a model can perform well on the easy fraud cases already present in the labels while missing harder cases that were never identified. He also describes rule-based systems for detecting programming languages, such as treating "go" as a programming language only when it appears as a noun. Comparing rule-based and learned models can reveal bad labels and suggest new rules. Warmerdam discusses cultural variation in labeling, noisy public datasets, and his packages human-learn, doubtlab, and scikit-lego. He is cautious about terms such as "data-centric AI" and prefers the clearer idea of iterating on data instead of models. ## Key ideas ### Understanding failures gives teams something they can fix [06:06](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=366s) Vincent Warmerdam says he studies machine learning failures because understanding how a system breaks makes it easier to improve. Failure stories are more useful to him than polished demos because they contain details people can apply in daily work. He describes writing about a system that kept showing baby-product advertisements to someone after a stillbirth. The company's technical lead contacted him and agreed that the behavior was an unintended failure worth investigating. Warmerdam says criticism works better when it is specific and aimed at fixing a failure, rather than delivered as a rant. ### System understanding matters more than chasing an optimum [09:23](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=563s) Warmerdam connects his preference for simpler systems to his background in operations research. A factory can be optimized perfectly for a particular price of raw materials, but that solution becomes non-optimal when prices change. He says optimality is a moving target, so teams should first understand the system and make assumptions that are likely to hold. A heuristic can then provide a useful starting point and benchmark. For an online-chat problem, a few rules might be enough to detect conversations that appear to be handled by a bot. A more complex model can be added later if it proves useful. ### Rule-based systems can handle obvious cases before machine learning [12:48](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=768s) For fraud detection, Warmerdam suggests starting with rules for cases that are clearly suspicious. Someone younger than 16 earning over a million dollars a year, or a private individual with more than 20 bank accounts and five addresses across three countries, can be sent for review without training a model. Such cases may be too rare for a model to learn from. He applies the same idea to detecting programming languages in text. Since "go" is usually an English verb, a rule can identify it as the language only when its part of speech is a noun. These rules are easy to explain, debug, and maintain. ### Interactive data inspection can turn patterns into usable rules [16:02](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=962s) Warmerdam describes human-learn as a way to build rule-based systems that work with scikit-learn. In an Iris-style example, a user can draw a circle around a cluster and assign everything inside it to a class. His broader point is that inspecting data interactively can reveal common patterns that become heuristics. In fraud data, a few SQL queries might show that people with a median income and one bank account are usually safe. That observation could form part of a first model without requiring a complex algorithm. He sees some machine learning work as a user-interface problem, where better ways to inspect data may produce better rules. ### Model disagreement is a practical way to find bad labels [17:54](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=1074s) Warmerdam recommends comparing a complex model with a simpler model, such as a gradient-boosted tree or deep learning system against logistic regression or domain rules. The useful examples are where both models are generally good but disagree. Some disagreements reveal incorrect labels. Others show that the learned model found a pattern missing from the rules. In programming-language detection, the models might reveal that version numbers written near "go" are informative. Teams can add that pattern as a rule, improve the labels, retrain, and repeat the comparison. Warmerdam says this kind of data iteration has been more useful in practice than grid search. ### Labels contain cultural and specialist disagreement [21:01](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=1261s) Warmerdam warns that labeling is difficult even when errors are not obvious. Sentiment and emotion depend on culture, so people from different parts of the world may interpret the same text differently. Medical labeling can vary because specialists notice different conditions in the same image. He cites research that found about 10 percent label error in Quick Draw and about 2 percent in the Amazon sentiment dataset. This means a model that reaches state-of-the-art performance may be fitting incorrect labels. He also says data scientists are sometimes prevented from inspecting labels because their time is considered too expensive, which makes the problem harder to detect. ### Fraud metrics can hide the cases the system never learned [26:21](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=1581s) Warmerdam gives a fraud example where a model performs well, enters production, and catches several criminals. The team may conclude that the problem is solved. He says the labeled fraud cases are probably the easy cases, because they were detected in the first place. Harder fraud may remain in the supposedly non-fraud data. A model can therefore look successful while missing the cases that matter most. He suggests investigating different types of fraud in turn, using domain knowledge to decide what to examine each month. This does not remove uncertainty, but it gives the search a manageable scope. ### Constraints can express business and fairness requirements [44:50](https://www.youtube.com/watch?v=4bbLiMXmFLI&t=2690s) Warmerdam argues that machine learning systems often reduce a complicated goal to one number, while real systems have several constraints. He gives a recommender example where a user could explicitly say they do not want romantic comedies, even if their click history suggests otherwise. He connects this idea to TensorFlow Lattice, which can impose monotonicity, such as treating smoking more as a consistently higher health risk. He also mentions fairness methods that constrain the difference in true positive rates between subgroups. These constraints are technically difficult and depend on how the requirement is defined, but he considers them a useful area to explore. ## Notable quotes - "If we understand how things fail, it's also easier for us to sort of figure out how things can be fixed." (06:48) - "Don't introduce any complexity unless you need to." (09:23) - "If two models that are good disagree, something interesting is happening." (18:45) - "Bad labels are everywhere, this is a problem folks." (21:09) - "I think those kinds of sounds seem good, but you always got to be a little bit careful with industry terms." (38:29) ## Tools & references mentioned - Explosion - Rasa - spaCy - Pi Data - human-learn - doubtlab - scikit-lego - scikit-learn - Prodigy - TensorFlow Lattice - Fairlearn - Quick Draw - Amazon sentiment corpus - Mechanical Turk ## Who should watch - You are building a fraud, recommender, or text-classification system and need a way to test whether its success reflects the real problem. - Your team is tuning models while label quality, business constraints, or production behavior remain poorly understood. - You want practical examples of when rules, heuristics, and model comparison can improve a machine learning workflow. ## Related talks - [Model Blind Spot Discovery for Better Models](https://mlopstalks.com/talks/model-blind-spot-discovery-for-better-models) (Pavol Bielik, LatticeFlow & David Garnitz, VectorFlow, 55:35) - [Monitoring Unstructured Data](https://mlopstalks.com/talks/monitoring-unstructured-data) (Aparna Dhinakaran & Jason Lopatecki, Arize AI, 13:12) - [Look At Your ****ing Data 👀](https://mlopstalks.com/talks/look-at-your-ing-data) (Kenny Daniel, Hyperparam, 1:05:26) - [Evaluating and Integrating ML Models](https://mlopstalks.com/talks/evaluating-and-integrating-ml-models) (Morgan McGuire & Anish Shah, Weights & Biases, 51:57) - [Visualize - Bringing Structure to Unstructured Data](https://mlopstalks.com/talks/visualize-bringing-structure-to-unstructured-data) (Markus Stoll, Renumics, 50:39)