Connie Yang says translating a business or engineering question into a precise data science problem is the hardest and most important step in the work.
2
Pallet combines manual heuristics with machine learning to classify and label job postings, rather than using a model for cases that simple rules can handle.
3
Pallet deployed Casper as a cloud-native API on Google Cloud Run, which increased the number of labeled job posts by 1,000% per day and by 600% on its 30-day moving average.
Summary
Connie Yang explains how data science fits into a product that already has business, engineering, and operational constraints. She focuses on the work after model selection: translating a product question into a data science problem, deploying the result, monitoring predictions, and maintaining the pipeline. At Pallet, her team built Casper, a multi-label, multi-class classifier that labels job postings for community job boards. The system combines YAML-based heuristics with machine learning, exposes predictions through a Flask API, and runs in a Docker container on Google Cloud Run. Connie describes how probability thresholds, manual review, and a performance dashboard help catch errors. She also discusses the difference between core, product, and hybrid data science teams, and why startup data scientists often need to work across the full pipeline. Her account is practical about the work after a proof of concept, including retraining, interpretability, quality control, and CI/CD.
Translating the product problem determines whether the model will matter
Connie says effective data science starts with two questions: what is the problem, and how can it be translated into a data science problem? She calls this translation the most difficult step, because an inaccurate formulation makes everything downstream less useful to the product or engineering team. Her approach is to assess the available data and design a flexible method that can generalize to the desired outcome. She spends less time on feature engineering and model tuning in this talk. Her focus is how a business or engineering need becomes a deployed system that affects the product.
Pallet uses classification to automate job labeling for community boards
Pallet needed a repository of jobs that were labeled according to its internal classification system so it could source suitable jobs for each community. Community curators understand their members, but recruiting is not their main responsibility. Pallet therefore built a multi-label, multi-class classifier to automate much of the labeling work. Curators receive a review pile of jobs, such as a posting classified as senior machine learning, and can accept suitable jobs into their board. The model handles text from job titles and descriptions, then returns categories and predicted probabilities.
Simple rules should handle obvious cases before machine learning is used
Connie deliberately combines heuristics with machine learning. If a scraped posting says software engineer, she does not want a model to rediscover that obvious classification. The manual rules live in a YAML file, where conditions map directly to labels. The model handles data that the rules do not recognize or data points the system has not seen before. She describes this combination as more effective than relying on machine learning for every case. The models were trained for text data, and the selected model was chosen for performance, interpretability, and its ability to return prediction probabilities.
Deployment made the labeling process useful to the product
Casper began as a proof of concept and was then packaged as a Flask API with a predict endpoint. The team sent post requests to the endpoint, received classification results, Dockerized the service, and put it behind a production server. They deployed the containerized API on Google Cloud Run, which was Pallet's main cloud environment. The Pallet backend calls the API to label and classify job postings automatically. This made the model part of the job-board workflow rather than an isolated research experiment.
Automation changed the volume of labeled jobs immediately
Before Casper, outsourced workers and people on the operations team labeled job posts manually. The maximum output was about 150 labeled posts per day, and the 30-day moving average reached about 70 posts. Connie says the deployment produced an immediate spike, with a 1,000% increase per day and a 600% increase in the 30-day moving average. She uses this comparison to show the product effect of automating repetitive work. The evaluation is about the labeling operation as a whole, rather than only the model's offline score.
Production monitoring needs probability thresholds and human checks
Pallet tracks the probability attached to each prediction and watches the distribution of confidence levels. Connie initially considered a 95% threshold for a true-positive prediction, then described looking at lower-confidence points for review. A person checks examples below a threshold with basic heuristics and matching, and the team investigates why a job was misclassified. The production system currently uses a dashboard that links to predictions below the threshold. Connie calls this approach rudimentary, but says it fits a model with high enough class-level performance and keeps the process interpretable.
A deployed model still needs retraining and maintenance
Connie says the work continues after deployment. Her team monitors quality, handles misclassified examples, maintains the pipeline, and decides when to retrain. Casper is retrained about once every two months. She is doing interpretability and quality-control work ad hoc, and says tools for tracking interpretability at scale would help. Her description is honest about the operational burden: a model can be finished as a proof of concept while the team still has to check that it behaves as expected over time.
Data science roles differ by how teams are organized
Connie describes core, product, and hybrid data science teams. On a core team, other data scientists are the main colleagues and the work is consulted into product groups. On a product team, the data scientist may be the only specialist among engineers and managers, with responsibility for applying data science to that product. In a hybrid structure, the scientist has data science peers but spends enough time in product work that performance is tied to the product's results. She says the broad pipeline is too wide and deep for most people to become experts in every part of it.
Startups require repeated MVPs and attention to production constraints
At a startup, Connie has had to cover business understanding, problem translation, research, deployment, monitoring, and maintenance. She recommends building an MVP quickly, even when it is imperfect, because the first version reveals whether the team chose the right question and metrics. She often goes through four or five MVPs, depending on the problem. Once the idea is clearer, the work shifts to scalability, compute needs, latency, and monitoring. Pallet's small team also uses CircleCI so changes merged to master rebuild and automatically deploy the service on Cloud Run.
"If something is very obvious, like for example if we see a scraped job that says software engineer, then we don't need a machine learning model to tell us, hey, classify this as a software engineer."Connie Yang20:25
Who should watch
You are turning a data or business request into an ML project and need a concrete way to define the problem before choosing a model.
Your classifier is already in production and you need ideas for confidence thresholds, manual review, retraining, and error analysis.
You are deciding between a core, product, or hybrid data science role, or you work at a small company where one person owns much of the pipeline.
