# Applications of Data Science

Connie Yang, Pallet | MLOps Meetup | Episode 95 | 50:35
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=rc8zLY15WZU
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/applications-of-data-science
Published: 2022-03-14
Tags: deployment, human-in-the-loop, model-serving, monitoring

## TL;DR
- Connie Yang says translating a business or engineering question into a precise data science problem is the hardest and most important step in the work.
- Pallet combines manual heuristics with machine learning to classify and label job postings, rather than using a model for cases that simple rules can handle.
- Pallet deployed Casper as a cloud-native API on Google Cloud Run, which increased the number of labeled job posts by 1,000% per day and by 600% on its 30-day moving average.

## Summary
Connie Yang explains how data science fits into a product that already has business, engineering, and operational constraints. She focuses on the work after model selection: translating a product question into a data science problem, deploying the result, monitoring predictions, and maintaining the pipeline. At Pallet, her team built Casper, a multi-label, multi-class classifier that labels job postings for community job boards. The system combines YAML-based heuristics with machine learning, exposes predictions through a Flask API, and runs in a Docker container on Google Cloud Run. Connie describes how probability thresholds, manual review, and a performance dashboard help catch errors. She also discusses the difference between core, product, and hybrid data science teams, and why startup data scientists often need to work across the full pipeline. Her account is practical about the work after a proof of concept, including retraining, interpretability, quality control, and CI/CD.

## Key ideas
### Translating the product problem determines whether the model will matter
[10:55](https://www.youtube.com/watch?v=rc8zLY15WZU&t=655s)
Connie says effective data science starts with two questions: what is the problem, and how can it be translated into a data science problem? She calls this translation the most difficult step, because an inaccurate formulation makes everything downstream less useful to the product or engineering team. Her approach is to assess the available data and design a flexible method that can generalize to the desired outcome. She spends less time on feature engineering and model tuning in this talk. Her focus is how a business or engineering need becomes a deployed system that affects the product.

### Pallet uses classification to automate job labeling for community boards
[13:58](https://www.youtube.com/watch?v=rc8zLY15WZU&t=838s)
Pallet needed a repository of jobs that were labeled according to its internal classification system so it could source suitable jobs for each community. Community curators understand their members, but recruiting is not their main responsibility. Pallet therefore built a multi-label, multi-class classifier to automate much of the labeling work. Curators receive a review pile of jobs, such as a posting classified as senior machine learning, and can accept suitable jobs into their board. The model handles text from job titles and descriptions, then returns categories and predicted probabilities.

### Simple rules should handle obvious cases before machine learning is used
[20:04](https://www.youtube.com/watch?v=rc8zLY15WZU&t=1204s)
Connie deliberately combines heuristics with machine learning. If a scraped posting says software engineer, she does not want a model to rediscover that obvious classification. The manual rules live in a YAML file, where conditions map directly to labels. The model handles data that the rules do not recognize or data points the system has not seen before. She describes this combination as more effective than relying on machine learning for every case. The models were trained for text data, and the selected model was chosen for performance, interpretability, and its ability to return prediction probabilities.

### Deployment made the labeling process useful to the product
[23:50](https://www.youtube.com/watch?v=rc8zLY15WZU&t=1430s)
Casper began as a proof of concept and was then packaged as a Flask API with a predict endpoint. The team sent post requests to the endpoint, received classification results, Dockerized the service, and put it behind a production server. They deployed the containerized API on Google Cloud Run, which was Pallet's main cloud environment. The Pallet backend calls the API to label and classify job postings automatically. This made the model part of the job-board workflow rather than an isolated research experiment.

### Automation changed the volume of labeled jobs immediately
[17:50](https://www.youtube.com/watch?v=rc8zLY15WZU&t=1070s)
Before Casper, outsourced workers and people on the operations team labeled job posts manually. The maximum output was about 150 labeled posts per day, and the 30-day moving average reached about 70 posts. Connie says the deployment produced an immediate spike, with a 1,000% increase per day and a 600% increase in the 30-day moving average. She uses this comparison to show the product effect of automating repetitive work. The evaluation is about the labeling operation as a whole, rather than only the model's offline score.

### Production monitoring needs probability thresholds and human checks
[26:01](https://www.youtube.com/watch?v=rc8zLY15WZU&t=1561s)
Pallet tracks the probability attached to each prediction and watches the distribution of confidence levels. Connie initially considered a 95% threshold for a true-positive prediction, then described looking at lower-confidence points for review. A person checks examples below a threshold with basic heuristics and matching, and the team investigates why a job was misclassified. The production system currently uses a dashboard that links to predictions below the threshold. Connie calls this approach rudimentary, but says it fits a model with high enough class-level performance and keeps the process interpretable.

### A deployed model still needs retraining and maintenance
[31:55](https://www.youtube.com/watch?v=rc8zLY15WZU&t=1915s)
Connie says the work continues after deployment. Her team monitors quality, handles misclassified examples, maintains the pipeline, and decides when to retrain. Casper is retrained about once every two months. She is doing interpretability and quality-control work ad hoc, and says tools for tracking interpretability at scale would help. Her description is honest about the operational burden: a model can be finished as a proof of concept while the team still has to check that it behaves as expected over time.

### Data science roles differ by how teams are organized
[33:17](https://www.youtube.com/watch?v=rc8zLY15WZU&t=1997s)
Connie describes core, product, and hybrid data science teams. On a core team, other data scientists are the main colleagues and the work is consulted into product groups. On a product team, the data scientist may be the only specialist among engineers and managers, with responsibility for applying data science to that product. In a hybrid structure, the scientist has data science peers but spends enough time in product work that performance is tied to the product's results. She says the broad pipeline is too wide and deep for most people to become experts in every part of it.

### Startups require repeated MVPs and attention to production constraints
[37:45](https://www.youtube.com/watch?v=rc8zLY15WZU&t=2265s)
At a startup, Connie has had to cover business understanding, problem translation, research, deployment, monitoring, and maintenance. She recommends building an MVP quickly, even when it is imperfect, because the first version reveals whether the team chose the right question and metrics. She often goes through four or five MVPs, depending on the problem. Once the idea is clearer, the work shifts to scalability, compute needs, latency, and monitoring. Pallet's small team also uses CircleCI so changes merged to master rebuild and automatically deploy the service on Cloud Run.

## Notable quotes
- Connie Yang: "The translation from that business question or engineering question to a data science problem, I would argue, is one if it's not the most important, it's definitely the top most." (12:40)
- Connie Yang: "If something is very obvious, like for example if we see a scraped job that says software engineer, then we don't need a machine learning model to tell us, hey, classify this as a software engineer." (20:25)
- Connie Yang: "This is always the challenge in productionalizing any type of machine learning models in my opinion, because there's the part that's like, okay, well everything is probabilistic." (28:01)
- Connie Yang: "Once it is deployed, how do we do monitoring and then how do we ensure, for example, how do we take care of the ones that are misclassified?" (31:55)
- Connie Yang: "If I have to make any changes, I just push to the repository and that's it." (49:46)

## Tools & references mentioned
- Pallet
- MLOps Community
- Microsoft
- George Box
- Greenhouse
- Lever
- Workable
- Flask
- Docker
- Google Cloud Run
- Google Cloud
- Postgres
- Google BigQuery
- CircleCI
- Azure

## Who should watch
- You are turning a data or business request into an ML project and need a concrete way to define the problem before choosing a model.
- Your classifier is already in production and you need ideas for confidence thresholds, manual review, retraining, and error analysis.
- You are deciding between a core, product, or hybrid data science role, or you work at a small company where one person owns much of the pipeline.

## Related talks

- [Luigi in Production](https://mlopstalks.com/talks/luigi-in-production) (Luigi Patruno, ML in Production, 47:23)
- [Practical MLOps](https://mlopstalks.com/talks/practical-mlops) (Noah Gift, Pragmatic AI Labs, 58:36)
- [Scaling Machine Learning with Data Mesh](https://mlopstalks.com/talks/scaling-machine-learning-with-data-mesh) (Shawn Kyzer, Thoughtworks, 53:54)
- [Turn Data Chaos into AI Strategy with Programmatic AI Data Development](https://mlopstalks.com/talks/turn-data-chaos-into-ai-strategy-with-programmatic-ai-data-development) (Elena Boiarskaia, Snorkel AI, 27:14)
- [How To Move From Barely Doing BI to Doing AI](https://mlopstalks.com/talks/how-to-move-from-barely-doing-bi-to-doing-ai) (Joe Reis, Ternary Data, 53:47)