# LinkedIn Job Recommendations

Alexandre Patry, LinkedIn | MLOps Coffee Sessions | Episode 60 | 51:41
Hosted by Skylar Payne

Source: https://www.youtube.com/watch?v=a0Jlh2o2jj4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/linkedin-job-recommendations
Published: 2021-10-25
Tags: evals, feature-engineering, monitoring, recommender-systems

## TL;DR
- Alexandre Patry's team moved personalization out of large models and into reusable activity features, which made models faster to train, deploy, and experiment with.
- LinkedIn treated user complaints about poor job recommendations as an intake and classification process, then worked with linguists to define and measure very bad recommendations.
- The team changed its objective from maximizing applications to improving the quality of applications and eventually helping people get hired, while designing infrastructure for repeated experimentation.

## Summary
Alexandre Patry describes how LinkedIn's job recommendation team changed its models, processes, and measures after large personalized models became difficult to operate. The team moved personalization into activity features that could be shared across job search, recommendations, notifications, and recruiter products. This reduced the time to put a trained model online from one to two weeks to a few hours and removed an earlier limit on concurrent experiments. The team also created a process for investigating user complaints, classified recurring failure types, and partnered with linguists to define very bad job recommendations across product surfaces. Patry explains why application counts were an incomplete objective and how LinkedIn began modeling the quality of applications instead. He also discusses feature skew, missing tracking data, HDFS failures, faster design reviews, and the need to treat machine learning work as a sequence of experiments that produces knowledge.

## Key ideas
### Large personalized models made growth and experimentation increasingly expensive
[02:52](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=172s)
When Alexandre Patry joined LinkedIn's Talent Solutions and Careers AI work, job recommendation relied on large linear models with member- and job-specific components. Those models had to be retrained daily, and the operational cost rose as more engineers and models were added. Once the team aimed to run about a dozen models, grid workflows could take more than a day, so engineers spent substantial time keeping pipelines alive instead of testing ideas. Skylar Payne describes this as a limit on experimental bandwidth: talented engineers could only test one or two ideas at a time.

### Moving personalization into features made models smaller and reusable
[05:14](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=314s)
The team found that model size, retraining frequency, and the difficulty of moving large artifacts caused many of its problems. Personalization had been stored inside each model's weights. They refactored the system so member activity and job activity became features supplied to experimental models. This created a shared feature engineering layer for job recommendation, job search, recruiter products, and notifications. It also allowed the team to move from linear models to TensorFlow models. Patry says the change reduced online availability time from one to two weeks to a couple of hours and removed the earlier limit of roughly six to eight models running at once.

### The migration succeeded through small validations and executive support
[08:57](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=537s)
Patry says the move to lighter TensorFlow models took about a year and carried real risk. Engineers leading the transition made small gains along the way and used them to validate the direction before asking others to accept it. Some colleagues were skeptical because LinkedIn had a long history of success and infrastructure around large personalized models. The team needed support from its own leadership, especially when senior people from other organizations questioned the change. Productivity gave them a concrete argument: machine learning requires many experiments, so increasing the number of experiments engineers can run can matter more than preserving exact parity at every step.

### Machine learning experiments produce knowledge even when metrics do not improve
[14:11](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=851s)
Patry compares machine learning development to an evolutionary process. Models keep changing, so an experiment is valuable when it improves a metric or teaches the team something that informs the next experiment. He says the goal is to understand the problem better and keep iterating rather than declare the work finished after writing code. The conversation also exposes a weakness in this process: much of the team's knowledge remained tribal. Engineers remembered attempts from years earlier without documentation, and Patry had not yet found a good central repository for the details of what had worked.

### User complaints became actionable after the team created one intake path
[17:24](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=1044s)
Poor job recommendations were more sensitive than poor movie or product recommendations because users expected LinkedIn to understand their professional profile. The team received emotional complaints about what it called "face palm jobs." These failures could come from candidate selection, features, or ranking, and previously issues were passed between teams without clear ownership. Patry assigned every complaint to candidate selection as an initial lens, made one team responsible for investigating them, and personally handled the first month. A taxonomy of recurring errors then emerged, including mismatch and seniority problems, which gave the team a consistent way to discuss fixes.

### Linguists helped define bad recommendations across product surfaces
[23:30](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=1410s)
The team struggled to measure progress because dismissals were biased, feedback widgets changed with product design, and people disagreed about what counted as a bad result. LinkedIn hired linguists to refine the definition of a very bad job recommendation and evaluate the product across different surfaces. Patry explains that linguists were useful because they could develop annotation guidelines and taxonomies, then check whether judgments were consistent. This process could not label millions of examples, so the team used it as a high-quality measurement point and began considering how to extend that knowledge into scalable model indicators.

### Optimizing applications alone could concentrate demand on a few jobs
[28:26](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=1706s)
The team had previously considered maximizing the number of job applications. Patry says that objective could cause good models to keep very good jobs at the top, leading to a few jobs receiving thousands of applicants while many other jobs received too few to be successful. LinkedIn began shifting toward matching people and making hiring happen. Since the company had incomplete and noisy visibility into downstream hiring outcomes, the team trained models to distinguish good applications from bad applications using available signals. The ranking model could then use those predicted application-quality scores instead of relying only on top-funnel application counts.

### Infrastructure choices have to support several months of iteration
[35:00](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=2100s)
Patry says a model design should be judged partly by whether engineers can continue experimenting with it for the next three or four months. The team began discussing deployment friction with infrastructure partners instead of treating modeling and infrastructure as separate concerns. One problem involved generating the same features offline and online. Offline training rebuilt a snapshot of the world, while online inference depended on caches and timeouts. Missing online features and cache misses could create inconsistencies, so the team added monitoring and made models more tolerant of those incidents.

### Fast feedback and rough end-to-end prototypes help engineers learn sooner
[44:45](https://www.youtube.com/watch?v=a0Jlh2o2jj4&t=2685s)
LinkedIn revised its AI design review process so engineers considered the opportunity, prior work, success measures, and baselines rather than only implementation details. Patry wanted reviews to work more like pull request reviews, with fast feedback instead of a process that took months. For uncertain projects, he recommends gathering peer feedback and building an end-to-end version quickly, even when it is incomplete. The purpose of the first version is to find the hardest part of the problem. He also advises junior engineers to inspect individual examples alongside aggregate metrics, plan a sequence of experiments, compare alternatives, and define success as a change in the product rather than completion of the code.

## Notable quotes
- Alexandre Patry: "The goal of each experiment, why you want to lift metric, is really to gather knowledge, better understand your problem and keep pushing and keep iterating and keep learning." (15:11)
- Alexandre Patry: "Every good machine learning engineer should be able to tell a story with the data." (31:52)
- Alexandre Patry: "Our work is not yourself, so try to detach yourself from it, try to be as critical as you can." (49:35)
- Alexandre Patry: "It's okay for an experiment to not work. It's not a reflection of your failure, it's just you learn something and you move on." (49:45)

## Tools & references mentioned
- LinkedIn
- TensorFlow
- HDFS
- Microsoft

## Who should watch
- You run large personalized models whose retraining and deployment work leaves engineers with little room for experiments.
- Your recommendation system gets complaints, but ownership and definitions of bad results are unclear.
- You are changing a mature machine learning system and need to align model objectives, infrastructure, design reviews, and team learning.

## Editor's note

Alexandre Patry says LinkedIn's machine learning knowledge often remained tribal, while engineers also had to keep pipelines alive instead of testing ideas. ZenML records each run's steps, inputs, outputs, and code version, so experiments can be traced and compared after the team moves from one infrastructure setup to another. That gives repeated experimentation a durable record.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [AI Careers Insights from Ex Meta Staff Eng](https://mlopstalks.com/talks/ai-careers-insights-from-ex-meta-staff-eng) (Ilya Reznik, Instructed Machines, LLC, 59:24)
- [MLOps Build or Buy, Startup vs. Enterprise?](https://mlopstalks.com/talks/mlops-build-or-buy-startup-vs-enterprise) (Aaron Maurer & Katrina Ni, Slack, 49:57)
- [Recommender System: Why They Update Models 100 Times a Day](https://mlopstalks.com/talks/recommender-system-why-they-update-models-100-times-a-day) (Gleb Abroskin, FunCorp, 49:01)
- [Evaluating and Integrating ML Models](https://mlopstalks.com/talks/evaluating-and-integrating-ml-models) (Morgan McGuire & Anish Shah, Weights & Biases, 51:57)
- [Building Recommender Systems with Large Language Models](https://mlopstalks.com/talks/building-recommender-systems-with-large-language-models) (Sumit Kumar, Meta, 11:31)