# Building Recommender Systems with Large Language Models

Sumit Kumar, Meta | LLMs in Production 2023 | 11:31

Source: https://www.youtube.com/watch?v=nq6ediVlDbs
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-recommender-systems-with-large-language-models
Published: 2023-08-02
Tags: fine-tuning, privacy, prompt-engineering, recommender-systems

## TL;DR
- Large language models can add external knowledge to recommender systems when user behavior data is sparse or missing.
- Current research uses LLMs either as embedding providers for downstream recommendation models or as generative systems driven by prompts and tuning.
- LLM recommenders still face problems with latency, prompt sensitivity, candidate validity, position bias, privacy, and unfair outputs.

## Summary
Sumit Kumar explains how large language models are being used in recommendation systems and why researchers are interested in them. LLMs can add web-scale knowledge that behavior logs may not contain, such as connecting turkeys with Thanksgiving. Their zero-shot and few-shot abilities may help with cold starts and sparse data, while their text representations can encode user profiles and item descriptions. Kumar separates the research into discriminative approaches, which use language-model embeddings, and generative approaches, which prompt or fine-tune an LLM to produce recommendations. He also describes uses in data augmentation, conversational recommendation, reranking, and direct generation. The limits are substantial. LLMs can recommend items outside the candidate set, respond poorly to prompt changes, add unacceptable latency, expose private data, favor popular items, change rankings based on item order, and produce biased or harmful content.

## Key ideas
### LLMs add outside knowledge that behavior logs may lack
[02:11](https://www.youtube.com/watch?v=nq6ediVlDbs&t=131s)
Kumar says LLMs encode a large amount of external knowledge that can supplement the user behavior data used by traditional recommenders. His example is Thanksgiving: an LLM might recommend turkeys because it knows the association, while a conventional system may have no click history connecting turkeys with Thanksgiving. He also links zero-shot and few-shot capabilities to cold-start and data-sparsity problems. LLM feature representations can encode user profiles, item descriptions, and other text-based information for recommendation models.

### Recommendation research divides into discriminative and generative methods
[02:55](https://www.youtube.com/watch?v=nq6ediVlDbs&t=175s)
Kumar describes two broad approaches. In discriminative systems, language models mainly provide embeddings for downstream recommendation tasks. BERT-style models generally fit here, with work split between fine-tuning and prompt tuning. His focus is the generative approach, where an LLM produces recommendations from a prompt or from a tuned model. Generative work includes methods that assume the model already has recommendation abilities and methods that train it with recommendation data.

### Prompts turn recommendation into a language task
[04:03](https://www.youtube.com/watch?v=nq6ediVlDbs&t=243s)
Kumar describes a prompting method evaluated by Alibaba that uses three parts: a natural-language task description, a behavior-injection section containing user and item interactions, and an output-format indicator. Few-shot versions add demonstration examples. Studies found that zero-shot and few-shot recommendations can beat random guessing and some simple heuristics, but they do not outperform a traditional recommendation model trained for the specific task with task-specific data.

### Fine-tuning frameworks convert interactions into instructions
[05:50](https://www.youtube.com/watch?v=nq6ediVlDbs&t=350s)
To address the limits of prompting, researchers have proposed frameworks that fine-tune LLMs with recommendation data. These systems use user-item interactions to create instructions for training. Kumar also discusses foundational-model approaches such as P5, which pre-trains on several recommendation tasks under a shared text-to-text language-model objective. The goal is to put different recommendation problems into a common language format.

### LLMs appear at several points in a recommender pipeline
[06:32](https://www.youtube.com/watch?v=nq6ediVlDbs&t=392s)
Kumar lists several uses beyond directly generating recommendations. LLMs can augment data and encode text features. They can power a conversational interface that decides whether to continue talking with a user or call a backend API to refine candidates. They can rerank items alongside a traditional retrieval system, and some research uses them to generate the final recommendation output. A chat interface also lets users express preferences directly in natural language.

### Generative recommenders have practical serving problems
[08:01](https://www.youtube.com/watch?v=nq6ediVlDbs&t=481s)
Kumar warns that an LLM may recommend an item outside the available candidate set, return the wrong format, or produce a verbose response when the system needs a rating or a simple yes-or-no answer. Prompt design is sensitive, including the number and type of demonstrations. Traditional recommendation systems also rely heavily on ID-like features, and incorporating those features into prompts remains difficult. Since online recommenders are time-sensitive services, prompt construction and LLM inference can add substantial cost in latency.

### Private behavior data does not fit easily into LLM context
[09:11](https://www.youtube.com/watch?v=nq6ediVlDbs&t=551s)
Kumar describes a gap between the general knowledge stored in an LLM's parameters and the specific behavior patterns found in private-domain data. Data security is another concern. Limited context windows make it difficult to include long behavior sequences in prompts. The model's broad knowledge may also distort ranking: popular items such as best-selling books can receive higher rankings because they appear more often in the model's pre-training data.

### Ranking can change with item order and can reproduce social bias
[10:01](https://www.youtube.com/watch?v=nq6ediVlDbs&t=601s)
Kumar says some LLMs show position bias, meaning that changing the order of input items changes the ranking output. This makes them less suitable for reranking. He also points to harmful content, social bias, and unfair treatment of sensitive attributes such as gender and race. He closes by noting that researchers are actively studying ways to reduce these problems, but the area remains active rather than settled.

## Notable quotes
- "One of the big motivations for using LLMs for recommendations is that LLMs encode a massive amount of external knowledge that can supplement the user behavior data that we commonly use in recommenders." (02:11)
- "Zero-shot and few-shot recommendations can beat random guessing or some carefully designed heuristics, but they still cannot surpass the performance of a traditional recommendation model that is trained specifically for a given task and task-specific data." (05:28)
- "Online recommender systems are real-time services and they are extremely time sensitive as well." (09:00)
- "Some LLMs have also been shown to have position bias, where changing the order of the input items significantly changes the LLM's ranking output as well." (10:06)

## Tools & references mentioned
- ChatGPT
- Alibaba
- P5
- BERT

## Who should watch
- You are evaluating whether an LLM belongs in a recommendation pipeline and need a compact account of possible insertion points.
- Your recommender has sparse behavior data, cold-start users, or rich text such as profiles and item descriptions.
- You are considering LLM reranking or generation and need to account for latency, candidate validity, privacy, ranking bias, and fairness.

## Related talks

- [Large Language Models in Production Round-table Conversation](https://mlopstalks.com/talks/large-language-models-in-production-round-table-conversation) (Diego Oppenheimer, Factory HQ & David Hershey, Unusual Ventures & Hannes Hapke, Digits & James Richards, Bountiful & Rebecca Qian, Facebook AI Research, 57:21)
- [Traditional vs LLM Recommender Systems: Are They Worth It?](https://mlopstalks.com/talks/traditional-vs-llm-recommender-systems-are-they-worth-it) (Arpita Vats, LinkedIn, 47:40)
- [Evaluating the Effectiveness of Large Language Models](https://mlopstalks.com/talks/evaluating-the-effectiveness-of-large-language-models) (Aniket Singh, Ultium Cells, 35:41)
- [The Next Revolution in AI: LLMs and Beyond](https://mlopstalks.com/talks/the-next-revolution-in-ai-llms-and-beyond) (, 13:47)
- [Incorporating LLMs in High-stake Use Cases](https://mlopstalks.com/talks/incorporating-llms-in-high-stake-use-cases) (Yada Pruksachatkun, Moonhub, 11:01)