Spotify's recommender systems use embeddings for candidate generation, then rank the retrieved tracks for a specific user and context.
2
Sanket Gupta describes foundational user embeddings as a shared layer that teams can fine-tune for products such as Search, Home, Autoplay, and Discover Weekly.
3
The hardest part is evaluating general-purpose embeddings and updating user taste quickly without forcing every system into fully real-time processing.
Summary
Sanket Gupta explains how Spotify builds recommendation systems around user and item embeddings. A track embedding combines audio characteristics such as tempo, pitch, instrumentation, and valence with playlist co-occurrence. Spotify stores these representations in vector indexes, retrieves a candidate set, and applies ranking models to decide what a user is likely to play. Gupta's team also builds foundational user representations that other product teams can tune for their own tasks. This avoids making every team build its own understanding of the same user. The conversation covers cold start, taste onboarding, stale user profiles, and a move from daily batch updates toward updates that can arrive within minutes. Gupta is direct about the evaluation problem. Embeddings cannot be judged by looking at their numbers, so Spotify tests nearest-neighbor behavior and passes embeddings into downstream prediction models. The right update speed depends on the product. Serving must stay fast, while the data behind it can be refreshed at a slower cadence.
Spotify recommendation retrieval has the same shape as RAG without generation
Sanket Gupta compares recommendation systems with retrieval-augmented generation because both use embeddings and nearest-neighbor lookup. For a song such as Ed Sheeran's "Bad Habits," Spotify can retrieve roughly 100 or 200 nearby items from an index, then rank those candidates to choose the songs a user may play next. The difference is the final output. A RAG system may generate an explanation or answer after retrieval, while Spotify returns tracks rather than a paragraph explaining their similarity. Gupta frames vector databases as an established recommendation technique that has recently become familiar through text and NLP systems.
A track embedding combines audio signals with playlist behavior
The track representation comes mostly from the song itself, including spectrogram-derived signals such as tempo, valence, instrumentation, and pitch. Spotify also adds a collaborative signal based on which songs people place together in playlists. The result is an embedding for each track in the catalog. Gupta describes this as a large system whose practical output is a numerical representation that can be indexed for nearest-neighbor retrieval. The embedding does not depend only on descriptive metadata. It combines properties of the audio with evidence about how listeners group tracks.
Foundational user embeddings give Spotify teams a shared understanding of listeners
Gupta says his team builds foundational embeddings so separate product teams do not interpret the same user from scratch. A listener may move from music in the morning, to a podcast in a car, to high-energy music at the gym. A shared representation can carry those patterns across products. Search, Home, Autoplay, personalized DJ, and Discover Weekly can then tune the shared representation for their own tasks. Gupta compares this with transfer learning for language models. The common embedding provides a starting point, while each downstream model adapts it to its product and signals.
Cold start comes from missing history and slow system updates
Gupta identifies two cold-start problems. A new user has no listening history, and Spotify's existing systems may take one or two days to process all users and dependencies. Spotify addresses the first with taste onboarding, where a new listener selects artists and languages. It addresses the second by moving toward near-real-time processing. After someone starts listening, the system can create a new taste profile and embedding within minutes, making the representation available to other models sooner.
User taste changes gradually through repeated inference
When a listener starts playing jazz, Spotify builds features from several time windows, such as the last six months, 28 days, and week. The new signal does not erase the rest of the profile. Gupta describes adding a small amount of jazz to an existing pop and rock representation, then expanding jazz and reducing other interests if the listener keeps engaging with it. Models decide how much weight older and newer behavior should receive. By default, recent behavior gets more weight, while sustained activity can bring an older interest back into the profile.
Embedding evaluation requires both neighborhood checks and downstream tests
Gupta says he has spent more time evaluating embeddings than building their models because the numerical vectors are too abstract for a person to judge directly. Intrinsic evaluation checks the nearest neighbors of a user embedding and compares them with heuristics about that user's interests. Extrinsic evaluation puts the embedding into a mock downstream model, such as a predictor of what the listener will play over the next seven days. The predicted behavior is compared with actual listening. That turns the problem into familiar classification or regression evaluation, where Spotify can use precision, recall, ROC AUC, and related metrics.
Vector databases and online feature stores overlap but support different consumers
For candidate generation, product teams call a vector database and ask for tracks near a user or item. They usually want the retrieved candidates, not the raw embedding values. For ranking, a team may fetch the actual user embedding and combine it with demographics or listening features in its own model. Gupta therefore describes the vector database as similar to an online feature store in one use case, while ranking teams treat it as a source of raw model features. Both paths must provide a representation of the user through a fast online store, even when the stored understanding is several days old.
Near-real-time updates are a trade-off between freshness and system cost
Gupta rejects the idea that Spotify must choose between daily batch processing and inference in 30 milliseconds. Serving recommendations still needs to be fast, but the user representation can be refreshed every few minutes. New listening events can wait in a queue with events from other users, then update the embedding and online store together. This creates a middle ground that responds quickly enough for many products without forcing every backend system to scale for per-event inference. The right freshness depends on the product. A Home shelf may need jazz to appear soon, while another search use case can tolerate a profile that is a couple of days old.
