Personalization in the Era of LLMs - Shivam Verma, Spotify

From Discover Weekly to AI DJ: why Spotify is changing the stack

Shivam Verma opens by framing Spotify’s next chapter in personalization: less about classic agent workflows, more about “context engineering on the modeling side.” He grounds it in Spotify’s scale — 750 million users, 100 million-plus tracks, roughly 400,000 audiobooks, millions of podcasts, 184 markets — and points to products like AI DJ, prompted playlists, and the new podcast-capable prompt playlists as signs that recommendations are becoming conversational and steerable.

The old recommender pipeline is breaking into silos

He gives the quick recommender-systems primer: traditional stacks shrink a huge catalog through candidate generation, then rankers, sometimes multiple ones, until you get a final list. The problem is organizational as much as technical — different surfaces like home, playlists, search, podcasts, and ads often end up with separate teams and separate models, which means uneven quality and fragmented features.

User embeddings: compressing your listening life into one vector

Verma’s team, the user representations group in Spotify’s AI Foundation org, builds the embeddings that tell Spotify who you are. He describes older approaches like autoencoders that compress a user’s features into a compact vector and reconstruct them, but says the company is moving toward sequential transformer-based modeling that treats user interactions as context, much closer to how LLMs work.

A map where users, tracks, and podcast episodes all live together

One of the coolest moments in the talk is a visualization of Spotify’s newer model: tracks in blue, podcast episodes in pink, users in green, all embedded in the same space. Verma points out his own embedding landing near the Big Technology podcast, making the point viscerally: once the model sees enough user context, it can place people and content on the same “hypersphere” and reason across modalities.

Teaching LLMs Spotify’s catalog with semantic IDs

Once you understand users, the next challenge is teaching an LLM the catalog itself. Spotify combines its own item embeddings with open-weight models like Llama and Qwen, then uses semantic IDs — compressed tokenized versions of item vectors — so the model can generate the “next item” autoregressively, not as plain text but as the next song or episode.

Why Ariana Grande and Bruno Mars start with the same tokens

Verma uses a concrete example: Spotify might represent artists with six semantic-ID tokens, and Ariana Grande and Bruno Mars share the first two because both live in a broad pop neighborhood. Later tokens diverge to capture finer-grained differences, creating a hierarchical structure that lets the model learn both broad similarity and niche distinctions.

Taste Profile and the final personalization trick: a user as a soft token

The last piece is user control. Taste Profile exposes a text summary of what Spotify thinks you like, then lets you edit it — maybe ask for more Justin Bieber, or tell Spotify to stop leaning on a certain podcast — and feed that signal back into the system.

Projecting the user into the LLM itself

Because you can’t train directly on every one of Spotify’s 750 million-plus users, Spotify injects personalization by projecting a user embedding into the LLM’s space as a soft token. Verma says this is already showing positive internal results, and if you’re getting next-episode recommendations in Spotify podcasts today, something like this is already in production.