← back
Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs - Eugene Yan, Amazon
Takeaway
Recsys is moving from hash-based IDs to learned multimodal semantic IDs that share representational space with LLMs.
Summary
- Three ideas to merge recsys with LLMs: semantic IDs (replace hash IDs to fix cold-start and sparsity), LLM-driven data augmentation, and unified models replacing separate retrieval/ranking stacks.
- Kuaishou's trainable multimodal semantic IDs combine ResNet visual embeddings, BERT text embeddings, and behavioral signals in a two-tower architecture — solves the hundreds-of-millions-of-new-items-per-day cold-start problem.
- Hash-based IDs encode nothing about content; semantic IDs let models generalize across items and survive the long-tail popularity bias.
- Architectural trend: from co-occurrence embeddings (2013) → GRUs → transformers → unified LLM-style models for recsys.
recsysembeddingsmultimodal
Original description
Recommendation systems and search have long adopted advances in language modeling, from early adoption of Word2vec for embedding-based retrieval to the transformative impact of GRUs, Transformers, and BERT on predicting user interactions. Now, the rise of large language models (LLMs) is inspiring innovations in model architecture, scalable system designs, and richer customer experiences. In this keynote, we'll dive into cutting-edge industry applications of LLMs in recommendation and search systems, exploring real-world implementations and measurable outcomes. Join us for an look at current trends and an exciting vision of how LLM-driven techniques will shape the future of content discovery and intelligent search. About Eugene Yan Eugene Yan is a Principal Applied Scientist at Amazon building recommendation systems and AI-powered products that serve customers at scale. He's led ML/AI teams at Alibaba, Lazada, and a Healthtech Series A. He writes about RecSys, LLMs, and engineering at eugeneyan.com. Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter Timestamps: [00:00] Introduction to Language Modeling in Recommendation Systems [01:31] Challenge 1: Hash-based Item IDs [02:14] Solution: Semantic IDs [05:37] Challenge 2: Data Augmentation and Quality [06:10] Solution: LLM-Augmented Synthetic Data [06:21] Indeed Case Study [10:37] Spotify Case Study [13:34] Challenge 3: Separate Systems and High Operational Costs [14:24] Solution: Unified Models [14:51] Netflix Case Study (Unicorn) [16:46] Etsy Case Study (Unified Embeddings) [20:26] Key Takeaways