[Full Workshop] Building Conversational AI Agents - Thor Schaeff, ElevenLabs

4.4K views · Jul 31, 2025 · 61:42 min · Watch on YouTube ↗

Takeaway

ElevenLabs' conversational stack favors text-mediated pipelines for transparency and supports 99 languages via colocated ASR/LLM/TTS components.

Summary

ElevenLabs DevEx walkthrough of conversational AI pipeline: ASR -> LLM brain (GPT-4o/Gemini etc.) -> TTS, all colocated to minimize latency; system tools include language detection and function calling.
Their new ASR model supports 99 languages with word-level timestamps, speaker diarization, audio event tagging (laughter, coughing).
Launched a 'text-to-bark' sound-effects model on April 1 (mistaken for an April Fools); also v3 multilingual TTS with new languages including Hindi, Tamil.
Argues their text-mediated pipeline beats sound-token-to-sound-token (OpenAI Realtime) for observability and reliability at scale despite some latency cost.

voiceconversational-aielevenlabs

Original description

In this workshop you will learn how to build multilingual Conversational AI agents that can automatically detect your user's spoken language and can seamlessly switch to their preferred language.

About Thor Schaef  
Thor is a software engineer who loves to teach and help developers build. 

Having grown up around the SAP headquarters in Germany, he started building on the web back in high-school, later studied Computer Science and Media across Germany, Ireland, and Switzerland, and interned with the Google Maps Team in London. 

He joined early Stripe in Dublin, building out various customer-facing engineering teams across Europe and Southeast Asia, contributing to open-source software, while mentoring and investing in early stage startups along the way. 

Settled in sunny Singapore since 2019, he helped grow Supabase from 800 to over a million databases, and recently joined ElevenLabs to help build the developer platform for AI audio!

Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter