← back
BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data
Takeaway
Pairing batch-based LLM evaluations with synthetic data generated from real support tickets is a fast path from POC chatbot to production-ready RAG.
Summary
- BotDojo is a low-code AI enablement platform for productionizing LLM apps with RAG, structured JSON outputs (Grok/Claude), and per-node tracing.
- Batches feature runs many test questions through a flow and scores them against evaluations (hallucination, retrieval coverage, etc.).
- Synthetic data trick that works for customers: extract Q&A pairs from real support tickets, retrieve relevant index content, generate documents in the same style, and add an inline citation check.
- Code nodes let users drop in TypeScript (Python coming) when low-code blocks aren't enough.
- Demo: adding generated data to the vector index improved eval scores from mixed red/green to all green.
botdojoevalssynthetic-data
Original description
So you've built an AI assistant, but how do you gain the confidence to deploy it to production? In this live demo we introduce BotDojo, a platform that helps you build and improve LLM applications. We'll demonstrate how it identifies gaps in your assistant's knowledge through targeted evaluations. Then, we'll show how to generate synthetic data from existing documentation and support interactions to improve performance. Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025 About Paul Paul Henry is the Founder and CEO of BotDojo, an AI enablement platform that assists companies in designing and deploying Large Language Model (LLM) applications. Prior to founding BotDojo, Paul served as the CTO of Auctane, where he led the Engineering team and contributed to the company's growth from 10 to 2,000 employees. During his tenure, he helped develop a logistics platform that now processes over three billion orders annually for more than a million customers, powering products such as ShipStation, ShipEngine, and Stamps.com. At BotDojo, Paul draws upon his experience to address the challenges associated with reliably deploying LLM applications to enhance internal operations and customer support. The platform supports the entire lifecycle of AI application development, from design and evaluation to deployment, allowing companies to deploy AI solutions with confidence.