← back
Turning Fails into Features: Zapier’s Hard-Won Eval Lessons — Rafal Willinski, Vitor Balocco, Zapier
Takeaway
Treat probabilistic agents like a data flywheel: instrument traces so any run becomes a replayable eval, then mine implicit feedback to drive continuous improvement.
Summary
- Rafal Willinski and Vitor Balocco (Zapier) describe the data flywheel for Zapier Agents: instrument code, capture traces (tool calls, errors, pre/post-processing), then make runs replayable as evals for free.
- Explicit thumbs-up/down feedback is rare — they boost it by asking in-context (post-run CTA: 'did this do what you expected?').
- Mine implicit signals: enabling a tested agent = strong positive feedback; copying responses, conversational cues, etc. supplement explicit feedback.
- Build evals from real failures, ship features that fix them, attract more users, get more failure data — the flywheel compounds product quality.
evalszapierfeedback-loops
Original description
Every agent failure can be a roadmap to your next breakthrough. This talk reveals how Zapier's evaluation system transforms frustrating user experiences into targeted improvements, creating a data flywheel that continuously strengthens our agents. You'll learn practical approaches for building the data flywheel, detecting implicit feedback signals, building solid evals, prioritizing metrics that actually matter, and why your most reliable evals might secretly be sabotaging your performance. About Rafal Wilinski Rafal Wilinski is the AI Tech Lead for Zapier Agents, where he builds intelligent systems that enable workflow automation for millions of users. I'm passionate about bringing products to life from 0 to 1. Began my career with an interest for AWS cloud, where I've spent my first decade helping startups and enterprises build robust infrastructure. When not working, I'm most likely climbing or drinking whiskey (but not simultaneously). About Vitor Balocco Vitor is a Staff Software Engineer on the AI R&D team at Zapier, involved in most of Zapier's AI initiatives: - Co-creator of Zapier Agents - Co-creator of Zapier MCP - Creator of the AI Zap builder (natural language to automation) - Co-creator of AI custom actions Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter