← back
Agents are Robots Too: What Self-Driving Taught Me About Building Agents — Jesse Hu, Abundant
Takeaway
Treat coding/browser agents as digital robots — invest in closed-loop sensing, sampling rates, and an offline simulation/eval stack, not just better models.
Summary
- Ex-Waymo/Google engineer maps self-driving lessons onto agent design: 1% model, 99% offline stack (eval, simulation, training, monitoring) wins.
- Agents are increasingly embodied: APIs/MCPs are hands, terminal/browser/VM are full bodies; need closed-loop feedback (e.g., observing bash output in real time) not just open-loop tool calls.
- Implicit design choice: agents wait turns instead of sampling the world at high frequency like a 50Hz robot — limits real-time reactivity to pop-ups and long-running processes.
- Stateful VMs and dagger-style out-of-distribution failures (e.g., browser agents on unseen pop-ups) require evals/simulation analogous to self-driving.
agentsroboticsevals
Original description
In this talk, I break down the surprising parallels between robotics and agents: embodiment, statefulness, simulation, and more. The main lesson from self-driving: everyone thought perception was hard and planning was easy. It took 8-10 years to learn we had it backwards. We're seeing the same pattern with agents today. Predictive models aren't action models. Perfect reasoning doesn't guarantee good execution. And just like in robotics, the company with the best infrastructure wins—not just the one with the best model. Whether you're building agents, training models, or just trying to understand why production agents are so hard, this talk covers the concepts from robotics (DAgger, MDPs, simulation, offline RL) that directly apply to making agents work at scale. Jesse has spent the last 10 years as an ML engineer, starting from research in computer vision and NLP, to working on deep learning and two-tower embedding recommender systems at YouTube, to transformer-based planning models for self-driving at Waymo. He is currently working on bringing large-scale RL and simulation techniques to coding agents at Abundant. --- Socials: - LinkedIn: https://www.linkedin.com/in/jessehu - GitHub: http://github.com/huyouare - Company: Abundant (https://abundant.ai)