← back
Hard Won Lessons from Building Effective AI Coding Agents – Nik Pash, Cline
Takeaway
In 2025, the leverage in coding agents has shifted from harness cleverness to building RL environments with outcome-only verifiers that train the next model generation.
Summary
- Frontier models bulldoze scaffolding — Gemini 3.0 topped Terminal-Bench out of the box with the unopinionated Terminus harness (no RAG, no graph search, just a terminal).
- Capability beats scaffolding; clever context tricks are mostly played-out content; the real bottleneck is benchmarks/RL environments that teach models new behaviors.
- Cline built an 'RL environments factory' that auto-converts real-world coding tasks into containerized RL envs: phase 1 sub-agents qualify origin/journey/outcome from PRs; phase 2 reconstructs both states locally, dockerizes, defines a verifier.
- Verifier analogy: a kettle whistle (pure outcome) beats prescribing the burner setting (process). Time to build one env dropped from 16 hours to <20 minutes; endgame is RL envs that train agents to build RL envs.
coding-agentsrl-environmentsverifiers
Original description
Most of what’s written about AI agents sounds great in theory — until you try to make them work in production. The seductive ideas (multi-agent orchestration, RAG, prompt stacking) often collapse under real-world constraints. Why? Because they optimize for the wrong thing. In this talk, Nik Pash shares hard-won lessons from building large-scale coding agents at Cline — what failed, what survived, and why the next leap forward won’t come from clever scaffolds, but from evals and environments that truly measure and improve reasoning. Attendees will walk away with a clearer sense of what actually drives progress — and what’s just noise. https://www.linkedin.com/in/nikpash