← back
Dark Factory: How OpenClaw Ships Faster Than You Can Read the Diff — Vincent Koc
Takeaway
Replace static benchmark evals with adaptive eval pipelines that evolve alongside your agents as you ship faster than you can review diffs.
Summary
- Comet's Vincent Koc argues static evals are 'dead' for agentic systems — you need malleable, adaptive evals that change with the system.
- Maps software-engineering practices (unit tests, chaos engineering, observability) onto AI: traditional pipelines miss non-deterministic failure modes.
- Advocates a 'dark factory' setup where evals continuously generate and retire test cases in step with shipping velocity.
- Live demos how Comet's Opik handles drifting agent behavior with traces and dynamic regression suites.
evalsagentsobservability
Original description
Static benchmarks made sense for static software. Agents that adapt to users, rewrite their own harnesses, and shift behavior over time break that assumption. This talk is about what evaluation looks like when the system you're measuring keeps changing underneath you. Vincent Koc traces the arc from prompt engineering to context engineering to intent engineering, where agents self-optimize toward what users actually want. The eval problem compounds at each step: production traces reveal behavioral drift, test suites go stale, and the 20% of edge cases that break your product rarely show up in handcrafted datasets. The alternative he proposes: define the end state, let agents curate their own suites from traces, and treat evals as a living system rather than a point-in-time snapshot. Speaker info: - https://x.com/vincent_koc