← back
How to Improve your Vibe Coding — Ian Butler
Takeaway
Vibe coding only works with explicit rules, careful context management, and thinking models — naive setups generate alert fatigue from false positives.
Summary
- Bismuth's SM-100 benchmark (released day before talk): 3/6 popular coding agents had ≤10% true positive rate; Cursor had 97% false-positive rate over 100+ repos, 1,200+ issues.
- Practical fixes for vibe coding: bug-focused rules files priming agents with OWASP Top 10 and named bug classes (auth bypass, SQL injection, prototype pollution); require test-pass-based fix validation.
- Context management is the dominant failure: when agents compact files, bug-detection collapses; feed diffs, prevent summarization of key files, ask for a component inventory first.
- Thinking models substantially outperform non-thinking on bug detection due to deeper chain-of-thought exploration.
- Bug findings still vary run-to-run — agents don't holistically read files like humans, a key current limitation.
vibe-codingbug-detectionbismuth
Original description
[last round of Attendee-Led 10min lightning talks] Are your vibes immaculate? - Vibe coding is the new hotness but everyone has a story of AI making really dumb choices. Let's talk about how you can improve your vibe coding so your vibes are safe and bug free and you spend more Ian Butler Recorded at the AI Engineer World's Fair in San Francisco. Stay up to date on our upcoming events and content by joining our newsletter here: https://www.ai.engineer/newsletter