← back
The ROI of AI: Why you need Eval Framework - Beyang Liu
Takeaway
Pick an explicit ROI evaluation framework matched to your business context — naive 'roles eliminated' math doesn't survive contact with software engineering reality.
Summary
- Sourcegraph CTO Beyang Liu argues measuring AI ROI for software dev is NP-hard because it reduces to measuring developer productivity — but tractable frameworks exist.
- Uses the 'beans' analogy: developers (Bob) grow beans, sales (Pat) sells, CFO (Alice) counts; tension between engineering and finance over AI tool ROI.
- Framework 1 'Roles Eliminated': common in customer support but rarely workable for software; speaker rejects pure headcount-reduction framing for engineering.
- Sourcegraph's Cody coding assistant uses their code-search engine as the context layer — sold into 1Password, Palo Alto, Leidos government contractors.
roievalsai-business
Original description
There has been a ton of hype about AI's applications in software development, but leaders must look beyond the hype to assess the ROI for their organizations. We'll cover some quantitative and qualitative frameworks that have proven useful for our customers, highlight the benefits and drawbacks of popular metrics in use, and share the themes that have emerged as important pillars of the value prop for AI dev tools. I'll also share a vision for where the next 18 months of AI will lead in software development. Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025 Beyang Liu is CTO and co-founder of Sourcegraph. Sourcegraph is the world's best code search and understanding engine, which enables developers to find and grok code across very large codebases and dependency graphs. Beyang is also the creator of Cody, the context-aware AI coding assistant, which uses Sourcegraph's search and understanding engine to generate code, answer questions, and enable a wide variety of AI automation in the context of your private codebase. Prior to Sourcegraph, Beyang built large-scale data analysis engines for Fortune 500 companies with complex codebases as an engineer at Palantir Technologies. Beyang's interest in AI sprang from his experience studying computer science at Stanford University, where he first encountered the Chomsky and Norvig models of intelligence and discovered a love for compilers while publishing research as a member of the Stanford AI Lab.