← back
Analyzing 10,000 Sales Calls With AI In 2 Weeks — Charlie Guo
Takeaway
AI-driven analysis of unstructured customer data is now a fortnight's solo project—if you spend on the smartest model and add prompt caching, structured outputs, and verifiable citations.
Summary
- Charlie Guo (Pulley) analyzed 10,000 sales calls in 2 weeks to refine ICP—manual analysis would have been ~625 person-days.
- Chose Claude 3.5 Sonnet over cheaper models because smaller models hallucinated company types and founder status; the worst outcome was bad data.
- Multi-layer hallucination defenses: RAG with internal+external sources, chain-of-thought prompts, structured JSON output with citation trails.
- Prompt caching cut costs ~90% and latency ~85%; experimental extended-output flag turned a $5k analysis into $500 and removed multi-turn output chaining.
llm-appsprompt-cachingclaude
Original description
AKA: The Data Goldmine You’re Probably Ignoring Most companies are sitting on mountains of customer data: sales calls, customer support tickets, product reviews, user feedback, and social media interactions. But the truth is that most of this valuable data remains untouched - or worse, unusable. In this case study, I'll share how our team leveraged Claude to analyze 10,000 sales call transcripts in a handful of days, extracting deep customer insights at scale. We'll cover the AI engineering challenges we faced, including model selection tradeoffs, reducing hallucinations with retrieval-augmented generation (RAG), and optimizing prompt caching to dramatically cut costs and latency (by up to 90% in some cases). This isn't theoretical - it's a practical blueprint with concrete ROI metrics. Perfect for AI engineers, data scientists, and anyone sitting on mountains of unstructured customer data they can't analyze at scale. Read more at https://www.ignorance.ai/