← back
Optimizing LLMs in Insurance with DSPy: Jeronim Morina
Takeaway
Treat LLM apps as compilable programs: define metrics on real production data, then let DSPy optimize prompts rather than hand-tweaking templates.
Summary
- AXA Germany's Data Innovation Lab uses DSPy to move from hand-tuned prompts to programmatic, optimizable LLM pipelines for insurance T&C chatbots.
- Critiques 'looks-good-to-me-at-10' eval culture; emphasizes first-principles thinking, mitmproxy inspection of frameworks (LangChain/Guardrails/Instructor) per Hamel Husain's 'show me the prompt'.
- Built proper eval set from production data with attention to data leakage; uses Arize Phoenix for tracing.
- DSPy reframes prompting as Signatures/Modules and optimizers (BootstrapFewShot, MIPRO) that compile prompts against metrics.
- Stresses defining problems with domain experts and avoiding fragile chains stacked with error-handling code.
dspyinsuranceprompt-optimization
Original description
In the insurance industry, LLMs promise efficiency but often get bogged down by manual tuning for optimal performance. DSPy changes the game. Traditional LLM deployment is a high-effort, error-prone process, demanding extensive prompt engineering and fine-tuning across multiple steps. Imagine deploying LLMs where manual optimizations are replaced by DSPy's automated, efficient prompt and weight optimization Recorded live in San Francisco at the AI Engineer World's Fair. See the full schedule of talks at https://www.ai.engineer/worldsfair/2024/schedule & join us at the AI Engineer World's Fair in 2025! Get your tickets today at https://ai.engineer/2025 About Jeronim I'm currently working at AXA, one of the biggest insurers of the world with over 100 billion euro revenue. I build critical Machine Learning infrastructure for AXA Germany to enable hundreds of Data Scientists working effectively every day. I provide core Machine Learning Platform toolings as well as help them deploy Machine Learning models to production reliably and efficiently. I'm also responsible for orchestrating and creating LLM applications that help our customer agents with claims management. Apart from that I have founded my own company bloomed AI in 2023 and help startups with their LLM infrastructure, as well as create a LLM based coverage check tool for customers to see if their insurance provides with with the right coverage and how to write a claim properly.