← back
LLM codegen fails and how to stop 'em — Danilo Campos, PostHog
Takeaway
Fight LLM codegen hallucination by tool-calling fresh markdown docs into context — context windows are big enough that RAG isn't required.
Summary
- PostHog Wizard ships ~15k integrations/month by giving the coding agent fresh markdown docs from posthog.com via tool calls — fixing model rot without RAG embeddings.
- Model rot is the #1 codegen failure: 6-18-month-old training data invents APIs, keys, and patterns that don't exist.
- Large context windows make 'shove the markdown in' competitive with RAG for keeping codegen agents current.
- Wizard converts 2 hours of integration misery into 8 minutes of pseudo-entertainment; trackable user satisfaction on social proof.
code-generationposthogdocs
Original description
Danilo Campos breaks down the most common failure modes in LLM code generation and the practical strategies PostHog uses to prevent them. Drawing from a system that helps 5,000+ users each month, he shares a playbook for making autonomous codegen more reliable, correct, and production-ready. Speaker info: - https://www.linkedin.com/in/danilocampos