Judging LLMs: Alex Volkov

Summary

• Alex Volkov (W&B AI evangelist, ThursdAI host) frames the talk as a courtroom drama judging hypothetical AI engineers, then teaches LLM eval best practices.
• Three eval methods: programmatic (numerical comparison, assertions like HumanEval — cheap, scales, misses multi-turn); human-in-the-loop (vibe testing during development); LLM-as-judge (most scalable for natural language).
• Case 'turbulence on production' references the real Air Canada chatbot lawsuit — programmatic assertions can't catch customer-facing reasoning failures; you need human-in-the-loop evals.
• Case 'premature fine-tuning' warns against jumping to fine-tuning before iterating on prompts/chain-of-thought/flow engineering/SPI/MOA — W&B Models integrates with OpenAI, Mistral, Together, Axolotl, HF Trainer fine-tuning.
• Promotes W&B Weave for one-line tracing/logging of LLM calls (decorator), tracking inputs/outputs, system prompts, temperature, multi-turn conversations.