Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

Summary

• Arize team workshop on 'prompt learning' — agents fail not from weak models but weak environment: missing tools, no planning, bad context engineering, brittle instructions.
• Defines a spectrum from RL (update weights from scalar reward, infeasible) to metaprompting (LLM rewrites prompt from score) to prompt learning (LLM rewrites prompt using rich English feedback explaining *why* outputs were wrong).
• Loop combines LLM-as-judge explanations and SME human-labeled failure annotations, then edits specific system-prompt instructions — the rich text signal is what other approaches like GA-based optimizers miss.
• Frames responsibility split: engineers own pipelines/perf, SMEs/PMs own product principles and evals; prompt learning brings both into a single feedback loop.