← back
Trust, but Verify: Shreya Rajpal
Takeaway
Treat every LLM call as untrusted and wrap it in a verification suite with reask/fix/refrain policies so correctness becomes a programmable property, not a hope.
Summary
- Shreya Rajpal (CEO Guardrails AI) argues GenAI apps have lower retention than traditional SaaS (cites Sequoia data) because LLMs are non-deterministic and 'worked in prototype, failed in prod' is the dominant failure mode.
- Proposes a 'trust but verify' paradigm: every LLM output passes through a verification suite checking hallucinations, structure, PII/PHI, profanity, competitor mentions, executable code, source-groundedness etc., and the model self-heals on failure.
- Quotes Alex Graveley (Copilot creator): add a constraint checker, on violation inject the violation back and regenerate — Guardrails AI implements this framework around any LLM call.
- Guards are constructed from declarative specs (XML/RAIL), Pydantic models or strings; runtime policies on failure include reasking, programmatic fix, fallback, refrain, or no-op-with-logging.
- Cites self-driving experience as motivation for layered verification rather than trusting any single ML model end-to-end.
guardrailsreliabilitysafety
Original description
Large Language Models (LLMs) such as ChatGPT have revolutionized AI applications, offering unprecedented potential for complex real-world scenarios. However, fully harnessing this potential comes with unique challenges such as model brittleness and the need for consistent, accurate outputs. These hurdles become more pronounced when developing production-grade applications that utilize LLMs as a software abstraction layer. In this talk, we will tackle these challenges head-on. We introduce Guardrails AI, an open-source platform designed to mitigate risks and enhance the safety and efficiency of LLMs. We will delve into specific techniques and advanced control mechanisms that enable developers to optimize model performance effectively. Recorded live in San Francisco at the AI Engineer Summit 2023. See the full schedule of talks at https://ai.engineer/summit/schedule & join us at the AI Engineer World's Fair in 2024! Get your tickets today at https://ai.engineer/worlds-fair About Shreya Shreya Rajpal is the creator and maintainer of Guardrails AI, an open source platform developed to ensure increased safety, reliability, and robustness of large language models in real-world applications. Her expertise spans a decade in the field of machine learning and AI. Most recently, she was the founding engineer at Predibase, where she led the ML infrastructure team. In earlier roles, she was part of the cross-functional ML team within Apple's Special Projects Group and developed computer vision models for autonomous driving perception systems at Drive.ai.