# Guardrails for LLMs: A Practical Approach Shreya Rajpal, Guardrails AI | LLMs in Production 2023 | 11:51 Source: https://www.youtube.com/watch?v=e_9o4los7DQ Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/guardrails-for-llms-a-practical-approach Published: 2023-08-11 Tags: guardrails, structured-outputs, tool-use ## TL;DR - LLM applications often become flaky in production because models can hallucinate, produce falsehoods, and fail to follow the required output structure. - Guardrails AI controls model outputs by combining an LLM with application-specific validators that define what a correct response means. - When validation fails, Guardrails AI can re-prompt the model with information about the error, or filter, reject, log, or programmatically fix the output. ## Summary Shreya Rajpal explains why prompts alone are a weak way to control LLM applications. Models are stochastic, so the same input can produce different outputs, and prompts cannot guarantee correctness. Fine-tuning can be expensive, while hosted model APIs give developers little control over model versions. Her proposed approach adds a verification suite after the model call. Small application-specific programs check properties such as executable SQL, the absence of private tables, or protection against dangerous predicates. Guardrails AI provides the framework, validators, orchestration, and specification language needed to run these checks. Rajpal walks through a text-to-SQL example in which an invalid table name fails verification, triggers an automatically constructed correction prompt, and produces a valid query on the second attempt. The library also supports filtering, refraining, exceptions, logging, and programmatic fixes when validation fails. ## Key ideas ### Production LLM applications fail in ways that prompts cannot prevent [01:30](https://www.youtube.com/watch?v=e_9o4los7DQ&t=90s) Rajpal says LLM applications can work during prototyping and become flaky in production. Common failures include hallucinations, falsehoods, and outputs that do not follow the required structure. Developers often try to control these behaviors by adding instructions to the prompt, such as telling the model not to use a word or to always respond in a certain format. She calls this approach insufficient because the model remains difficult to control when correctness matters. ### Prompts do not guarantee the same result for the same input [02:30](https://www.youtube.com/watch?v=e_9o4los7DQ&t=150s) Rajpal describes three ways to control an LLM: prompts, the model itself, and guardrails. Prompts are limited because LLMs are stochastic, so the same input may produce different outputs. Model-level control can require expensive or time-consuming training and fine-tuning. Hosted APIs also leave developers without control over model versions, and the provider may change the model without notice. ### Verification programs define correctness for a specific application [03:11](https://www.youtube.com/watch?v=e_9o4los7DQ&t=191s) The Guardrails approach combines an LLM with output verification. After the model produces an answer, small independent programs check whether it meets the application's requirements. The checks depend on the use case. They might reject personally identifying information or profanity, prevent competitor mentions, test whether generated code is executable, or check whether a summary stays faithful to its source. ### A failed check can trigger a corrective model call [05:12](https://www.youtube.com/watch?v=e_9o4los7DQ&t=312s) If every verification test passes, the application receives the output. If a test fails, Guardrails can build a new prompt that includes context about what went wrong and ask the LLM to correct itself. Rajpal presents this as an orchestration flow from prompting to verification to re-prompting, rather than as a single prompt that has to handle every requirement. ### Guardrails AI combines custom validators with ready-made components [05:48](https://www.youtube.com/watch?v=e_9o4los7DQ&t=348s) Rajpal describes Guardrails AI as an open-source library for creating custom validators and connecting them to the application flow. It handles the prompting, verification, and re-prompting steps. It also includes validators for common use cases and a specification language for communicating with LLMs. Developers can build checks that match their own application's definition of a safe or correct response. ### Text-to-SQL needs checks for execution, privacy, and dangerous actions [06:29](https://www.youtube.com/watch?v=e_9o4los7DQ&t=389s) In the text-to-SQL example, a user asks which department has the most employees. The generated query must be executable against the database, must not expose private tables, and must not contain risky predicates such as commands to drop tables. These checks are separate, so a query can pass the privacy and risk checks while still failing because it refers to a table that does not exist. ### Guardrails can repair an invalid SQL query through re-prompting [08:02](https://www.youtube.com/watch?v=e_9o4los7DQ&t=482s) The example produces a query using a nonexistent Departments table. Verification catches the execution failure while the other checks pass. Guardrails then constructs a re-asking prompt with the relevant error context and sends it to the LLM. The model returns a query with the correct table name, which passes the verification suite and is sent to the user. ### Validation failures have several possible responses [09:41](https://www.youtube.com/watch?v=e_9o4los7DQ&t=581s) Rajpal says re-asking is only one way to handle a failed check. Guardrails can filter the output, refrain from returning an answer, raise an exception, log the failure, or try to fix the output programmatically. The choice depends on what the application needs when a response violates its validators. ## Notable quotes - "LLMs are awesome but they are brittle and hard to control." (01:30) - "The guardrails approach to controlling LLMs and offering guarantees is combining an LLM with output verification." (03:11) - "The prompt just does not guarantee correctness." (02:30) - "Guardrails AI is a fully open source library that offers a bunch of functionality." (05:48) ## Tools & references mentioned - Guardrails AI - GitHub - Twitter - ChatGPT ## Who should watch - You are building an LLM feature whose output must follow a structure or satisfy application-specific rules. - Your prototype works, but production responses are flaky and you need checks beyond prompt instructions. - You are evaluating whether failed LLM outputs should be retried, filtered, logged, rejected, or repaired in code. ## Related talks - [The Confidence Checklist for LLMs in Production](https://mlopstalks.com/talks/the-confidence-checklist-for-llms-in-production) (Rohit Agarwal, portkey.ai, 32:34) - [The Emerging Toolkit for Reliable, High-quality LLM Applications](https://mlopstalks.com/talks/the-emerging-toolkit-for-reliable-high-quality-llm-applications) (Matei Zaharia, Databricks, 31:01) - [Incorporating LLMs in High-stake Use Cases](https://mlopstalks.com/talks/incorporating-llms-in-high-stake-use-cases) (Yada Pruksachatkun, Moonhub, 11:01) - [LLM Evaluation with Arize AI's Aparna Dhinakaran](https://mlopstalks.com/talks/llm-evaluation-with-arize-ais-aparna-dhinakaran) (Arize AI's Aparna Dhinakaran, 55:48) - [Pitfalls and Best Practices: 5 Lessons from LLMs in Production](https://mlopstalks.com/talks/pitfalls-and-best-practices-5-lessons-from-llms-in-production) (Raza Habib, Humanloop, 30:27)