# Controlled and Compliant AI Applications

Daniel Whitenack, Prediction Guard | LLMs in Production 2023 | 25:13

Source: https://www.youtube.com/watch?v=xPwGyWugyVs
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/controlled-and-compliant-ai-applications
Published: 2023-07-31
Tags: enterprise, guardrails, security, structured-outputs

## TL;DR
- Large language model applications need structured output because inconsistent text blobs are difficult to use in reliable software.
- Companies are managing hallucinations, cost, data leakage, compliance, and injection risks through wrappers, private hosting, validation, and model checks.
- Daniel Whitenack argues for a standardized API that works across open and closed models while enforcing output types, factuality, consistency, and toxicity checks.

## Summary
Daniel Whitenack describes the practical problems that appear when large language models move from chat demos into production systems. Their output is often inconsistent, unstructured text, while corporate users also worry about hallucinations, token costs, HIPAA and other compliance requirements, leaked personal or business data, and injection attacks. He reviews approaches such as Guardrails, Guidance, regex-constrained decoding, self-consistency checks, and language models that evaluate other model outputs. These tools can work well, but they often behave differently across models, especially open access models. Whitenack argues that developers need a standardized API for open and closed models, with compliant hosting for open models and familiar controls for output types, factuality, consistency, and toxicity. He demonstrates this approach with Prediction Guard, using categorical, integer, and JSON outputs alongside factuality and consistency checks.

## Key ideas
### Unstructured model output is difficult to build into reliable software
[02:30](https://www.youtube.com/watch?v=xPwGyWugyVs&t=150s)
Whitenack starts with a common experience: a language model produces an impressive but inconsistent text blob. A sentiment prompt may return a sentiment label followed by extra explanation, and the label may even be wrong. Developers can strip the extra text or add a stop token, but those fixes do not provide a dependable interface. He argues that software needs a predictable result type rather than output that only looks right in a chat window.

### Corporate teams worry about more than hallucinations
[03:53](https://www.youtube.com/watch?v=xPwGyWugyVs&t=233s)
Whitenack says lawyers, finance departments, and security professionals are concerned about several parts of language model use. APIs charge by tokens, which can make extraction workloads expensive. Companies may not know how commercial APIs use submitted intellectual property or personal information. Compliance requirements such as HIPAA and SOC 2 can conflict with sending sensitive data to an external service. Injection vulnerabilities add another security concern. In the best case, the customer likes the new experience while internal teams object. In the worst case, incorrect answers also damage the customer relationship.

### Wrappers can validate and repair structured output
[06:46](https://www.youtube.com/watch?v=xPwGyWugyVs&t=406s)
One common response is to put a wrapper around the model call. The wrapper can use regular expressions, keyword filtering, a framework, or a schema. Whitenack describes Guardrails, which uses XML-like specifications to request structured JSON, validate the result, and ask the model again when the result does not fit. He also mentions Guidance from Microsoft. These approaches work well with OpenAI in his experience, but open access models such as Falcon may need additional integrations.

### Constrained decoding limits which tokens the model can produce
[08:44](https://www.youtube.com/watch?v=xPwGyWugyVs&t=524s)
Whitenack presents reLLM, also called regex LLM, as a different approach. Instead of generating an answer and validating it afterward, the system examines a regular expression or context-free grammar and masks tokens that cannot be valid next choices. Parser LLM applies a similar idea with context-free grammars. This can control the output during generation, although developers then have to manage the added complexity of regular expressions or grammars.

### Consistency and evaluator models can reduce unreliable answers
[09:55](https://www.youtube.com/watch?v=xPwGyWugyVs&t=595s)
For hallucinations, Whitenack describes running a prompt several times at a higher temperature and taking a majority vote. This is easier for classes or single values than for free-form text, where outputs may need comparison through embeddings or another method. He also discusses using one language model to evaluate another model's output. Inspired Critique can estimate factual consistency between a source and a reference text, and can perform toxicity checks. LlamaIndex documentation describes comparing retrieval-chain output with its context.

### Compliance choices each have a cost
[12:45](https://www.youtube.com/watch?v=xPwGyWugyVs&t=765s)
Whitenack groups current compliance responses into three patterns. Some teams ignore the risks and proceed, which he describes as a setup for disaster. Others stop using language models or forbid certain data from being passed to them, which limits possible benefits and may create business pressure. A third option is hosting private models, which keeps more control inside the organization but requires GPU hosting and scaling knowledge. None of these choices removes the operational work.

### A model-by-model toolchain becomes too hard to maintain
[13:48](https://www.youtube.com/watch?v=xPwGyWugyVs&t=828s)
Whitenack says a production application may involve multiple open-source projects, several models evaluating other models, different query formats, regular expressions, XML, model hosting, and private hosting. Each model can require its own integration because the surrounding tools do not work the same way everywhere. His concern is that this amount of custom plumbing becomes more than one person can manage, especially when the next project needs a different model.

### Prediction Guard combines familiar output controls behind one API
[14:39](https://www.youtube.com/watch?v=xPwGyWugyVs&t=879s)
Whitenack proposes a standardized API for open and closed models. Open models can be hosted in a compliant way, while developers use an API similar to the OpenAI API. In his Prediction Guard examples, developers request categorical, integer, float, Boolean, or JSON output. They can also turn on consistency and factuality checks. A request for product names and prices returns a parsed Python dictionary or JSON response instead of an unstructured text result.

## Notable quotes
- "I can't really build robust systems with inconsistent unstructured text blob output." (02:50)
- "This starts to become more than a single human can manage, I think." (14:18)
- "I can say, well, I just don't want any sort of output here. I actually want a certain type of output." (18:24)
- "Just because we can structure output doesn't mean that it is factual or consistent." (19:31)

## Tools & references mentioned
- Prediction Guard
- Practical AI
- MLOps Community
- OpenAI
- HIPAA
- SOC 2
- Guardrails
- Shreya
- Guidance
- Microsoft
- Falcon
- reLLM
- Regex LLM
- Parser LLM
- Matt Rickard
- regular expressions
- context-free grammars
- Chain of Thought prompting
- Inspired Critique
- Carnegie Mellon University
- Graham Neubig
- BART
- LlamaIndex
- LangChain
- Camel 5 billion model

## Who should watch
- You are building a language model feature that needs categorical, numeric, or JSON output instead of free-form text.
- Your legal, finance, or security teams are asking how model calls will handle sensitive data, compliance, cost, or injection risks.
- You are comparing wrappers and evaluation tools across several open and closed models and want a single developer interface.

## Related talks

- [Harnessing AI APIs for Safer, Accurate, & Reliable Applications](https://mlopstalks.com/talks/harnessing-ai-apis-for-safer-accurate-reliable-applications) (Ron Heichman, SentinelOne, 1:08:14)
- [Data Privacy and Security](https://mlopstalks.com/talks/data-privacy-and-security) (Diego Oppenheimer, Factory & Gevorg Karapetyan, ZERO Systems & Vin Vashishta, V Squared & Saahil Jain, U.com & Shreya Rajpal, 25:44)
- [GenAI: An Unreliable Information Store](https://mlopstalks.com/talks/genai-an-unreliable-information-store) (Noble Ackerson, Venta, 27:55)
- [Age of Industrialized AI](https://mlopstalks.com/talks/age-of-industrialized-ai) (Dan Jeffries, AI Infrastructure Alliance, 29:56)
- [The Truth About AI Agents](https://mlopstalks.com/talks/the-truth-about-ai-agents) (Silen Naihin, AutoGPT, 31:40)