# Evolving AI Governance for an LLM World

Diego Oppenheimer, Factory | LLMs in Production 2023 | 14:47

Source: https://www.youtube.com/watch?v=C15RxW_mtoI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/evolving-ai-governance-for-an-llm-world
Published: 2023-08-14
Tags: governance, guardrails, human-in-the-loop, workflows

## TL;DR
- Traditional AI governance is model-centric, while LLM applications require governance at the workflow level.
- Teams should document workflow failure modes, error costs, error frequency, guardrails, and the outer boundaries of possible failure.
- LLM workflows need change management, audit logs, corrective actions, and documented paths to human experts.

## Summary
Diego Oppenheimer explains why governance frameworks built around individual machine learning models do not map cleanly to LLM applications. Traditional governance tracks model provenance, training data, algorithms, accuracy, drift, access, and audit records. LLM applications often use general-purpose models from external providers inside larger workflows, so the important unit of review becomes the complete workflow and its components. Oppenheimer recommends cataloging the models, versions, providers, uses, guardrails, and risks involved in each workflow. Teams should define possible failure modes and the outer boundaries of stochastic behavior, then decide which risks they will accept. They also need to record provider changes, fine-tuning, model-version changes, error states, corrective actions, and human interventions. A documented fallback to human experts is needed when a workflow cannot meet its quality guarantees. The goal is to increase assurance in LLM workflows enough to use them in regulated and high-impact areas.

## Key ideas
### AI governance exists to assign responsibility and control risk
[03:12](https://www.youtube.com/watch?v=C15RxW_mtoI&t=192s)
Oppenheimer says governance matters because organizations need to understand how AI is used and who is accountable for its actions. It provides safeguards for privacy and personal data, along with checks for safety, robustness, and risk. He frames this as a practical way to use powerful technology in areas such as medicine, government, and financial services. The work builds on older governance concerns around security, investment decisions, compliance, fairness, transparency, and accountability.

### Traditional governance follows the model through its lifecycle
[04:57](https://www.youtube.com/watch?v=C15RxW_mtoI&t=297s)
Older machine learning governance usually required a model catalog, risk documentation, data-source descriptions, training and prediction details, and the methods used to build the model. Organizations applied different levels of scrutiny depending on the model's use, such as credit decisions compared with marketing decisions. They also tracked data origins, training, deployment into existing systems, monitoring, accuracy, data consistency, model changes, access, and audit logs. This approach is centered on how a particular model was built and operated.

### LLM applications break the assumption that one model maps to one risk level
[07:07](https://www.youtube.com/watch?v=C15RxW_mtoI&t=427s)
Oppenheimer points out that LLM governance has weak provenance because teams may not know exactly where all training data came from or how a foundation model was trained. The same general-purpose model can appear in many different applications with different risks, so a uniform high-risk or low-risk model classification no longer fits. LLMs also create practical integration problems because organizations must connect changing tools and specialized hardware to legacy systems.

### The workflow is the right unit for LLM governance
[09:41](https://www.youtube.com/watch?v=C15RxW_mtoI&t=581s)
Instead of cataloging only the models, teams should catalog the complete workflow. That includes the model and version used at each step, the provider, the purpose of each component, the guardrails around every model call, and the risks associated with that component. Oppenheimer says the review should focus less on how the model was built and more on what can go wrong in the workflow. Error type, error cost, and error frequency matter at this level.

### Teams should define and accept the outer boundaries of failure
[10:52](https://www.youtube.com/watch?v=C15RxW_mtoI&t=652s)
Because LLM behavior is stochastic, teams may not be able to predict the exact outcome of every run. They can still describe the outer bounds of where a workflow might go wrong. Oppenheimer recommends documenting the workflow's potential failure modes and deciding whether to accept or reject those boundaries. This turns an uncertain model behavior problem into an explicit risk decision for the application.

### Provider and model changes require continuous validation
[11:25](https://www.youtube.com/watch?v=C15RxW_mtoI&t=685s)
LLM workflows change when an external provider fine-tunes a model, adds new training data, or releases a new version. Oppenheimer says governance needs constant change management and validation to understand how those changes affect the workflow. Guardrails should add structure, type, and quality guarantees to model outputs. When a workflow crosses a defined boundary, the organization should have a corrective action ready.

### Human fallback and detailed logs make the workflow reviewable
[12:36](https://www.youtube.com/watch?v=C15RxW_mtoI&t=756s)
An LLM workflow should include a documented path to human experts when it cannot provide the required quality guarantees. Teams should log guardrails, error states, corrective actions, corrected components, and human interventions across the workflow. These records allow the organization to review what happened, explain its risk boundaries, and show regulators how the workflow is controlled.

## Notable quotes
- "The general idea here is that, for those who don't know me, I'm Diego Oppenheimer, currently a partner at an AI-focused fund." (02:27)
- "It's much more important right now to catalog the workflow." (10:01)
- "The type of error, the cost of the errors, and the frequency of the errors is what matters, but at the workflow level." (10:40)
- "We should understand and have the ability to fall back on a documented path of human experts when the workflows can't provide certain quality guarantees." (12:36)
- "Ultimately, the power of LLMs is there, and what we really want to do is increase AI assurance of those workflows." (13:58)

## Tools & references mentioned
- MLOps Community
- Demetrios Brinkmann
- Algorithmia
- DataRobot
- Factory
- Slack
- Twitter
- LinkedIn

## Who should watch
- You are building an LLM application for a regulated or high-impact setting and need a practical governance approach.
- Your existing model inventory and risk process says little about prompts, providers, workflow steps, or human review.
- You need to define failure boundaries and build audit records before putting an LLM workflow into production.

## Editor's note

Diego Oppenheimer says LLM governance should catalog the complete workflow, including each model version, provider, guardrail, purpose, and risk, rather than only the model. ZenML records each pipeline run's steps, inputs, outputs, and code version, so teams can trace an artifact or model back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Building LLM Applications for Production](https://mlopstalks.com/talks/building-llm-applications-for-production) (Chip Huyen, Claypot AI, 35:23)
- [Fireside Chat - The Future of LLMs](https://mlopstalks.com/talks/fireside-chat-the-future-of-llms) (David Hershey, Unusual Ventures & Daniel Jeffries, AI Infrastructure Alliance, 36:07)
- [Evaluating LLMs for AI Risk](https://mlopstalks.com/talks/evaluating-llms-for-ai-risk) (Finn Howell, Robust Intelligence, 10:38)
- [Guiding LLMs While Staying in the Driver's Seat](https://mlopstalks.com/talks/guiding-llms-while-staying-in-the-drivers-seat) (Jacob van Gogh, Adept AI, 10:02)
- [From Building Self-driving Cars to Building LLM Applications](https://mlopstalks.com/talks/from-building-self-driving-cars-to-building-llm-applications) (Effy Zhang, Baserun, 10:45)