Traditional AI governance is model-centric, while LLM applications require governance at the workflow level.
2
Teams should document workflow failure modes, error costs, error frequency, guardrails, and the outer boundaries of possible failure.
3
LLM workflows need change management, audit logs, corrective actions, and documented paths to human experts.
Summary
Diego Oppenheimer explains why governance frameworks built around individual machine learning models do not map cleanly to LLM applications. Traditional governance tracks model provenance, training data, algorithms, accuracy, drift, access, and audit records. LLM applications often use general-purpose models from external providers inside larger workflows, so the important unit of review becomes the complete workflow and its components. Oppenheimer recommends cataloging the models, versions, providers, uses, guardrails, and risks involved in each workflow. Teams should define possible failure modes and the outer boundaries of stochastic behavior, then decide which risks they will accept. They also need to record provider changes, fine-tuning, model-version changes, error states, corrective actions, and human interventions. A documented fallback to human experts is needed when a workflow cannot meet its quality guarantees. The goal is to increase assurance in LLM workflows enough to use them in regulated and high-impact areas.
AI governance exists to assign responsibility and control risk
Oppenheimer says governance matters because organizations need to understand how AI is used and who is accountable for its actions. It provides safeguards for privacy and personal data, along with checks for safety, robustness, and risk. He frames this as a practical way to use powerful technology in areas such as medicine, government, and financial services. The work builds on older governance concerns around security, investment decisions, compliance, fairness, transparency, and accountability.
Traditional governance follows the model through its lifecycle
Older machine learning governance usually required a model catalog, risk documentation, data-source descriptions, training and prediction details, and the methods used to build the model. Organizations applied different levels of scrutiny depending on the model's use, such as credit decisions compared with marketing decisions. They also tracked data origins, training, deployment into existing systems, monitoring, accuracy, data consistency, model changes, access, and audit logs. This approach is centered on how a particular model was built and operated.
LLM applications break the assumption that one model maps to one risk level
Oppenheimer points out that LLM governance has weak provenance because teams may not know exactly where all training data came from or how a foundation model was trained. The same general-purpose model can appear in many different applications with different risks, so a uniform high-risk or low-risk model classification no longer fits. LLMs also create practical integration problems because organizations must connect changing tools and specialized hardware to legacy systems.
Instead of cataloging only the models, teams should catalog the complete workflow. That includes the model and version used at each step, the provider, the purpose of each component, the guardrails around every model call, and the risks associated with that component. Oppenheimer says the review should focus less on how the model was built and more on what can go wrong in the workflow. Error type, error cost, and error frequency matter at this level.
Teams should define and accept the outer boundaries of failure
Because LLM behavior is stochastic, teams may not be able to predict the exact outcome of every run. They can still describe the outer bounds of where a workflow might go wrong. Oppenheimer recommends documenting the workflow's potential failure modes and deciding whether to accept or reject those boundaries. This turns an uncertain model behavior problem into an explicit risk decision for the application.
Provider and model changes require continuous validation
LLM workflows change when an external provider fine-tunes a model, adds new training data, or releases a new version. Oppenheimer says governance needs constant change management and validation to understand how those changes affect the workflow. Guardrails should add structure, type, and quality guarantees to model outputs. When a workflow crosses a defined boundary, the organization should have a corrective action ready.
Human fallback and detailed logs make the workflow reviewable
An LLM workflow should include a documented path to human experts when it cannot provide the required quality guarantees. Teams should log guardrails, error states, corrective actions, corrected components, and human interventions across the workflow. These records allow the organization to review what happened, explain its risk boundaries, and show regulators how the workflow is controlled.
"We should understand and have the ability to fall back on a documented path of human experts when the workflows can't provide certain quality guarantees."12:36
Who should watch
You are building an LLM application for a regulated or high-impact setting and need a practical governance approach.
Your existing model inventory and risk process says little about prompts, providers, workflow steps, or human review.
You need to define failure boundaries and build audit records before putting an LLM workflow into production.
