# Unified Data + AI Governance with Unity Catalog

Michelle Leon & Victoria Bukta, Databricks | DE4AI 2024 | 24:56

Source: https://www.youtube.com/watch?v=a62QCLUCvGk
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/unified-data-ai-governance-with-unity-catalog
Published: 2024-10-09
Tags: data-engineering, governance, open-source

## TL;DR
- A multi-vendor data platform often has separate catalogs, formats, access policies, audit logs, and lineage systems.
- Unity Catalog is presented as one catalog for tables, files, models, vector indexes, functions, and other data and AI assets.
- Unity Catalog uses open interfaces, including its REST API and the Iceberg REST Catalog API, so external tools can access governed assets without copying data.

## Summary
Michelle Leon and Victoria Bukta describe how data platforms become difficult to govern when teams use several catalogs, storage formats, engines, and AI tools. Victoria walks through an architecture with object storage, Hive, Iceberg REST Catalog, Glue, Delta, structured files, and unstructured assets. In that setup, discovery, permissions, auditing, and lineage are split across systems. Michelle presents Unity Catalog as a single governance layer for data and AI assets. The proposed catalog supports multiple table formats, non-tabular files, models, vector indexes, functions, and other AI resources. It can provide a unified view, end-to-end lineage, auditing, and temporary credentials for external tools. The speakers also describe access through the Unity REST API and Iceberg REST Catalog API, with examples involving DuckDB, Trino, MLflow, Unstructured IO, LangChain, and LlamaIndex. They close by explaining that Unity Catalog is open source and inviting contributions through GitHub, Slack, and community meetups.

## Key ideas
### A data catalog gives assets names, locations, formats, schemas, and metadata
[03:08](https://www.youtube.com/watch?v=a62QCLUCvGk&t=188s)
Victoria defines a data catalog as infrastructure that acts as the source of truth for critical data and machine learning assets and their metadata. Without one, an organization may have object storage containing Parquet, JSON, CSV, Delta, and Iceberg data, alongside logs, audio, video, images, and other unstructured files. Different applications write and process these assets, while engines such as Spark, MLflow, Snowflake, and Trino create refined datasets. The catalog records how assets can be interpreted and accessed.

### Without a unified catalog, discovery and lineage are spread across storage and permissions systems
[05:42](https://www.youtube.com/watch?v=a62QCLUCvGk&t=342s)
When data sits directly in object storage, users have to traverse folders to find it. Access depends on IAM permissions and groups, while teams need separate answers about who accessed an asset, when it was used, which engine handled it, and which datasets produced another dataset. Victoria says a Hive metastore improves this by registering tables, locations, formats, schemas, and properties. That approach becomes less complete when organizations add Iceberg REST Catalog, Glue, Delta, and unstructured assets.

### Multiple catalogs create duplicated policies and gaps in governance
[09:41](https://www.youtube.com/watch?v=a62QCLUCvGk&t=581s)
With Hive, Glue, and an Iceberg REST Catalog in the same organization, metadata becomes siloed. Victoria describes fragmented discovery, governance, auditing, and lineage. Access may use file paths and IAM roles in one system, while another system relies on catalog permissions. Administrators must reconcile different logs, and lineage tools may not work across every catalog. The result is a fragmented view of data and AI assets, along with more opportunities for security and audit errors.

### Organizations want one governed view while keeping their choice of tools
[11:07](https://www.youtube.com/watch?v=a62QCLUCvGk&t=667s)
Michelle says customers want to manage data and AI assets in one place and govern them through a single source of truth. She describes the burden of requesting permissions from several administrators just to access a dataset for analysis or model training. At the same time, teams want to use specialized engines and platforms for different workloads. The proposed open lakehouse model keeps the underlying data owned by the customer and lets different tools work with a single copy.

### Unity Catalog is designed to govern tables, files, and AI assets together
[14:23](https://www.youtube.com/watch?v=a62QCLUCvGk&t=863s)
Michelle describes Unity Catalog as an open catalog that creates a unified view across tables, unstructured and semi-structured data, volumes, functions, models, and vector indexes. It is intended to support discovery across these asset types, trace lineage from data to trained models and vector databases, and provide monitoring and auditing across the environment. The catalog is described through three pillars: multiformat support, multimodal asset coverage, and unified governance.

### Open interfaces let external engines access governed data
[18:47](https://www.youtube.com/watch?v=a62QCLUCvGk&t=1127s)
Unity Catalog is presented as an interoperability layer for a broad ecosystem of engines and platforms. Michelle says it implements the Unity REST API and the Iceberg REST Catalog API. Engines that support the Iceberg interface, such as Trino, can access Unity Catalog tables. She also describes temporary credential vending, where Unity Catalog gives a tool a credential scoped to the files and metadata for a particular table instead of exposing the whole storage system.

### Temporary credentials provide the access primitive behind higher-level policies
[20:27](https://www.youtube.com/watch?v=a62QCLUCvGk&t=1227s)
For a DuckDB example, Michelle explains that DuckDB can talk to Unity Catalog, receive a temporary credential scoped to the files for a table, and read the table through the Unity REST APIs. She calls temporary credential vending the underlying primitive on which higher-level controls such as rules-based and attribute-based access control can be built. This approach avoids separately connecting every tool to the catalog and the underlying storage.

### The same governance model extends to volumes and agent tools
[21:27](https://www.youtube.com/watch?v=a62QCLUCvGk&t=1287s)
Michelle describes volumes as a named abstraction for sets of files, with metadata that tools such as MLflow and Unstructured IO can use for data ingestion, training, and serving. She also says Python and SQL functions can be registered in Unity Catalog with names, descriptions, parameters, and other metadata. Frameworks such as LangChain and LlamaIndex can then use those registered functions in agent applications while retaining shared governance, lineage, auditability, and monitoring.

### Unity Catalog is open source and invites community participation
[23:29](https://www.youtube.com/watch?v=a62QCLUCvGk&t=1409s)
Michelle closes by saying Unity Catalog became open source during the summer before this session. She directs viewers to the Unity Catalog website, GitHub repository, Slack, GitHub discussions, and community meetups for documentation, integrations, roadmap information, and discussion of additional data and AI asset types. She presents open source as a way to give users freedom from vendor lock-in and to encourage contributions to the project.

## Notable quotes
- Victoria Bukta: "The metadata for these different assets, because of these different metastores and catalogs, my metadata is siloed." (09:41)
- Michelle Leon: "People will ultimately just want to manage their data and AI assets in one place." (11:23)
- Michelle Leon: "We want to provide that vision and deliver on that end-to-end vision of having this single catalog which can govern access across your entire data estate." (18:19)
- Michelle Leon: "This is basically the core primitive of governance, upon which higher-level governance such as rules-based access control or attribute-based access control is built upon." (20:47)
- Michelle Leon: "Unity Catalog is open source." (23:29)

## Tools & references mentioned
- Unity Catalog
- Databricks
- Delta Lake
- Apache Iceberg
- Hive metastore
- Glue
- Amazon S3
- Azure Blob Storage
- Google Cloud Storage
- Apache Spark
- MLflow
- Snowflake
- Trino
- DuckDB
- MLOps Community
- LangChain
- LlamaIndex
- Linux Foundation AI and Data Foundation
- Unstructured IO

## Who should watch
- You are responsible for governance in a data platform that has multiple catalogs, storage formats, engines, or cloud environments.
- Your data science or AI teams need access to shared data without creating separate copies or managing storage permissions for every tool.
- You are evaluating open interfaces and want to understand how Unity Catalog handles tables, files, models, functions, lineage, and temporary credentials.

## Related talks

- [Data Quality = Quality AI](https://mlopstalks.com/talks/data-quality-quality-ai) (Samuel Partee, Redis & Chad Sanderson, Gable & Joe Reis, Ternary Data & Maria Zhang, Proactive AI Lab Inc & Pushkar Garg, Clari Inc, 27:15)
- [Is Open Source Software Actually Secure?](https://mlopstalks.com/talks/is-open-source-software-actually-secure) (Hudson Buzby, JFrog, 59:23)
- [Responsible Autonomy: Building Governance Frameworks for AI That Act in the Real World via MCP](https://mlopstalks.com/talks/responsible-autonomy-building-governance-frameworks-for-ai-that-act-in-the-real) (Saurabh Mishra, Optum, 27:55)
- [Omnigent: Composition, Control, and Collaboration for AI Agents](https://mlopstalks.com/talks/omnigent-composition-control-and-collaboration-for-ai-agents) (Denny Lee, Databricks, 58:17)
- [AI-Powered Data Unification for Data Platforms](https://mlopstalks.com/talks/ai-powered-data-unification-for-data-platforms) (Shelby Heinecke, Salesforce, 13:16)