Medallion architecture simplifies data work into bronze, silver, and gold, but AI workflows often need data from any layer and do not fit neatly into those boxes.
2
Machine learning teams need visibility into the cleaning and transformations applied to each layer because cleaning can remove signals or introduce bias.
3
AI engineers and data engineers should agree where cleaned data, feature stores, and inference outputs belong in a shared lakehouse process.
Summary
Simon Whiteley explains why bronze, silver, and gold are useful shorthand for lakehouse design but poor as rigid architectural rules. Traditional analytics pipelines move from raw data through cleaning into reusable business models. Machine learning workflows have different needs. They may use raw, cleaned, or business-level data, depending on whether earlier transformations removed useful predictive signals. Feature engineering, feature stores, and inference outputs also sit awkwardly beside the usual Medallion layers. Simon describes designs with additional stages for structured landed data, canonical models, feature stores, analytical models, and semantic models. The right number of layers depends on the use case. A small system may need two layers, while a large enterprise may use six. His practical advice is for AI and data teams to share lineage, document transformations, and decide together where reusable data products and inference results should live.
Simon describes two common backgrounds for data engineers. Software engineers moved toward data to build scalable, repeatable pipelines for training datasets and machine learning models. Traditional SQL and analytics engineers built generic models, such as star schemas, that could answer many business questions. Both groups use engineering practices to prepare and transform data, but they optimize for different kinds of reuse. One may build a pipeline for a particular model, while the other creates shared data structures for many consumers.
Bronze, silver, and gold repeat an older warehouse pattern
The Medallion architecture gives new names to a familiar process. Bronze contains raw data, which may include duplicates, bad values, or the wrong source data. Silver applies cleaning, filtering, and standardization. Gold contains tidy, business-level aggregates. Simon compares this with older warehouse work, where teams landed data, cleaned it, and then built facts and dimensions. Reusing one cleaned dataset across many dimensions was already a central idea.
A model can use bronze, silver, or gold data, depending on the problem. Silver is convenient because someone has already cleaned it, standardized addresses, or removed personally identifiable information. For fraud detection in insurance, that cleaning may remove predictive signals. The team may therefore need raw data and its own cleaning process. Gold can also be useful when business logic and product relationships have already been assembled. The choice depends on what the model needs.
Lineage tells model builders whether cleaning is safe to reuse
Simon says teams should be able to see what changed between layers. A model builder needs to know which transformations were applied, whether data was removed or standardized, and whether those changes could affect the model. This lets the team decide whether to reuse an existing cleaned dataset or start from a less processed source. Without that visibility, a data scientist may receive data that already contains assumptions or bias without knowing it.
Feature stores and inference outputs do not fit neatly into three layers
Feature extraction, feature engineering, and feature stores often sit beside the standard Medallion flow. A batch inference result may be an augmented table with an extra column rather than a finished business data model. It may technically belong in gold because it has been curated, but it does not necessarily fit the usual meaning of that layer. This mismatch explains why data science teams can struggle when engineers treat bronze, silver, and gold as complete descriptions of the architecture.
Mature lakehouses often need more than three layers
Simon gives a broader progression: raw files can be landed, structurally validated, and turned into standard tables without changing their values. Text documents may be chunked for a vector database. A later layer can clean values, and another can combine multiple ERP systems into a canonical model. Feature stores, analytical models, and semantic models may follow. A simple use case may need two layers, while a large enterprise may use as many as six.
AI engineers should discuss with data engineers where cleaned data belongs, how a feature store will be managed, and whether batch inference can be integrated into the standard ETL process. Simon argues that teams should work in the same process while accepting that a strict bronze-silver-gold design may need to be extended. The architecture should leave room for reuse without forcing every activity into an unsuitable box.
Documented transformations let teams judge bias for themselves
The closing discussion returns to communication and governance. Cleaning can be acceptable when it is documented and the data scientist knows what happened. A transformation may have introduced bias, or it may have completed work the model builder would have done anyway. Teams need enough information to make that decision and repeat the process. Simon treats this as a shared working agreement rather than a problem that one team can solve alone.
"In reality if I'm saying what are my different AI personas and which bit of The Medallion did they go to, well all of them, but for different reasons and different use cases."07:35
Who should watch
You are designing a lakehouse and are unsure whether bronze, silver, and gold are enough for your machine learning workflows.
Your models depend on cleaned or aggregated data, but you cannot see which transformations were applied before you received it.
You work across data engineering and AI teams and need a practical discussion about feature stores, inference outputs, and shared lineage.
