# Putting the AI back in Medallion Lake Design Simon Whiteley, Advancing Analytics | MLOps Community | 13:28 Source: https://www.youtube.com/watch?v=OqsXI1U0Xls Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/putting-the-ai-back-in-medallion-lake-design Published: 2024-10-09 Tags: data-engineering, feature-engineering, feature-stores ## TL;DR - Medallion architecture simplifies data work into bronze, silver, and gold, but AI workflows often need data from any layer and do not fit neatly into those boxes. - Machine learning teams need visibility into the cleaning and transformations applied to each layer because cleaning can remove signals or introduce bias. - AI engineers and data engineers should agree where cleaned data, feature stores, and inference outputs belong in a shared lakehouse process. ## Summary Simon Whiteley explains why bronze, silver, and gold are useful shorthand for lakehouse design but poor as rigid architectural rules. Traditional analytics pipelines move from raw data through cleaning into reusable business models. Machine learning workflows have different needs. They may use raw, cleaned, or business-level data, depending on whether earlier transformations removed useful predictive signals. Feature engineering, feature stores, and inference outputs also sit awkwardly beside the usual Medallion layers. Simon describes designs with additional stages for structured landed data, canonical models, feature stores, analytical models, and semantic models. The right number of layers depends on the use case. A small system may need two layers, while a large enterprise may use six. His practical advice is for AI and data teams to share lineage, document transformations, and decide together where reusable data products and inference results should live. ## Key ideas ### Data engineering grew from two different needs [01:18](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=78s) Simon describes two common backgrounds for data engineers. Software engineers moved toward data to build scalable, repeatable pipelines for training datasets and machine learning models. Traditional SQL and analytics engineers built generic models, such as star schemas, that could answer many business questions. Both groups use engineering practices to prepare and transform data, but they optimize for different kinds of reuse. One may build a pipeline for a particular model, while the other creates shared data structures for many consumers. ### Bronze, silver, and gold repeat an older warehouse pattern [02:43](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=163s) The Medallion architecture gives new names to a familiar process. Bronze contains raw data, which may include duplicates, bad values, or the wrong source data. Silver applies cleaning, filtering, and standardization. Gold contains tidy, business-level aggregates. Simon compares this with older warehouse work, where teams landed data, cleaned it, and then built facts and dimensions. Reusing one cleaned dataset across many dimensions was already a central idea. ### Machine learning may need data from any layer [05:17](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=317s) A model can use bronze, silver, or gold data, depending on the problem. Silver is convenient because someone has already cleaned it, standardized addresses, or removed personally identifiable information. For fraud detection in insurance, that cleaning may remove predictive signals. The team may therefore need raw data and its own cleaning process. Gold can also be useful when business logic and product relationships have already been assembled. The choice depends on what the model needs. ### Lineage tells model builders whether cleaning is safe to reuse [06:47](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=407s) Simon says teams should be able to see what changed between layers. A model builder needs to know which transformations were applied, whether data was removed or standardized, and whether those changes could affect the model. This lets the team decide whether to reuse an existing cleaned dataset or start from a less processed source. Without that visibility, a data scientist may receive data that already contains assumptions or bias without knowing it. ### Feature stores and inference outputs do not fit neatly into three layers [07:47](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=467s) Feature extraction, feature engineering, and feature stores often sit beside the standard Medallion flow. A batch inference result may be an augmented table with an extra column rather than a finished business data model. It may technically belong in gold because it has been curated, but it does not necessarily fit the usual meaning of that layer. This mismatch explains why data science teams can struggle when engineers treat bronze, silver, and gold as complete descriptions of the architecture. ### Mature lakehouses often need more than three layers [09:17](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=557s) Simon gives a broader progression: raw files can be landed, structurally validated, and turned into standard tables without changing their values. Text documents may be chunked for a vector database. A later layer can clean values, and another can combine multiple ERP systems into a canonical model. Feature stores, analytical models, and semantic models may follow. A simple use case may need two layers, while a large enterprise may use as many as six. ### AI and data teams should design the flow together [11:11](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=671s) AI engineers should discuss with data engineers where cleaned data belongs, how a feature store will be managed, and whether batch inference can be integrated into the standard ETL process. Simon argues that teams should work in the same process while accepting that a strict bronze-silver-gold design may need to be extended. The architecture should leave room for reuse without forcing every activity into an unsuitable box. ### Documented transformations let teams judge bias for themselves [12:04](https://www.youtube.com/watch?v=OqsXI1U0Xls&t=724s) The closing discussion returns to communication and governance. Cleaning can be acceptable when it is documented and the data scientist knows what happened. A transformation may have introduced bias, or it may have completed work the model builder would have done anyway. Teams need enough information to make that decision and repeat the process. Simon treats this as a shared working agreement rather than a problem that one team can solve alone. ## Notable quotes - "The problem with The Medallion architecture is it's oversimplified, is trying to put things into three boxes and everyone always believes it." (09:19) - "I should be able to see between those different layers what cleaning has been done." (06:47) - "In reality if I'm saying what are my different AI personas and which bit of The Medallion did they go to, well all of them, but for different reasons and different use cases." (07:35) - "We should be talking to each other, working in the same way, but also understanding that if it's just bronze silver gold you're going to have to break the rules." (11:11) ## Tools & references mentioned - Advancing Analytics - Databricks - Delta ## Who should watch - You are designing a lakehouse and are unsure whether bronze, silver, and gold are enough for your machine learning workflows. - Your models depend on cleaned or aggregated data, but you cannot see which transformations were applied before you received it. - You work across data engineering and AI teams and need a practical discussion about feature stores, inference outputs, and shared lineage. ## Related talks - [Building a Data Infrastructure for AI/ML](https://mlopstalks.com/talks/building-a-data-infrastructure-for-ai-ml) (Vinoth Chandar, Onehouse, 29:41) - [AI Operations Without Fundamental Engineering Discipline](https://mlopstalks.com/talks/ai-operations-without-fundamental-engineering-discipline) (Nikhil Suresh, Hermit Tech, 49:28) - [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12) - [How To Move From Barely Doing BI to Doing AI](https://mlopstalks.com/talks/how-to-move-from-barely-doing-bi-to-doing-ai) (Joe Reis, Ternary Data, 53:47) - [Engineering Your AI Platform](https://mlopstalks.com/talks/engineering-your-ai-platform) (Tobias Macey, Massachusetts Institute of Technology (MIT) & Daniel Svonava, Superlinked & Colleen Tartow, VAST Data, 30:09)