# Building a Data Infrastructure for AI/ML

Vinoth Chandar, Onehouse | MLOps Community | 29:41

Source: https://www.youtube.com/watch?v=EgUyPqvcevc
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-a-data-infrastructure-for-ai-ml
Published: 2024-10-09
Tags: data-engineering, embeddings, feature-engineering, open-source

## TL;DR
- AI and ML systems depend on data quality, governance, scale, and a durable storage layer.
- An open lakehouse on cloud storage keeps data usable across changing engines and frameworks.
- Lakehouse pipelines can reduce vector embedding costs by processing only the latest state of changed data.

## Summary
Vinoth Chandar argues that AI and ML projects need a durable data platform underneath them. Models depend on data quality, governance, scale, and careful control of training inputs. He recommends using cloud storage as a source of truth, with open storage and table formats that keep data available to different warehouses, engines, and ML frameworks. The talk then applies this architecture to feature engineering, model training, production features, vector embeddings, and retrieval applications. Chandar explains how Apache Hudi supports incremental processing when data changes, so teams do not repeatedly scan complete datasets. For generative AI, he describes a lakehouse as the place to generate and manage embeddings, with vector databases used where low-latency serving is needed. He also addresses cost. Point-in-time views can reduce repeated embedding calls, while less frequently used vectors can move out of expensive online databases. The talk makes a practical case for treating the lakehouse as a shared data foundation rather than tying data to one engine or serving system.

## Key ideas
### AI projects still depend on ordinary data platform work
[00:30](https://www.youtube.com/watch?v=EgUyPqvcevc&t=30s)
Chandar starts with infrastructure, which he calls foundational even though it is often invisible. The models are only as good as the data, so data quality, governance, storage, and platform basics still matter when projects move into production. Scale also needs attention early because new AI use cases consume data that was not previously processed in the same way. He warns that training or fine-tuning on the wrong dataset can waste a large budget. The surrounding tool ecosystem is still changing, so teams also need to expect gaps and new tools.

### Cloud storage and open formats preserve future choices
[05:28](https://www.youtube.com/watch?v=EgUyPqvcevc&t=328s)
Chandar recommends putting structured and unstructured data on cloud storage and treating it as the source of truth. Data should remain interoperable with warehouses, lake engines, and ML frameworks because compute systems will change over time. He also advises using open-source or cloud-agnostic components and open data and table formats. His concern is lock-in: a company with a large dataset in one system may be unable to use a newer engine without expensive movement or redesign. Keeping data in an open lakehouse preserves options and can lower storage costs.

### A lakehouse architecture connects ingestion, processing, analytics, and AI
[07:57](https://www.youtube.com/watch?v=EgUyPqvcevc&t=477s)
The example architecture begins with files, application events, and databases. Teams can use change data capture from databases, messaging or streaming platforms for application events, and cloud storage for file data. The resulting data can support business intelligence, AI, and data engineering transformations. Chandar says the lakehouse does not have to replace every specialized tool. Different engines fit different workloads, and many can query the lakehouse directly or receive data through simple transfers. Keeping the main copy in the lakehouse preserves optionality and avoids unnecessary duplication.

### Incremental processing keeps feature pipelines current
[11:06](https://www.youtube.com/watch?v=EgUyPqvcevc&t=666s)
For machine learning, teams join data from many sources, transform it into features, create training data, train models, and serve predictions. Chandar uses Uber examples such as ETA prediction and recommendation products. A lakehouse provides scalable storage and compute for these workloads and gives teams a central repository from which they can rebuild or rescore models. Apache Hudi was created partly to handle changing data incrementally. Instead of rereading and rescanning an entire dataset, pipelines can compute the changes to features as the underlying tables change.

### Vector embeddings add a data type that needs the same management controls
[14:39](https://www.youtube.com/watch?v=EgUyPqvcevc&t=879s)
Chandar describes a typical generative AI application as a process that maps source data into vector embeddings, uses similarity search to find relevant objects, and then supplies those results to a prompt or application. Vector databases can store and serve the embeddings, especially when low latency matters. However, embeddings also need the familiar data platform capabilities: storage and compute separation, update management, and interoperability. He argues that vector data should become part of the broader data stack instead of being scattered across disconnected systems.

### The lakehouse and vector database can have separate jobs
[17:17](https://www.youtube.com/watch?v=EgUyPqvcevc&t=1037s)
Embedding generation is a pipeline problem, while serving similarity searches in real time is a serving problem. Chandar says the lakehouse can sit between those components. It can hold the broader, managed copy of the data and feed a downstream vector database when an application needs low-latency queries. This arrangement lets teams use one open data foundation for analytics and AI while choosing a specialized serving system for the interactive path. He also describes use cases such as periodic AI summaries where a vector database may not be needed at all.

### Point-in-time views can reduce embedding and online database costs
[22:13](https://www.youtube.com/watch?v=EgUyPqvcevc&t=1333s)
In the questions, Chandar explains how cost can rise when a document changes repeatedly and each edit triggers embedding calls. A lakehouse pipeline can produce a point-in-time view of the latest state, so the system embeds one aggregated copy instead of making calls for every change. He also says teams often keep too much data in online vector databases because there is no suitable lower-cost store. Moving less frequently used data into the lakehouse can reduce the size of the online database and the number of required instances.

### A managed lakehouse can make open infrastructure accessible to smaller companies
[26:32](https://www.youtube.com/watch?v=EgUyPqvcevc&t=1592s)
Chandar says lakehouses were historically associated with large companies such as Facebook and LinkedIn, but data science, ML, and AI are now common across company sizes. The obstacle for smaller teams is the work needed to combine and operate multiple open-source projects. He describes Onehouse as an attempt to provide managed ease of use while retaining open data and support for multiple workloads. His broader recommendation is to make the lakehouse the default data architecture, rather than forcing teams to choose between a convenient proprietary warehouse and an open system they must build themselves.

## Notable quotes
- "The models are only as good as your data." (03:23)
- "Data is the only permanent thing." (05:59)
- "The data remaining at the lakehouse preserves optionality for you, lowers your cost, keeps your data free." (08:59)
- "The lakehouse is your base house for data infrastructure." (20:32)
- "You make one call versus hundreds or tens or thousands, so that saves you a lot of cost." (23:20)

## Tools & references mentioned
- Apache Hudi
- Onehouse
- Uber
- LinkedIn
- Oracle
- Confluent
- Apache Spark
- Apache Kafka
- Flink
- Delta Lake
- Iceberg
- Ray
- PostgreSQL
- OpenAI
- Databricks
- Snowflake

## Who should watch
- You are designing an AI or ML platform and need to decide where structured, unstructured, and vector data should live.
- Your feature or embedding pipelines repeatedly rescan changing data and drive up compute or model API costs.
- You want open data formats and several specialized engines without copying your main dataset into every system.

## Related talks

- [Putting the AI back in Medallion Lake Design](https://mlopstalks.com/talks/putting-the-ai-back-in-medallion-lake-design) (Simon Whiteley, Advancing Analytics, 13:28)
- [Engineering Your AI Platform](https://mlopstalks.com/talks/engineering-your-ai-platform) (Tobias Macey, Massachusetts Institute of Technology (MIT) & Daniel Svonava, Superlinked & Colleen Tartow, VAST Data, 30:09)
- [The Role of Infrastructure in ML Leveraging Open Source](https://mlopstalks.com/talks/the-role-of-infrastructure-in-ml-leveraging-open-source) (Niels Bantilan, Union.ai, 1:05:25)
- [Data Engineering + ML + Software Engineering](https://mlopstalks.com/talks/data-engineering-ml-software-engineering) (Satish Chandra Gupta, Slang Labs, 57:05)
- [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12)