Iceberg separates data files, metadata, and catalog information so different query engines can work with the same data on object storage.
2
DuckDB and Python could make lakehouse data easier for data scientists to query, but the current integration had read, write, and authentication gaps.
3
AWS Athena and PyIceberg worked for parts of the demo, while DuckDB could not be made to query the AWS Glue catalog successfully.
Summary
Eric, Hossam, and Demetrios explain how analytical databases differ from transactional databases, then build toward a serverless lakehouse using S3, Apache Parquet, Iceberg, and the AWS Glue catalog. The talk focuses on separating storage from compute and reducing vendor lock-in. Iceberg stores Parquet data alongside metadata that supports table discovery, efficient reads, transactions, and time travel. The demonstration creates an S3 bucket and Glue database with Pulumi, registers New York City taxi data with PyIceberg, and queries it through Athena. The speakers also test DuckDB as a lighter Python-friendly alternative to Spark. That attempt exposes the current friction: DuckDB lacks Iceberg write support, the Glue catalog connection fails with authentication errors, and PyIceberg upserts fail on the example data. The conclusion is cautious. Athena and Spark remain practical choices, while the broader Python and C++ ecosystem around Iceberg still needs more mature integrations.
Analytical workloads need different storage and compute than transactional applications
Eric contrasts PostgreSQL and MySQL with lakehouse systems. Transactional databases handle small numbers of rows, joins, and fast application requests. Analytics queries scan long periods of historical data and calculate over many rows. Companies often use both: a transactional database for an application and a lakehouse for reporting, model training sets, inference data, and historical analysis. Snowflake and Databricks provide this at large scale, but their cost and operational model can be excessive for smaller projects.
Separating storage from compute reduces infrastructure lock-in
The talk traces a shift from servers where storage and compute were tied together to systems that keep data in S3 or another object store and provision compute when needed. Storage can grow independently, while query compute can scale down when no queries are running. Open table formats extend that separation. If data is stored in an open format on blob storage, Spark, Athena, DuckDB, Python, or another compatible engine can query it instead of leaving the data tied to one vendor.
Parquet makes analytical reads cheaper by storing typed columns
Eric explains why Parquet is more suitable for analytical data than plain CSV. Parquet stores values by column, includes types, and compresses the data. A query that calculates a median or other aggregate can read the relevant column rather than stepping through every row. Parquet also has pages and other internal structures that help engines read only the parts they need. Its broad library support makes it a practical standard across languages and vendors.
Iceberg adds database-like metadata and time travel around Parquet files
Iceberg keeps the data in Parquet files and adds metadata plus a catalog that maps tables to those files. Metadata can contain summaries that let a query avoid scanning every object in S3. Iceberg also tracks table history, so a reader can query the state of a table before later files were added. The format supplies database-style properties such as atomic writes and consistency when clients follow its read and write rules.
Versioned lakehouse tables can reduce the cost of reproducing model training data
For machine learning, the speakers connect Iceberg time travel to reproducibility. An audited model may require the exact data, code, and libraries used during training. Eric describes data version control as copying the full training set for each version, which consumes storage. With an Iceberg table, a training run could record a timestamp and query the table as it existed at that point, although the SQL used to produce the final training set may also need to be saved.
Duck Lake puts table metadata in a database for small writes
The discussion introduces Duck Lake, which uses a PostgreSQL database for table metadata rather than keeping every piece of metadata in object-store files. The motivation is that small writes can create many Parquet and metadata files, and object storage is slow and costly when a workload requires many round trips. The speakers describe Duck Lake as mostly based on the same ideas as Iceberg, with a different choice for where metadata lives.
A serverless AWS lakehouse can combine S3, Glue, PyIceberg, and Athena
The demo uses New York City taxi Parquet files, S3 for physical storage, and AWS Glue as the Iceberg catalog. PyIceberg registers the table schema and location, then writes data and metadata to S3. Athena can query the resulting table without first running a Glue crawler because the catalog metadata was registered directly. Athena supplies distributed SQL compute while the table data remains in object storage.
DuckDB and Python are promising for last-mile work, but the ecosystem is immature
Eric argues that many data science workloads can reduce a large source to a smaller training set before running Python on one machine. This avoids Spark cluster setup, job-submission overhead, and unfamiliar data-frame APIs. The demo could not complete the desired DuckDB path: authentication to the Glue catalog returned a 403 error, DuckDB had no Iceberg write support, and PyIceberg upserts failed on the taxi data. The speakers therefore describe the approach as promising but unfinished.
"So what I'm trying to get at is I think spark is sort of like it's a necessary evil for many problems that involve big data because the ecosystem's not there and the DX isn't as strong."Eric44:33
Who should watch
You are deciding whether an open table format can reduce dependence on Snowflake, Databricks, or Spark.
You want to understand how S3, Parquet, Iceberg, and the AWS Glue catalog fit together in a working data path.
You are a data scientist considering Python or DuckDB for lakehouse queries and want to see the integration problems before adopting it.
