# Why DuckDB is the Future of Data

Prof. Dr. Hannes Mühleisen, DuckDB Labs | MLOps Community | 31:56

Source: https://www.youtube.com/watch?v=bi0XhmbkqU8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/why-duckdb-is-the-future-of-data
Published: 2024-10-09
Tags: data-engineering, deployment, edge, open-source

## TL;DR
- DuckDB runs inside an application process, which removes the need to operate a separate database server and makes data movement with Python, pandas, and Arrow direct.
- DuckDB can replace parts of heavier data architectures by running in operational servers, lambdas, application servers, browsers, and mobile devices.
- DuckDB is designed to handle serious analytical workloads, with broad SQL support, persistence, transactions, cloud storage access, parallel execution, and out-of-core processing.

## Summary
Prof. Dr. Hannes Mühleisen presents DuckDB as an analytical query engine that can run wherever data processing is needed. Its in-process design removes the need for a separate database server, containers, or database administration for many workloads. DuckDB still supports serious analytical features, including complex SQL, transactions, persistence, cloud storage, Parquet, Iceberg, Delta, and clients for several languages. Mühleisen describes uses ranging from interactive analysis in Python to enterprise pipelines and serverless processing. He argues that DuckDB can simplify data architectures by moving work closer to operational databases, app servers, browsers, and devices. The talk also covers direct integration with pandas and Arrow, browser-based SQL through WebAssembly, and processing data larger than memory. In the discussion, he explains partial downloads from Parquet, running DuckDB near S3, and why DuckDB can be more efficient than Spark on a single node, while acknowledging that some format support is still developing.

## Key ideas
### DuckDB is designed to make large-scale data work less intimidating
[02:52](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=172s)
Mühleisen says DuckDB grew from a mission to improve people's difficult relationship with data. He describes the fear that starts when data no longer fits in Excel, when a pandas workflow might need to become a Spark platform, or when database administrators become gatekeepers. DuckDB's design focuses on the full user experience, including ingesting data, querying it, and getting results back out. He contrasts this with systems built mainly around query algorithms and benchmark performance. The goal is for people to work confidently with very large datasets instead of treating data tools as hostile or unwieldy.

### The in-process architecture removes much of the work around running a database
[05:54](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=354s)
DuckDB runs inside the process chosen by the user, often a Python process, instead of on a separate database server. Mühleisen says this avoids setting up a server, running a Docker container, upgrading a database server, or asking for installation permission. Users can install it through package managers such as pip, brew, npm, or system packages. Because the database and application share a process, data can move without sockets or serialization. This makes the architecture simple to deploy while keeping the database close to the code that needs the data.

### DuckDB combines a small deployment model with a broad analytical feature set
[10:50](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=650s)
Mühleisen rejects the idea that an in-process database is only useful for small experiments. DuckDB supports complex SQL, transactions, persistence, and direct reading of CSV, Parquet, and JSON. It connects to cloud storage such as S3, Azure, and Google Cloud Storage, and reads formats including Iceberg and Delta. It also connects to PostgreSQL and MySQL and has clients for languages including Python, R, and Java. The engine automatically parallelizes transformations across available CPU cores and can use disk when an operation runs out of memory.

### DuckDB supports both interactive analysis and large data pipelines
[10:20](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=620s)
Mühleisen describes three broad patterns. Users can load DuckDB in a Python shell, query files, and persist results on a laptop. Companies also use it as a component inside large enterprise pipelines. A third pattern places DuckDB inside new kinds of deployments, including lambdas and WebAssembly applications. The Python integration is especially important to the project. Mühleisen says Python is DuckDB's largest API by download count, with around six million downloads per month at the time of the talk. DuckDB can read pandas data frames and Arrow data directly within the process.

### DuckDB can simplify the data lake path by doing work earlier
[15:25](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=925s)
Mühleisen proposes adding DuckDB alongside operational databases rather than replacing them. A PostgreSQL plugin called PG DuckDB can run DuckDB inside a PostgreSQL server. This allows teams to pre-aggregate, filter, enrich, and encode data into Parquet before uploading it to a data lake. The approach can remove a separate transformation step and reduce the complexity of the architecture. He cites Fivetran as an example of a company using DuckDB in its data lake writer for transformations when writing to Iceberg.

### Serverless and application-local processing can replace fixed analytical clusters
[17:27](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=1047s)
DuckDB has a startup time of a few milliseconds and a binary of roughly 20 megabytes, according to Mühleisen. That makes it possible to use lambdas or a small virtual machine for work that might otherwise require a continuously running Spark or Redshift cluster. He describes Boiling Data using a fleet of lambdas for parallel SQL queries and Octar running DuckDB in lambdas after ingesting data into S3. DuckDB can also run inside an application server, allowing requests to transform data close to the client and avoiding a separate database protocol.

### Running DuckDB in browsers and devices changes where analytical queries can happen
[20:20](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=1220s)
DuckDB can run in desktop applications, browsers through WebAssembly, and iOS or Android apps. Mühleisen says browser execution can reduce dashboard response times from multi-second network round trips to millisecond-scale local results. He shows examples involving Observable, Hugging Face datasets, and a web shell. Hugging Face uses the WebAssembly version for its dataset viewer, and Mühleisen says its datasets can be queried through a SQL button in the browser. On-device processing can also keep data available when the network is down.

### DuckDB can process data beyond available memory and does not require local copies of everything
[27:47](https://www.youtube.com/watch?v=bi0XhmbkqU8&t=1667s)
In the discussion, Mühleisen explains that pandas itself requires data to fit in memory, but DuckDB can process inputs, outputs, and intermediate results that exceed available memory. Its joins, aggregates, and sorts support out-of-core execution. For remote datasets, Parquet can allow partial downloads so a query retrieves only the data it needs. Users can also run DuckDB on virtual machines near S3 instead of downloading large datasets to a local computer. He says DuckDB usually outperforms Spark on a single node, while noting that Iceberg support was still developing at the time.

## Notable quotes
- "We want to actually build this confidence that dealing with data isn't necessarily something that you have to fear." (04:33)
- "This in-process architecture is something that we didn't pick at random. It's something that we picked because we wanted to actually build something that's easy to use." (06:50)
- "DuckDB can be everywhere in your stack." (22:37)
- "You don't have to have a centralized data engine anymore and we can still put that anywhere." (24:08)

## Tools & references mentioned
- DuckDB
- DuckDB Labs
- PG DuckDB
- Python
- pandas
- Apache Arrow
- CSV
- Parquet
- JSON
- Amazon S3
- Azure
- Iceberg
- Delta Lake
- PostgreSQL
- MySQL
- Fivetran
- Spark
- Redshift
- Boiling Data
- Octar
- WebAssembly
- Hugging Face
- Observable
- MotherDuck
- Dart

## Who should watch
- You are deciding whether an analytical workload needs a separate database server or cluster, and you want to see what an in-process option changes.
- Your Python or data pipeline work moves between pandas, Parquet, cloud storage, and heavier systems such as Spark.
- You are building a browser, mobile, serverless, or embedded application that needs local SQL and analytical processing.

## Related talks

- [DuckDB is fast for analytics, but what can it do for AI?](https://mlopstalks.com/talks/duckdb-is-fast-for-analytics-but-what-can-it-do-for-ai) (Mehdi Ouazza, MotherDuck, 12:49)
- [Small Data, Big Impact: The Story Behind DuckDB](https://mlopstalks.com/talks/small-data-big-impact-the-story-behind-duckdb) (Hannes Mühleisen, DuckDB Labs & Jordan Tigani, MotherDuck, 1:08:35)
- [Office Hours on DuckDB, AWS Glue, and Iceberg](https://mlopstalks.com/talks/office-hours-on-duckdb-aws-glue-and-iceberg) (, 1:57:51)
- [Deep in the Heart of Data](https://mlopstalks.com/talks/deep-in-the-heart-of-data) (Carl Steinbach, LinkedIn, 55:27)
- [Python Power: How Daft Embeds Models and Revolutionizes Data Processing](https://mlopstalks.com/talks/python-power-how-daft-embeds-models-and-revolutionizes-data-processing) (Sammy Sidhu, Eventual, 51:30)