# Small Data, Big Impact: The Story Behind DuckDB

Hannes Mühleisen, DuckDB Labs & Jordan Tigani, MotherDuck | MLOps Podcast | Episode 202 | 1:08:35
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Vs2nTt36Zd0
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/small-data-big-impact-the-story-behind-duckdb
Published: 2024-01-09
Tags: data-engineering, developer-experience, open-source

## TL;DR
- DuckDB began as a response to the poor experience of teaching large-scale data systems for workloads that often involved only hundreds of megabytes or a few gigabytes.
- DuckDB chose to be proudly single-node and in-process, which simplifies data movement and avoids many problems found in distributed systems.
- MotherDuck builds a managed cloud service around DuckDB while DuckDB Labs remains focused on developing the open-source database.

## Summary
Hannes Mühleisen explains that DuckDB started after researchers saw people struggle with large data systems while working with much smaller datasets. The project took an unusual path for 2018: it was designed to run on one machine and inside another process. Jordan Tigani recognised the opportunity after working on BigQuery and SingleStore, where scaling distributed systems down could be difficult. He describes MotherDuck as a separate company that provides a managed cloud service, with DuckDB Labs concentrating on the open-source database. Both guests return to the same design concern: the full user experience matters, from installation and CSV parsing to query execution and returning results. MotherDuck extends the local DuckDB experience into the cloud through local clients, hybrid execution, visualisation, and controls for data location. They also discuss Python integration, vector data, machine learning workflows, and planned text-to-SQL work.

## Key ideas
### DuckDB started by questioning the assumption that every workload needed a distributed system
[08:08](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=488s)
Hannes Mühleisen says the idea came from teaching people to use big data systems and seeing that the experience was poor. The team gradually questioned whether the prevailing model was necessary. Practitioners often had a few hundred megabytes of CSV files, yet there was little software designed for that situation. DuckDB became a deliberate departure from the belief that a database was worthless unless it scaled out. The researchers decided to solve a problem they had actually observed rather than inventing an abstract research challenge.

### DuckDB made single-node and in-process execution central architectural choices
[15:03](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=903s)
Hannes Mühleisen defines single-node execution as running on one computer, which can still be a large computer, instead of a collection of machines. DuckDB is also in-process, so applications interact with the database without a socket protocol. This makes data transfer with application code and third-party libraries more efficient. Hannes says these choices were unusual in 2018, but he remains happy with them. Jordan Tigani adds that distributed systems bring difficult coordination problems, including distributed two-phase commit, while a single-node system can move faster and avoid much of that complexity.

### Much of the data called big data is used in smaller working sets
[18:49](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=1129s)
Jordan Tigani describes how his experience with BigQuery changed his view of workload size. Even large customers often queried cleaned-up or summarised data rather than their full log tables. He says almost all BigQuery queries were below a terabyte, and most were below 100 megabytes. Large datasets may exist in storage, but users often work with recent hot data or a smaller derived table. This weakens the case for designing every workload around a distributed system. A single large machine can have substantial memory and many processors, while avoiding the operational cost of a cluster.

### DuckDB treats installation, ingestion, and results as part of database design
[39:23](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=2363s)
Hannes Mühleisen says DuckDB focuses on the complete experience around a query. The project has no external dependencies, so users can install it without asking an administrator and can compile it with a compiler. CSV files became an early test of this approach because users commonly start with them. DuckDB's team wrote a research paper about CSV reading and assigned a PhD researcher to the parser because getting data into the system matters as much as query execution. Hannes argues that a fast join is irrelevant if users cannot install the system or import their data.

### Community feedback gives DuckDB a practical stream of problems to solve
[52:41](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=3161s)
DuckDB Labs has people responsible for collecting signals from the community, including issue reports and Discord discussions. Hannes Mühleisen does not read every report, but repeated complaints make recurring problems visible. He also describes a meetup where someone raised an issue directly, and he worked on it soon afterward because the complaint was valid and actionable. The team knows that every reported problem may represent many silent users with the same experience. That makes feedback useful even when the volume is difficult to process.

### MotherDuck and DuckDB split responsibilities across two companies
[30:44](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=1844s)
Jordan Tigani initially planned to hack on a serverless or cloud version of DuckDB. After three days, he contacted the DuckDB team, and the groups agreed to partner rather than have DuckDB Labs build the service itself. DuckDB remains the open-source database, while MotherDuck is the managed service. They have separate staff and leadership, but the relationship includes a co-founder share and a development agreement. This structure lets DuckDB Labs focus on building the database while MotherDuck focuses on a differentiated cloud product. Jordan acknowledges that their incentives could diverge if competitors adopt DuckDB, so the partnership depends on trust.

### MotherDuck keeps DuckDB in the client so local and cloud data can work together
[57:00](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=3420s)
Jordan Tigani says every MotherDuck client includes a local DuckDB. The database can run in the browser through WebAssembly, and the same general client model works across DuckDB connectors. MotherDuck uses this setup for hybrid execution, such as joining a Pandas data frame in a Jupyter notebook with data stored in the cloud. It also supports highly reactive visualisations because the client can run database operations locally. The service adds capabilities expected from a cloud data warehouse, including user management, teamwork, larger datasets, durability, and time travel, while retaining the feel of local DuckDB.

### DuckDB fits machine learning workflows through Python, local execution, and efficient data handling
[1:03:06](https://www.youtube.com/watch?v=Vs2nTt36Zd0&t=3786s)
Hannes Mühleisen points out that DuckDB can run inside the same Python process as a machine learning model, which makes moving data between the database and the model inexpensive. He says this helps with tasks that machine learning frameworks often handle poorly, such as reading data and managing updates and consistency. DuckDB also added a fixed-size list type for vectors, avoiding repeated storage of each vector's length. Jordan Tigani adds that many machine learning algorithms scale up better than they scale out, making a local scale-up database useful. MotherDuck is also working on text-to-SQL features and a DuckDB-specific language model with Number Station AI.

## Notable quotes
- Hannes Mühleisen: "We were kind of building our system that was really proudly single node and proudly in process." (08:48)
- Jordan Tigani: "Most people don't have big data and this is sort of something that I'd seen in the real world." (21:12)
- Hannes Mühleisen: "If people can't get their data into your system, it doesn't matter how good your join operator is." (48:12)
- Jordan Tigani: "Every time you're talking to MotherDuck, there's a DuckDB locally." (57:42)
- Hannes Mühleisen: "We don't have to integrate a large language model ourselves, and I don't think we want to." (1:05:02)

## Tools & references mentioned
- DuckDB
- DuckDB Labs
- MotherDuck
- Google BigQuery
- SingleStore
- Looker
- Oracle
- Snowflake
- Amazon Web Services
- Python
- Pandas
- Jupyter
- WebAssembly
- Number Station AI
- S3
- CSV

## Who should watch
- You are choosing between a local analytical database and a distributed warehouse for datasets that may fit on one machine.
- You are building a data or machine learning product and want to improve the experience around installation, ingestion, querying, and results.
- You want to understand how an open-source database project can partner with a separate company building a managed cloud service.

## Related talks

- [DuckDB is fast for analytics, but what can it do for AI?](https://mlopstalks.com/talks/duckdb-is-fast-for-analytics-but-what-can-it-do-for-ai) (Mehdi Ouazza, MotherDuck, 12:49)
- [Why DuckDB is the Future of Data](https://mlopstalks.com/talks/why-duckdb-is-the-future-of-data) (Prof. Dr. Hannes Mühleisen, DuckDB Labs, 31:56)
- [Office Hours on DuckDB, AWS Glue, and Iceberg](https://mlopstalks.com/talks/office-hours-on-duckdb-aws-glue-and-iceberg) (, 1:57:51)
- [Machine Learning at Reasonable Scale](https://mlopstalks.com/talks/machine-learning-at-reasonable-scale) (Jacopo Tagliabue, Coveo, 1:04:32)
- [Building for Small Data Science Teams](https://mlopstalks.com/talks/building-for-small-data-science-teams) (James Lamb, SpotHero, 52:26)