# MLOps Coffee Sessions #14 Conversation with the Creators of Dask

Hugo Bowne & Matthew Rocklin, Coiled | MLOps Coffee Sessions | Episode 14 | 56:27

Source: https://www.youtube.com/watch?v=tXckHqfkPqg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/mlops-coffee-sessions-14-conversation-with-the-creators-of-dask
Published: 2020-10-13
Tags: data-engineering, deployment, open-source, orchestration

## TL;DR
- Dask grew from an effort to parallelize the Python data stack, then became a general-purpose task scheduler used by other libraries and domains.
- Coiled packages the operational work around Dask so data scientists can use larger cloud or on-premise clusters without managing every infrastructure detail.
- Dask keeps Python's local, exploratory workflow while adding distributed execution, fault tolerance, and access to larger datasets.

## Summary
Matthew Rocklin and Hugo Bowne explain why Dask was created and how Coiled grew around it. Rocklin describes Dask's original goal, parallel versions of NumPy, pandas, and scikit-learn, and its later role as a general-purpose dynamic task scheduler. Bowne connects the project to his work helping researchers gain access to data skills and tools. The conversation focuses on the gap between exploratory data science and the operational demands of clusters, containers, networking, security, and resource management. Rocklin argues that Dask should make a cluster feel more like a local Python environment, while Coiled handles infrastructure around the open-source project. They also discuss commercial open source, enterprise support, governance, contributions, and the risk of large companies competing with open-source businesses. The episode closes with Dask's use across more fields and the goal of making distributed computing available to smaller teams and less specialized users.

## Key ideas
### Dask began by extending the Python data stack beyond one machine
[03:30](https://www.youtube.com/watch?v=tXckHqfkPqg&t=210s)
Matthew Rocklin says Python data science worked well on a single machine, but NumPy, pandas, and early machine learning tools did not scale easily to multiple machines, multiple cores, or datasets larger than memory. While at Anaconda, he saw this as a structural problem for the ecosystem. Dask was first designed as a parallel version of NumPy, with the expectation that the rest of the stack would build on it. That plan changed as the project was adapted to fit many existing libraries. Rocklin says this work reached cancer researchers, genomicists, climate scientists, and other users through the libraries they already used.

### Coiled aims to remove infrastructure work from ordinary data science
[01:59](https://www.youtube.com/watch?v=tXckHqfkPqg&t=119s)
Hugo Bowne describes Coiled as a company building hosted deployments for data scientists and machine learning practitioners. The goal is to make it easier to run data science work on cloud and on-premise clusters. He connects this to his earlier work in education and research, where people often had data but lacked the tools or skills to analyze it. Bowne says a working data scientist may need software engineering and DevOps knowledge, including Kubernetes and Docker, but many would rather focus on analysis. Coiled is intended to lower that operational barrier while keeping the open-source Python workflow accessible.

### Tools can carry practices to people who do not know them yet
[10:04](https://www.youtube.com/watch?v=tXckHqfkPqg&t=604s)
Rocklin says software can encode a practice after engineers have repeated a task enough times to build a library around it. It can also communicate practices to users who do not know them, such as security or authentication procedures that data scientists may not understand in detail. Bowne extends this idea to products that use machine learning. A hiring system may not need to teach an HR leader k-fold cross-validation, but the user should understand false positives, false negatives, confusion matrices, and bias. Their concern is that powerful products can make users dangerous if education and safe defaults are absent.

### Visibility lets teams use compute without hiding its costs
[14:44](https://www.youtube.com/watch?v=tXckHqfkPqg&t=884s)
Rocklin says giving data scientists more compute should come with visibility into what the system is doing. Users need to understand why work is fast or slow. IT teams need to see cluster utilization and decide whether controls are needed. The person paying for the system may want to know how much was spent parsing CSV files and what a different format could save. He points to Dask's visual dashboard, which was originally built to help him develop scheduling algorithms. Once users could see the same information, the dashboard also helped them understand the system. Bowne connects this visibility to sharing knowledge and reducing dependence on one person.

### Exploratory data science is difficult to reproduce on a cluster
[19:32](https://www.youtube.com/watch?v=tXckHqfkPqg&t=1172s)
Bowne calls data science an amorphous collection of techniques, tools, methods, and questions rather than one fixed activity. Rocklin says exploratory machine learning depends on novelty. Practitioners may move between Jupyter, Excel, CPUs, GPUs, FPGAs, phone calls, and other resources while solving a new problem. Recreating that flexible laptop experience on a cloud cluster introduces containers, web interfaces, proxies, storage, and deployment concerns. Dask tries to make a group of machines feel more like a local Python process. Coiled addresses the infrastructure problems that remain around that abstraction.

### Commercial open-source companies can fill the support and operations gap
[24:07](https://www.youtube.com/watch?v=tXckHqfkPqg&t=1447s)
Rocklin says open source removed parts of older software stacks, including support contracts, distribution systems, funded maintenance, and direct relationships with developers. Commercial open-source companies can provide those missing pieces through support, funded development, and hosted services. They can also communicate with both enterprise buyers and open-source communities, which may otherwise struggle to work together contractually. Bowne describes a related gap between individual contributors and the relationships between IT, management, and data science. He says companies can address those organizational connections with support, training, and products.

### Dask is both a Python data API and a general task scheduler
[37:29](https://www.youtube.com/watch?v=tXckHqfkPqg&t=2249s)
Rocklin explains that Dask can scale familiar NumPy, pandas, and scikit-learn-style work from a single machine to larger memory systems or clusters. Its internal engine is a dynamic distributed task scheduler. Users can submit ordinary Python functions, add dependencies between tasks, manage data placement, respond to worker failures, and update graphs as new files arrive. Many users care more about this engine than about the high-level data APIs, so other libraries build on it. Rocklin describes Dask as lower level than Spark and less opinionated, which lets people build different kinds of applications without replacing the wider Python ecosystem.

### Parallelism is a scheduling problem as well as a systems problem
[39:19](https://www.youtube.com/watch?v=tXckHqfkPqg&t=2359s)
Rocklin uses an Amish barn raising to explain parallelism. Many people work toward one outcome, so someone must decide which worker does which task, when tasks can start, and how to coordinate them. The analogy also raises questions about failures and downtime. In computing, the problem includes planning work and dealing with hardware, networks, compression libraries, and cluster systems. Rocklin places Dask between high-level systems that impose a complete application model and low-level tools such as MPI, ZeroMQ, sockets, threads, and queues. Dask handles common coordination patterns without requiring users to manage every message and failure themselves.

### Dask's lightweight design helps it fit the existing Python community
[50:01](https://www.youtube.com/watch?v=tXckHqfkPqg&t=3001s)
Rocklin says Dask is easy to install, runs on laptops by default, and can grow to large systems. Its lightweight design also makes it easier for other library authors to integrate. He contrasts this with Spark, which builds a separate stack with Spark data frames and machine learning libraries. Dask adds parallelism to existing Python libraries instead of rebuilding the whole ecosystem. Bowne says this matters for people who learned Python after working in Excel. Dask can meet them with APIs resembling pandas and NumPy, while still allowing the same Python code to run across larger systems.

## Notable quotes
- Matthew Rocklin: "Dask was actually originally designed to be like a parallel NumPy." (04:27)
- Hugo Bowne: "Data science isn't a single thing. It's an amorphous blob of ad-hoc techniques and tools and methodologies and questions that have kind of been thrown into this giant bucket." (19:26)
- Matthew Rocklin: "Dask was really designed to pretend that your cluster of machines was like your laptop to a certain extent." (23:23)
- Matthew Rocklin: "Dask is also sort of a low-level general-purpose library for parallel computing." (38:25)
- Matthew Rocklin: "That's parallelism, right? You've got all these resources, you've got this thing you want to do, how are you going to plan things together?" (40:22)

## Tools & references mentioned
- Dask
- Coiled
- NumPy
- pandas
- scikit-learn
- Jupyter
- TensorFlow
- PyTorch
- Theano
- Anaconda
- Kubernetes
- Docker
- Weights & Biases
- Neptune AI
- Spark
- Hadoop
- MapReduce
- MPI
- ZeroMQ
- Prefect
- BlazingDB
- MongoDB
- Confluent
- Apache Arrow
- Kubeflow
- Snowflake
- Google

## Who should watch
- You are a Python data scientist who needs larger-than-memory datasets or cluster compute but does not want to rebuild your workflow around a separate data platform.
- Your team is evaluating a hosted service around an open-source project and wants to understand the trade-offs between community software, commercial support, and possible cloud-provider competition.
- You maintain Python data tools and want to understand why a lightweight task scheduler can be easier to integrate than a complete replacement stack.

## Editor's note

Matthew Rocklin says recreating a flexible laptop experience on a cloud cluster brings containers, web interfaces, proxies, storage, and deployment concerns. ZenML lets teams write workflows as Python pipelines while choosing the infrastructure underneath, so the same pipeline code can run on a laptop or on Kubernetes. Each run records its steps, inputs, outputs, and code version for later tracing.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Scalable Python for Everyone, Everywhere](https://mlopstalks.com/talks/scalable-python-for-everyone-everywhere) (Matthew Rocklin, Coiled Computing, 57:07)
- [Python and Dask: Scaling the DataFrame](https://mlopstalks.com/talks/python-and-dask-scaling-the-dataframe) (Dan Gerlanc, Enplus Advisors, 1:28:56)
- [MLOps at DoorDash](https://mlopstalks.com/talks/mlops-at-doordash) (Hien Luu & DoorDash Leads, DoorDash, 45:20)
- [Airflow Sucks for MLOps](https://mlopstalks.com/talks/airflow-sucks-for-mlops) (Stephen Bailey, Whatnot, 1:05:53)
- [MLOps Investments](https://mlopstalks.com/talks/mlops-investments) (Sarah Catanzaro, Amplify Partners, 46:18)