# Python and Dask: Scaling the DataFrame

Dan Gerlanc, Enplus Advisors | MLOps Meetup | Episode 25 | 1:28:56
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=wUokn3RQ9_A
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/python-and-dask-scaling-the-dataframe
Published: 2020-07-14
Tags: data-engineering, workshop

## TL;DR
- Dask scales familiar pandas-style DataFrame work across partitions, cores, and machines while keeping the computation in Python.
- Dask builds a lazy task graph and executes it only when you call compute, which lets it find opportunities for parallel work.
- Distributed computing adds coordination and data-transfer costs, so an in-memory pandas solution is usually faster when the data fits on one machine.

## Summary
Dan Gerlanc gives a practical introduction to Dask DataFrames. He explains how Dask keeps the pandas mental model while splitting a logical DataFrame into smaller pandas DataFrames, or partitions, that can run across workers. Dask builds a directed acyclic graph of operations instead of executing each statement immediately. Calling compute schedules that graph, while persist keeps computed partitions in memory for later work. The workshop uses New York City payroll data to demonstrate reading multiple CSV files, inspecting partitions, calculating maxima, and grouping data. Dan also uses the Dask dashboard to show workers, task execution, waiting time, and data transfers. He compares Dask with Spark and PySpark, describing Dask as easier to start with for Python teams and more flexible for arbitrary Python functions. He is clear about the cost: distributed execution adds overhead, so data that fits in RAM should usually stay in memory.

## Key ideas
### Dask keeps the DataFrame model by partitioning pandas DataFrames
[06:03](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=363s)
Dask DataFrames are made from multiple smaller pandas DataFrames. One larger logical DataFrame can be split into partitions in memory on one computer or across a cluster. The partitions provide the source of parallelism and let Dask work through data that is too large for one machine's memory. Dask handles the chunking, so users do not have to manually read, process, and combine one piece at a time. In the demonstration, Dask reads New York City payroll CSV files and creates three partitions by default, then creates 22 partitions when Dan sets a smaller block size of half a megabyte.

### Dask delays execution by building a directed acyclic graph
[18:45](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=1125s)
A Dask operation creates a directed acyclic graph of functions rather than immediately producing a result. Reading a CSV creates separate read tasks for the partitions. Selecting a column and calculating its maximum adds more tasks to that graph. The independent file reads and per-partition maxima can run in parallel, while the final aggregation waits for those results. Dask only performs the work after the user calls compute. This lets the scheduler inspect the whole computation and find parallel work without the user describing how to divide it across workers.

### Dask can run locally before it runs across a cluster
[12:05](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=725s)
Dan starts Dask's distributed scheduler by importing the client and instantiating it. The same approach can create a local cluster on a laptop or connect to a larger cluster. In his example, the local setup exposes four workers, eight threads, and the laptop's available memory. Dask can also run through Binder, locally with JupyterLab, on HPC clusters, on existing YARN clusters, or across instances connected with SSH. Dan describes it as lightweight because it does not require HDFS or YARN, although those systems can be used when available.

### compute performs the graph, while persist caches its partitions
[21:16](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=1276s)
Calling compute turns a lazy Dask graph into an answer and makes the scheduler execute its tasks. Dan points out that Dask does not automatically keep the data in memory. After showing the head of a DataFrame, a later maximum calculation reads the CSV again because the earlier result was not persisted. Calling persist tells Dask to keep the computed partitions in memory on the cluster. The resulting graph contains the in-memory partitions instead of the original file-reading tasks, so later calculations can start from those partitions.

### Distributed execution is slower when the data already fits in memory
[33:02](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=1982s)
Dan compares pandas and Dask on the same kind of calculation. Pandas is fast after the data is loaded into memory, while Dask carries the extra cost of coordinating workers and serializing data. He says that a distributed solution should be used when it solves a real size or compute problem. If the dataset fits in RAM, the in-memory approach has less work to do. The Dask documentation describes parallelism and distributed computing as expensive ways to accelerate an application that could instead use memory, a faster algorithm, or compiled code.

### Dask gives Python teams arbitrary functions and a simpler execution model than Spark
[43:33](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=2613s)
Dan contrasts Dask with PySpark and Koalas. Spark runs on the JVM and Python calls can involve a separate Python process with serialized data, while Dask is written in Python from the workers through the scheduler. Dask can use regular Python functions and integrate with pandas, NumPy, and scikit-learn. Its lower-level primitives give users more control than Spark's higher-level operators, although that flexibility can give the scheduler fewer obvious opportunities for optimization. Dan prefers starting with Dask when a team needs Python-based distributed processing and does not have Spark specialists.

### The dashboard shows whether workers are computing or waiting
[1:12:14](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=4334s)
Dask's dashboard shows the task stream as a timeline. Each rectangle is an individual task, each color usually identifies a function, and each bar corresponds to a worker or execution thread. Filled areas show computation, while white space shows a worker waiting. Red areas indicate data transfer between workers. Dan uses the dashboard to explain why a computation can finish with one worker holding the final result after intermediate values move across the cluster. A graph with frequent gaps can mean the tasks are too small, because network and scheduling overhead takes longer than the computation.

### Task size determines whether parallelism helps
[1:20:48](https://www.youtube.com/watch?v=wUokn3RQ9_A&t=4848s)
Dan changes the chunk size of a random-array calculation to show poor worker utilization. If tasks are too small, workers finish quickly and spend most of their time waiting for the next task or transferring results. Adding two numbers is cheap compared with moving those numbers across the network, while summing a much larger batch can make the transfer cost worthwhile. The dashboard makes this visible through white space and data-transfer blocks. The practical point is to choose work units large enough to keep workers busy without creating partitions that are too large for available memory.

## Notable quotes
- Dan Gerlanc: "Dask is a library that you can think of for parallel execution or asynchronous execution on task graphs." (09:30)
- Dan Gerlanc: "Dask DataFrames under the hood are just pandas DataFrames except they are partitioned into multiple smaller DataFrames." (15:28)
- Dan Gerlanc: "You can't have it both ways. You can't store everything in memory and then expect it to also be able to work out of memory." (25:44)
- Dan Gerlanc: "Parallelism and distributed computing are expensive ways to accelerate your application that could otherwise be solved in memory or using a faster algorithm or compiled code." (39:59)
- Dan Gerlanc: "If the tasks are too small, they are going to be finishing too quickly and most of their time is just going to be waiting to get the next task assigned." (1:21:12)

## Tools & references mentioned
- Dask
- Python
- pandas
- NumPy
- scikit-learn
- Spark
- PySpark
- Koalas
- BlazingSQL
- RAPIDS AI
- NVIDIA
- Binder
- JupyterLab
- GitHub
- Hadoop MapReduce
- Chan Zuckerberg Foundation

## Who should watch
- You work with pandas DataFrames and need to process data that no longer fits comfortably in one machine's memory.
- Your team wants Python-native distributed processing and does not have deep Spark expertise.
- You need to understand why a distributed job is slow, especially when the dashboard shows waiting workers or heavy data transfer.

## Related talks

- [Scalable Python for Everyone, Everywhere](https://mlopstalks.com/talks/scalable-python-for-everyone-everywhere) (Matthew Rocklin, Coiled Computing, 57:07)
- [MLOps Coffee Sessions #14 Conversation with the Creators of Dask](https://mlopstalks.com/talks/mlops-coffee-sessions-14-conversation-with-the-creators-of-dask) (Hugo Bowne & Matthew Rocklin, Coiled, 56:27)
- [Dataframes Are All You Need: MLOps on Easy Mode](https://mlopstalks.com/talks/dataframes-are-all-you-need-mlops-on-easy-mode) (Jay Chia, Eventual, 57:25)
- [Python Power: How Daft Embeds Models and Revolutionizes Data Processing](https://mlopstalks.com/talks/python-power-how-daft-embeds-models-and-revolutionizes-data-processing) (Sammy Sidhu, Eventual, 51:30)
- [Scaling your data and AI from 0-100 with open source](https://mlopstalks.com/talks/scaling-your-data-and-ai-from-0-100-with-open-source) (Maarten Breddels, Pycafe & Pranav Aurora, Mooncake & Simba Khadder, Featureform, 1:09:51)