# Modern Data Science with Vaex

Maarten Breddels, Vaex.io & Jovan Veljanoski, Tiqets | MLOps Meetup | Episode 97 | 1:11:16
Hosted by Ben Epstein

Source: https://www.youtube.com/watch?v=p__sVdwz8v8
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/modern-data-science-with-vaex
Published: 2022-04-18
Tags: data-engineering, feature-engineering, open-source

## TL;DR
- Vaex lets users explore datasets larger than RAM on a single machine through memory mapping, column-based storage, lazy expressions, and out-of-core computation.
- Vaex can process a 1.1 billion-row, 100 GB taxi dataset interactively, including filtering, grouping, statistics, and visualizations.
- Vaex ML stores feature engineering, model predictions, and post-processing in a computational graph that can be saved and applied to new data.

## Summary
Maarten Breddels and Jovan Veljanoski introduce Vaex, an open-source Python DataFrame library for datasets that do not fit in memory. Vaex uses memory-mapped files, column-based storage, immutable data, lazy expressions, and chunked algorithms so users can work locally without configuring a cluster. Jovan demonstrates the approach with a New York City taxi dataset containing 1.1 billion rows and occupying 100 GB on disk. The demo covers filtering bad values, calculating statistics, building histograms and heat maps, grouping data, and interactive geographic exploration. The speakers then show Vaex ML, where feature transformations and model predictions remain expressions in a computational graph. An incremental scikit-learn model can train over chunks of data, and the complete state can be saved to cloud storage and applied to another DataFrame with the same schema. Maarten closes with dashboard, caching, cloud storage, and production examples.

## Key ideas
### Vaex is designed for large local DataFrames without a cluster
[07:52](https://www.youtube.com/watch?v=p__sVdwz8v8&t=472s)
Maarten describes Vaex as a high-performance out-of-core DataFrame library. It is intended for data larger than RAM, including terabyte-scale datasets when a machine has much less memory. The API resembles pandas, but Vaex is not built on pandas. Installation is intended to be simple, with no cluster or administration required. Maarten says users can work with about a billion rows on a single laptop or machine. Vaex is open source under the MIT license.

### Memory mapping avoids copying the whole dataset into application memory
[09:05](https://www.youtube.com/watch?v=p__sVdwz8v8&t=545s)
A normal disk read copies data into the operating system cache and then into memory allocated by the application. With memory mapping, Vaex gets a pointer to the operating system cache instead. The kernel can release cached pages when another process needs memory and read them again later. This also allows multiple processes or Jupyter kernels to share cached data. The trade-off is that the on-disk format must be directly usable by the CPU.

### Lazy expressions keep filtering and derived columns cheap
[14:46](https://www.youtube.com/watch?v=p__sVdwz8v8&t=886s)
Vaex treats the source data as immutable and records filters instead of copying matching rows into a new array. A derived column is also stored as an expression, such as x plus y multiplied by 10, rather than being calculated for every row immediately. When a result is needed, Vaex evaluates the expression in chunks. This lets users build up transformations over a billion rows without allocating the full result in RAM.

### Column storage fits analytical workloads
[11:51](https://www.youtube.com/watch?v=p__sVdwz8v8&t=711s)
Maarten explains that analytical work often uses only a few columns from a much wider table. Column-based storage lets Vaex read the columns needed for an operation instead of scanning every field. He discusses HDF5, Apache Arrow, and Apache Parquet. HDF5 provides contiguous arrays that can be memory mapped, Arrow improves interoperability between processes, and Parquet offers compression that helps with slower I/O at the cost of decompression work.

### The taxi demonstration shows interactive exploration at billion-row scale
[22:08](https://www.youtube.com/watch?v=p__sVdwz8v8&t=1328s)
Jovan opens a 100 GB taxi file with more than a billion rows on a machine that does not have 100 GB of RAM. He previews rows and columns quickly because Vaex reads only what the display needs. The exploration then filters passenger counts, trip distances, locations, durations, speeds, and fares. Value counts, histograms, group-bys, and two-dimensional heat maps expose invalid values and geographic patterns. The same filtered data is used to inspect taxi activity by hour and day.

### Vaex can accelerate expensive expressions with Numba and GPU backends
[29:22](https://www.youtube.com/watch?v=p__sVdwz8v8&t=1762s)
Jovan calculates the distance between pickup and drop-off points using an expression with trigonometric operations. The default NumPy execution takes about 12 seconds for the billion-row dataset in his example. Compiling the expression with Numba reduces the example to about two seconds. Vaex can also execute expressions on a GPU through CUDA, and Jovan mentions Metal support for suitable newer Mac hardware.

### Vaex ML keeps feature engineering and predictions in the same DataFrame graph
[47:04](https://www.youtube.com/watch?v=p__sVdwz8v8&t=2824s)
Jovan uses Vaex ML to create features for predicting taxi trip duration. The features include geographic transformations, cyclical encodings for time, standard scaling, distance, and direction. Vaex ML provides an API similar to scikit-learn, while storing transformed columns as expressions. Model predictions are treated as expressions too, so diagnostics, error analysis, stacking, averaging, and post-processing can happen in the same DataFrame.

### Incremental training makes a large-data pipeline reusable
[52:36](https://www.youtube.com/watch?v=p__sVdwz8v8&t=3156s)
The example uses scikit-learn's SGDRegressor, which supports incremental learning. Vaex streams chunks of rows into the model instead of putting the full training matrix in memory. The trained model, feature transformations, filters, and prediction clipping are captured in a computational graph. Maarten and Jovan save that state and apply it to a test DataFrame with the same schema. They also show a production option that disables exploratory filters so rows are not silently removed during inference.

### Vaex supports dashboards and shared production workloads
[1:04:26](https://www.youtube.com/watch?v=p__sVdwz8v8&t=3866s)
Maarten shows a Dash application backed by Vaex that visualizes 120 million taxi trips. Users can select airports, hours, days, and map regions, with the plots recomputed interactively. He describes caching for repeated dashboard queries, shared memory across processors, support for Flask and FastAPI, and cloud storage through S3 and Google Cloud Storage. Vaex also has separate packages so deployments can install only the parts they need.

## Notable quotes
- Ben Epstein: "Vaex is an out of core data frame library that enables users to process enormous amounts of data locally on their laptops without needing clusters or any kind of intense configuration." (04:06)
- Maarten Breddels: "We consider that immutable, the data is the data and that stays on disk and we don't touch that, but we keep track of the filters that you're doing." (14:55)
- Jovan Veljanoski: "The idea that everything is an expression in the modeling comes that when we do a normal job with scikit-learn or XGBoost, when we do model.predict we get an in-memory array or data frame or series with the predictions." (55:41)
- Jovan Veljanoski: "All we need to do is to take this state and apply it to any other data frame that has the same schema as this trained data frame and we'll basically get the entire pipeline applied to the new data frame." (59:06)

## Tools & references mentioned
- Vaex
- Vaex ML
- Python
- Jupyter
- pandas
- Dask
- Modin
- Koalas
- PySpark
- RAPIDS
- HDF5
- Apache Arrow
- Apache Parquet
- NumPy
- Numba
- CUDA
- scikit-learn
- XGBoost
- LightGBM
- Keras
- River
- Dash
- Flask
- FastAPI
- Google Cloud Storage
- Amazon S3
- Plotly
- Galileo
- Gaia
- Space Telescope Science Institute

## Who should watch
- You need to inspect files larger than local RAM and want to avoid starting a distributed cluster for early analysis.
- Your current feature engineering and inference code is split across notebooks and production scripts, and you want to preserve a reusable computation graph.
- You are building an interactive dashboard over large tabular data and need repeated queries to share cached data and computation.

## Related talks

- [Vector Databases and Large Language Models](https://mlopstalks.com/talks/vector-databases-and-large-language-models) (Samuel Partee, Redis, 13:10)
- [Retrieval Augmented Generation](https://mlopstalks.com/talks/retrieval-augmented-generation) (Syed Asad, KiwiTech, 44:10)
- [DevTools for Language Models: Unlocking the Future of AI-Driven Applications](https://mlopstalks.com/talks/devtools-for-language-models-unlocking-the-future-of-ai-driven-applications) (Diego Oppenheimer, Factory, 29:55)
- [Democratizing AI](https://mlopstalks.com/talks/democratizing-ai) (Yujian Tang, Zilliz, 54:18)
- [Python Power: How Daft Embeds Models and Revolutionizes Data Processing](https://mlopstalks.com/talks/python-power-how-daft-embeds-models-and-revolutionizes-data-processing) (Sammy Sidhu, Eventual, 51:30)