Vaex lets users explore datasets larger than RAM on a single machine through memory mapping, column-based storage, lazy expressions, and out-of-core computation.
2
Vaex can process a 1.1 billion-row, 100 GB taxi dataset interactively, including filtering, grouping, statistics, and visualizations.
3
Vaex ML stores feature engineering, model predictions, and post-processing in a computational graph that can be saved and applied to new data.
Summary
Maarten Breddels and Jovan Veljanoski introduce Vaex, an open-source Python DataFrame library for datasets that do not fit in memory. Vaex uses memory-mapped files, column-based storage, immutable data, lazy expressions, and chunked algorithms so users can work locally without configuring a cluster. Jovan demonstrates the approach with a New York City taxi dataset containing 1.1 billion rows and occupying 100 GB on disk. The demo covers filtering bad values, calculating statistics, building histograms and heat maps, grouping data, and interactive geographic exploration. The speakers then show Vaex ML, where feature transformations and model predictions remain expressions in a computational graph. An incremental scikit-learn model can train over chunks of data, and the complete state can be saved to cloud storage and applied to another DataFrame with the same schema. Maarten closes with dashboard, caching, cloud storage, and production examples.
Vaex is designed for large local DataFrames without a cluster
Maarten describes Vaex as a high-performance out-of-core DataFrame library. It is intended for data larger than RAM, including terabyte-scale datasets when a machine has much less memory. The API resembles pandas, but Vaex is not built on pandas. Installation is intended to be simple, with no cluster or administration required. Maarten says users can work with about a billion rows on a single laptop or machine. Vaex is open source under the MIT license.
Memory mapping avoids copying the whole dataset into application memory
A normal disk read copies data into the operating system cache and then into memory allocated by the application. With memory mapping, Vaex gets a pointer to the operating system cache instead. The kernel can release cached pages when another process needs memory and read them again later. This also allows multiple processes or Jupyter kernels to share cached data. The trade-off is that the on-disk format must be directly usable by the CPU.
Lazy expressions keep filtering and derived columns cheap
Vaex treats the source data as immutable and records filters instead of copying matching rows into a new array. A derived column is also stored as an expression, such as x plus y multiplied by 10, rather than being calculated for every row immediately. When a result is needed, Vaex evaluates the expression in chunks. This lets users build up transformations over a billion rows without allocating the full result in RAM.
Maarten explains that analytical work often uses only a few columns from a much wider table. Column-based storage lets Vaex read the columns needed for an operation instead of scanning every field. He discusses HDF5, Apache Arrow, and Apache Parquet. HDF5 provides contiguous arrays that can be memory mapped, Arrow improves interoperability between processes, and Parquet offers compression that helps with slower I/O at the cost of decompression work.
The taxi demonstration shows interactive exploration at billion-row scale
Jovan opens a 100 GB taxi file with more than a billion rows on a machine that does not have 100 GB of RAM. He previews rows and columns quickly because Vaex reads only what the display needs. The exploration then filters passenger counts, trip distances, locations, durations, speeds, and fares. Value counts, histograms, group-bys, and two-dimensional heat maps expose invalid values and geographic patterns. The same filtered data is used to inspect taxi activity by hour and day.
Vaex can accelerate expensive expressions with Numba and GPU backends
Jovan calculates the distance between pickup and drop-off points using an expression with trigonometric operations. The default NumPy execution takes about 12 seconds for the billion-row dataset in his example. Compiling the expression with Numba reduces the example to about two seconds. Vaex can also execute expressions on a GPU through CUDA, and Jovan mentions Metal support for suitable newer Mac hardware.
Vaex ML keeps feature engineering and predictions in the same DataFrame graph
Jovan uses Vaex ML to create features for predicting taxi trip duration. The features include geographic transformations, cyclical encodings for time, standard scaling, distance, and direction. Vaex ML provides an API similar to scikit-learn, while storing transformed columns as expressions. Model predictions are treated as expressions too, so diagnostics, error analysis, stacking, averaging, and post-processing can happen in the same DataFrame.
Incremental training makes a large-data pipeline reusable
The example uses scikit-learn's SGDRegressor, which supports incremental learning. Vaex streams chunks of rows into the model instead of putting the full training matrix in memory. The trained model, feature transformations, filters, and prediction clipping are captured in a computational graph. Maarten and Jovan save that state and apply it to a test DataFrame with the same schema. They also show a production option that disables exploratory filters so rows are not silently removed during inference.
Vaex supports dashboards and shared production workloads
Maarten shows a Dash application backed by Vaex that visualizes 120 million taxi trips. Users can select airports, hours, days, and map regions, with the plots recomputed interactively. He describes caching for repeated dashboard queries, shared memory across processors, support for Flask and FastAPI, and cloud storage through S3 and Google Cloud Storage. Vaex also has separate packages so deployments can install only the parts they need.
"The idea that everything is an expression in the modeling comes that when we do a normal job with scikit-learn or XGBoost, when we do model.predict we get an in-memory array or data frame or series with the predictions."Jovan Veljanoski55:41
Who should watch
You need to inspect files larger than local RAM and want to avoid starting a distributed cluster for early analysis.
Your current feature engineering and inference code is split across notebooks and production scripts, and you want to preserve a reusable computation graph.
You are building an interactive dashboard over large tabular data and need repeated queries to share cached data and computation.
