Watch on YouTubeReproducing a data pipeline requires keeping the same code, data, environment, and, when needed, hardware.
A data lake needs separate but connected layers for versioned data, reproducible compute, and versioned code.
Nessie and Apache Iceberg provide time travel and branching for data, while containerized cloud runtimes preserve the execution environment.
Ciro Greco explains why reproducing a failed data pipeline is harder than reproducing a software failure. A pipeline may have changed code, data, dependencies, or cloud infrastructure between the original run and the investigation. His checklist is to preserve the same code, data, environment, and, when needed, hardware. He organizes the solution into three layers: a data layer that versions tables, a compute layer that records the runtime, and a code layer managed through Git. Bauplan uses Parquet files represented as Apache Iceberg tables and stored in Nessie, which adds time travel and zero-copy branches to object storage. The runtime is containerized and runs in the cloud, so individual pipeline steps can be reproduced with their original dependencies. Git-like commands let developers create branches, run computations, inspect changes, and merge data artifacts back into production.
Greco says a failed pipeline may be caused by the code, data, environment, or cloud architecture. By the time someone investigates, the data may have changed, new code may be in production, and new dependencies may have altered the environment. That makes it difficult to reproduce the exact failure. He argues that deterministic investigation needs the same inputs and execution conditions that existed when the problem occurred.
Greco gives a reproducibility checklist: use the same code, the same data, and the same environment. For a more exact reproduction, hardware also needs to match. His example pairs a particular data frame and size with a specific code commit and a containerized runtime whose dependencies are frozen. Cloud execution can help control the hardware used by the pipeline.
Greco supports building on data lakes and object storage because they are widespread in enterprise systems and work with open formats. He also says these systems can become messy because their pieces were not always designed together for reproducibility. Data, metadata, compute, and code may be scattered across different systems, leaving teams to patch missing capabilities together.
Greco proposes a hierarchy of abstractions. The data layer stores data and decides how it is versioned. The compute layer records the runtime, such as a Spark cluster, Python runtime, Airflow deployment, or SQL engine. The code layer uses a Git system to version application logic. These layers need to be engineered together so a pipeline can be reproduced across time.
Bauplan stores Parquet files as Apache Iceberg tables. Iceberg adds a metadata layer that supports table operations such as transactions and incremental updates. Greco says Nessie provides the catalog and lets teams version tables, travel back in time, and create branches without copying the full data lake. Those branches can act as sandboxes for debugging.
The runtime can vary, but Greco says every function should be containerized. In Bauplan's example, functions run in cloud containers that can be checkpointed over time. If a pipeline fails, each node can be reproduced with the libraries and dependencies used for its original execution. Combined with versioned data and code, this completes the reproducibility checklist.
Greco describes a small command-line interface modeled on Git. Developers can list catalog versions, create a branch, check it out, explore the tables copied by reference into that branch, and run computations against it. After producing new artifacts or changing data, they can check out the main branch and merge the work back into the production data environment.
"The main reason why reproducibility is effectively a major problem in data projects is because reproducing stuff on a data system requires a bit more than what we're usually used to in software engineering."01:56