# How to Make Your Data Science Reproducible (and Why You Should Care)

Ciro Greco, Bauplan | MLOps Community | 11:59

Source: https://www.youtube.com/watch?v=geTJ7AAImmE
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-to-make-your-data-science-reproducible-and-why-you-should-care
Published: 2024-10-09
Tags: data-engineering, data-quality, debugging

## TL;DR
- Reproducing a data pipeline requires keeping the same code, data, environment, and, when needed, hardware.
- A data lake needs separate but connected layers for versioned data, reproducible compute, and versioned code.
- Nessie and Apache Iceberg provide time travel and branching for data, while containerized cloud runtimes preserve the execution environment.

## Summary
Ciro Greco explains why reproducing a failed data pipeline is harder than reproducing a software failure. A pipeline may have changed code, data, dependencies, or cloud infrastructure between the original run and the investigation. His checklist is to preserve the same code, data, environment, and, when needed, hardware. He organizes the solution into three layers: a data layer that versions tables, a compute layer that records the runtime, and a code layer managed through Git. Bauplan uses Parquet files represented as Apache Iceberg tables and stored in Nessie, which adds time travel and zero-copy branches to object storage. The runtime is containerized and runs in the cloud, so individual pipeline steps can be reproduced with their original dependencies. Git-like commands let developers create branches, run computations, inspect changes, and merge data artifacts back into production.

## Key ideas
### Data pipelines change while engineers are trying to reproduce them
[01:38](https://www.youtube.com/watch?v=geTJ7AAImmE&t=98s)
Greco says a failed pipeline may be caused by the code, data, environment, or cloud architecture. By the time someone investigates, the data may have changed, new code may be in production, and new dependencies may have altered the environment. That makes it difficult to reproduce the exact failure. He argues that deterministic investigation needs the same inputs and execution conditions that existed when the problem occurred.

### Reproducibility requires matching code, data, and execution conditions
[03:12](https://www.youtube.com/watch?v=geTJ7AAImmE&t=192s)
Greco gives a reproducibility checklist: use the same code, the same data, and the same environment. For a more exact reproduction, hardware also needs to match. His example pairs a particular data frame and size with a specific code commit and a containerized runtime whose dependencies are frozen. Cloud execution can help control the hardware used by the pipeline.

### Object storage makes data versioning harder when the stack is fragmented
[04:17](https://www.youtube.com/watch?v=geTJ7AAImmE&t=257s)
Greco supports building on data lakes and object storage because they are widespread in enterprise systems and work with open formats. He also says these systems can become messy because their pieces were not always designed together for reproducibility. Data, metadata, compute, and code may be scattered across different systems, leaving teams to patch missing capabilities together.

### A reproducible data system has data, compute, and code layers
[05:39](https://www.youtube.com/watch?v=geTJ7AAImmE&t=339s)
Greco proposes a hierarchy of abstractions. The data layer stores data and decides how it is versioned. The compute layer records the runtime, such as a Spark cluster, Python runtime, Airflow deployment, or SQL engine. The code layer uses a Git system to version application logic. These layers need to be engineered together so a pipeline can be reproduced across time.

### Apache Iceberg and Nessie add table history and branches to object storage
[06:57](https://www.youtube.com/watch?v=geTJ7AAImmE&t=417s)
Bauplan stores Parquet files as Apache Iceberg tables. Iceberg adds a metadata layer that supports table operations such as transactions and incremental updates. Greco says Nessie provides the catalog and lets teams version tables, travel back in time, and create branches without copying the full data lake. Those branches can act as sandboxes for debugging.

### Containerized runtimes preserve the dependencies behind each pipeline step
[08:06](https://www.youtube.com/watch?v=geTJ7AAImmE&t=486s)
The runtime can vary, but Greco says every function should be containerized. In Bauplan's example, functions run in cloud containers that can be checkpointed over time. If a pipeline fails, each node can be reproduced with the libraries and dependencies used for its original execution. Combined with versioned data and code, this completes the reproducibility checklist.

### Git-like APIs hide the complexity of versioned data workflows
[08:48](https://www.youtube.com/watch?v=geTJ7AAImmE&t=528s)
Greco describes a small command-line interface modeled on Git. Developers can list catalog versions, create a branch, check it out, explore the tables copied by reference into that branch, and run computations against it. After producing new artifacts or changing data, they can check out the main branch and merge the work back into the production data environment.

## Notable quotes
- "The main reason why reproducibility is effectively a major problem in data projects is because reproducing stuff on a data system requires a bit more than what we're usually used to in software engineering." (01:56)
- "It is essentially same code, same data, same environment, and for the pedantic, same hardware." (03:12)
- "These three layers require then to be somehow engineered as one." (06:34)
- "The essential part is make sure that the runtime is containerized so you have the possibility of traveling back in time and reproduce the environment." (08:06)

## Tools & references mentioned
- Bauplan
- Nessie
- Apache Iceberg
- Parquet
- Git
- Docker
- Spark
- Airflow
- Databricks
- Dremio
- Starburst
- Sigma

## Who should watch
- You are debugging data pipelines whose inputs, code, or dependencies have changed since the failure.
- Your data platform uses object storage and needs table history or isolated branches for testing.
- You are comparing data workflow tools and want a concrete model based on versioned tables, containers, and Git-like commands.

## Editor's note

Ciro Greco says reproducing a failed data pipeline requires the same code, data, environment, and, when needed, hardware. ZenML records each pipeline run's steps, inputs, outputs, and code version, so a model or artifact can be traced to the data and code that produced it. Its stack configuration also lets the same pipeline run across different infrastructure.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- ['Git for Data' - Who, What, How and Why?](https://mlopstalks.com/talks/git-for-data-who-what-how-and-why) (Luke Feeney & Gavin Mendel-Gleason, TerminusDB, 57:44)
- [Durable Data Discovery: Making Exploratory Analysis Stick](https://mlopstalks.com/talks/durable-data-discovery-making-exploratory-analysis-stick) (James Campbell, Superconductive, 58:26)
- [Operationalizing Machine Learning at a Large Financial Institution](https://mlopstalks.com/talks/operationalizing-machine-learning-at-a-large-financial-institution) (Daniel Stahl, Regions Bank, 1:04:58)
- [DataOps and Data Versioning in ML](https://mlopstalks.com/talks/dataops-and-data-versioning-in-ml) (Dmitry Petrov, Iterative.AI, 1:01:55)
- [All Data Scientists Should Learn Software Engineering Principles](https://mlopstalks.com/talks/all-data-scientists-should-learn-software-engineering-principles) (Catherine Nelson, Freelance Data Scientist, 52:55)