# 'Git for Data' - Who, What, How and Why?

Luke Feeney & Gavin Mendel-Gleason, TerminusDB | MLOps Meetup | Episode 52 | 57:44
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=r5uxntl_hWg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/git-for-data-who-what-how-and-why
Published: 2021-02-19
Tags: data-engineering, data-quality

## TL;DR
- Git for data tools fall into four broad groups: versioning layers, data catalogs, data pipeline versioning, and version-controlled databases.
- Data needs more than file versioning to support collaboration. A useful system also needs branching, provenance, querying, schema management, rollback, and push, pull, and clone workflows.
- TerminusDB argues for storing structured data in a revision-controlled database so teams can query data, manage schema and instance changes, run experiments on branches, and merge useful results back into shared environments.

## Summary
Luke Feeney surveys the open-source Git for data ecosystem and groups tools into versioning layers, data catalogs, data pipeline versioning, and version-controlled databases. He argues that many products called Git for data provide only part of Git's collaboration model. Gavin Mendel-Gleason then explains why ordinary Git struggles with large or structured data. Line-based diffs do not understand schemas, cell-level changes, binary data, or the meaning of a merge. A data-focused system should support querying, typed schemas, programmatic updates, provenance, authorship, branching, rollback, cloning, and pipeline stages similar to software CI/CD. TerminusDB's approach stores schema and instance changes in a revision-controlled database. The speakers connect this design to machine learning workflows, where teams need repeatable data curation, safe experimentation, and a way to merge predictions or enriched labels back into shared data.

## Key ideas
### Git for data includes several different kinds of tools
[04:29](https://www.youtube.com/watch?v=r5uxntl_hWg&t=269s)
Luke Feeney groups Git for data products into four broad buckets. Versioning layers sit on top of systems such as S3, Google Cloud, or Postgres. Data catalogs focus on discovery and management. Data pipeline versioning reproduces an entire pipeline and its relationship to data. Version-controlled databases provide append-only data with branching, diff, and merge across both schema and data. The boundaries are hazy, and some projects are active while others are dormant. This classification gives engineers a way to compare tools that use the same label for very different capabilities.

### Versioning alone does not provide Git-style collaboration
[06:40](https://www.youtube.com/watch?v=r5uxntl_hWg&t=400s)
Luke argues that simply versioning data is insufficient. A Git-like system should bring coordination and collaboration workflows to data scientists and machine learning engineers. He is skeptical of end-to-end platforms that describe themselves as GitHub for data without first implementing the underlying Git concepts. The speakers prefer choosing the best tool at each stage of an ML pipeline, while acknowledging that an end-to-end platform can fit some use cases. The concern is that users may become locked into one platform and lose access to more focused tools.

### Data is harder to diff and merge because line-based tools lack meaning
[24:30](https://www.youtube.com/watch?v=r5uxntl_hWg&t=1470s)
Gavin Mendel-Gleason explains that Git's line-based model is workable for relatively small text files, yet it becomes awkward for large data. A binary file cannot be handled like source code, and changing one cell in a CSV raises a different question from changing an entire row. Large numbers of changes cannot be displayed or merged sensibly without querying the affected data. Gavin says structured data and schemas could make merges more intelligent because the system would know which changes are valid, although thousands of conflicts would still require programmatic resolution.

### The value of Git comes from collaboration, provenance, and safe change
[34:36](https://www.youtube.com/watch?v=r5uxntl_hWg&t=2076s)
Gavin describes Git's important contribution as distributed collaboration, rather than revision history alone. Separate repositories can push, pull, merge, and share state across multiple nodes. Data workflows also need provenance, authorship, and a record of when changes happened. Branching and rollback make people more willing to try a different approach because a bad experiment can be abandoned. Gavin also wants data workflows to adopt more of the staged checks used in software CI/CD, such as tests, benchmarks, intermediate environments, and controlled movement into production.

### A data revision system needs schemas, queries, and programmatic updates
[44:57](https://www.youtube.com/watch?v=r5uxntl_hWg&t=2697s)
Gavin lists the features he expects from a Git for data system. Data should be discoverable through queries rather than requiring the whole dataset to be loaded. Users should be able to update it programmatically, work with typed values, and manage schema changes over time. The system must scale to the datasets in question while preserving provenance, authorship, commit time, branching, rollback, and collaboration. Fast cloning is useful because a team can copy production data into a branch, experiment without changing the live environment, and later merge an accepted result.

### TerminusDB stores schema and data changes for revision transitions
[51:22](https://www.youtube.com/watch?v=r5uxntl_hWg&t=3082s)
Gavin says TerminusDB stores changes to schema and instance data as delta encodings. A commit records what was added and removed in both the schema and the instance graph. He mentions that another tool versions SQL transformations themselves, which is an alternative approach, though it requires understanding how each transformation is coupled to a commit. TerminusDB's approach keeps the transitions in the database's revision structure. The hard part remains merging structured changes, but schema and query access provide more information than plain text lines.

### Machine learning teams should branch data experiments and merge useful results
[52:35](https://www.youtube.com/watch?v=r5uxntl_hWg&t=3155s)
Gavin argues that machine learning data is often stored in the format most convenient for the current algorithm, such as CSV files, rather than in a logical structure that can be maintained over time. He prefers a database-backed source that supports extraction, enrichment, and repeatable updates. Branches let teams run experiments without polluting another environment. If an experiment works, its changes can move back into the shared pipeline. He also wants predictions and labels to flow back into the data environment smoothly. The intended users extend beyond machine learning to data engineers dealing with curated data and integration problems.

## Notable quotes
- Luke Feeney: "You need to allow for that coordination and collaboration that Git facilitates amongst software engineers and really bring that to ML engineers or data scientists." (06:58)
- Gavin Mendel-Gleason: "The collaboration's key, I think the second is the provenance." (41:39)
- Gavin Mendel-Gleason: "Merge on data is hard. It's hard on code, but code has small numbers of lines." (47:52)
- Gavin Mendel-Gleason: "I think branching helps here because you can do experiments and you don't have to pollute somebody else's environment." (54:03)

## Tools & references mentioned
- TerminusDB
- Git
- GitHub
- Data Hub
- Instabase
- Lightweight
- Fluree
- SQLite
- Noms
- Replicash
- Dolt
- DoltHub
- Dotmesh
- Dot Science
- Meltano
- Singer
- dbt
- Gretel Synthetics
- Grist
- DVC
- DAGsHub
- Pachyderm
- AI Infrastructure Alliance
- Query
- Quilt
- AWS S3
- Keepsake
- Google Cloud Storage
- Irmin
- OCaml
- LakeFS
- Jupyter notebooks
- Git LFS
- ZFS
- Subversion
- CVS

## Who should watch
- You are comparing tools that claim to provide Git for data and need a simple way to distinguish file versioning, pipeline versioning, catalogs, and databases.
- Your ML workflow relies on CSV extracts and lacks clear lineage, repeatability, or a safe way to test data changes before production.
- You are designing a revision-controlled data system and want to think through schemas, queries, branching, merges, cloning, and data CI/CD.

## Related talks

- [DataOps and Data Versioning in ML](https://mlopstalks.com/talks/dataops-and-data-versioning-in-ml) (Dmitry Petrov, Iterative.AI, 1:01:55)
- [How to Make Your Data Science Reproducible (and Why You Should Care)](https://mlopstalks.com/talks/how-to-make-your-data-science-reproducible-and-why-you-should-care) (Ciro Greco, Bauplan, 11:59)
- [Data Engineering for ML](https://mlopstalks.com/talks/data-engineering-for-ml) (Chad Sanderson, Convoy, 57:54)
- [The Rise of Modern Data Management](https://mlopstalks.com/talks/the-rise-of-modern-data-management) (Chad Sanderson, Gable.ai, 57:53)
- [Just Fetch the Data and then...](https://mlopstalks.com/talks/just-fetch-the-data-and-then) (David Bayliss, LexisNexis Risk Solutions, 51:56)