# Durable Data Discovery: Making Exploratory Analysis Stick

James Campbell, Superconductive | MLOps Meetup | Episode 86 | 58:26
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=DKtQZPmhfMw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/durable-data-discovery-making-exploratory-analysis-stick
Published: 2021-11-19
Tags: data-quality, testing

## TL;DR
- Great Expectations can turn exploratory questions into reusable expectations and documentation that stay with a dataset as it changes.
- A data asset depends on both the data being collected and the purpose for which someone is using it, so the same underlying data can have many valid views.
- Profilers capture repeatable workflows for asking questions across batches, which helps teams preserve the reasoning behind data quality checks and model updates.

## Summary
James Campbell argues that exploratory data analysis often contains knowledge that disappears when a project moves into production or changes hands. Great Expectations can preserve that knowledge by turning questions about data into expectations, documentation, and reusable profiling workflows. He defines a data asset through the intersection of collected data and its purpose. The same New York City taxi data can support different views, such as monthly analysis, streaming analysis, or analysis of all available records. A batch is also defined by how the data is being considered, including the state of a warehouse at a point in time or a month of taxi records. Campbell demonstrates how Great Expectations groups monthly files into meaningful assets and batches, then uses profilers to ask repeatable questions across those batches. The resulting expectations expose schema changes, unusual fare values, and other surprises while preserving the assumptions behind the investigation.

## Key ideas
### MLOps workflows need to preserve feedback between groups and across time
[05:13](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=313s)
Campbell starts from the shared feature of MLOps, DevOps, and related practices: several groups interact through time, so feedback has to be part of the workflow. He describes testing as a way to define things and communicate with other people, including a future version of yourself. Great Expectations brings automated software testing ideas into data, beginning with declarations such as expecting a column to exist. Expectations can also describe nulls, uniqueness, value ranges, regular expressions, date formats, aggregate statistics, and distributions. Campbell's point is that these checks capture information about what the data means instead of treating testing as a separate cost.

### A data asset is defined by the data and the purpose for using it
[10:57](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=657s)
Campbell defines a data asset as something at the intersection of collected data and the purpose for using it. He uses New York City taxi trip data to show why this matters. The same records could be used to study COVID's impact, understand Uber and Lyft's effect on taxis, or examine a policy change. Each purpose can lead someone to inspect the data differently. Data assets also inherit assumptions from broader categories. Event data is temporal and often lacks complete context, while report data combines information into a more contextual record such as an invoice. Expectations can begin at an abstract level and become tighter as the purpose becomes more specific.

### A batch is created by the way someone looks at data
[18:19](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=1099s)
Campbell says attention is what makes a batch. A batch of invoices might be the state of the data when a person views a dashboard. In a pipeline, it might be the state of a warehouse when the pipeline runs. For New York City taxi data, a batch might be one month, which makes sense for studying changes over time. The batch depends on the purpose and access pattern. In a streaming setting, a batch may refer to data that was available at a particular point and then disappeared after a statistic was stored. Great Expectations uses this framing to connect data assets and batches to the way engineers actually work with data.

### Data discovery should be repeatable when new data arrives
[21:45](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=1305s)
Exploratory analysis asks questions about columns, quantiles, outliers, distributions, correlations, and model assumptions. Campbell says those questions change as new data arrives or as the dataset's purpose changes. An engineer should be able to keep the earlier work rather than start over for every new month, provider, or dataset. He calls this durable data discovery. The setup needs to make the intended view explicit, such as monthly yellow taxi data, so later batches can use the same investigation. This preserves the reasoning behind feature selection, anomaly checks, and other decisions that might otherwise remain only in the original engineer's notebook.

### Great Expectations can group files into meaningful data assets and batches
[24:24](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=1464s)
In the demo, Campbell downloads New York City taxi files and creates a Great Expectations project with a pandas data source. An initial configuration sees hundreds of individual files as separate assets. He then uses the filename convention, including the yellow trip data prefix and year-month captures, to group the files into assets with monthly batches. The revised view produces assets such as yellow, green, and fhvhv, with batches identified by year and month. Some files do not match the pattern because they use a different naming or compression arrangement. The example shows that defining a batch requires understanding the actual data and expressing the intended grouping.

### Profilers turn exploratory questions into reusable rules
[33:48](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=2028s)
Campbell describes profilers as configurable workflows for exploring data. A profiler does not directly state that data must have a fixed value. It defines how Great Expectations should create expectations by asking questions of one or more batches. A rule can inspect the first or last several batches, calculate statistics such as means, and use those observations to set an expectation. For anomaly detection, Great Expectations can use a multi-batch bootstrapped metric distribution parameter builder to estimate confidence intervals and create a starting false-positive threshold. The workflow remains visible and editable, so changing the threshold also records a change in how the team understands the data.

### Validation turns surprising observations into follow-up investigation
[44:22](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=2662s)
When Campbell validates newer taxi data against expectations informed by earlier batches, the results expose a change in column names. Later files use names such as rate code ID where earlier files used a different form. The validation also finds fare values outside the earlier observed range, including a negative fare and a much larger fare than expected. Campbell does not claim to know why those values occurred. They might reflect credits, data processing, or another business rule. The useful outcome is that the system gets the team to the question quickly and lets them add expectations after investigating. Those expectations can apply to raw data and to data after transformation.

### The data scientist packages assumptions with the data or answer
[50:29](https://www.youtube.com/watch?v=DKtQZPmhfMw&t=3029s)
Campbell says a data scientist may use Great Expectations when sharing raw data, models, or an answer to a business question. Expectations package assumptions with what is being shared. For an analysis of COVID's effect on taxi revenue, a scientist must separate that effect from the effect of Uber and Lyft. Great Expectations helps expose the assumptions before modeling those interacting changes. It also gives the scientist more confidence that the data means what they think it means. Campbell describes the project's focus as helping people understand and communicate about data without making the quality process a black box.

## Notable quotes
- James Campbell: "A data asset is specifically a thing that lives in the intersection of data that you collect and your purpose in using that data." (10:57)
- James Campbell: "So fundamentally it's event data, it's New York City taxi trip data, but it's being looked at through a particular lens and there's lots of different lenses." (17:19)
- James Campbell: "You want to not have to start from scratch and do this huge project in order to be able to pick up and keep using the insights that you've gained about a data set." (23:44)
- James Campbell: "The result of a run of a profiler is a set of expectations." (40:25)
- James Campbell: "We're really trying to keep the emphasis on your ability to understand and communicate about data." (54:39)

## Tools & references mentioned
- Superconductive
- Great Expectations
- New York City Taxi and Limousine Commission
- S3
- pandas
- Uber
- Lyft
- Arize AI
- Monte Carlo
- Soda

## Who should watch
- You are building an ML pipeline and the reasoning from initial data exploration is likely to disappear when the pipeline or its owner changes.
- Your team needs to compare new data batches with earlier observations while keeping the assumptions behind those comparisons visible.
- You are evaluating data quality tools and want an open-source workflow that combines profiling, expectations, validation, and generated documentation.

## Editor's note

James Campbell says exploratory analysis often contains knowledge that disappears when a project moves into production or changes hands. ZenML records each pipeline run's steps, inputs, outputs, and code version, so the workflow behind a data check or model update can be traced back to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [From Expectations to Synthetic Data Generation](https://mlopstalks.com/talks/from-expectations-to-synthetic-data-generation) (Fabiana Clemente, YData, 55:39)
- [Mid-Scale Production Feature Engineering](https://mlopstalks.com/talks/mid-scale-production-feature-engineering) (Dr. Venkata Pingali, Scribble Data, 1:01:35)
- [Data Engineering + ML + Software Engineering](https://mlopstalks.com/talks/data-engineering-ml-software-engineering) (Satish Chandra Gupta, Slang Labs, 57:05)
- [Eliminating Garbage In/Garbage Out for Analytics and ML](https://mlopstalks.com/talks/eliminating-garbage-in-garbage-out-for-analytics-and-ml) (Roy Hasson & Santona Tuli, Upsolver, 50:38)
- [Speed Up Data-Driven Value](https://mlopstalks.com/talks/speed-up-data-driven-value) (Delina Ivanova, HelloFresh, 53:47)