Great Expectations can turn exploratory questions into reusable expectations and documentation that stay with a dataset as it changes.
2
A data asset depends on both the data being collected and the purpose for which someone is using it, so the same underlying data can have many valid views.
3
Profilers capture repeatable workflows for asking questions across batches, which helps teams preserve the reasoning behind data quality checks and model updates.
Summary
James Campbell argues that exploratory data analysis often contains knowledge that disappears when a project moves into production or changes hands. Great Expectations can preserve that knowledge by turning questions about data into expectations, documentation, and reusable profiling workflows. He defines a data asset through the intersection of collected data and its purpose. The same New York City taxi data can support different views, such as monthly analysis, streaming analysis, or analysis of all available records. A batch is also defined by how the data is being considered, including the state of a warehouse at a point in time or a month of taxi records. Campbell demonstrates how Great Expectations groups monthly files into meaningful assets and batches, then uses profilers to ask repeatable questions across those batches. The resulting expectations expose schema changes, unusual fare values, and other surprises while preserving the assumptions behind the investigation.
MLOps workflows need to preserve feedback between groups and across time
Campbell starts from the shared feature of MLOps, DevOps, and related practices: several groups interact through time, so feedback has to be part of the workflow. He describes testing as a way to define things and communicate with other people, including a future version of yourself. Great Expectations brings automated software testing ideas into data, beginning with declarations such as expecting a column to exist. Expectations can also describe nulls, uniqueness, value ranges, regular expressions, date formats, aggregate statistics, and distributions. Campbell's point is that these checks capture information about what the data means instead of treating testing as a separate cost.
A data asset is defined by the data and the purpose for using it
Campbell defines a data asset as something at the intersection of collected data and the purpose for using it. He uses New York City taxi trip data to show why this matters. The same records could be used to study COVID's impact, understand Uber and Lyft's effect on taxis, or examine a policy change. Each purpose can lead someone to inspect the data differently. Data assets also inherit assumptions from broader categories. Event data is temporal and often lacks complete context, while report data combines information into a more contextual record such as an invoice. Expectations can begin at an abstract level and become tighter as the purpose becomes more specific.
A batch is created by the way someone looks at data
Campbell says attention is what makes a batch. A batch of invoices might be the state of the data when a person views a dashboard. In a pipeline, it might be the state of a warehouse when the pipeline runs. For New York City taxi data, a batch might be one month, which makes sense for studying changes over time. The batch depends on the purpose and access pattern. In a streaming setting, a batch may refer to data that was available at a particular point and then disappeared after a statistic was stored. Great Expectations uses this framing to connect data assets and batches to the way engineers actually work with data.
Data discovery should be repeatable when new data arrives
Exploratory analysis asks questions about columns, quantiles, outliers, distributions, correlations, and model assumptions. Campbell says those questions change as new data arrives or as the dataset's purpose changes. An engineer should be able to keep the earlier work rather than start over for every new month, provider, or dataset. He calls this durable data discovery. The setup needs to make the intended view explicit, such as monthly yellow taxi data, so later batches can use the same investigation. This preserves the reasoning behind feature selection, anomaly checks, and other decisions that might otherwise remain only in the original engineer's notebook.
Great Expectations can group files into meaningful data assets and batches
In the demo, Campbell downloads New York City taxi files and creates a Great Expectations project with a pandas data source. An initial configuration sees hundreds of individual files as separate assets. He then uses the filename convention, including the yellow trip data prefix and year-month captures, to group the files into assets with monthly batches. The revised view produces assets such as yellow, green, and fhvhv, with batches identified by year and month. Some files do not match the pattern because they use a different naming or compression arrangement. The example shows that defining a batch requires understanding the actual data and expressing the intended grouping.
Profilers turn exploratory questions into reusable rules
Campbell describes profilers as configurable workflows for exploring data. A profiler does not directly state that data must have a fixed value. It defines how Great Expectations should create expectations by asking questions of one or more batches. A rule can inspect the first or last several batches, calculate statistics such as means, and use those observations to set an expectation. For anomaly detection, Great Expectations can use a multi-batch bootstrapped metric distribution parameter builder to estimate confidence intervals and create a starting false-positive threshold. The workflow remains visible and editable, so changing the threshold also records a change in how the team understands the data.
Validation turns surprising observations into follow-up investigation
When Campbell validates newer taxi data against expectations informed by earlier batches, the results expose a change in column names. Later files use names such as rate code ID where earlier files used a different form. The validation also finds fare values outside the earlier observed range, including a negative fare and a much larger fare than expected. Campbell does not claim to know why those values occurred. They might reflect credits, data processing, or another business rule. The useful outcome is that the system gets the team to the question quickly and lets them add expectations after investigating. Those expectations can apply to raw data and to data after transformation.
The data scientist packages assumptions with the data or answer
Campbell says a data scientist may use Great Expectations when sharing raw data, models, or an answer to a business question. Expectations package assumptions with what is being shared. For an analysis of COVID's effect on taxi revenue, a scientist must separate that effect from the effect of Uber and Lyft. Great Expectations helps expose the assumptions before modeling those interacting changes. It also gives the scientist more confidence that the data means what they think it means. Campbell describes the project's focus as helping people understand and communicate about data without making the quality process a black box.
"You want to not have to start from scratch and do this huge project in order to be able to pick up and keep using the insights that you've gained about a data set."James Campbell23:44
Who should watch
You are building an ML pipeline and the reasoning from initial data exploration is likely to disappear when the pipeline or its owner changes.
Your team needs to compare new data batches with earlier observations while keeping the assumptions behind those comparisons visible.
You are evaluating data quality tools and want an open-source workflow that combines profiling, expectations, validation, and generated documentation.
