# Introducing Data Downtime: From Firefighting to Winning

Barr Moses, Monte Carlo | MLOps Coffee Sessions | Episode 19 | 1:00:51
Hosted by Vishnu Rachakonda

Source: https://www.youtube.com/watch?v=IMyI5eKQxMI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/introducing-data-downtime-from-firefighting-to-winning
Published: 2020-11-23
Tags: data-quality, monitoring, observability

## TL;DR
- Barr Moses defines data downtime as the period when data is wrong, missing, stale, or otherwise unreliable for the people and systems using it.
- Data observability applies software observability ideas to data through five pillars: freshness, volume, schema, distribution, and lineage.
- Teams can gain adoption by connecting data reliability to business outcomes, showing each group what it has to gain, and assigning responsibility across the whole company.

## Summary
Barr Moses explains how her work in the Israeli Air Force, consulting, and Gainsight led her to Monte Carlo. At Gainsight, she saw that giving people dashboards and data did not answer basic questions about trust, freshness, ownership, or changes in values. She calls the resulting failures data downtime. These failures can affect board reports, customer-facing websites, public health systems, and government payments. Moses compares data observability with the observability practices developed for software systems. Her proposed framework covers freshness, volume, schema, distribution, and lineage. She also discusses the organisational side of reliability. Data teams need to connect downtime to lost customers, bad campaigns, compliance risk, or wasted time. Ownership is shared across the people responsible for models, pipelines, data, and customers. Monte Carlo connects to existing systems without requiring code or pipeline changes, monitors data at rest, detects anomalies, and helps teams trace and resolve incidents.

## Key ideas
### Trust has to be established before data can drive a culture
[06:23](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=383s)
Barr Moses says companies often announce a push to become data-driven, buy tools, hire data scientists, and give thousands of people access to reports. The trouble starts when those users ask basic questions: whether the data can be trusted, when it was last updated, where it came from, who owns it, and why its values changed. She argues that these fundamentals must be addressed before a company can create a real data-driven culture. Monte Carlo's mission came from this problem, with data downtime acting as a barrier to wider use of data.

### Data failures can affect decisions far beyond the data team
[09:27](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=567s)
Moses describes a board-report incident from her own experience. On the weekend before a Monday board meeting, her team discovered that the report had been scrambled. They had to trace the problem through refreshes, formulas, upstream tables, and third-party sources. She also cites a case where the US government sent more than a billion dollars in checks to dead people because a table had not been updated. Vishnu Rachakonda mentions a UK COVID-19 tracking failure caused by an Excel sheet reaching its row limit. These examples show why knowing the full path of data matters.

### Data downtime brings software-style reliability requirements to data
[15:50](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=950s)
Moses defines data downtime as the set of situations in which data is wrong or unavailable. She compares it with application downtime. Software teams developed methods for measuring availability, including five nines, because applications became central to daily activity. Moses expects data to require similar treatment as companies rely on it for decisions. Teams can measure how much data downtime they have, how quickly they detect issues, and how quickly they resolve them. She says this gives data culture a concrete way to discuss whether trust is improving.

### Business outcomes make reliability work easier to fund
[19:07](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=1147s)
Moses says the hardest part of many data initiatives is getting people to care. Her advice is to connect reliability work to outcomes instead of describing only the technical failure. A schema change that breaks a report can cause a customer to lose trust. A data problem can make a marketing campaign's return appear wrong or cause sales teams to target the wrong customers. She also points to the growth of standalone data organisations that report to senior executives. At the team level, each group needs a clear answer to the question of what data downtime means for its own work.

### Data organisations often move between centralised and embedded models
[22:57](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=1377s)
Moses says there is no single structure that fits every company. Early teams may hire one or two people who combine analysis, some data science, and project management. Their first tasks can be basic questions such as revenue, customer counts, churn cohorts, and what makes customers successful. As the company grows, data specialists may embed in finance, marketing, sales, and customer success. Later, a central data platform team may consolidate architecture and make data self-serve. Organisations can decentralise again as they grow, while events such as preparing for an IPO can create pressure to align on shared practices.

### Data observability needs its own definition and framework
[27:30](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=1650s)
Moses says software observability has established concepts such as traces, metrics, and logs, while data teams still have to define what observability means for their systems. Monte Carlo studied hundreds of data teams, collecting horror stories, root causes, symptoms, and resolution methods. From that work, Moses describes five pillars of data observability: freshness, volume, schema, distribution, and lineage. She says observing these together gives teams visibility into the health of their data and can help prevent problems rather than merely report them afterward.

### Monitoring tools may follow people and responsibilities rather than one universal product
[32:56](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=1976s)
When Vishnu Rachakonda asks why Monte Carlo focused on data monitoring instead of covering software, data, and model monitoring, Moses says the areas are more similar than different and can be connected. She also says the personas are still changing. Someone called a BI professional might write ETL and do data science, while an ML engineer may handle other work before a dedicated ML function exists. For that reason, she expects tools to develop around specific users and their pain points. She does not commit to one tool handling every type of monitoring.

### Everyone connected to a data product shares responsibility for its failures
[36:12](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=2172s)
Moses compares data ownership with an earlier question she faced in customer success: who owns an unhappy or churning customer when sales, customer success, and product all contributed? Her answer is that everyone has a part in the outcome. The same applies when data downtime affects a model or customer-facing product. The person responsible for the model, the person responsible for its input data, and the customer who finds the issue may all be involved. Moses still wants teams to define who owns each part, while accepting that no group can simply avoid the problem.

### Lineage and anomaly detection reduce manual investigation
[43:30](https://www.youtube.com/watch?v=IMyI5eKQxMI&t=2610s)
Moses lists common causes of failure: more data sources, more complex pipelines, larger specialised teams, schema changes, changed field types, syntax errors, partial transfers, and datasets filled with nulls. She recalls manually comparing each stage of a pipeline and creating whiteboarded checks for every morning. Monte Carlo's approach connects to existing systems, monitors data at rest, uses machine learning to learn the environment, and looks for anomalies without requiring users to set thresholds. It also provides lineage, triage, and runbook support. Moses says the information changes behaviour because people can see what a change may break before making it.

## Notable quotes
- Barr Moses: "If you don't have the fundamentals right everything else is incredibly hard." (08:41)
- Barr Moses: "We need to develop the methodologies in the same way that we have in DevOps to support data engineers and to support folks in data to make sure that their data is up and they have five nines of availability." (17:47)
- Barr Moses: "The first is freshness, the second is volume, third is schema, fourth is distribution and fifth is lineage." (31:13)
- Barr Moses: "Hope is not a strategy." (49:54)
- Barr Moses: "When you put sort of the right technology and information and data in the hands of people you actually get that cultural change." (57:02)

## Tools & references mentioned
- Monte Carlo
- Gainsight
- Bain & Company
- Stanford
- Israeli Air Force
- Percy Diaconis
- Math Magic
- Facebook
- Mixpanel
- DevOps
- New Relic
- Datadog
- PagerDuty
- Excel
- COVID-19
- GDPR
- CCPA

## Who should watch
- You run reports, models, or customer-facing products and need to find out when upstream data has become stale or wrong.
- Your data team spends time tracing incidents by hand and wants a framework for deciding what to monitor.
- You are trying to get other teams to fund data reliability work and need to connect it to customers, revenue, campaigns, or compliance.

## Related talks

- [Scaling Data Reliably: A Journey in Growing Through Data Pain Points](https://mlopstalks.com/talks/scaling-data-reliably-a-journey-in-growing-through-data-pain-points) (Miriah Peterson, 16:04)
- [Data Quality Management Techniques - The Complete Guide](https://mlopstalks.com/talks/data-quality-management-techniques-the-complete-guide) (, 28:07)
- [Data Contracts: The Missing Piece of the Data Puzzle](https://mlopstalks.com/talks/data-contracts-the-missing-piece-of-the-data-puzzle) (Mark Freeman, Humu, 13:40)
- [The Rise of Modern Data Management](https://mlopstalks.com/talks/the-rise-of-modern-data-management) (Chad Sanderson, Gable.ai, 57:53)
- [Durable Data Discovery: Making Exploratory Analysis Stick](https://mlopstalks.com/talks/durable-data-discovery-making-exploratory-analysis-stick) (James Campbell, Superconductive, 58:26)