# Scaling Data Reliably: A Journey in Growing Through Data Pain Points Miriah Peterson | DE4AI 2024 | 16:04 Source: https://www.youtube.com/watch?v=mqXawOz-OqA Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/scaling-data-reliably-a-journey-in-growing-through-data-pain-points Published: 2024-10-09 Tags: data-engineering, data-quality, observability, reliability ## TL;DR - Data downtime includes partial, erroneous, or missing data, and it can break dashboards, model training, inference, and financial operations. - Data reliability engineering applies engineering practices to data quality, with metrics for data volume, sources, accuracy, value, and throughput. - SLAs, SLOs, and SLIs should match the stakeholder, while severity rules help teams respond to active business impact without treating every issue as an outage. ## Summary Miriah Peterson explains data downtime as any period when data is partial, erroneous, or missing. The effects reach beyond broken dashboards. They include inaccurate model training, failed inference, and financial operations based on incorrect warehouse data. She frames data reliability engineering as an engineering mindset that treats data quality as something teams can measure, improve, and repair quickly. Peterson describes the five Vs as a way to discuss data volume, variety, veracity, value, and velocity with the business. She then applies reliability metrics such as latency, traffic, errors, and saturation to services built on top of data. SLAs, SLOs, and SLIs need to reflect the needs of each stakeholder, from researchers using historical data to end users expecting a near-instant application response. Her practical advice is to define severity by business impact and use clear remediation paths so engineers can address urgent failures without responding to every problem as an emergency. ## Key ideas ### Data downtime includes wrong data, not only unavailable systems [01:45](https://www.youtube.com/watch?v=mqXawOz-OqA&t=105s) Miriah Peterson defines data downtime as a period when data is partial, erroneous, or missing. It can affect data used for training and inference, as well as dashboards, tables, files, and APIs. The result may be inaccurate model training, false results, broken financial operations, or an unavailable service. This differs from ordinary software downtime because the system may still be running while the data itself is incorrect. Duplicate data can also create problems, including training a model on repeated records. ### Reliable systems should fail quickly and recover quickly [02:39](https://www.youtube.com/watch?v=mqXawOz-OqA&t=159s) Peterson uses reliability, maintainability, and operability as the foundation for software and data systems. Systems will fail, and teams should be able to touch and change them without treating every failure as a reason to avoid innovation. Reliability comes from fixing failures quickly rather than pretending failures can be eliminated. She connects fast recovery to high uptime and says the goal is to minimize downtime whenever a system breaks. ### Data reliability engineering treats data quality as an engineering problem [05:00](https://www.youtube.com/watch?v=mqXawOz-OqA&t=300s) Peterson describes data reliability engineering as a field and mindset rather than a job title. The work is to iterate on the data itself and improve the things built from it. Since data supports AI and machine learning, she argues that teams need to improve that foundation with the same engineering approach used for other systems. Reliability concerns can appear across serving, analytics, storage, transformation, security, operations, orchestration, and software engineering. ### The five Vs help teams explain the value and condition of data [05:48](https://www.youtube.com/watch?v=mqXawOz-OqA&t=348s) Peterson uses volume, variety, veracity, value, and velocity to measure data. Volume covers how much data flows through streams, transformations, training, APIs, and warehouses. Variety covers the sources, such as APIs, databases, and tables. Veracity asks whether the data produces the expected insights. Value asks whether data is being used enough to justify its storage cost. Velocity covers throughput and how quickly teams can obtain new data, train or update models, create embeddings, and put changes into products. ### Reliability metrics apply to services built on top of data [08:05](https://www.youtube.com/watch?v=mqXawOz-OqA&t=485s) For data-powered services, Peterson uses the four reliability pillars from Google's Site Reliability Engineering book: latency, traffic, errors, and saturation. A service team needs to know how often an API or model is called, how long requests take, how many errors occur, and whether capacity is sufficient. She gives her Twitch bot as an example. A response taking 30 seconds to a minute can fit the normal rhythm of chat, while a five-minute response is too slow and calls for a change. ### Service objectives should match the person using the data [09:09](https://www.youtube.com/watch?v=mqXawOz-OqA&t=549s) Peterson explains SLAs, SLOs, and SLIs through a batch data pipeline. A freshness SLA might say that data must never be more than 24 hours old. The objective is to extract and transform it daily, while an error, timeout, or alert indicates that the pipeline needs attention. Different stakeholders need different contracts. A data scientist may care about historical completeness and accuracy. An executive may need a warehouse refreshed within 24 hours. An end user of a machine learning application may expect a response within a few seconds. ### Severity should reflect business impact and give engineers room to work [12:22](https://www.youtube.com/watch?v=mqXawOz-OqA&t=742s) A database being unavailable can be more severe than duplicate values in a dataset, depending on who is affected. Peterson suggests deciding whether an issue requires immediate action, can wait until the next business day, or belongs in a future sprint. The important question is whether the failure is actively causing money to be lost or preventing someone from using a service. Clear severity levels reduce constant interruptions and set expectations with stakeholders. ### Data reliability engineers connect the customer experience to the data underneath [13:40](https://www.youtube.com/watch?v=mqXawOz-OqA&t=820s) Peterson describes the data reliability engineer as someone who connects what a customer needs to the data and systems that support it. This involves understanding how people use the data, setting suitable alerts, defining reliability expectations, and using SLAs to guide action. Her closing advice is to add data-specific complexity to ordinary reliability engineering, measure the data with appropriate metrics, confirm that systems on top of it are available, and choose urgency based on the actual impact of downtime. ## Notable quotes - "Any system you design, so any piece of software you build, which is literally everything we do, has to be built with the mindset of reliability, maintainability and operability." (02:52) - "Data reliability engineering is treating data quality as an engineering problem." (05:27) - "A 30 second delay is usually about how long it takes for a normal person to respond in Twitch chat." (08:48) - "Sometimes a downtime does not mean an outage." (12:22) - "We all have data downtime, and the way to minimize our data downtime is with reliability engineering." (14:25) ## Tools & references mentioned - Designing Data-Intensive Applications - Database Reliability Engineering - Data Engineering Podcast - Google's Site Reliability Engineering book - Twitch ## Who should watch - You own data pipelines that feed dashboards, machine learning systems, APIs, or financial operations and need a practical definition of what can go wrong. - Your team receives frequent requests about missing or incorrect data and needs service targets, alerts, and severity rules to reduce interruptions. - You are introducing reliability practices to a data organization and want a simple way to connect data quality metrics with stakeholder expectations. ## Related talks - [Introducing Data Downtime: From Firefighting to Winning](https://mlopstalks.com/talks/introducing-data-downtime-from-firefighting-to-winning) (Barr Moses, Monte Carlo, 1:00:51) - [SRE for ML Infra](https://mlopstalks.com/talks/sre-for-ml-infra) (Todd Underwood, Google, 1:11:50) - [Data Engineering + ML + Software Engineering](https://mlopstalks.com/talks/data-engineering-ml-software-engineering) (Satish Chandra Gupta, Slang Labs, 57:05) - [Data Quality Management Techniques - The Complete Guide](https://mlopstalks.com/talks/data-quality-management-techniques-the-complete-guide) (, 28:07) - [Build Reliable Systems with Chaos Engineering](https://mlopstalks.com/talks/build-reliable-systems-with-chaos-engineering) (Benjamin Wilms, Steadybit, 46:58)