Partnering with Product for Effective, Quality Data Ingestion & Training Data

Daniela Santisteban18:43 · Oct 2024 · 62 views
Thumbnail for Partnering with Product for Effective, Quality Data Ingestion & Training Data Watch on YouTube
TL;DR
  1. 1

    Data engineering can show more business value by partnering with product managers on measurement plans for products and features.

  2. 2

    Without early planning, teams discover missing tracking and data after launch, creating urgent work and disrupting sprints.

  3. 3

    Data engineers can act like data PMs by finding product road maps, defining requirements with product and analytics, and mapping dependencies into the delivery plan.

Summary

Daniela Santisteban explains how data engineering teams can work more closely with traditional product managers. Data teams often build pipelines and training data without staying connected to the product that eventually creates revenue. This makes their value less visible and causes problems when a product launches without the tracking needed to measure success. Daniela suggests that data engineers take on some data PM responsibilities when their organization does not have that role. They should find the product road map, identify launches where measurement matters, and meet with the product manager and analytics team before development is complete. Together, they can define success, data requirements, tracking changes, dependencies, and delivery timing. The data work does not always need to finish before launch, but it needs to be planned. Product managers can then include the work in their road maps and coordinate dependencies with other teams.

Key ideas
01:59

Data engineering becomes more visible when it connects its work to business outcomes

Daniela asks how data engineering teams can continue to justify their headcount and project costs after their initial pipelines are running. She says one answer is to attach data engineering work to high-value initiatives, including machine learning training data. That helps the team become more visible, but she says data engineers can go further by partnering with teams that measure whether those initiatives succeed. Product is her main example, although the same approach can apply to marketing. The aim is to connect data work to the end product and to the revenue it helps create.

03:36

Traditional product managers are a useful bridge between data teams and the end product

Daniela describes traditional product managers as people who understand business needs, work with development teams, research customers, build business cases, and communicate with sales, marketing, partnerships, and executives. They care about meeting release timelines and avoiding a product failure with customers. A data PM has a similar need to prove that data work matters, but many organizations do not have one. When that role is missing, data engineers can work directly with traditional product managers and take on part of the data PM role themselves.

05:55

Waiting until after launch creates missing data and urgent work

If data engineering stays focused on data scientists and analysts, contact with product management is usually reactive. Daniela says product teams then discover that analytics cannot show whether a newly launched feature succeeded because the required data was never planned or built. This creates bad surprises, urgent requests, sprint disruption, and organizational chaos. Planning measurement before a launch gives both teams time to identify the required tracking and pipelines. It also gives product managers a way to prove whether their work is successful.

07:41

The data PM approach starts by understanding the product manager's customer and pain

Daniela's framework has three goals: understand the customer, find where data can reduce that customer's pain, and bring the solution into production. In this setting, the customer is usually the traditional product manager, although the framework can also be used with other internal stakeholders. Product managers spend much of their time on execution and firefighting, while success measurement often receives less planning time. Data engineering can help by taking responsibility for measurement planning rather than waiting for the product manager to request it at the end.

12:16

The product road map reveals where data engineering should get involved

Daniela recommends finding the product road map instead of asking a product manager for a broad explanation of upcoming work. The road map may be in Jira or Productboard and can show delivery timelines, involved teams, and product areas that affect consumers. Data engineers should identify high-impact launches, products that use machine learning, and places where measurement planning can begin before release. She then recommends a meeting with the product individual contributor, the analytics team, and a data engineering representative.

13:13

Measurement planning requires defining success before deciding on the data work

In the discovery meeting, the group should determine what success means for the product, product manager, or feature. They should also decide when and how the results will be shown. Those answers become data requirements and a measurement plan. The work may include schema changes, new tables, and updates to tracking plans. Daniela provides a separate resource with questions for discovering these needs, especially when the internal customer is less technically experienced than the data team's usual stakeholders.

14:41

Data work needs to appear in the product delivery plan, including its dependencies

Daniela says data engineering should map its work into the broader product development plan. That means identifying the team's tasks, sequencing them, finding dependencies, and creating tickets. An API team may need to expose new variables or calls, or the product development team may need to make changes before data can be collected. Analytics work may finish after the product launch, but the timing still needs to be agreed. The product manager should include the data work in the product epic or road map and coordinate dependencies with other teams.

16:53

The partnership improves through repeated planning and feedback

Daniela recommends asking the product manager how the measurement building blocks can be shown in the road map. She presents her framework as guidance rather than a fixed process because product development changes through iteration. Data teams should ask product counterparts for feedback, invite them to bring new products and ideas for measurement, and create a recurring cadence for those conversations. Her final test is whether the data organization can connect its work from the start of the product equation to the point where money flows in.

"If you don't have a data PM to kind of take on this role and manage, you can still partner with these traditional product managers and put on this hat or mask of a data PM yourself."06:59
Who should watch
  • You work on data engineering or data platforms and product teams often ask for tracking or measurement after a launch.
  • Your organization has no dedicated data PM, but you need a practical way to work with product managers before development is complete.
  • You want data work to appear in product plans and connect more clearly to the outcomes that matter to the business.