# Building the World's First Data Engineering Conference

Pete Soderling, Data Council and Data Community Fund | MLOps Coffee Sessions | Episode 92 | 41:58
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=RyeoFSctI-M
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-the-worlds-first-data-engineering-conference
Published: 2022-04-11
Tags: developer-experience, open-source, product-strategy, startups

## TL;DR
- Data Council grew from a 2013 meetup for a barely recognized profession into a full-stack data conference where adjacent technical roles learn to work together.
- Pete Soderling wants to help 1,000 engineers start companies, using his community and investing work to support founders building tools for engineers.
- Open source founders need to understand how community adoption, promotion, and revenue fit together before turning a project into a company.

## Summary
Pete Soderling describes how Data Council began as a 2013 meetup inside Spotify's New York office and grew into a conference covering data infrastructure, data engineering, science, models, analytics, and AI. He connects that community-building work to his goal of helping 1,000 engineers start companies. His investing focus is on founders with a specific insight grounded in personal experience. Pete also explains why consulting work can reveal and test a broadly useful tool, using dbt and Great Expectations as examples. The discussion then turns to open source business models. Pete says founders should distinguish open source adoption from commercial revenue and should not casually mix the two. Community building itself involves promotion, explanation, and gentle sales work. On no-code tools, he supports broader access to data systems but prefers low-code systems that let technical users inspect or change the underlying logic.

## Key ideas
### Data Council grew by defining and connecting a new technical community
[05:12](https://www.youtube.com/watch?v=RyeoFSctI-M&t=312s)
Pete says Data Council started in 2013 as a meetup inside the Spotify office in New York. At the time, data engineering was barely recognized as a job or discipline. The group claimed the term and began asking who data engineers were, what they did, and how they could learn from one another. By 2015, data engineers were joined by data scientists, analysts, and AI researchers. Their shared interest was learning to work better with adjacent roles across the stack. The meetup became a conference covering data infrastructure, engineering, science, models, analytics, and AI products. The 2022 Austin event brought those groups together again after the pandemic.

### The return to in-person events exposed the data ecosystem's growth
[04:52](https://www.youtube.com/watch?v=RyeoFSctI-M&t=292s)
Pete describes the 2022 Austin conference as the first Data Council since the pandemic, with 600 people attending. He attributes the strong response to growing interest in data tooling, data systems, and new open source projects, along with the excitement of meeting in person again. Attendees followed the event with Twitter activity and blog posts, and Pete says people were visibly happy to see one another. He was also struck by the number of investors who attended. Their presence gave some technical attendees a first view of how much the data ecosystem had expanded and what was helping fuel that expansion.

### Pete's investing goal follows from years of building tools for engineers
[09:33](https://www.youtube.com/watch?v=RyeoFSctI-M&t=573s)
Pete describes himself as an engineer who became a founder. He started two companies in New York before 2010 and two more in San Francisco afterward. The users or customers for all of them were engineers, so his work has consistently involved building tools, communities, or other products for that audience. He says one of his life goals is to help 1,000 other engineers start companies. Data Council showed him that a community centered on data engineering could produce interesting startups, including dbt, while also giving him a way to see founders and their work before they were widely known. Investing became a natural extension of that community work.

### The strongest early-stage company ideas usually come from a founder's experience
[21:28](https://www.youtube.com/watch?v=RyeoFSctI-M&t=1288s)
Pete says he looks for the insight that justifies creating an entire company. At the pre-seed and seed stage, he may be considering the first outside check, so he needs to believe that the founder has a specific understanding of a problem. He says this insight almost always comes from personal experience and is difficult to manufacture convincingly. Teams leaving large technology companies often attract investors because they have solved a data problem at scale and may carry a valuable insight with them. Pete does not limit this pattern to infrastructure companies. His test for any engineer-founder is whether the founder can explain the core idea that makes the company worth building.

### Consulting work can reveal a product that has already faced real use
[18:17](https://www.youtube.com/watch?v=RyeoFSctI-M&t=1097s)
Pete says venture investors generally do not invest in consulting companies, but consulting work can expose a tool that has wider value. Fishtown Analytics used dbt while working for consulting clients and developed it through those projects. Great Expectations followed a related path. Superconductive began as a small consulting company doing data work in healthcare, then grew Great Expectations from that experience. Pete calls these tools battle tested because they emerged from real client work and showed demand beyond the original team. A consulting practice does not automatically identify the right product, but it can reveal a process that produces a tool useful to a broader part of the data engineering community.

### Open source founders need to separate adoption from commercial revenue
[25:11](https://www.youtube.com/watch?v=RyeoFSctI-M&t=1511s)
Pete says founders should understand the difference between an open source go-to-market model and a commercial one because the revenue flows are very different. He warns against trying to combine the two too early, since that can create confusion about strategy and customers. Open source projects can build a valuable wedge when engineers use them repeatedly in a specific workflow, as dbt did in analytics engineering. That adoption creates a surface area that a company may later monetize. The difficult question is whether the people using the open source project are also the people who will pay. Pete points to Docker as another company that encountered this problem.

### Building an open source community requires deliberate promotion
[29:05](https://www.youtube.com/watch?v=RyeoFSctI-M&t=1745s)
Pete says he once assumed open source projects appeared through purely organic community activity. He later learned that developer evangelism and promotion are part of making a project successful. Engineers may dislike self-promotion and sales, but an open source leader still has to help people discover the project, understand its value, and gather around its purpose. That work can include blog posts, content marketing, explanations in bullet points, and public statements of the community's vision. Pete describes this as a gentler form of sales and marketing. He also sees it as a useful test for future founders because company building requires the same ability to explain, promote, and sometimes sell.

### Pete prefers low-code systems that keep technical control visible
[33:46](https://www.youtube.com/watch?v=RyeoFSctI-M&t=2026s)
Pete is skeptical that every data engineering or machine learning workflow should have a no-code version. He supports democratizing access to data systems for less technical users, but he wants engineers to retain a way to inspect and influence the logic a tool creates. He therefore describes himself as more of a low-code believer. He worries that no-code systems often omit versioning, explicit logic, stated assumptions, and testing. Adding a Python interpreter to a spreadsheet does not bring software engineering practices to the resulting macros. In his view, useful abstractions must still leave room for technical users to understand decisions and apply engineering discipline.

## Notable quotes
- Pete Soderling: "One of my life goals is to help a thousand other engineers start companies." (10:31)
- Pete Soderling: "If you have a consulting platform you might be able to step into a process that on the end sort of outputs a tool that's actually useful not just to you but a broader section of the data engineering world." (19:54)
- Pete Soderling: "When it comes to building a successful open source project, we need to realize that open source and sort of community development at least require some promotion." (29:23)
- Pete Soderling: "I'm much more of a low code believer than a no code believer." (34:40)
- Pete Soderling: "The question is just, are your open source users actually the people who are going to end up paying for your product in the end?" (27:14)

## Tools & references mentioned
- Data Council
- Data Community Fund
- Spotify
- dbt
- Snowflake
- Palantir
- Great Expectations
- Superconductive
- Fishtown Analytics
- Databricks
- Docker
- AngelList
- Peter Thiel
- Zero to One
- Kafka
- Retool
- Anvil

## Who should watch
- You are building an open source tool and need to decide whether community adoption can support a company.
- You are an engineer considering a startup and want a practical test for whether your idea comes from a real insight.
- You are designing a technical community and want to understand the promotion and communication work that sits behind organic participation.

## Related talks

- [Building Better Data Teams](https://mlopstalks.com/talks/building-better-data-teams) (Leanne Fitzpatrick, Financial Times, 1:01:40)
- [The Rise of Modern Data Management](https://mlopstalks.com/talks/the-rise-of-modern-data-management) (Chad Sanderson, Gable.ai, 57:53)
- [From A Coding Startup to AI Development in the Enterprise](https://mlopstalks.com/talks/from-a-coding-startup-to-ai-development-in-the-enterprise) (Ryan Carson, Intel, 58:16)
- [The Only Constant is (Data) Change](https://mlopstalks.com/talks/the-only-constant-is-data-change) (Benjamin Rogojan, Seattle Data Guy & Chad Sanderson, Gable & Christophe Blefari, NAO & Maggie Hays, Acryl Data, 40:50)
- [Collaboration and Strategy](https://mlopstalks.com/talks/collaboration-and-strategy) (Vin Vashishta, V Squared, 51:54)