DataOps combines data work with development and operations so teams can build and use data products together.
2
DataOps depends on shared tools, shared processes, data democratization, and immutable data that teams can discover without constant coordination.
3
Privacy requirements can conflict with immutable data platforms, so organizations must plan how they will delete personal data before collecting it.
Summary
Lars Albertsson explains DataOps as a way of working across data engineering, data science, development, and operations. He traces its roots through waterfall, agile, DevOps, and big data, arguing that the important change is how teams work together rather than which tools they buy. At Spotify, shared infrastructure helped teams build data products and enabled bottom-up innovation. Albertsson connects this foundation to machine learning, where repeatable results require data collection, processing, training, validation, monitoring, and retraining. Privacy creates a direct tension with DataOps because immutable data lakes make collaboration easier while making deletion harder. He recommends separating personal data from non-personal data or planning for reprocessing. Compliance should also be built into code review, approvals, deployment, and audit processes instead of handled through occasional manual checks.
DataOps grew from combining data work with development and operations
Albertsson presents DataOps as the next step in a longer shift away from isolated phases of work. Waterfall separated requirements, design, coding, testing, and operations. Agile brought requirements and implementation together, while DevOps combined development and operations. Big data made analysis part of product development because teams could collect and measure large amounts of data. DataOps then combines data work with development and operations. Its main point is cooperation across disciplines toward the same goal, even when the people involved have different skills.
Spotify used shared data capabilities to spread innovation across the company
When Albertsson joined Spotify, the company had a large Hadoop cluster, but effective work with data depended on tribal knowledge. The team worked to let more than a few groups build data pipelines and use data in their engineering processes. Over time, most engineering teams could process data and use it in their work. He cites the team behind Discover Weekly, which said its prototype succeeded because Spotify had enabled bottom-up innovation. Teams could build on existing recommendation algorithms and infrastructure instead of waiting for a central strategic decision.
DataOps adoption depends on teams sharing environments and definitions of success
Albertsson says organizations need people with different competences to work together in the same teams, at the same location where that applies, and toward the same definition of success. Their shared goal affects processes and tools. Separate teams with separate environments can prevent cooperation. Notebooks such as Jupyter and SageMaker are useful for explaining work, but they should not be placed directly into production. At the same time, data scientists are unlikely to work comfortably in a typical Java and Maven environment. Teams need common ground that works for everyone involved.
Data democratization requires discoverable and immutable data
Teams can innovate without synchronizing with other teams when they can find and use the data they need. Albertsson describes putting data into a data lake as one way to reduce requests for direct database access. The data should preferably be immutable. If data changes underneath a team, that team must coordinate again to understand the change. Immutability makes the data easier to share because users know it will not change. Albertsson says organizations often miss this requirement when they try to democratize data.
Machine learning needs an operational data foundation before it can deliver repeatable value
Albertsson uses Monica Rogati's data science hierarchy of needs to explain why DataOps matters to machine learning. Before a company can build advanced AI systems, it must collect data, process and refine it, draw conclusions, measure results, and run tests such as A/B tests. These capabilities form a foundation for sustainable machine learning. A team can build an isolated chatbot or another small experiment without that foundation. It cannot produce results in a repeatable and predictable way. Machine learning also adds training, validation, monitoring, and regular retraining.
Privacy rules conflict with the immutability that makes data platforms useful
Older database systems often kept data in separate silos, so correcting or removing a user could mean changing or deleting a row. Data lakes favor large, immutable files, which makes removing one person's information difficult. Albertsson describes a case where an organization keyed large volumes of data by username and later discovered that it could not meet deletion requirements. Recovering from that design took tens of engineering years. His advice is to plan for deletion before collection, either by separating personal data from non-personal data or by designing for reprocessing.
Privacy compliance needs organized data and a controlled change process
Organizations must know where personal information exists if they need to delete it or provide all information held about a person. Albertsson recommends keeping personal data in a small number of manageable locations. A DataOps culture helps control the heterogeneity and entropy of a data platform, while a lack of organization can make privacy work grow out of control. He also says compliance should be part of the process for changing systems. Code review, approvals, recorded changes, and deployment through a reviewed continuous delivery process can make each system change auditable.
The value of technology comes from changing how people work together
Albertsson is skeptical of choosing technology for its feature list. He says clients often ask for machine learning platforms, feature stores, or streaming systems before defining what they want to do. GitHub changed collaboration by allowing people who did not share an organization or trust relationship to work together. Hadoop mattered partly because its limitations prevented users from mutating data, which pushed them toward data lakes, flows, and pipelines. He argues that early adopters benefited from learning new working methods, while late adopters sometimes asked vendors to add transactions and SQL so they could preserve old workflows.
