# Building 12-Factor Data Apps with Kedro

Ivan Danov, QuantumBlack | MLOps Meetup | Episode 90 | 1:22:18
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=j-qPZkdJREw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/building-12-factor-data-apps-with-kedro
Published: 2021-12-21
Tags: data-pipelines, deployment, orchestration

## TL;DR
- Machine learning projects need a shift from one-off analysis toward repeatable machine learning products that can run on demand or keep a dashboard up to date.
- The 12-Factor App principles provide useful engineering guidance for data applications, especially around dependencies, configuration, backing services, process state, disposability, and development-production parity.
- Kedro structures data pipelines as Python applications with nodes, pipelines, configuration, and data catalogs, while orchestration and monitoring remain the responsibility of tools such as Airflow or Kubeflow.

## Summary
Ivan Danov argues that data teams are repeating problems the web software industry faced during its rapid growth. One-off analyses are becoming machine learning products that must run repeatedly, support business decisions, and operate reliably. He introduces the 12-Factor App methodology as a set of cultural practices rather than a strict checklist. The principles cover one code base, declared dependencies, deploy-specific configuration, swappable backing services, separated build and release stages, stateless processes, disposability, development-production parity, event-stream logs, and one-off admin processes. Danov then maps these ideas onto Kedro. Kedro gives teams a Python project structure, pure-function nodes, pipelines, a configuration system, and a data catalog that keeps storage details out of code. Kedro authors pipelines, while orchestrators run and monitor them. The comparison with web frameworks is intended to help data teams reuse established software engineering ideas instead of inventing separate production practices.

## Key ideas
### Data teams are moving from one-off analysis to repeatable machine learning products
[07:35](https://www.youtube.com/watch?v=j-qPZkdJREw&t=455s)
Danov distinguishes projects that draw insights once from machine learning products that repeat the same process at scale. Clients increasingly want recommendations generated on demand or dashboards that stay current. These products support decisions in large enterprises, where failures have higher consequences. He compares the change to moving from a small shed to a multi-storey building. The engineering mindset has to change because the system must be reliable, repeatable, and suitable for ongoing operation rather than useful only during a short investigation.

### The 12-Factor App methodology grew from the web industry's need to scale
[10:06](https://www.youtube.com/watch?v=j-qPZkdJREw&t=606s)
Danov describes the web industry's growth from the early 2000s to 2010, when companies needed to support more users and more frequent interactions. Portability, interoperability, modularity, standardization, and DevOps culture helped teams scale applications. Heroku observed which applications were easier to scale and described twelve factors that could guide their design. Danov presents these factors as part of a changing engineering culture, rather than as a standard that teams must follow perfectly. He says frameworks embedded many of the practices so developers received much of the discipline automatically.

### Configuration should describe deploy-specific resources, while application wiring stays in code
[19:13](https://www.youtube.com/watch?v=j-qPZkdJREw&t=1153s)
For Danov, configuration includes anything that changes between deploys, such as credentials and the location of a database or S3 resource. Internal application wiring belongs in code, rather than scattered YAML files that must change at runtime. This separation lets operations teams configure an application without understanding all its internal details. He also describes backing services as swappable resources. A local file system and S3 can provide the same role when the application selects between them through configuration. Kedro applies this separation through its configuration folders and data catalog.

### Data applications benefit from explicit build, release, and run stages
[21:15](https://www.youtube.com/watch?v=j-qPZkdJREw&t=1275s)
Danov says interpreted languages make data scientists forget that applications still have build and release stages. The build stage turns code into an executable artifact. The release stage combines that artifact with configuration. The run stage starts the application in its environment and applies the runtime configuration. Keeping these stages separate makes it clearer when changes should happen and makes debugging easier. He connects this practice with Docker and with the wider software culture that made applications easier to package, deploy, and operate.

### Stateless and disposable processes protect long-running data jobs
[23:37](https://www.youtube.com/watch?v=j-qPZkdJREw&t=1417s)
The 12-Factor process model treats applications as stateless processes. Data that must survive a run belongs in a stateful backing service, not in process memory or a local cache. Danov connects disposability directly to data work, where jobs can run for hours and fail overnight. Jobs should be re-entrant, use transactions or idempotent operations, and shut down cleanly, so a restart does not corrupt data or force all work to begin again. He says data scientists and engineers already understand this problem because long-running jobs have always been vulnerable to failure.

### Kedro authors data pipelines, while orchestrators run and monitor them
[37:20](https://www.youtube.com/watch?v=j-qPZkdJREw&t=2240s)
Danov positions Kedro as a Python framework for structuring machine learning and data projects, not as another workflow orchestrator. Kedro helps teams author a pipeline across ingestion, cleaning, feature engineering, model training, testing, and deployment. Tools such as Airflow and Kubeflow focus on scheduling, execution, failure handling, and monitoring. He describes orchestrators as task-centric because they leave data loading to the application. Kedro is data-centric: the pipeline describes processing, while configuration describes where data comes from and where it goes.

### Kedro makes pipeline structure and data locations explicit
[43:07](https://www.youtube.com/watch?v=j-qPZkdJREw&t=2587s)
A Kedro project includes build configuration, application configuration, runtime data configuration, source code, tests, and a pipeline registry. A node wraps a normal Python function with named inputs and outputs. Danov recommends pure functions that read their arguments and return results rather than performing hidden file operations. The data catalog maps dataset names to formats and locations, such as a CSV file. Moving from a local file to S3 can therefore require a configuration change without changing the pipeline code. Kedro-Viz derives a graph from the pipeline registry, which also gives teams a way to explain the work to non-technical stakeholders.

### The web application analogy offers reusable design ideas for MLOps
[54:15](https://www.youtube.com/watch?v=j-qPZkdJREw&t=3255s)
Danov compares web applications built around MVC with data pipelines built around ETL. In both cases, a graph connects data or requests to functions that produce changed outputs. Kedro nodes resemble web routes that direct inputs to functions, while datasets have some similarities to models and schemas. The analogy is deliberately broad, and Danov acknowledges that web site maps are not always directed acyclic graphs. His practical conclusion is that data teams can draw from established web engineering practices instead of creating a separate production culture from scratch.

## Notable quotes
- Ivan Danov: "The focus of Kedro is for you to produce something that is worth being run often." (39:32)
- Ivan Danov: "You don't need to change a single line in your code base, it's just configuration." (52:32)
- Ivan Danov: "If you squint your eyes hard enough, these two patterns are very, very similar." (57:39)
- Ivan Danov: "If we can remove the technical barriers for getting things to production, then I'm pretty sure that can move the needle quite a bit." (1:15:26)

## Tools & references mentioned
- Kedro
- Kedro-Viz
- QuantumBlack
- McKinsey
- Heroku
- Docker
- Kubernetes
- Airflow
- Kubeflow
- MLflow
- Django
- Ruby on Rails
- Spring Boot
- Play
- dbt
- Pachyderm
- Spark
- Dask
- React

## Who should watch
- You are building data or machine learning pipelines that started as notebooks or one-off analyses and now need repeatable deployments.
- Your team is unsure where a pipeline framework ends and orchestration, scheduling, and monitoring begin.
- You want a practical way to apply software engineering practices such as dependency isolation, deploy-specific configuration, stateless processes, and development-production parity to data projects.

## Editor's note

Ivan Danov argues that data teams need to move from one-off analysis to repeatable machine learning products that can run on demand or keep dashboards current. ZenML can structure such workflows as Python pipelines whose runs record steps, inputs, outputs, and code version, so a model or artifact can be traced to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Building a Python-Centric Feature Platform to Power Production AI Applications](https://mlopstalks.com/talks/building-a-python-centric-feature-platform-to-power-production-ai-applications) (Matt Bleifer, Tecton, 27:11)
- [Applying DevOps Practices in Data and ML Engineering](https://mlopstalks.com/talks/applying-devops-practices-in-data-and-ml-engineering) (Antoni Ivanov, VMWare, 1:04:43)
- [2 tools = 90% operational ML](https://mlopstalks.com/talks/2-tools-90-operational-ml) (Michael Del Balso, Tecton & Willem Pienaar, Feast & David Aronchick, Kubeflow, 56:39)
- [DataOps is a Software Engineering Challenge](https://mlopstalks.com/talks/dataops-is-a-software-engineering-challenge) (Micha Kunze, Maersk, 57:57)
- [Why You Need More Than Airflow](https://mlopstalks.com/talks/why-you-need-more-than-airflow) (Ketan Umare, Union.ai, 1:11:12)