Micha Kunze argues that data teams can improve speed and reliability by applying ordinary software engineering practices to data work.
2
His team runs production-scale experiments, uses generated data and automated tests, and compares changes against the same production snapshot.
3
Data quality checks, runtime metrics and circuit breakers stop bad data or forecasts from reaching Maersk's automated decision-making processes.
Summary
Micha Kunze describes how Maersk's forecasting team operates five machine learning products that produce about 3 million forecasts each day. The team has to work with constantly changing global logistics data, legacy systems and modern infrastructure. Kunze's approach is to treat DataOps as an engineering problem. The team keeps development loops short, runs experiments at production scale, uses simple containers on Kubernetes and blob storage, and avoids unnecessary environments and services. Generated test data lets engineers test complete jobs locally, while schema checks expose downstream breakage before deployment. Runtime validation checks inputs and outputs, records metrics and stops pipelines when data or forecasts look wrong. Kunze is direct about the limits of sample data. A partial view of a global operation can produce misleading conclusions, so changes need to be compared against production data and the same snapshot. He also explains why the team prefers open source components and in-house code when a small engineering effort can provide the required control.
Maersk's forecasting team turns system-of-record data into forecasts about future container demand and equipment movements. Operations use those forecasts to plan capacity and make decisions, so the team operates the data, models and production systems together. The group has five machine learning products and produces roughly 3 million forecasts each day over a time horizon of about a quarter. The difficulty comes from global logistics, changing customer behavior and events such as port lockdowns and congestion. Maersk is also 120 years old, with infrastructure ranging from mainframes to Kafka. Kunze says the team must handle both business complexity and the work of connecting old and new systems.
Kunze defines his team's DataOps work as delivering high-quality, reliable data as quickly as possible when new features are needed. When he joined, some important datasets arrived on time only 75% of the time. The team improved that to 99.x% and added systematic validation to its pipelines. It still prevents incidents, including cases where an ARIMA model would have produced a bad forecast. The team has roughly 5 to 20 production changes per day and treats continuous integration and continuous deployment as part of its normal work. Kunze's point is practical: reliability is built into the delivery process rather than checked manually after the fact.
Industrialisation replaces manual checks with repeatable systems
Kunze compares DataOps with moving from a hand tool to an industrial machine. Manual work and basic scripts may help an individual make progress, but operating machine learning products every day requires repeatable, high-quality execution at scale. Automatic notifications and built-in tooling reduce the need for people to inspect forecasts manually. The team's stack is deliberately simple. It uses blob storage, Kubernetes, jobs running scikit-learn, PySpark or Spark, R, occasional Streamlit dashboards and scripts. There is no SaaS tool, data warehouse or feature store in the setup he describes. Kunze calls it a metal stack because it stays close to the underlying infrastructure.
Kunze rejects the idea that teams must choose between moving quickly and producing reliable systems. The team optimises for short development loops, automates common problems and runs full-scale experiments as first-class work. Local development should take seconds or minutes, followed by CI/CD and production or production-scale execution. He cites the principle that 'only production counts' because sample data and partial views can hide problems in a global forecasting system. The team therefore aims to reach production quickly and learn from realistic results. Tests move checks into the fast local loop, where failures cost minutes instead of hours or days.
The team uses generated data instead of relying on small samples copied from transaction logs. Generated scenarios make it easier to express different cases, reproduce bugs and prevent regressions. Kunze prefers testing a complete job, or sometimes a short sequence of jobs, through stable input and output interfaces. Unit tests are still used for particularly complicated transformations, but testing every internal function can create maintenance work when the order of operations changes. Data classes define columns and Spark schemas, which lets the team generate data and test downstream dependencies in parallel. These checks run locally and on pull requests, so engineers can learn about schema breakage before deploying.
For a data or model change, the team runs the changed path alongside the production path and compares the results. Feature flags or shadow deployments let engineers change an upstream step, such as filtering, and inspect its effect on later predictions. Each dataset has a specific snapshot, so both paths use the same underlying data and the comparison is between like-for-like results. Kunze says this makes it possible to decide whether a change is useful before adopting it. Running the experiment in a development environment with different data would take much longer and would not provide the same evidence about production behavior.
The team tests both code and data. Each job can validate its input and output separately from the job logic. A validation failure triggers an automatic alert and puts the job into a failed state, which prevents bad data from continuing through an automated decision process. The checks include schemas, expected values and custom metrics. Kunze shows Great Expectations defining expectations in JSON for a job's data. The team also tracks cases such as missing UTC timestamps when a local timestamp is used as a fallback. That metric reveals data drift that might otherwise remain hidden downstream. The goal is to catch bad forecasts, corrupted data and changes in source quality before they are published.
A single data snapshot can look acceptable while the underlying behavior changes over time. The team saves standard data-quality metrics and custom metrics with job metadata, so engineers can inspect what happened for an exact snapshot and follow trends across runs. Simple metrics go to DataDog, while more complicated accuracy views are handled with jobs and Streamlit dashboards. Kunze says this helps explain why an alert fired and how a threshold evolved, rather than leaving engineers to inspect data manually. The process creates a feedback loop: the team finds unexpected data, fixes the pipeline or the upstream source, and adds protection against the same problem returning.
