The UN Global Platform helped national statistical offices use big data to create official statistics and support the Sustainable Development Goals.
2
Mark Craddock's small core team built a multi-cloud platform because no single cloud provider met the needs of users across different countries.
3
Wardley Maps helped the team decide where to build, where to wait for commodity services, and how to organize teams around different types of work.
Summary
Mark Craddock describes the United Nations Global Platform, which supported statistical offices in using big data alongside traditional statistics, administrative sources, and geospatial information. The platform served users across 194-plus statistical offices and ran across Amazon, Microsoft, Google, and Alibaba because technology, security, and geopolitical needs differed by country. A core team of five built the platform while working with more than 50 people overall. Mark explains how Wardley Maps exposed which capabilities were novel, custom-built, productized, or available as commodity services. This helped the team avoid building its own Kubernetes and container stack when cloud providers began offering those services. The talk also covers encrypted computation, synthetic data, data lineage with Apache NiFi, and the use of flight and ship data for official statistics and economic analysis. The platform processed 600 million flight records per day and 40 million ship records per day.
The UN platform was built for statistical offices that needed to turn big data into official statistics
The platform's vision was global collaboration to use data for better lives. It supported the United Nations statistical community and national statistical offices in using big data to create official statistics. The work also supported the Sustainable Development Goals, including the goal of removing poverty by 2030. Mark describes the platform as a place to learn how to use big data, choose suitable algorithms and methods, and combine those methods with traditional statistical work. The users were public organizations whose statistics inform government policy and spending.
A small core team built a multi-cloud platform for users with different national constraints
The core team grew from two people to five, with four people focused on technical work. More than 50 people were involved overall, including administration, governance, and work inside the UN. The final platform ran across Amazon, Microsoft, Google, and Alibaba. Mark says this was because no single cloud met all user needs globally. The reasons included geopolitical and security concerns as well as technology. The team started with a minimal viable platform, tested tools and techniques, released the platform to users, collected feedback, and iterated around user needs.
The platform's design included a commitment to reach users who were furthest behind
Mark describes one of the platform's principles as leaving no one behind and trying to reach the furthest behind first. In practice, that meant spending more effort bringing less advanced users up to the same level as everyone else. This affected how the team thought about the platform and its users. The platform had to support statistical offices with different levels of technical capability, different infrastructure, and different access to data. The principle was tied to the UN context, where the platform had to work across many countries rather than for one uniform organization.
The Platform Design Toolkit focused the team on the business model around the digital platform
Mark distinguishes the platform business model from the underlying digital platform. The business model asks what users need, what kind of platform is being created, and how participants connect within an ecosystem. He describes platforms as bringing providers and consumers together, like a dating site, while reducing friction for participants. The Platform Design Toolkit supplied templates, processes, and material about user needs and network effects. Mark says a platform should allow people to do things its creators did not expect, since those uses can reveal new markets and new ways to meet user needs.
Wardley Maps showed where the team should build and where it should wait
Mark explains that a Wardley Map places capabilities along an evolutionary path from genesis to custom-built, product, and commodity or utility. New and unusual capabilities sit on the left. Familiar, pay-per-use capabilities such as cloud compute sit on the right. The map helped the UN team avoid spending time and money building infrastructure that cloud providers would soon offer as a service. Mark says the team waited for managed services instead of building its own Kubernetes stack and saved about a million by doing so. The map predicts movement toward the right, although it does not predict the exact timing.
Different stages of evolution called for different teams and skills
Mark says teams can also be organized from a Wardley Map. Work on the right involves known services that people configure and connect. Work on the left involves new capabilities that require people who can write code, handle uncertainty, accept failure, and adapt quickly. He contrasts people who glue existing services together with people who develop new components. The map also exposed inertia between stages, such as objections from teams that were reluctant to move from owned data-center infrastructure to cloud services. Mark says the Wardley Mapping material includes ways to understand and address those forms of inertia.
Data and statistical methods also move from unfamiliar work toward reusable services
Mark applies the same evolutionary model to data. A familiar, well-defined dataset can be consumed as a service, while new or poorly understood data needs custom work. He warns that one tool cannot handle data at every stage. The UN team preferred data that providers had already structured and defined, since this reduced the time spent trying to understand an unfamiliar dataset. Statistical methods remained custom-built within statistical offices because they required substantial rigor, but their standards and outputs could be reused by other parts of the platform.
Privacy-preserving computation could let organizations share data without exposing raw records
The platform identified multi-party authentication, multi-party computation, and homomorphic encryption as capabilities that would matter over time. These methods allow calculations on encrypted data while keeping the source records hidden. Mark gives a height example: a group could calculate an average or identify the tallest person without revealing each person's height. He mentions OpenMined, Andrew Trask, PyTorch, and work from Cybernetica as examples of this area. He also describes efforts to use machine learning to create synthetic mobile-phone datasets that look real enough for analysis while protecting people's privacy.
Streaming location data gave the platform material for public statistics and economic analysis
The platform received around 40 million ship records per day and 600 million flight records per day. Flight data included identifiers, location, speed, and altitude. Ship data included vessel identity, position, heading, and speed. Apache NiFi collected, processed, routed, and stored the streams, while Kafka, HBase, Spark, GeoMesa, GeoServer, and Jupyter were part of the wider stack. NiFi also provided lineage, so users could trace where data came from and what operations had been applied. During COVID-19, flight data helped identify countries that might see more cases, while ship movements helped track economic activity and inform policy decisions.
"You could do the calculation, you can get the average height and you get the choice and the mean or whatever, but you would never know, you would never see the raw data because it would always be encrypted."Mark Craddock18:30
Who should watch
You are designing a data or machine learning platform for users in several countries and need to account for cloud, security, and geopolitical constraints.
Your team is deciding which infrastructure to build and which capabilities to consume from cloud providers.
You work with sensitive data and want practical examples of lineage, encrypted computation, synthetic data, or large location-data streams.
