# UN Global Platform

Mark Craddock, Global Certification and Training Ltd (GCATI) | MLOps Meetup | Episode 42 | 58:48
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=6oFCqGE_2-U
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/un-global-platform
Published: 2020-11-16
Tags: data-engineering, data-pipelines, platform-teams, privacy

## TL;DR
- The UN Global Platform helped national statistical offices use big data to create official statistics and support the Sustainable Development Goals.
- Mark Craddock's small core team built a multi-cloud platform because no single cloud provider met the needs of users across different countries.
- Wardley Maps helped the team decide where to build, where to wait for commodity services, and how to organize teams around different types of work.

## Summary
Mark Craddock describes the United Nations Global Platform, which supported statistical offices in using big data alongside traditional statistics, administrative sources, and geospatial information. The platform served users across 194-plus statistical offices and ran across Amazon, Microsoft, Google, and Alibaba because technology, security, and geopolitical needs differed by country. A core team of five built the platform while working with more than 50 people overall. Mark explains how Wardley Maps exposed which capabilities were novel, custom-built, productized, or available as commodity services. This helped the team avoid building its own Kubernetes and container stack when cloud providers began offering those services. The talk also covers encrypted computation, synthetic data, data lineage with Apache NiFi, and the use of flight and ship data for official statistics and economic analysis. The platform processed 600 million flight records per day and 40 million ship records per day.

## Key ideas
### The UN platform was built for statistical offices that needed to turn big data into official statistics
[05:11](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=311s)
The platform's vision was global collaboration to use data for better lives. It supported the United Nations statistical community and national statistical offices in using big data to create official statistics. The work also supported the Sustainable Development Goals, including the goal of removing poverty by 2030. Mark describes the platform as a place to learn how to use big data, choose suitable algorithms and methods, and combine those methods with traditional statistical work. The users were public organizations whose statistics inform government policy and spending.

### A small core team built a multi-cloud platform for users with different national constraints
[06:20](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=380s)
The core team grew from two people to five, with four people focused on technical work. More than 50 people were involved overall, including administration, governance, and work inside the UN. The final platform ran across Amazon, Microsoft, Google, and Alibaba. Mark says this was because no single cloud met all user needs globally. The reasons included geopolitical and security concerns as well as technology. The team started with a minimal viable platform, tested tools and techniques, released the platform to users, collected feedback, and iterated around user needs.

### The platform's design included a commitment to reach users who were furthest behind
[08:11](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=491s)
Mark describes one of the platform's principles as leaving no one behind and trying to reach the furthest behind first. In practice, that meant spending more effort bringing less advanced users up to the same level as everyone else. This affected how the team thought about the platform and its users. The platform had to support statistical offices with different levels of technical capability, different infrastructure, and different access to data. The principle was tied to the UN context, where the platform had to work across many countries rather than for one uniform organization.

### The Platform Design Toolkit focused the team on the business model around the digital platform
[08:24](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=504s)
Mark distinguishes the platform business model from the underlying digital platform. The business model asks what users need, what kind of platform is being created, and how participants connect within an ecosystem. He describes platforms as bringing providers and consumers together, like a dating site, while reducing friction for participants. The Platform Design Toolkit supplied templates, processes, and material about user needs and network effects. Mark says a platform should allow people to do things its creators did not expect, since those uses can reveal new markets and new ways to meet user needs.

### Wardley Maps showed where the team should build and where it should wait
[13:31](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=811s)
Mark explains that a Wardley Map places capabilities along an evolutionary path from genesis to custom-built, product, and commodity or utility. New and unusual capabilities sit on the left. Familiar, pay-per-use capabilities such as cloud compute sit on the right. The map helped the UN team avoid spending time and money building infrastructure that cloud providers would soon offer as a service. Mark says the team waited for managed services instead of building its own Kubernetes stack and saved about a million by doing so. The map predicts movement toward the right, although it does not predict the exact timing.

### Different stages of evolution called for different teams and skills
[32:49](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=1969s)
Mark says teams can also be organized from a Wardley Map. Work on the right involves known services that people configure and connect. Work on the left involves new capabilities that require people who can write code, handle uncertainty, accept failure, and adapt quickly. He contrasts people who glue existing services together with people who develop new components. The map also exposed inertia between stages, such as objections from teams that were reluctant to move from owned data-center infrastructure to cloud services. Mark says the Wardley Mapping material includes ways to understand and address those forms of inertia.

### Data and statistical methods also move from unfamiliar work toward reusable services
[36:03](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=2163s)
Mark applies the same evolutionary model to data. A familiar, well-defined dataset can be consumed as a service, while new or poorly understood data needs custom work. He warns that one tool cannot handle data at every stage. The UN team preferred data that providers had already structured and defined, since this reduced the time spent trying to understand an unfamiliar dataset. Statistical methods remained custom-built within statistical offices because they required substantial rigor, but their standards and outputs could be reused by other parts of the platform.

### Privacy-preserving computation could let organizations share data without exposing raw records
[17:33](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=1053s)
The platform identified multi-party authentication, multi-party computation, and homomorphic encryption as capabilities that would matter over time. These methods allow calculations on encrypted data while keeping the source records hidden. Mark gives a height example: a group could calculate an average or identify the tallest person without revealing each person's height. He mentions OpenMined, Andrew Trask, PyTorch, and work from Cybernetica as examples of this area. He also describes efforts to use machine learning to create synthetic mobile-phone datasets that look real enough for analysis while protecting people's privacy.

### Streaming location data gave the platform material for public statistics and economic analysis
[47:02](https://www.youtube.com/watch?v=6oFCqGE_2-U&t=2822s)
The platform received around 40 million ship records per day and 600 million flight records per day. Flight data included identifiers, location, speed, and altitude. Ship data included vessel identity, position, heading, and speed. Apache NiFi collected, processed, routed, and stored the streams, while Kafka, HBase, Spark, GeoMesa, GeoServer, and Jupyter were part of the wider stack. NiFi also provided lineage, so users could trace where data came from and what operations had been applied. During COVID-19, flight data helped identify countries that might see more cases, while ship movements helped track economic activity and inform policy decisions.

## Notable quotes
- Mark Craddock: "The core team was five." (07:30)
- Mark Craddock: "You need to understand that you know Kubernetes and the MLOps type tools, they're going to move really quickly to the right." (24:48)
- Mark Craddock: "You could do the calculation, you can get the average height and you get the choice and the mean or whatever, but you would never know, you would never see the raw data because it would always be encrypted." (18:30)
- Mark Craddock: "The one thing Apache NiFi gives you is the lineage." (52:38)
- Mark Craddock: "The flight record data is 600 million records a day." (48:27)

## Tools & references mentioned
- United Nations Global Platform
- United Nations Statistics Division
- Sustainable Development Goals
- Platform Design Toolkit
- Wardley Maps
- Simon Wardley
- Amazon
- Microsoft
- Google
- Alibaba
- Kubernetes
- Docker
- Algorithmia
- OpenMined
- Andrew Trask
- PyTorch
- Apache NiFi
- Apache Kafka
- HBase
- Spark
- Jupyter
- GeoMesa
- GeoServer
- DSTL
- Forterra
- Synthetic data
- Data mesh architecture

## Who should watch
- You are designing a data or machine learning platform for users in several countries and need to account for cloud, security, and geopolitical constraints.
- Your team is deciding which infrastructure to build and which capabilities to consume from cloud providers.
- You work with sensitive data and want practical examples of lineage, encrypted computation, synthetic data, or large location-data streams.

## Editor's note

Mark Craddock says no single cloud met all user needs globally, so the UN platform ran across Amazon, Microsoft, Google, and Alibaba. ZenML lets teams configure a pipeline's stack so the same Python workflow can run on different infrastructure, including cloud services and Kubernetes. That addresses the operational split without rewriting the pipeline.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [How Data Platforms Affect ML & AI](https://mlopstalks.com/talks/how-data-platforms-affect-ml-ai) (Jake Watson, The Oakland Group, 39:12)
- [The Future of ML and Data Platforms](https://mlopstalks.com/talks/the-future-of-ml-and-data-platforms) (Michael Del Balso, Tecton, 55:17)
- [Aggressively Helpful Platform Teams](https://mlopstalks.com/talks/aggressively-helpful-platform-teams) (Stefan Krawczyk, Stitch Fix, 51:52)
- [The Future of Data Science Platforms is Accessibility](https://mlopstalks.com/talks/the-future-of-data-science-platforms-is-accessibility) (Skylar Payne, 52:17)
- [The intersection between DataOps and privacy](https://mlopstalks.com/talks/the-intersection-between-dataops-and-privacy) (Lars Albertsson, Scling, 33:26)