Mercado Libre moved from separate machine learning projects toward a shared platform after different teams built their own cloud pipelines and services.
2
Fury gives developers a cloud-independent way to build and deploy microservices, with repositories, continuous integration, infrastructure, monitoring, logs, and metrics built in.
3
Fury Data Apps gives data scientists controlled data access, suitable machines for ETL and training, process versioning, GPUs, JupyterLab instances, and configurable REST API deployment for models.
Summary
Carlos de la Torre describes how Mercado Libre's machine learning work grew from separate fraud detection and recommendation projects into a platform effort. Different teams had been choosing their own cloud services, databases, queues, and pipelines. Fury helped standardise application development and deployment while hiding infrastructure details from developers. Fury Data Apps extends that platform for machine learning. It provides secure access to data sources, machines sized for ETL and model training, GPU access, versioning for ETL and training processes, and a way to create a scalable REST API from a trained model through configuration. Carlos also explains that the platform does not try to own every data system. Teams can keep services such as Spark clusters and expose them for use by others. The same approach applies to a feature catalog being developed by the fraud team, with representations of customers, payments, and other entities updated almost in real time.
Mercado Libre started with separate machine learning projects owned by individual teams
Carlos says Mercado Libre had machine learning teams working on production problems such as fraud detection and recommendation systems. Different teams followed different paths. Some considered IBM Watson, while others built their own pipelines in the cloud. These were specific solutions for separate problems, so the company accumulated repeated work and inconsistent approaches. Carlos and his colleagues had already developed their own processes and products for machine learning, including a way to wrap models in REST APIs. They began working with Mercado Libre about five years before the recording, and the platform effort had been under development for about two years.
Fury hides infrastructure details behind a common application framework
Fury is a framework for developing and deploying microservices. When a developer starts an application, Fury provides a repository with standard interfaces, Docker support, and automatic continuous integration processes. The platform abstracts infrastructure and works across cloud providers, so developers do not need to know where an application is deployed. It also provides monitoring, logs, and metrics. Carlos connects this abstraction to Mercado Libre's growth from a few hundred developers to roughly three thousand during the period he describes. The platform team handles the operational work while application developers use the resulting interface.
Fury Data Apps adds machines and data access for machine learning work
Fury Data Apps is an in-house extension of Fury for data science and machine learning. Its early pipeline includes an ETL task and a separate training step. A user can request an ETL machine through the interface and work inside a secure network environment with permissions to reach the required data sources. The platform also provides machines with enough CPU, memory, or GPU capacity for large data sets and model training. This addresses a limitation of ordinary microservice infrastructure, where the available machine sizes may not fit data processing workloads.
The platform versions data preparation and training together
Fury Data Apps versions ETL processes and training processes, linking the training process to the ETL process that produced its data. The training environment can include GPUs and other infrastructure intended for model training. The output is a model, and Carlos says users are not restricted to one machine learning technology as long as it fits the platform's requirements. The platform then automates creation of a REST API from the model. A data scientist can configure the deployment and receive infrastructure that automatically scales model serving.
Fury Data Apps uses existing data services instead of rebuilding them
Carlos says data is spread across Mercado Libre, with other teams already maintaining curated data and processing infrastructure. Fury Data Apps can trigger queries against those systems rather than processing everything inside a notebook or ETL machine. He gives Spark clusters as another example of infrastructure that can remain with the team that operates it. The platform team calls this external innovation. If another team has built a useful service and has the resources and budget to maintain it, Fury Data Apps should make that service available to users instead of taking ownership of every system.
The platform plans to connect to a feature catalog built by the fraud team
Carlos describes a team in the fraud area building a feature catalog with strong representations of entities such as payments and customers. These representations are intended for machine learning and are completed almost in real time. The fraud team is solving the problem because it needs the capability for its own work. Fury Data Apps aims to integrate with that service. Carlos is explicit that the platform team does not want to build everything itself, because each problem can require a dedicated team and its own budget.
"Fury first is called Fury because lots of nerdy guys and a bigger stuff, and you know Nick Fury is such a badass."Carlos de la Torre02:23
Who should watch
You are building an internal platform for data scientists who need governed access to data and larger machines than ordinary application services provide.
Your machine learning teams have separate pipelines, deployment methods, and cloud infrastructure, and you want to standardise the common parts.
You already have data or feature services owned by other teams and need a platform that can use them without taking over their operation.
