# Machine Learning at Scale in Mercado Libre

Carlos de la Torre, Mercado Libre | MLOps Meetup | Episode 11 | 59:28
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=ypySVdT9U7Q
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/machine-learning-at-scale-in-mercado-libre
Published: 2020-05-15
Tags: build-vs-buy, model-serving, monitoring, platform-teams

## TL;DR
- Mercado Libre built Fury Data Apps as an extension of its Fury platform because teams needed shared environments for data access, ETL, training, and model serving.
- Fury Data Apps gives data scientists configurable machines for ETL and training, versioning, access to company data, model packaging, and automatically scaled REST APIs.
- Carlos argues that machine learning deployment still lacks standard solutions, so Mercado Libre is automating repeatable platform work while keeping deployment responsibility explicit.

## Summary
Carlos de la Torre explains how Mercado Libre extended its Fury platform to support machine-learning work across the company. Before Fury Data Apps, teams used different vendors, technologies, data sources, and pipelines. The new platform gives data scientists controlled access to data, machines for ETL and training, versioning, notebook environments, model packaging, and configurable REST APIs that scale automatically. It supports different Python-based technologies rather than forcing one machine-learning framework. Carlos also describes the limits of the platform. Some data remains spread across APIs, databases, and a curated data lake, while very large workloads require direct work with infrastructure teams. Model monitoring is being built separately, with input and output collection, statistical and business checks, daily reports, and future plans for larger-scale processing. He is candid that building everything in-house created a large backlog, and says existing open-source projects could have solved parts of the problem faster.

## Key ideas
### Mercado Libre moved from separate machine-learning projects toward shared infrastructure
[07:27](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=447s)
Mercado Libre had machine-learning teams working on recommendations, delivery estimates, moderation, fraud detection, and other problems. Each team had developed its own approach. Some bought services such as IBM Watson, while others built pipelines with Python. After Carlos's previous company joined Mercado Libre, the teams remained focused on their individual products, while a cross-area group began working with infrastructure, cloud, and business intelligence teams on shared machine-learning infrastructure. The group had grown to 15 people after a year and a half. Carlos describes the goal as making Mercado Libre the best place to work on machine learning in Latin America.

### Fury abstracts cloud infrastructure so application teams do not need their own DevOps work
[23:13](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=1393s)
Fury is Mercado Libre's platform for developing and deploying microservices. Developers start from a repository with standard interfaces and Docker support, then use automatic continuous integration and platform-managed infrastructure. The platform is cloud independent, provides monitoring, logs, and metrics, and hides deployment details from application developers. Carlos says this approach let the company grow from roughly 300 or 400 developers to about 3,000. Fury Data Apps extends the same idea to data science, where workloads need different amounts of CPU, memory, and GPU capacity than ordinary microservices.

### Fury Data Apps gives data scientists a repeatable path from data preparation to a serving API
[23:58](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=1438s)
Fury Data Apps lets a user create an ETL machine with access to approved data sources, then create a training machine with the resources needed for model development. ETL and training processes can be versioned, and the output of training is a model. Users can work with different Python-based technologies instead of being limited to TensorFlow or scikit-learn. The platform can package a model as a REST API and deploy infrastructure that scales automatically. It also provides configurable Jupyter environments. Carlos describes this as the current core of FDA: access to data, places to process and train, and a simple way to deploy a model.

### The platform relies on other teams for data systems instead of trying to own every layer
[25:34](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=1534s)
Data at Mercado Libre is spread across APIs, databases, and a curated data lake. The data lake is useful when one-day-old data is sufficient, but real-time use cases need direct access to APIs and other services. Carlos says FDA should use infrastructure built by other teams when those teams already operate it. He gives the example of a feature service that exposes a representation for a user. FDA can call that service during ETL, training, or serving if network access and authorization are configured. This lets an independent feature system or monitoring system integrate through clear interfaces without being forced into the same user interface.

### Large workloads expose the limits of simple platform abstractions
[27:47](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=1667s)
Carlos gives examples of training datasets with hundreds of gigabytes, billions of rows, thousands of features, and datasets reaching 1.5 terabytes. These workloads make ordinary microservice infrastructure inadequate. The monitoring project also faces this issue. Its current system collects model inputs and outputs, stores them in S3, and runs checks daily, but some models produce enough data that a single machine cannot process it. Carlos expects the monitoring system to need a Dask or Spark cluster, or a similar distributed processing system, before it can handle the largest models.

### FDA automates preparation and serving while leaving deployment responsibility explicit
[44:25](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=2665s)
Automation is a core feature of Fury Data Apps. Carlos says ETL and training should be automated, and the team is building batch inference so a model can produce outputs for a known set of inputs. Building a model API and scaling its serving infrastructure are also largely automated. The team does not automate the final deployment decision or the deployment itself. Mercado Libre treats deployment as a business and operational responsibility, so someone must be clearly accountable when a model goes live. Carlos says offline metrics alone do not make a model safe to deploy, and deployment strategies still need more work.

### Monitoring starts with model-specific checks and is being designed for reuse
[50:35](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=3035s)
The monitoring service collects model inputs and outputs through an API or through FDA configuration. Carlos and the user define checks for each model, including expected value ranges, input and output distributions, business rules, and suspicious repeated sequences. The current system runs these checks once a day and produces reports showing which checks passed or failed. The longer-term design would give users reusable base checks for common model types, while allowing data scientists to add checks for their own models. The team also needs to connect monitoring to Mercado Libre's alerting system and scale processing beyond the current MVP.

### Building the whole platform in-house created flexibility and a large backlog
[38:18](https://www.youtube.com/watch?v=ypySVdT9U7Q&t=2298s)
Carlos says FDA uses many open-source components, including Terraform, but the team started by extending building blocks that already existed in Fury. Because FDA is tied to Mercado Libre's internal platform and business logic, it cannot be released as a complete open-source product. He is direct about the trade-off: if he started again, he would use existing products to solve parts of the machine-learning platform problem. The team is considering open-sourcing smaller libraries, such as a component that wraps a model in a REST API. Carlos also prefers contributing engineers to existing open-source projects when the company can support their maintenance.

## Notable quotes
- Carlos de la Torre: "We want professionals in Latin America deciding to work with us if they want to do machine learning, real machine learning in production." (13:33)
- Carlos de la Torre: "We don't have standards yet, there's no the way to do things." (16:04)
- Carlos de la Torre: "You can do whatever technology to your data science as long as it is Python." (24:40)
- Carlos de la Torre: "We don't want right now to automate it because this is a big business to keep alive, so we can't figure it somebody if anything." (45:32)
- Carlos de la Torre: "Open source is a double-edged sword." (41:56)

## Tools & references mentioned
- Mercado Libre
- Fury
- Fury Data Apps
- IBM Watson
- Terraform
- Hive
- Presto
- Spark
- Amazon Kinesis
- S3
- Jupyter

## Who should watch
- You are building an internal machine-learning platform and need to separate shared infrastructure from team-owned services.
- Your teams use different frameworks and data systems, and you want a platform that supports them without forcing one model format.
- You need a realistic account of where automation helps and where model deployment and monitoring still require human ownership.

## Editor's note

Carlos de la Torre says Mercado Libre built Fury Data Apps because teams had separate approaches to data, training, and deployment. ZenML lets teams write ML workflows as Python pipelines and run the same code on different infrastructure through a configured stack. Each run records its steps, inputs, outputs, and code version, so the resulting model can be traced back to its source.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Fury Platform and Fury Data Apps at Mercado Libre](https://mlopstalks.com/talks/fury-platform-and-fury-data-apps-at-mercado-libre) (Carlos de la Torre, Mercado Libre, 10:30)
- [ML Platform Tradeoffs and Wondering Why to Use Them](https://mlopstalks.com/talks/ml-platform-tradeoffs-and-wondering-why-to-use-them) (Javier Mansilla, Mercado Libre, 53:57)
- [Scaling ML Capabilities in Large Organizations](https://mlopstalks.com/talks/scaling-ml-capabilities-in-large-organizations) (Bertjan Broeksema & Axel Goblet, BigData Republic, 1:02:47)
- [Machine Learning at Reasonable Scale](https://mlopstalks.com/talks/machine-learning-at-reasonable-scale) (Jacopo Tagliabue, Coveo, 1:04:32)
- [ML Scalability Challenges](https://mlopstalks.com/talks/ml-scalability-challenges) (Waleed Kadous, Anyscale, 1:00:03)