# Recommender System: Why They Update Models 100 Times a Day

Gleb Abroskin, FunCorp | MLOps Coffee Sessions | Episode 123 | 49:01
Hosted by Jake Noble

Source: https://www.youtube.com/watch?v=mgZTm1ZG04k
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/recommender-system-why-they-update-models-100-times-a-day
Published: 2022-09-16
Tags: feature-engineering, feature-stores, monitoring, orchestration, recommender-systems

## TL;DR
- FunCorp gives data scientists broad control over model and feature deployment through a thin, modular platform rather than a full feature store.
- The iFunny recommender system retrains and deploys about 20 models every hour because meme content changes quickly.
- Business metrics, especially retention and user actions, matter more for monitoring than technical metrics, but slow feedback makes experiments difficult.

## Summary
Gleb Abroskin describes the recommender system behind FunCorp's iFunny app, where user-generated meme content changes quickly and models need frequent updates. A small engineering group supports a larger data science team by giving researchers control over Airflow pipelines, Kubernetes jobs, features, and model deployment. FunCorp chose a thin, modular feature-store approach instead of imposing one unified platform. Engineers provide Spark interfaces, operators, storage tools, and an ONNX-based boundary for model serving. Client logging flows through a Java service, Kafka, and ClickHouse, supporting online events, training data, and experiment analysis. Gleb says the team can take a new model with new features from initial work to production in about a month, then retrain and deploy models hourly. The conversation is honest about the costs. Monitoring is still incomplete, experiments can fail because of missing or bad logs, and a successful hypothesis may still be rejected when it does not improve business results.

## Key ideas
### A small team gives data scientists broad control over production work
[05:35](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=335s)
Gleb says FunCorp has three software engineers, six data scientists, one partly allocated DevOps engineer, and business analytics support. The ratio works because data scientists have almost complete control over features and can deploy models without waiting for a separate engineering team. Airflow runs in Kubernetes, and the data science team can deploy jobs on whatever schedule it needs. Gleb attributes this arrangement to trust backed by heavy monitoring, confidence in the databases, and shared knowledge about deployment and scaling. He does not call the system sophisticated, but he credits the team with producing good results despite its size.

### FunCorp chose a thin feature platform because its requirements were still changing
[08:01](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=481s)
Data scientists create features through ClickHouse queries that write into MongoDB, while engineers mainly use Spark and a set of interfaces that make pipeline code faster to write. Gleb says the team rejected a single platform for every feature because it could not gather stable requirements on the first attempt. Instead, engineers built small pieces of code that provide individual features. The current approach includes copied pipeline code and separate tools for batch and streaming work. The plan is to abstract those pieces behind an API after the team understands the recurring requirements.

### The feature system grows from concrete needs instead of starting as a full product
[10:59](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=659s)
Gleb expects the platform to become more opinionated later. His proposed direction is to package each model with the features it needs, make those features available in the online store, and let the inference service load them dynamically without custom code. He agrees that a full feature store can be excessive for a team of five. The immediate work is more monitoring, including additional Grafana dashboards, along with Airflow operators, Spark tooling, and CI/CD pipelines that shorten deployment work. This reflects a deliberate decision to add abstraction only when the current system creates enough pressure.

### Business metrics come before technical monitoring
[18:56](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=1136s)
When Jake Noble asks where a new recommender system should invest monitoring effort, Gleb puts business metrics first. He names click-through rates, action rates, and retention because they provide direct feedback from users. Technical monitoring follows after the team knows that business measurements are working. Retention is slow, with important measures arriving after 30 days, so the team also uses daily, weekly, and some hourly metrics to check experiments sooner. The discussion makes the tradeoff clear. Fast metrics are useful for iteration, while long-term retention is closer to the business goal but arrives too late for quick experiment decisions.

### Client logging can quietly damage features, labels, and experiments
[26:02](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=1562s)
FunCorp collects user events through a Java service, sends them through Kafka, and writes them to ClickHouse on an hourly basis. The events include actions such as likes and opening the comment section. Gleb describes missing Android logs, absent logs from an exploration feed, and offline clients sending old events with timestamps that no longer make sense. These failures can degrade a recommender system slowly, making the cause hard to see. Jake adds that front-end changes to logging or experiment systems can make back-end metrics fall without an obvious connection. FunCorp has found bugs in these pipelines while observing A/B tests.

### The serving boundary lets a data scientist produce one portable model artifact
[36:24](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=2184s)
FunCorp uses an inference service written in Kotlin with Spring Boot. Models are stored or used in ONNX format, and the data science team checks that the ONNX model produces the same results as the original Python model. Gleb says this abstraction removes the need to manage the model's runtime environment inside the serving system. A researcher produces the artifact and sends it to the service through an HTTP request. The service also uses Memcached and in-process Java hash maps for several caching layers, along with other services, caches, and HTTP endpoints around the user-facing path.

### Meme recommendations require hourly retraining and deployment
[38:33](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=2313s)
Gleb explains that one model artifact is never enough for FunCorp because the meme domain changes quickly. The team deploys around 20 models every hour, retrained on data from the previous hour or two. Each model needs a continuous Airflow pipeline. The data scientist's deliverable is a directed graph with documentation, models in S3, and features in MongoDB. Gleb describes a best-case path of about two weeks for new feature engineering, database and service integration, and deployment work, while the model development itself can take about a month. Bugs, missing data, and repeated A/B tests still add delays.

### A successful experiment can still be rejected when it has no business value
[44:24](https://www.youtube.com/watch?v=mgZTm1ZG04k&t=2664s)
FunCorp's A/B testing setup uses multiple ClickHouse instances and many manually reviewed dashboards. The team slices results by content categories and by how long users have been on the platform. Gleb gives an example where nearest-neighbor search reduced dislikes by removing disliked content from recommendations. The test proved the hypothesis, but the team did not launch it to all users because the result did not justify the business cost. He says experiments often examine changes in user behavior before the team decides what to do next. Fast model iteration matters because a correct idea can still fail the launch decision.

## Notable quotes
- Jake Noble: "The name of the game here is, how long does it take you to try your idea." (03:53)
- Gleb Abroskin: "We decided against the single platform for all the features and to end with this approach." (10:00)
- Gleb Abroskin: "The most important part would be the business metrics like click-through rates, action rates and all the other stuff which allows to receive direct feedback from users." (19:13)
- Gleb Abroskin: "Every hour we deploy around 20 models which were retrained on the data from the previous hour or two hours ago." (38:33)
- Gleb Abroskin: "Sometimes even a proven hypothesis won't be enough to roll out the tests on 100% of users." (45:45)

## Tools & references mentioned
- FunCorp
- iFunny
- Airflow
- Kubernetes
- MongoDB
- ClickHouse
- Kafka
- Spark
- Grafana
- ONNX
- Memcached
- S3
- Spring Boot
- Spotify
- YouTube
- DoorDash
- Tecton
- Jimmy Nilsson
- Applying Domain-Driven Design And Patterns: With Examples in C# and .NET

## Who should watch
- You are building a recommender system and need to connect client events, feature pipelines, model serving, and A/B testing into one operating process.
- Your data science team is blocked by model deployment work and you are weighing a thin collection of tools against a full feature platform.
- You need a realistic account of how missing logs, delayed retention metrics, and failed experiments affect recommendation work in production.

## Editor's note

Gleb Abroskin says FunCorp deploys around 20 models every hour because meme content changes quickly, with each model needing a continuous Airflow pipeline. ZenML records each pipeline run's steps, inputs, outputs, and code version, so the team can trace a model or artifact to the data and code that produced it.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- ["Real-Time" ML: Features and Inference](https://mlopstalks.com/talks/real-time-ml-features-and-inference) (Sasha Ovsankin & Rupesh Gupta, LinkedIn, 51:55)
- [Product Enrichment and Recommender Systems](https://mlopstalks.com/talks/product-enrichment-and-recommender-systems) (Marc Lindner & Amr Mashlah, eezylife Inc., 56:12)
- [From Robotics to Recommender Systems](https://mlopstalks.com/talks/from-robotics-to-recommender-systems) (Miguel Fierro, Microsoft, 55:54)
- [LinkedIn Job Recommendations](https://mlopstalks.com/talks/linkedin-job-recommendations) (Alexandre Patry, LinkedIn, 51:41)
- [RecSys at Spotify](https://mlopstalks.com/talks/recsys-at-spotify) (Sanket Gupta, Spotify, 50:25)