# How to Avoid Suffering in MLOps/Data Engineering Role

Igor Lushchyk, Adyen | MLOps Meetup | Episode 55 | 57:52
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=7L1W6Y1G-sI
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/how-to-avoid-suffering-in-mlops-data-engineering-role
Published: 2021-03-12
Tags: data-engineering, monitoring, open-source, platform-teams

## TL;DR
- Igor Lushchyk advises people learning MLOps to understand small pieces of a system before adopting large frameworks.
- Data and machine learning systems should be designed with production requirements in mind, even when the first version is exploratory.
- Migrations work better when teams maintain feature parity, compare old and new systems, and keep communicating with data scientists.

## Summary
Igor Lushchyk talks about the habits that have helped him avoid avoidable pain in data engineering and MLOps. He recommends learning systems from small components, looking beneath abstractions, and admitting when he does not know something. At Adyen, the MLOps team grew from an initiative into a dedicated team connected to data infrastructure and data scientists. Igor describes the problems caused by treating interconnected data systems as temporary proofs of concept. A homegrown scheduler had been adopted by users and supported by one person, which made later migration difficult. He argues that teams should think about production constraints early, use open source where possible, and choose tools that cover most of their needs without forcing them to build everything themselves. Migration requires communication, feature parity, testing, and a controlled switch from the old installation to the new one. He also discusses model monitoring, regulated deployment, context switching, and confusing product names.

## Key ideas
### Small components make systems easier to understand
[08:05](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=485s)
Igor recommends avoiding large frameworks while learning how a system works. He uses web development as an example, contrasting a full framework such as Django with smaller frameworks where a learner builds pieces such as routing and authentication. Looking inside the components helps people understand how tools work, why they work that way, and how data flows through them. He also says this understanding matters in interviews. He asked candidates about the internal behavior of tools instead of focusing only on the tools they had used.

### Teaching exposes gaps in your own understanding
[10:19](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=619s)
Igor taught Python programming for two years and found that preparing lectures forced him to explain familiar subjects clearly. He spent a full day preparing some one-hour lectures. Simple questions from students sometimes exposed areas where he did not have a correct answer. His response was to say that he did not know and then find the answer for a later session or in Slack. He treats this honesty as part of learning rather than as a failure to hide.

### An MLOps team needs links to infrastructure and data scientists
[13:23](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=803s)
At Adyen, the dedicated MLOps team had only recently been formally created. Before that, people worked on MLOps as an initiative within a larger data infrastructure group. Igor wanted the team close to infrastructure while also keeping frequent contact with data scientists, who are its internal clients. The team had five people, drawn from data infrastructure engineers and technically oriented data scientists. Igor says the group needs regular alignment sessions and an open communication loop so engineers can understand pain points and respond when released features fail.

### Systems that start as proofs of concept can become hard to replace
[16:49](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=1009s)
Adyen had an in-house scheduling system that had been built around three years earlier, with some ideas taken from Airflow. It worked, but one person supported it. The original mistake was treating an interconnected system as a proof of concept. Machine learning systems connect infrastructure, CI/CD, artifact storage, and model training, so teams need to consider availability, redundancy, fault tolerance, and future load early. Once users build workflows and habits around a temporary system, replacing it becomes painful because their code and practices depend on it.

### Production thinking should start before users adopt the system
[20:38](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=1238s)
Igor says data and machine learning teams should forget the idea of a proof of concept when building shared systems. A developer can use a laptop to test whether something works, but a system offered to users should cover the technical and end-user cases it needs to support. If people start creating features and models on a temporary platform, later migration has to account for the use cases they have already adopted. That makes the initial design more important than it might appear.

### Use open source carefully and add only what the use case needs
[24:54](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=1494s)
Igor describes Adyen as an on-premises company, which limits some technology choices. The team used open source tools including Horton Data Platform, Spark, JupyterHub, Airflow, and MLflow. He found MLflow's user interface restrictive for some on-premises needs, especially around role-based access, experiment views, and custom functionality, although he still valued its API. His advice is to use an officially supported or community-supported open source tool that solves most of the problem, then add internal tools for the remaining needs. He also recommends checking the community's experience before choosing a tool.

### Migration needs feature parity and a feedback loop
[41:02](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=2462s)
Igor describes a migration pattern in which the old installation continues running while the new installation is built alongside it. The process can last much longer than two weeks, so the team must keep asking data scientists and other users which features need to move and what the change means for them. Feature parity comes first. The team should compare results and data between the two installations, test the new one, and only then switch off the old system. He also says migration should improve the implementation instead of blindly lifting and shifting it.

### Monitoring and retraining have separate problems
[45:58](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=2758s)
For model monitoring, Igor's team uses dashboards built on existing infrastructure. Prometheus collects metrics and Grafana displays them. The monitoring covers server and hardware health as well as model behavior, such as inference performance and degradation relative to an older model. Grafana remains useful, although the interface becomes difficult to manage when many services, machines, data centers, and model versions are involved. Igor distinguishes automatic training from automatic deployment. Airflow can schedule or trigger training, while production deployment remains harder because financial institutions require compliance controls and separation of responsibilities.

### Focus reduces operational mistakes
[54:10](https://www.youtube.com/watch?v=7L1W6Y1G-sI&t=3250s)
Igor's practical warning is to avoid constant context switching. He gives an example of deleting data from HDFS with trash skipped and then failing to check what had been removed. He also warns teams to choose product names carefully. At Adyen, a component called a router was actually a server, while a gateway was actually a router. He says the company allows people to make mistakes and learn from them, although he personally made mistakes during migrations partly because he was too optimistic about how quickly work would improve.

## Notable quotes
- Igor Lushchyk: "Don't use huge frameworks when you don't understand how they do work. Try to understand everything from smaller pieces." (08:05)
- Igor Lushchyk: "If I don't know something it's better for me to say I don't know and to look to try to find what to explain next time." (12:16)
- Igor Lushchyk: "When you're working in data world, like in machine learning world, forget about POC." (21:03)
- Igor Lushchyk: "You need to have feature parity between two installations." (42:32)
- Igor Lushchyk: "Guys, don't do context switching." (54:10)

## Tools & references mentioned
- Adyen
- Django
- Horton Data Platform
- Spark
- JupyterHub
- Airflow
- MLflow
- Databricks
- Prometheus
- Grafana
- HDFS
- Weights & Biases
- Kafka Connect
- DataStax
- Slack

## Who should watch
- You are building an internal data or machine learning platform and need to decide how much to build yourself.
- Your team has a homegrown system that users already depend on, and you are planning a migration.
- You work with model training or serving in a regulated environment and need practical guidance on monitoring and deployment controls.

## Editor's note

Igor Lushchyk says treating an interconnected machine learning system as a proof of concept makes it painful to replace once users depend on it. ZenML lets teams keep pipeline code while choosing the orchestrator and other infrastructure through configuration, so the same workflow can move between environments without being rewritten for each one.

Written by the MLOps Talks editors (the ZenML team), not by the speaker.

## Related talks

- [Organisational Challenges of MLOps](https://mlopstalks.com/talks/organisational-challenges-of-mlops) (Adam Sroka, Origami Energy, 54:29)
- [MLOps Insights](https://mlopstalks.com/talks/mlops-insights) (David Aponte-Demetrios Brinkmann-Vishnu Rachakonda, 37:47)
- [MLOps in Practice: Common Challenges and Lessons Learned](https://mlopstalks.com/talks/mlops-in-practice-common-challenges-and-lessons-learned) (Marouen Hizaoui & Mo Basirati, Machine Learning Reply, 1:03:01)
- [Practical MLOps Part 2](https://mlopstalks.com/talks/practical-mlops-part-2) (Alfredo Deza, Author and Speaker, 1:01:38)
- [Most Underrated MLOps Topics](https://mlopstalks.com/talks/most-underrated-mlops-topics) (Marian Ignev, CloudStrap.io & SashiDo.io, 53:56)