Igor Lushchyk advises people learning MLOps to understand small pieces of a system before adopting large frameworks.
2
Data and machine learning systems should be designed with production requirements in mind, even when the first version is exploratory.
3
Migrations work better when teams maintain feature parity, compare old and new systems, and keep communicating with data scientists.
Summary
Igor Lushchyk talks about the habits that have helped him avoid avoidable pain in data engineering and MLOps. He recommends learning systems from small components, looking beneath abstractions, and admitting when he does not know something. At Adyen, the MLOps team grew from an initiative into a dedicated team connected to data infrastructure and data scientists. Igor describes the problems caused by treating interconnected data systems as temporary proofs of concept. A homegrown scheduler had been adopted by users and supported by one person, which made later migration difficult. He argues that teams should think about production constraints early, use open source where possible, and choose tools that cover most of their needs without forcing them to build everything themselves. Migration requires communication, feature parity, testing, and a controlled switch from the old installation to the new one. He also discusses model monitoring, regulated deployment, context switching, and confusing product names.
Small components make systems easier to understand
Igor recommends avoiding large frameworks while learning how a system works. He uses web development as an example, contrasting a full framework such as Django with smaller frameworks where a learner builds pieces such as routing and authentication. Looking inside the components helps people understand how tools work, why they work that way, and how data flows through them. He also says this understanding matters in interviews. He asked candidates about the internal behavior of tools instead of focusing only on the tools they had used.
Igor taught Python programming for two years and found that preparing lectures forced him to explain familiar subjects clearly. He spent a full day preparing some one-hour lectures. Simple questions from students sometimes exposed areas where he did not have a correct answer. His response was to say that he did not know and then find the answer for a later session or in Slack. He treats this honesty as part of learning rather than as a failure to hide.
An MLOps team needs links to infrastructure and data scientists
At Adyen, the dedicated MLOps team had only recently been formally created. Before that, people worked on MLOps as an initiative within a larger data infrastructure group. Igor wanted the team close to infrastructure while also keeping frequent contact with data scientists, who are its internal clients. The team had five people, drawn from data infrastructure engineers and technically oriented data scientists. Igor says the group needs regular alignment sessions and an open communication loop so engineers can understand pain points and respond when released features fail.
Systems that start as proofs of concept can become hard to replace
Adyen had an in-house scheduling system that had been built around three years earlier, with some ideas taken from Airflow. It worked, but one person supported it. The original mistake was treating an interconnected system as a proof of concept. Machine learning systems connect infrastructure, CI/CD, artifact storage, and model training, so teams need to consider availability, redundancy, fault tolerance, and future load early. Once users build workflows and habits around a temporary system, replacing it becomes painful because their code and practices depend on it.
Production thinking should start before users adopt the system
Igor says data and machine learning teams should forget the idea of a proof of concept when building shared systems. A developer can use a laptop to test whether something works, but a system offered to users should cover the technical and end-user cases it needs to support. If people start creating features and models on a temporary platform, later migration has to account for the use cases they have already adopted. That makes the initial design more important than it might appear.
Use open source carefully and add only what the use case needs
Igor describes Adyen as an on-premises company, which limits some technology choices. The team used open source tools including Horton Data Platform, Spark, JupyterHub, Airflow, and MLflow. He found MLflow's user interface restrictive for some on-premises needs, especially around role-based access, experiment views, and custom functionality, although he still valued its API. His advice is to use an officially supported or community-supported open source tool that solves most of the problem, then add internal tools for the remaining needs. He also recommends checking the community's experience before choosing a tool.
Migration needs feature parity and a feedback loop
Igor describes a migration pattern in which the old installation continues running while the new installation is built alongside it. The process can last much longer than two weeks, so the team must keep asking data scientists and other users which features need to move and what the change means for them. Feature parity comes first. The team should compare results and data between the two installations, test the new one, and only then switch off the old system. He also says migration should improve the implementation instead of blindly lifting and shifting it.
For model monitoring, Igor's team uses dashboards built on existing infrastructure. Prometheus collects metrics and Grafana displays them. The monitoring covers server and hardware health as well as model behavior, such as inference performance and degradation relative to an older model. Grafana remains useful, although the interface becomes difficult to manage when many services, machines, data centers, and model versions are involved. Igor distinguishes automatic training from automatic deployment. Airflow can schedule or trigger training, while production deployment remains harder because financial institutions require compliance controls and separation of responsibilities.
Igor's practical warning is to avoid constant context switching. He gives an example of deleting data from HDFS with trash skipped and then failing to check what had been removed. He also warns teams to choose product names carefully. At Adyen, a component called a router was actually a server, while a gateway was actually a router. He says the company allows people to make mistakes and learn from them, although he personally made mistakes during migrations partly because he was too optimistic about how quickly work would improve.
