# Applying DevOps Practices in Data and ML Engineering Antoni Ivanov, VMWare | MLOps Meetup | Episode 111 | 1:04:43 Hosted by Ben Epstein Source: https://www.youtube.com/watch?v=TSh9KI8WAjI Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks. Page: https://mlopstalks.com/talks/applying-devops-practices-in-data-and-ml-engineering Published: 2022-10-26 Tags: data-engineering, deployment, orchestration, platform-teams ## TL;DR - Antoni Ivanov argues that data teams need self-service tools while operations teams retain control over infrastructure, security, and governance. - Versatile Data Kit provides an SDK and control service for creating, running, monitoring, deploying, and reverting data jobs in Kubernetes. - The demonstration builds ingestion and transformation jobs, then adds an anonymization plugin without changing the job's core data logic. ## Summary Antoni Ivanov describes a tension between operations teams, which manage infrastructure and reliability, and data teams, which need to deliver analytics and models quickly. He applies familiar DevOps practices to data engineering through Versatile Data Kit, an open-source framework from VMWare. The framework has an SDK for writing SQL and Python ingestion or transformation jobs, plus a control service for deploying and managing those jobs in Kubernetes. It automates much of the build, test, release, and deployment cycle, while allowing platform teams to add organization-wide plugins and policies. The hands-on session creates an API ingestion job that writes exchange-rate data to SQLite, builds a processing job, deploys it, and reviews job history and rollback. Antoni also demonstrates an anonymization plugin that changes ingested data without requiring changes to the data job itself. He is clear that VDK focuses on batch jobs and does not currently version data. ## Key ideas ### Data and operations teams have different goals that need a shared delivery model [02:39](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=159s) Antoni Ivanov separates the people who manage infrastructure from the people who build data applications. Operations teams provision virtual machines, containers, networks, databases, and Kubernetes environments. They care about keeping systems working, secure, and maintainable. Data practitioners bring business knowledge, join datasets, build models, and answer requests from marketing or executives. Their priority is speed because businesses make decisions quickly. Antoni compares this tension with the earlier relationship between software development and operations, where both groups depended on each other while measuring success differently. ### Versatile Data Kit gives data teams self-service without exposing infrastructure details [08:40](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=520s) Antoni presents Versatile Data Kit as a data engineering framework for creating data applications and jobs. It sits where ingestion, transformation, business data, and reporting meet. Data teams can use it to bring data from external systems or APIs into a database, run SQL or Python transformations, and support data-driven products. The framework hides infrastructure details from data users. Antoni gives Kafka configuration as an example: a data team should not need to know how messages must be sized or configured for ingestion. Platform teams can control those settings while data practitioners work through simpler interfaces. ### VDK has an SDK for job code and a control service for automation [14:30](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=870s) The SDK is a Python library for data engineers, analysts, and other data practitioners. At the time of the talk, it supports SQL and Python, with plugins available for other languages. Its focus is data ingestion and transformation, with data lineage and data quality capabilities also present. The control service manages the creation, deployment, and operation of jobs in Kubernetes. Jobs are versioned, monitored, and assigned different error targets for infrastructure failures and user or data errors. Antoni describes the SDK as the part focused on the data journey, while the control service automates the DevOps cycle around it. ### Organizations can extend the platform through shared plugins and policies [17:54](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=1074s) VDK lets an organization build a custom SDK from reusable pieces and add its own plugins. Platform teams can use this to apply governance, security, database, and scripting practices consistently across jobs. Antoni shows how plugins can run during build and test, perhaps to execute system tests, check security practices, or restrict execution privileges. The same SDK can be configured across jobs, giving teams a consistent experience while still allowing job-specific overrides. The platform is intended to remain open and extensible because data systems and database requirements change frequently. ### The demo builds a batch ingestion job from an API [21:13](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=1273s) In the practitioner session, Antoni uses a shared environment and asks participants to add a unique lowercase prefix to their job names. The example fetches data from an API and writes it to SQLite because the session did not set up a larger cloud database. VDK takes the ingestion object, creates the destination table, and populates its columns. The job extracts selected fields from the source data. Antoni runs it with the VDK command-line interface, then queries the database and shows exchange-rate records with dates and bid and ask prices. ### Transformation jobs can combine SQL and Python steps [40:12](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=2412s) The second hands-on example creates a processing job from several steps. It sets variables, creates an output table, and performs transformations over the exchange-rate data. Antoni says Python in this example is intended for ordinary processing rather than massive distributed computation. When more distributed processing is needed, VDK can help orchestrate systems such as Spark rather than replace them. The completed job produces aggregate values such as average and maximum bids for each month. Templates can hide repeated or complex logic behind reusable methods. ### Deployment automates the build, test, release, and provisioning cycle [46:17](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=2777s) To productionize a job, the user supplies configuration such as a schedule and runs the deployment command. VDK takes the source code through dependency installation, image creation, configured tests, release, and Kubernetes resource creation. The control service runs this process asynchronously. Its interface shows deployed versions, deployment history, and audit information. Antoni also explains that a previous version can be restored by retrieving its commit and using a revert command. This gives data teams a single command while platform teams control the shared deployment setup. ### VDK is aimed at batch data jobs and does not currently version data [27:48](https://www.youtube.com/watch?v=TSh9KI8WAjI&t=1668s) In response to Ben Epstein's questions, Antoni says VDK is currently focused on batch data jobs rather than streaming or sub-200-millisecond feature serving. It is used for batch machine learning predictions and can create training data or features. He says the framework supports monitoring, operation, and tracking around those jobs, but it does not currently version the data. He describes data versioning as something needed for comparing model results, and says it could be addressed through integration with another tool. The streaming use case remains future work. ## Notable quotes - Antoni Ivanov: "The goal is to make sure that everything works, otherwise clients are not happy and business suffers." (04:45) - Antoni Ivanov: "We think that they need self-service capabilities to speed up their time to deliver data applications." (11:20) - Antoni Ivanov: "The control service is meant to be multi-tenant so you can have multiple teams cooperating together." (17:30) - Antoni Ivanov: "VDK is currently focused on batch data jobs, not on streaming." (28:30) - Antoni Ivanov: "It does not version the data, which I know is important for ML." (30:41) ## Tools & references mentioned - Versatile Data Kit - VMWare - Kubernetes - Kafka - SQLite - Spark - MySQL - My Binder - GitHub ## Who should watch - You are building a data platform and want data teams to deploy jobs without managing every infrastructure detail themselves. - Your organization needs shared security, governance, or database practices that can be applied through platform-managed plugins. - You are evaluating a batch data engineering framework and want to see how ingestion, SQL or Python transformations, deployment, monitoring, and rollback fit together. ## Editor's note Antoni Ivanov shows how data teams can deploy and revert jobs without taking control of infrastructure away from operations teams. ZenML runs Python pipelines on a configured stack, so the same workflow can move from a laptop to Kubernetes or a cloud service without rewriting the pipeline code. Each run records its steps, inputs, outputs, and code version for later tracing. Written by the MLOps Talks editors (the ZenML team), not by the speaker. ## Related talks - [DataOps and Data Versioning in ML](https://mlopstalks.com/talks/dataops-and-data-versioning-in-ml) (Dmitry Petrov, Iterative.AI, 1:01:55) - [DataOps is a Software Engineering Challenge](https://mlopstalks.com/talks/dataops-is-a-software-engineering-challenge) (Micha Kunze, Maersk, 57:57) - [DevOps, Security, and Observability in ML](https://mlopstalks.com/talks/devops-security-and-observability-in-ml) (Luke Marsden, MLOps Consulting, 32:46) - [How A Manager Became a Believer in DevOps for Machine Learning](https://mlopstalks.com/talks/how-a-manager-became-a-believer-in-devops-for-machine-learning) (Keith Trnka, 98.6, 55:49) - [The Future of ML and Data Platforms](https://mlopstalks.com/talks/the-future-of-ml-and-data-platforms) (Michael Del Balso, Tecton, 55:17)