Testing in machine learning covers code, data, models, training, and serving, so teams should begin with simple checks that build confidence without overengineering.
2
Production ML systems need versioned metadata, automated validation, clear environments, and a rollback process that fits the use case.
3
Good MLOps tooling makes the correct path easy for data scientists, while community processes need to preserve the MLOps Community's helpful culture as it grows.
Summary
This conversation covers practical testing and production work in machine learning. David Aponte explains that ML systems need more than unit tests. Teams may test data quality, run a short training job, check serving infrastructure, and use end-to-end tests. He recommends starting simply and accepting trade-offs between test coverage, monitoring, and the uncertain future of research projects. Vishnu Rachakonda argues that tests should be understandable to people who did not write them and points to Martin Fowler's continuous delivery guidance for ML. The group then discusses new jobs, foundation models, the data stack, and faster access to baseline models. David describes onboarding production models at Microsoft with environments, pull requests, CI/CD, Azure Machine Learning, and Azure DevOps. They also discuss metadata for rollbacks, manual intervention for an R&D system, and new MLOps Community projects such as tool comparison pages and system design reviews.
David Aponte says tests are easy to postpone because they are difficult and less exciting than other work. ML adds several testing areas beyond ordinary software unit tests. A team may need data quality checks, an integration test that trains a model for a few epochs, a smoke test that checks whether a serving web service returns a 200 response, and an end-to-end test. Data quality tests can depend on a stored sample of production data, which becomes difficult to maintain when the data changes quickly.
Simple tests are a better starting point than full coverage
David Aponte frames testing as a way to gain confidence that the system works as expected. He recommends beginning with basic checks, such as whether training runs correctly, functions behave as expected, and data types are correct. A separate testing environment can provide a place to try changes before merging them into production. He questions whether 100 percent coverage is worth the time for every ML project, since monitoring and alerting may provide more value in some cases.
ML engineers need testing explained in accessible terms
Vishnu Rachakonda says many ML engineers enter industry from data science, mathematics, or research backgrounds and may not have written tests before. He argues that tests should be simple enough for someone else to understand. He recommends Martin Fowler's article on continuous delivery for machine learning, which discusses tests for data, models, and code, alongside unit, integration, and end-to-end tests. David adds that tests also protect the author during development and document what a function is meant to do.
Foundation models may make baseline performance faster to obtain
Vishnu Rachakonda expects teams to get useful baselines faster as large pre-trained models become easier to access through APIs. He gives OpenAI models such as Codex and GPT-3 as examples, along with cloud providers exposing models through API calls. His preferred workflow is to start with a model known to perform well, then decide whether deeper model design work is necessary. He also expects new data-stack tools to make data more visible and accessible across businesses.
Production onboarding needs environments, ownership, and documentation
David Aponte describes his work at Microsoft, where he is responsible for onboarding production ML projects onto an architecture with CI/CD, pull requests, separate development, pre-production, and production environments, and documentation. He is also designing an on-call process for growing numbers of models. Azure Machine Learning provides ML pipelines, while Azure DevOps handles CI/CD for testing and deployment. His goal is to let data scientists move quickly without having to manage infrastructure reliability themselves.
Automation can make the correct release path hard to avoid
David Aponte gives a release example in which automated tests run whenever code changes are made to a web service. A pre-production branch can receive changes first, while a production branch receives them only after validation. Pull requests can require review and information that helps future engineers reproduce a problem or recover the previous configuration. These checks happen through source control and CI/CD rather than relying on someone to remember each manual step.
Rollback decisions depend on versioned metadata and the use case
David Aponte recommends tracking model versions, dataset versions, framework versions, environments, and other pipeline assets in registries with metadata about their origins. Vishnu Rachakonda describes a model release system that chose manual rollbacks for an R&D setting. A failed smoke test created an alert with the relevant Git and data hashes, then an ML engineer inspected the situation. He says processes should clarify responsibility without becoming a burden, and the right level of automation depends on the system.
Community growth requires deliberate attention to its culture
Demetrios Brinkmann says the MLOps Community has grown from roughly 600 Slack members a year earlier to more than 6,000. He is proud that members are polite, helpful, and not pretentious, and worries that growth could weaken that atmosphere. Vishnu Rachakonda is expanding the volunteer team to support the newsletter, blog, podcast, reading group, and other activities. Demetrios is also using the Campfire Slack onboarding flow to learn what members work on and offer short conversations with them.
