# Testing machine learning systems A pack of 8 sessions from the MLOps Community YouTube channel, in the order to watch them. 5h 59m of video. Page: https://mlopstalks.com/packs/testing-machine-learning-systems Your code tests pass and the new model scores better, but an important customer scenario has got worse. Testing an ML system means checking several different promises: that the code runs correctly, that the evaluation resembles future use, and that the model behaves acceptably in the situations you support. Start by taking an inventory of those promises, then define the model's operating conditions. Make the production code testable with small inputs before choosing representative evaluation data. Scenario-level comparisons expose regressions hidden by averages, and inspecting errors gives you new cases to preserve. Finish by testing the surrounding system under controlled failures. The result should be a set of specific expectations you can check again, rather than another reassuring score that nobody knows how to challenge. ## This pack is for you if - A higher-scoring model has regressed on a customer scenario you thought was covered. - Your unit tests pass, but nobody has defined the conditions in which the model must work. - Changing feature code requires a full training run before you can spot a basic mistake. ## The talks, in order ### 1. The ML Test Score Skylar Payne | 57:48 | MLOps Reading Group Video: https://www.youtube.com/watch?v=sKF2jtmrMMM Summary: https://mlopstalks.com/talks/the-ml-test-score.md Why first: Payne turns production readiness into an inventory of missing checks. His LinkedIn account treats the rubric as a discussion aid, not a mandatory scorecard. Start with the gaps and simple invariants before deciding which more demanding model tests your system needs. ### 2. ML Tests Svet Penkov, Efemarai | 40:24 | MLOps Coffee Sessions Video: https://www.youtube.com/watch?v=c59Wmg2-jpA Summary: https://mlopstalks.com/talks/ml-tests.md Why second: Penkov defines the missing promise behind many test suites: where the model is expected to work. Brightness, position and other allowed variations become properties to test together. This gives the inventory a model-behavior target instead of assuming that passing code checks proves the model is useful. ### 3. Code Quality in Data Science Laszlo Sragner, Hypergolic | 23:18 | MLOps Meetup Video: https://www.youtube.com/watch?v=vlG2PZeJ5ig Summary: https://mlopstalks.com/talks/code-quality-in-data-science.md Why here: Sragner makes those checks practical to run. The same business code should work locally and in production, with external services replaceable in tests. That keeps the test from exercising a convenient rewrite while the deployed implementation remains coupled to a database or unavailable hardware. ### 4. Impact of SWE in ML Projects Laszlo Sragner & Tim Blazina | 55:42 | MLOps Reading Group Video: https://www.youtube.com/watch?v=3iiB3zxU5jc Summary: https://mlopstalks.com/talks/impact-of-swe-in-ml-projects.md Why here: Small data can reveal broken code without producing a useful model. Sragner and Blazina distinguish that fast check from statistical evaluation, and favor reference outputs for transformations over copying the calculation into its test. Once the implementation is testable, choose assertions that can actually disagree with it. ### 5. Continuous Evaluation & Model Experimentation Danny Ma, Sydney Data Science | 1:00:46 | MLOps Meetup Video: https://www.youtube.com/watch?v=i4QNpM20QOc Summary: https://mlopstalks.com/talks/continuous-evaluation-model-experimentation.md Why here: Ma questions the evaluation data itself. A correct transformation can still feed a misleading experiment when target timing or a random split leaks future information. His time-based examples establish what the test set must represent before the next two talks examine performance within it. ### 6. Build a Culture of ML Testing and Model Quality Mohamed Elgendy, Kolena | 51:14 | MLOps Coffee Sessions Video: https://www.youtube.com/watch?v=2Ln-cVE1W9o Summary: https://mlopstalks.com/talks/build-a-culture-of-ml-testing-and-model-quality.md Why here: Elgendy's detector example shows why a representative test set still needs scenario-level pass criteria. The higher aggregate score hid an important regression. Versioned cases and their result history preserve the product knowledge that would disappear if you kept only the winning model's average. ### 7. Authoring Interactive, Shareable AI Evaluation Reports with Zeno Alex Cabrera, Carnegie Mellon University | 22:28 | LLMs in Production 2023 Video: https://www.youtube.com/watch?v=fS3taBVKu_c Summary: https://mlopstalks.com/talks/authoring-interactive-shareable-ai-evaluation-reports-with-zeno.md Why here: Cabrera demonstrates how to discover cases you have not yet named, using audio inputs, model outputs and intersecting metadata filters. His 2023 demonstration makes reusable evaluation slices concrete; turning discovered patterns into regression tests is the next step he proposes, rather than an automatic guarantee of the tool. ### 8. Build Reliable Systems with Chaos Engineering Benjamin Wilms, Steadybit | 46:58 | MLOps Podcast Video: https://www.youtube.com/watch?v=hUXVTLfkSUU Summary: https://mlopstalks.com/talks/build-reliable-systems-with-chaos-engineering.md Why last: The model can pass every scenario while the service fails around it. Wilms tests explicit expectations under delayed data, unavailable dependencies and other controlled faults. Adding those experiments in a safe environment completes the move from testing predictions to testing the system that delivers them. ## Editor's note Elgendy's scenario history and Sragner's insistence on testing the deployed implementation both depend on knowing which code and data produced a result. ZenML records pipeline steps, inputs, outputs and code versions for each run. That gives a team's model tests a traceable place in the workflow, while the team still defines the cases and pass criteria. Written by the MLOps Talks editors (the ZenML team), not by any of the speakers.