Software engineering practices give ML teams control over their work by making code reproducible, testable, reviewable, and easier to change.
2
ML projects need different kinds of tests at different stages, including unit tests for code logic, schema checks, functional tests, and statistical analysis of model behaviour.
3
Teams must budget for ongoing model maintenance because data distributions change, models need monitoring, and a small team cannot maintain an unlimited number of production models.
Summary
The discussion covers the book "Object Design Style Guide" and applies its software design ideas to machine learning projects. Laszlo Sragner and Tim Blazina argue that version control, testing, code review, refactoring, and clear abstractions prevent ML work from becoming difficult to reproduce or maintain. They distinguish tests that fit ordinary software from problems that need data analysis, such as a model's performance changing after a new data batch. Schema checks can stop a pipeline when upstream data changes, while feature engineering code can use functional tests and small datasets for fast iteration. The group also discusses BigQuery, feature stores, dbt, monitoring, model rollbacks, and the cost of maintaining production models. Tim describes using abstractions and dependency inversion around external services, while Laszlo argues that refactoring and testing let developers make changes with more confidence.
Software engineering practices made the ML team faster
Laszlo Sragner describes four years at a startup where the work began as ordinary software engineering and later moved into heavy machine learning. His team learned test-driven development and other practices, then broke them down into first principles and applied them to ML. After training a team of about five people, he says they became more productive and moved faster. Tim Blazina reached a similar conclusion from a data science background. Tests had saved his teams from mistakes, while projects without these practices often felt productive during experimentation and became difficult to maintain when they reached production.
Version control and testing give developers control over their work
Laszlo says that getting trapped in a messy project is a sign of a broken process. Without tests and version control, developers worry about losing notebook progress or forgetting how a processing step worked. He argues that code is produced for the company, so it needs to be maintained, reviewed, and recorded. In one previous company, the reviewer merged the code and became responsible for it. He connects this to Jeff Bezos's distinction between reversible and irreversible decisions: version-controlled and tested work makes more decisions easy to undo.
Experiments need a record even when their results are negative
Tim says data scientists often describe their work as experimentation and do not think of themselves as writing software. His view is that once someone writes code, uses version control, and commits it, they are writing software. Laszlo adds that teams should record the steps and reasoning behind experiments, including failed attempts. This preserves the chain of evidence and stops another person from repeating the same failed approach. The group rejects the idea that experimentation excuses poorly written or undocumented code.
Early ML projects can test correctness with small data
Laszlo does not recommend tracking every early result as a full experiment. At the start, teams should track the steps they take and maintain code quality through refactoring and testing. A small dataset can test whether the code is correct even though it cannot produce a good model. The pipeline can run quickly, which lets the team refactor and rerun it. As the project develops, the team adds more data until it can record experiments based on a statistically relevant solution.
Feature code should be testable before it reaches shared infrastructure
Tim describes a proof of concept for sales forecasting with promotions. His team had been writing ad hoc SQL against BigQuery and assembling notebooks or Python scripts. He tried to make the work modular so they could test that they were pulling the right data. The team also considered a feature store because useful features were difficult to share between projects. Laszlo places the feature store at the infrastructure layer. Feature-generating code should run against a CSV, pandas data, or BigQuery and be tested independently before the resulting features are published to a shared system.
Functional tests fit many ML transformations better than literal unit tests
For feature engineering, Laszlo describes recording the expected output and comparing the current code with that reference. When a change produces different values, the team can decide whether the change is correct and update the reference. Reimplementing a numerical feature inside a unit test can duplicate the same logic and provide little value. Tim still sees a place for unit tests around logic, mocked data, and edge cases. Laszlo agrees that errors, zeros, and negative numbers should be tested, while numerical model specifications often need a different approach.
Data changes require checks and analysis beyond ordinary code tests
Tim says dbt helped his team test transformations in BigQuery. Great Expectations can also test data, but he does not have a complete answer for verifying every stage of an ML pipeline. A data change may leave the pipeline running while producing strange results. Laszlo recommends codifying schema expectations so the pipeline stops when a column is added or removed. Teams still need to run the full pipeline on representative data and inspect nulls, empty values, distributions, and other signs that the data is no longer understood.
Production models need a maintenance budget and repeated review
Laszlo argues that a model in production includes preprocessing, modeling, and postprocessing. Schema and missing-value problems can be handled in the pipeline, but gradual distribution changes require analysis and domain knowledge. He rejects the assumption that a five-person team can maintain hundreds of models without allocating time for them. Each new model needs time over its lifecycle for regular drift checks and reviews. Tim adds that pipelines can rot when they are not run regularly, so a team may discover after six months that retraining no longer works.
Abstractions and refactoring make design decisions easier to change
Tim's main takeaway from "Object Design Style Guide" is to use abstractions around external services and dependency inversion from the SOLID principles. He separates service objects, which perform tasks or return information, from value objects or entities, which represent things. Laszlo frames dependency inversion as useful when it improves testability and decouples code from production systems, rather than as a rule to apply everywhere. Tim also accepts that early design decisions will be imperfect. Regular refactoring and tests make those mistakes manageable.
"The second you start writing code and using version control and committing it, you're writing software there whether you want to believe that or not."Tim Blazina10:25
Who should watch
You work on ML projects that start as notebooks or ad hoc scripts and become hard to reproduce when they reach production.
Your team is deciding which parts of feature engineering, data processing, and model development should have tests.
You maintain production models and need a practical boundary between software tests, data checks, monitoring, and statistical analysis.
