Ben Wilson's ML engineering roadmap starts with business planning and ends with deployment and evaluation, rather than stopping when a model reaches production.
2
Machine learning code should use small, testable functions, mocked data, and unit tests for transformations and decision logic.
3
Teams need shared definitions and lightweight governance for feature stores, while code reviews should inspect real predictions in addition to passing tests.
Summary
Machine Learning Engineering in Action is the book Ben Wilson discusses in this reading group. Wilson presents the book as a record of mistakes and lessons from building machine learning systems for more than a decade. He walks through an ML engineering roadmap that starts with planning, business goals, and measures of success. Scoping, research, staffing, experimentation, development, deployment, and evaluation follow. He argues for time-boxed experiments and a simple MVP before code hardening. The discussion then covers adapting Agile practices to ML work, especially showing results to business users early. Wilson explains how TDD for ML differs from writing tests around ordinary application logic. Small mocked inputs should test transformations and feature logic, while larger tests belong to integration stages. He warns against turning refactoring into a rigid framework. The group also discusses wireframes for large systems, feature-store governance, code complexity, and ML code review. Wilson's advice is practical and candid, especially about involving software engineers and testing production-like predictions.
Planning must begin with the business problem and its measure of success
Wilson says the first phase of the roadmap is planning. Before writing code or collecting data, the team should ask the business unit what it is trying to solve and how it will measure whether the solution works. Small teams may need a data scientist to take on product or project management duties, while larger companies may have dedicated project managers. The people who will work through later phases should join these conversations, and their questions should focus on the end goal rather than technology. Wilson calls planning the make-or-break point of a project because a production system that solves the wrong problem only succeeded by luck.
Scoping requires the right people and early discovery of hidden business rules
During scoping, Wilson recommends gathering requirements from the business, interviewing people who understand the data, researching prior work, and deciding what support the project needs. A team may need data engineers, front-end engineers, or back-end engineers who own the serving data. Their time should be agreed with their managers rather than assumed. Wilson also recommends talking to subject-matter experts who perform the current process. These conversations expose business rules that management may not mention during planning. He gives the example of finding several unusual rules before development rather than discovering them after the project is already underway.
Time-boxed experiments lead to a simpler MVP and prevent expensive overbuilding
Wilson says experiments should have a fixed period, such as a two- or three-week sprint. The team can compare algorithms, feature sets, and simpler alternatives, including plain if-else logic, without trying to produce polished code. This helps establish what the MVP needs to be and whether the data can support the proposed solution. He criticizes building a highly complex model and codebase before attempting production. Edge cases and new functionality then force more additions, and development slows as the code becomes harder to modify. His preferred sequence is a simple MVP, early results, and later improvements based on what the system actually needs.
Deployment includes controls for bad predictions and business evaluation
Wilson describes deployment as more than choosing batch, streaming, semi-real-time, or real-time serving. The system also needs monitoring, failover logic, checks for nonsensical predictions, and a way to turn off a customer-facing model quickly if necessary. Teams should consider feature-data problems and prediction bias. After deployment, evaluation asks whether the system is producing enough value to justify its cost. Wilson says ML teams should be able to explain to company leaders why money was spent and whether the project is recovering that expense. Evaluation is one of the most important phases after planning because technical delivery alone does not establish that the project worked.
Frequent demonstrations give business users a reason to support the system
When adapting Agile principles to ML work, Wilson places particular weight on getting feedback early and often. Teams should schedule demonstrations during development instead of waiting for a finished artifact. Business users can report bugs, request features, and point out when the system does not match how they work. Wilson recommends sharing examples of predictions and showing how they would appear in the consuming product, even when the model performs well on held-out data. He says this also has a political purpose: people who feel included develop a personal investment in the project and are more likely to use it and help improve it.
ML testing should isolate data logic with small mocked examples
Wilson does not describe pure test-driven development as a universal fit for ML work. He recommends testing the logic applied to data, similar to tests in data engineering. Transformations, conditional rules, and feature-vector handling should be tested with one or two hard-coded rows instead of passing a full training dataset through a unit test. Tests should use mocked data rather than real data when possible. Time-series transformations need more deliberate fixtures, such as generated seasonal patterns or a sine wave, because random values will not validate the intended behavior. Larger runs with many rows and columns are integration tests, not unit tests.
Refactoring should start with functions before introducing deeper abstractions
Wilson warns that an engineer can make a wall of text worse by replacing it with an elaborate processing framework. Generic classes, inheritance, configuration layers, and rigid abstractions can leave data scientists unable to maintain their own code. His first refactoring step is simpler: split a thousand-line script into functions, keep them in one file if that is sufficient, and call them in order. Each function can then be tested independently. Only after that should the team look for genuinely shared behavior and move it into utility packages. The goal is code that can be changed by the people responsible for the model, not a framework that hides the implementation.
Feature stores need shared definitions with room for teams to move
Wilson describes two bad extremes for feature-store governance. With no controls, different teams may create features with the same name but different meanings. He uses profit as an example because finance, marketing, and operations may calculate it differently. With total central control, every feature requires approval from an architecture group, which can block a project for months. His middle ground is a technical person who understands the business and can coordinate definitions across teams. Before registering a feature, the team should record a qualified name, its calculation logic, and example SQL, Python, Java, or pseudocode. That gives teams consistency without making every change wait for a large committee.
Code complexity can be tested by asking a newer teammate to explain it
Wilson's test for over-engineering is to remove comments and ask a reasonably new team member to explain the code. If the person cannot follow the naming, control flow, or abstractions, the implementation may be too clever or too complex. He also asks whether someone can trace a unit test through the code without spending more than about 15 minutes finding the relevant path. The test is not aimed at someone who joined three days ago. It is aimed at a teammate who is onboarded and already maintains the team's systems. Wilson says senior engineers can mistake familiarity for simplicity, so another person's ability to read the code is useful evidence.
"The book is a summation of everything that I've screwed up over a period of more than a decade of building machine learning solutions or statistical inference solutions for multiple different companies."00:55
Who should watch
You are starting an ML project and need a concrete sequence from business planning through evaluation.
Your data science team has production code that is hard to test, hand over, or change safely.
You are deciding how much governance to put around shared features without blocking every project.
