Good engineering practices are prerequisites for building robust machine learning systems because individual algorithms cannot handle every failure mode.
2
A catalog built from academic and industry literature contains 29 general engineering practices, later expanded to 45 with trustworthiness practices, and measures their adoption across more than 500 practitioner teams.
3
Practice adoption can be used to estimate effects such as agility, software quality, team effectiveness, and traceability, while also suggesting which improvements are worth the effort for a particular team.
Summary
Alex Serban presents research into engineering practices for software that contains machine learning components. His team reviewed academic and industry literature, organized the findings into a catalog, and surveyed more than 500 teams about adoption and effects. The catalog covers data, training, coding, deployment, teams, and governance. The survey found that adoption tends to rise with team size and experience, although teams with more than five years of experience adopted fewer practices than expected. Machine learning-specific practices were adopted more often than traditional software engineering practices. Training practices were common, while feature ownership, shadow deployment, and several deployment practices were less common. Serban also describes statistical models that connect practice adoption to agility, quality, team effectiveness, and traceability. Trustworthiness practices, especially around robustness and security, had the lowest adoption because they often lack clear technical solutions and tooling. The project aims to turn the catalog into assessment tools that give teams practical improvement suggestions.
Robust machine learning depends on the whole system
Alex Serban separates robustness into algorithmic and system-wide concerns. An algorithm may fail on noisy or adversarial inputs, while the surrounding system can reduce harm through redundancy, decision rules, and operational safeguards. He also connects robustness with fairness, privacy, and transparency, since these boundaries overlap in deployed machine learning. The practical reason to study engineering is that no algorithm is completely robust. Teams therefore need engineering practices before they can address broader quality attributes. This matters especially when models support high-stakes decisions or systems that require high uptime.
The practice catalog combines literature with practitioner evidence
The research process starts with academic papers, workshop articles, company blog posts, white papers, and conference videos. The team records the findings in an open reading list and turns them into a practice catalog. The initial catalog contained 29 practices and later grew to 45 after trustworthiness practices were added. Each entry follows a design-pattern style with an intent, motivation, applicability, description, category, and difficulty. The practices are grouped around data, training, coding, deployment, teams, and governance. The catalog also reports adoption by team experience, team size, data type, and organization type.
Practice adoption varies by team and by practice type
The survey asks teams about their background, their level of adoption for each practice, and the effects they observe. More than 500 teams provided complete answers from around the world. Technology companies had the highest adoption, non-technology companies followed, and research organizations had the lowest adoption, especially for deployment practices. Adoption generally increased with team size and experience, but teams with more than five years of experience adopted fewer practices than expected. Machine learning teams also adopted machine learning-specific practices more often than traditional software engineering practices. Adoption was largely independent of whether teams used tabular data, text, images, or video.
Training practices are common while deployment discipline lags
The most adopted practices relate to training. Examples include expressing the training objective as a metric that is easy to measure and understand, sharing that objective within the team, and writing reusable scripts for data cleaning and merging. Less adopted practices include assigning an owner to each feature and documenting its rationale, archiving unused features, and using shadow deployment. Serban explains that shadow deployment is held back by the difficulty of finding, adopting, or building suitable tooling. Teams using products such as SageMaker or H2O.ai are more likely to use the practice when it is already part of their toolchain.
Practice adoption predicts several engineering effects
The study examines agility, software quality, team effectiveness, and traceability. Agility includes the ability to experiment, deploy, and roll back quickly. Quality includes testing, static analysis, and maintainable code. Traceability means being able to connect production outcomes to model configuration. Statistical analysis found linear relationships between some practice groups and these effects, while classifiers found stronger non-linear relationships. The team used practice importance to suggest improvements. Examples include automating model deployment and running training experiments in parallel for agility, plus continuous integration and automated regression tests for software quality.
Trustworthiness practices are difficult because many lack direct tooling
The trustworthiness extension maps 14 new practices to requirements discussed by the European Commission, including transparency, privacy, data governance, technical robustness, and risk management. Examples include testing training data for social bias and checking for bias within subgroups, rather than only across broad groups. These practices had the lowest adoption, especially those related to robustness and security. Serban attributes this partly to the absence of direct technical solutions and usable tools. It is difficult to define what makes a data set fully robust against adversarial examples. Auditing by outside experts is one governance practice that can help when the concern is difficult to formalize.
A maturity assessment can turn survey answers into improvement advice
The team applies item response theory from psychometrics to estimate a machine learning team's engineering ability from its practice adoption. The method also estimates how difficult it is to move from partial to complete adoption for each practice. Serban compares individual interview subjects with the broader benchmark and uses practice importance together with adoption difficulty to suggest possible improvements. This gives teams a way to decide whether a difficult practice is worth pursuing for the effect they want. The project is also exploring assessment tools that can provide immediate feedback instead of only reporting a score.
The project aims to make research usable inside engineering workflows
Serban says the team wants practitioners to find and understand the material quickly, rather than relying only on a research paper. They publish the reading list, practice catalog, adoption reports, and related work online. Future directions include yearly reports, a community that contributes reading and open-source material, and tools that can check practices inside continuous integration pipelines. He is careful about the limits of automation. Some practices can be checked automatically, while others need interpretation, governance, or expert review.
