Hannes Hapke says data preprocessing must receive the same software engineering care as model training, especially when working with terabytes of data.
2
Digits connects the preprocessing graph to the trained model so changes to preprocessing trigger retraining and redeployment, avoiding training-serving skew.
3
Digits automates its pipeline through model registration and code review, with one human check before deployment so a data scientist can judge model quality and the cost of backfilling data.
Summary
Hannes Hapke explains how Digits uses machine learning to process accounting data for small businesses and give accountants a more current view of their books. He focuses on the work around the model: clean and queryable data, parallel processing, preprocessing, testing, model evaluation, deployment, and feedback from users. Digits generates a preprocessing graph during training and connects it to the model in production. This keeps training and serving transformations aligned. Their pipeline runs through continuous integration, produces reports and comparisons with the previous model, registers the model, and sends a Slack notification. A data scientist reviews the results before a configuration change triggers deployment. Hapke also describes why similarity learning creates a difficult scaling problem. Each transaction needs an embedding and nearest-neighbor comparisons, so ingestion, embedding generation, and neighbor search have to be heavily parallelized and optimized. He argues that designers, backend engineers, data engineers, and ML engineers need to work together because product design affects the quality of the data and the usefulness of the model.
A technical book should teach the workflow behind the tools
Hannes Hapke explains that Building Machine Learning Pipelines began as an attempt to give companies a concise, end-to-end view of machine learning operations. The workflow runs from data ingestion and validation through model training, validation, and serving. He says the missing piece was often the glue code connecting separate tools. The specific tools and pinned versions will age, so the lasting lesson is the mental model: coordinate pipelines with an orchestrator, track work in a metadata store, and understand why that infrastructure exists. The book later updated its example code to TFX 1.0, although the printed copy could not change.
Parallel data processing matters as much as model training
Hannes says Apache Beam helped him understand how machine learning pipelines process large data sets. Each TFX component is based on Apache Beam, and custom components need to process data in a way that can be parallelized. Digits uses Dataflow for this work. He argues that the difficult engineering work is often around ingestion and preprocessing, since training may require repeated passes over terabytes of data. Digits generates a preprocessing graph and attaches it to the trained model. When the model reaches production, raw input passes through the same graph before prediction. Changes made during training then carry through to deployment.
A preprocessing graph prevents training-serving skew
Hannes gives feature normalization as an example. TensorFlow Transform makes one pass to calculate a feature's mean and standard deviation, then another pass to normalize the values. The resulting computation graph records how raw values become normalized values. Digits sends that graph with the model, so production inputs receive the same transformation used during training. In an earlier job, Hannes changed embedding generation without coordinating with another data scientist, and the deployed model produced bad results because preprocessing no longer matched the model. At Digits, a preprocessing change triggers retraining and redeployment automatically.
Digits keeps model deployment behind code review and one human check
Digits starts its pipeline from data in BigQuery, then generates statistics and a schema with TensorFlow Extended. Transform code is written in Python and can have unit tests that compare raw inputs with expected outputs. The team tests parts of the model architecture and runs quantitative and qualitative data sets against each model. Reports compare the new model with the last deployed one. The model enters a registry and sends a Slack message. A data scientist reviews the reports, decides whether the result is expected and useful, then commits a configuration change. Continuous integration handles the deployment. Hannes says people cannot deploy models manually.
Digits does similarity learning rather than only ordinary classification. When a new model is released, the team has to generate an embedding for every record and compare each embedding with the others. Hannes says that approach does not scale for a fast-growing startup. Digits improved the backfill process by parallelizing ingestion and embedding generation, optimizing nearest-neighbor searches for transactions, and improving post-processing. Once the process became automated, registering a model could trigger the system to infer settings, recompute nearest neighbors, and handle the backfill without a manual sequence of steps.
Digits uses machine learning to make accounting information more current
Hannes describes Digits as a company building finance tools for small businesses. The system ingests books from banks and accounting platforms, processes unstructured data, extracts information, and suggests annotations, vendors, categories, and similar transactions. Its goal is to give accountants and business owners a more current view than a monthly review can provide. Digits Boost watches accountants' work and can flag an unexpected change, such as a transaction categorized as PayPal after similar transactions were assigned to Venmo. Hannes calls accounting a useful machine learning setting because classifications can be subjective and users can provide feedback.
Product design changes the data a machine learning system receives
Hannes says a model can make a product more useful, but only if the product guides users toward the intended behavior. Digits works with designers, backend engineers, frontend engineers, and data engineers in feature-focused work streams. The team discusses how a model changes the design and how the design changes the model. In past jobs, a model was misunderstood by users, who clicked the wrong controls and generated bad data. At Digits, the product includes explanations for recommendations and gives users a way to report problems. Those actions become feedback for the machine learning system.
Scaling continues after the first model is trained
Hannes argues that the hard machine learning work continues after the first model is trained. Digits ingests millions of transactions and new transactions each day, so the deployed model and its surrounding pipelines must scale together. The team uses autoscaling, but still has to find the right throughput across ingestion, embedding generation, neighbor search, and model use. He describes this as a systems problem. A trained model alone does not make a machine learning system. The system also needs a feedback loop, and users need a way to tell the team when a recommendation is wrong.
"The pre-processing of the data is something that we're not just starting to emphasize in industry applications, it's also something I'm starting to see in the actual education of ML engineers now."Hannes Hapke19:59
Who should watch
You are building a machine learning pipeline and need a concrete example of validation, preprocessing, evaluation, registry, review, and deployment.
Your similarity or embedding system is becoming expensive to backfill, and you need ideas for parallel ingestion and nearest-neighbor search.
Your product team is treating machine learning as a separate engineering function, while user actions and interface design affect the quality of your training data.
