Hendrik Brackmann's first production model used R for training and F# to reproduce inference inside a C# backend, with the full training and deployment flow automated.
2
Event-driven design removed slow queries against operational databases, allowed features to be reused across models, and made precomputed feature lookup fast enough for real-time use cases.
3
A commercial feature store became worthwhile when Tide needed fresh, reusable features for real-time predictions, while a warehouse-based process was enough for less demanding freshness requirements.
Summary
Hendrik Brackmann describes how his approach to machine learning production systems changed across three jobs. His first credit-risk system trained models in R and manually transferred an XGBoost-style model into F# and C#. The process was automated, but feature testing was difficult because data and transformations crossed team boundaries. At his second job, Python prediction services simplified deployment, while SQL queries against a read replica made feature generation slow and tightly coupled to the backend schema. The team replaced this with event-driven design and local projections. At Tide, the same pattern supports risk models and invoice matching, with features reused across models and stored for quick retrieval. Hendrik defines real time as less than a second and describes a 500-millisecond transaction-monitoring deadline. He recommends a feature store when warehouse freshness is insufficient, while warning that it adds coupling, versioning work, and operational complexity.
The first system had to bridge R training and a C# production backend
Around 2014, Hendrik worked on risk assessment for companies with little or no existing history. Their operational data was ahead of the data warehouse, and production deployment was difficult because the backend was written in C# while model training was written in R. The team used F# type providers to connect scripts with the existing codebase. They extracted the weights from an XGBoost-style model, pickled the trees, and reimplemented inference in F#. The full process was automated: extract a dataset, run training, create the model artifact, and deploy a C# library. The wider company's contract was a C# interface that accepted a large object from the backend and returned model features and scores.
Organizational boundaries made feature changes and testing harder than model deployment
Hendrik says the main obstacles were organizational. The backend team owned the object passed to the data team, while the data team needed additional sources for its models. Adding a source meant getting it through a warehouse batch process, training with it, and then persuading the backend team to expose it in the production object. This separation made feature pipelines hard to test. The team could generate test data and compare model implementations, but feature calculations crossed database logic, application code, and returned data. Hendrik says they never fully solved feature testing in that architecture. He also notes that the training data was batch-generated, while the backend used a library for live inference and supplied a feature vector of roughly 160 values.
SQL feature calculation against operational databases created unacceptable coupling
At the second company, Python microservices made model deployment easier because the prediction code used the same language as the models. Feature acquisition was worse. SQL queries and stored procedures ran against a read replica of a backend database, and generating features took minutes. The same parameterized SQL queries did have one useful property: IDs and timestamps could be injected to generate historical training data and current inference features from related logic. Hendrik kept that idea while rejecting the database design. Queries against operational data made the backend schema part of the model contract, so backend changes could break model code. The team later moved to domain-driven design and communication through events.
Event-driven design lets the same transformations produce historical and current features
The replacement architecture used events and local projections. Historical events could be transformed to create training datasets, while current events updated projections used for inference. Because the features were calculated ahead of time, the final request became a database lookup and took less than 100 milliseconds. Hendrik explains that transformations turn raw inputs, often JSON documents, into numerical values for a classical machine learning model. They can extract values from credit reports, calculate differences between events, or vectorize text. The same transformation logic can support training and inference, which helps avoid a mismatch between the data used to train a model and the data supplied when it runs.
Teams with mixed skills work better than one person owning the whole MLOps stack
Hendrik does not expect one MLOps engineer to cover the complete system. He says it is difficult to find someone who can do everything, so he prefers teams that combine application engineers, data engineers, and data scientists. Their output is a production service that accepts an idea and returns a prediction. The team should control how data is obtained and whether it is good enough, because an external team can otherwise break the service by changing a pipeline. At Tide, the backend uses Java, while other companies used C#, Ruby, and Python. Hendrik prefers putting an API around a model instead of translating model code across languages. For high data volumes, he uses Spark or PySpark.
Tide reuses event-derived features across risk and invoice-matching models
Tide uses models for onboarding risk, fraud risk, credit risk, and invoice default risk. Another model matches invoices to incoming transactions so small businesses spend less time doing that work manually. Features such as invoicing patterns and risky transaction patterns can be reused across these models. Tide uses events from the backend to build features and historical training datasets. The team also uses the data warehouse where information does not need to be fresh, joining warehouse-based processing with event-driven processing. This is a larger version of the earlier event-driven design: features are shared across several models rather than being built mainly for one model.
Real-time features should be precomputed when a response deadline is hard
Hendrik generally calls anything under one second real time and anything above one second near real time. Transaction monitoring has a hard external deadline of 500 milliseconds, so the system cannot calculate every feature during the request. Instead, it precomputes features and updates them with information from the transaction itself. He prefers batch transformations when the use case allows them because they are often stateless and easier to reason about. Event-driven systems add state, ordering questions, and possible race conditions. The choice therefore depends on the use case and its freshness requirement. Batch is the default when it works; real-time processing is reserved for cases that need it.
A feature store earns its place when warehouse freshness is insufficient
Tide first built a feature store internally, briefly considered open-source options, and then chose Tecton. Hendrik says some open-source tools looked more like feature registries, where features had to be calculated before registration. Tide wanted a system that could help generate features for real-time serving and batch training data. The feature store reduced the effort required to create and deploy features, allowed data scientists to create some real-time features without waiting for engineers, and made monitoring easier because features passed through one system. Hendrik's practical threshold is freshness. A data warehouse with suitably fresh tables can be enough. A commercial solution becomes useful when the warehouse cannot meet the required freshness.
Feature stores add coupling and require deliberate version management
Hendrik is direct about the costs of a feature store. It adds another system and can tightly couple prediction services to a pipeline, so a pipeline failure can affect many downstream services. Feature versioning also needs active management. Changing a feature can change its distribution and make an existing model unsuitable. His approach is to create a new feature for each iteration rather than silently changing the old one. He also sees discovery and metadata as part of the value of a feature store. Teams should be able to find a feature, understand its definition, see its code, and identify its owner. He treats ownership as an organizational requirement that becomes harder as the company grows.
