Machine learning experiments include changing hyperparameters, algorithms, feature pipelines, input variables, and target variables.
2
Generating the target variable is often the hardest part of a supervised learning project, and its timing must match the business action the model supports.
3
Training, validation, and test data should reflect how the model will be used, which often means splitting data across time instead of making a random split.
Summary
Danny Ma explains experimentation from the parts of machine learning that happen before model serving. He covers hyperparameter search, algorithm comparison, feature engineering, and changes to input data, then spends most of the session on target variables. In supervised learning, the target is often created from business rules and joined data rather than handed to the data scientist. Its timing determines whether a prediction can lead to an action, while bias, drift, class imbalance, and data splitting affect whether the experiment reflects reality. Ma recommends building experimentation inside the production data framework when possible, so a successful proof of concept does not need to be rebuilt for deployment. He also argues that machine learning engineers should understand where model outputs are used and work with data scientists and business teams to define useful problems.
An experiment can change the whole learning setup, not only a hyperparameter
Ma defines machine learning experiments broadly. Teams can vary hyperparameters with grid search, random search, or Bayesian tuning. They can compare algorithms such as logistic regression and deep learning models. They can also change feature engineering pipelines, input variables, and labels or targets. He prefers random search in practice because it can explore the space more cheaply through parallel runs and may find useful regions that a fixed grid misses. Feature experiments can include transformations, polynomial combinations, removing correlated features, handling nulls, and choosing different encodings.
Feature work should be designed with production in mind
Ma describes a common gap between a data scientist's sandbox and the production system. A team may find a promising feature set in a small proof of concept, then face a long rebuild when it tries to deploy the work. He argues that experimentation should happen inside the production framework when the data, pipelines, and teams are mature enough. Then moving a successful experiment into production can be close to turning it on. He is also honest about the obstacle: large companies often have siloed teams, inconsistent definitions of production data, and separate strategies that do not connect.
The target variable is usually created from business logic
For supervised learning, the target is the column the model learns to predict. In real projects, it is rarely provided as neatly as it is in a competition. Teams usually generate it by joining tables, applying business rules, handling effective and expiry dates, and choosing records from different systems. Ma says most label generation he has seen is done in SQL or a similar data-processing language. He considers this work both the hardest and the most important part of the project. A strong model with an average input dataset can still work well when the label is sound, while an incorrect label can undermine everything else.
A prediction needs a time window that supports an action
Ma uses insurance lapse prediction to show why the target needs a defined time period. Instead of asking only whether a customer lapsed, a useful target asks for the probability of lapsing within the next 30 days or another relevant interval. That lets the business decide what to do, such as contacting the customer or making an offer. The window depends on the application. Marketing may act over 30 days, while an industrial failure prediction may need to support action within a minute. The target definition must match when someone can respond to the prediction.
Target construction has to account for timing, drift, and context
The date used for a label affects which features are valid. Features must come from the past relative to that date, since using future information creates leakage. Ma also describes drift and contextual changes across time. Consumer behavior during the global financial crisis may differ from later behavior, and data from the COVID period can carry a different context. For machinery, temperature, rainfall, or humidity may change the relationship being modeled. Recognizing these effects requires domain understanding and logical reasoning before it requires more technical methods.
Imbalanced targets require deliberate sampling or label logic
Real prediction problems rarely have the balanced classes often found in teaching examples or competitions. A rare-event problem might contain a very small fraction of positive cases. Ma mentions downsampling the majority class and using SMOTE to upsample the minority class with generated examples. Another approach is to change the label-generation logic so that more positive cases enter the dataset. These choices alter the training data, so they belong to the experimental design rather than being treated as an afterthought.
Random data splits can let future information influence the past
Ma explains that a random split across a dataset can mix different time periods. That can allow information from the future to influence a model intended to predict the past. For forward-looking problems, he prefers an out-of-time split, such as using earlier years for training and later periods for validation and testing. The validation and test sets should come from the same distribution, because the model is selected against validation performance and then judged on the test set. Additional forward testing can provide more evidence before a model is released.
Model quality has to be connected to the decision it changes
Ma repeatedly returns to application. A model can have strong metrics and still fail because it answers a question the business cannot use. He has seen teams spend months tuning a model before discovering that the people meant to use it could not act on its output. Machine learning engineers should ask where outputs go, which process they change, and how the result will be measured. Working with data scientists can add engineering knowledge to experiments, while learning the business context makes it easier to build and explain something useful.
"If you're a machine learning engineer and you're just focused on the pipeline or you're focused on getting things out, yeah, that's great, but what is this model going to be doing once it's out?"Danny Ma59:24
Who should watch
You are building supervised learning pipelines and need to define labels from operational data rather than receiving a clean target column.
Your model experiments work in a sandbox, but the production pipeline, data sources, and business process are disconnected.
You want a practical explanation of time-based validation, drift, leakage, and rare-event targets without starting from advanced statistics.
