Jeremy Jordan uses rules to encode threat knowledge before enough labeled data exists to train a machine learning model.
2
High-precision rules handle immediate blocking decisions, while machine learning generalizes from larger datasets and finds threats that rules miss.
3
Jordan writes blog posts to learn technical subjects, expose gaps in his understanding, and create material he can use to relearn a topic later.
Summary
Jeremy Jordan discusses machine learning infrastructure for threat detection at Duo Security, drawing on earlier work at Proofpoint. He explains why rules remain useful in cybersecurity. A new threat may need protection before a large labeled dataset exists, and a rule can also quickly suppress a false positive from a deployed model. Jordan prefers rules with high precision and limited interaction, while using machine learning to generalize and find patterns the rules miss. He also describes product designs for uncertain cases, such as Proofpoint's isolation browser and Duo's risk-based authentication, which add friction without simply blocking the user. The conversation then turns to testing machine learning systems. Jordan distinguishes evaluation metrics from behavioral tests that check whether a model follows expected patterns. He is candid that his testing blog post describes an ideal approach he was not fully using. Writing is his main learning method, since explaining a subject in his own words exposes gaps and creates a useful reference for the future.
Normconf focuses on practical machine learning work
Jordan says Normconf began with a joke tweet from Vicky Boykiss about creating a conference for the practical work that gets less attention than machine learning and AI hype. The virtual conference is free and features practitioners discussing day-to-day subjects. Jordan planned to speak about rules engines working alongside machine learning. He rejects the common idea that teams start with heuristics and eventually replace them all with models. In cybersecurity, he has found that rules usually remain part of the system, with machine learning added alongside them.
Rules provide protection before labeled data exists
Jordan explains that threat detection teams often learn about a new threat before they have much data about it. They still want to protect users immediately, so they encode their existing domain knowledge in rules. Cybersecurity changes as threat actors respond to improved detection and blocking. Those actors operate with incentives similar to a business, searching for new loopholes or social engineering attacks when their existing methods become less effective. Rules let defenders respond to new patterns before they can collect and label enough examples for model training.
A deployed machine learning model can introduce false positives and block benign content. Jordan says a rule can remediate that problem faster than adding examples to a dataset, labeling related cases, and starting another training run. The longer model improvement process should still happen, but the rule gives the team a quicker response while that work continues. In his systems, heuristics are effectively Python functions that follow agreed conventions for receiving and processing data. They can then be deployed with other rules and run in a distributed way.
Jordan's preferred way to control rules sprawl is to make detection rules highly precise. If every rule has high precision, the system can flag and block content when any rule fires, without building many interactions between noisy rules and suppressions. He acknowledges the tradeoff: optimizing for precision can reduce recall. Machine learning complements this approach by learning from a large labeled dataset, generalizing beyond the explicit rules, and finding threats those rules missed. Those model findings can also reveal where a new rule would improve coverage.
Uncertain threats need graduated product responses
Jordan says blocking decisions need very high precision because blocking a legitimate login or piece of content creates friction. Some cases fall into a gray area where the system cannot confidently call them benign or malicious. Proofpoint's isolation browser allowed users to visit a suspicious link in a more restricted environment instead of blocking it outright. At Duo Security, risk-based authentication can restrict the authentication factors available for a risky login. A user might need a YubiKey or Touch ID instead of an SMS code.
Behavioral tests reveal model changes that metrics hide
Jordan distinguishes model evaluation from model testing. Evaluation uses metrics and plots to summarize performance on validation or test data. Testing makes explicit checks for behaviors the model should follow, similar to unit tests in ordinary software. A new model can improve an aggregate precision or accuracy number while changing behavior in an important slice of the data. Behavioral tests can fail during a model update and show which expected property changed. Jordan describes this as a way to build a more detailed report of what the model does well and where it does poorly.
Foundation models are difficult to test across every use
Jordan does not claim to have a comprehensive testing method for foundation models. Their broad purpose makes the problem different from testing a model inside one defined product. He has seen evaluation approaches that enumerate possible applications and use separate datasets for those tasks. That gives a better view of downstream behavior than inspecting the loss function from foundation-model training. He expects the field to keep working out how to characterize these models as they are used in more settings.
Writing is a practical way to find gaps in understanding
Jordan usually writes to learn a subject rather than to document everything he is already applying at work. He gives Terraform as an example: he wanted to understand a technology he used at Duo Security for infrastructure, so he wrote about it after studying it. His typical audience is himself six months earlier. Writing forces him to teach the subject, build a coherent explanation, and notice where he cannot connect the ideas. The resulting post also becomes a reference he can revisit when his knowledge fades. He prefers having the explanation in his own voice because that makes relearning more efficient.
"Rules are great for encoding our domain knowledge or our understanding of a given threat before we have this huge labeled data set that we can go and train a machine learning model."Jeremy Jordan10:11
Who should watch
You are designing a threat detection system and need to decide where rules, models, and graduated user responses belong.
Your model metrics look good, but you need tests that catch changes in specific behaviors before a rollout.
You want a concrete learning practice for unfamiliar tools and technical subjects, especially when formal study has no exam or feedback loop.
