Machine learning security includes ordinary software security plus threats that target the data, model behavior, and training process.
2
External data, pre-trained models, and public packages can introduce security risks before a model reaches production.
3
Teams need to assess risks across roles, accept that some defenses reduce accuracy, and build dedicated model security skills and tooling.
Summary
Sahbi Chaieb explains why machine learning security needs attention before a model reaches production. Machine learning systems inherit ordinary software and cybersecurity risks, then add threats involving training data, model outputs, privacy, and model theft. He discusses extracting memorized personal information from large models, bypassing content filters with adversarial inputs, copying a model through its API, and poisoning federated learning through compromised devices. Sahbi describes defenses including differential privacy, data augmentation, homomorphic encryption, and secure multiparty computation. He assigns responsibility across data scientists, machine learning engineers, product owners, managers, and decision makers. Security choices can affect accuracy and deployment decisions, so they need to be made during development. The conversation also covers emerging AI red teams and tools from Microsoft and Facebook. Sahbi is honest that the field still requires substantial scientific knowledge, although practical tools and challenges are beginning to make model security more accessible.
Production constraints should shape model choices from the start
Sahbi describes an early project that produced a high-performing model for an international customer. Only after the work was complete did his boss ask him to put it into production. The team then had to confront choices made without deployment in mind, such as querying data in real time while using an offline model and combining models in an architecture that was too complex to deploy. Sahbi says security belongs in the same category. Data scientists often fetch large amounts of data, scrape websites, use open data, or download pre-trained models from GitHub. Each decision can create a security risk that would have been easier to address before the model was built.
Machine learning adds model-specific threats to ordinary software security
Sahbi says a machine learning system is still a piece of software, so standard software and cybersecurity practices apply. It also has threats tied to the model and its data. Data extraction can expose information memorized during training, which creates privacy problems. Attackers can corrupt a model or manipulate inputs so it produces false predictions. Sahbi gives the example of bypassing a fake-news or harmful-content filter on social media. Model theft is another concern. Someone who can repeatedly query a model through an API may be able to reproduce it, avoid paying for the service, or give the copy to a competitor.
The whole organization has a role in assessing model risk
Sahbi says a data scientist should know whether the data and models used during development come from reliable sources. A corrupted pre-trained model can corrupt the resulting system. Machine learning engineers, managers, product owners, and decision makers then take responsibility at later stages. A product owner can evaluate the risks with the data scientist and decide whether the model should go into production as it is or receive changes. Those changes can reduce accuracy or affect other objectives. Sahbi leaves accountability as an open organizational question, but his practical position is clear: each person should understand the risks attached to their part of the system.
Differential privacy trades memorization against accuracy
Sahbi explains that large models can memorize details from their training data. Researchers prompted Google's Smart Compose system with selected words and extracted names, phone numbers, email addresses, and physical addresses. Differential privacy limits how much information the model can memorize. Sahbi describes an epsilon setting as a control that lets a team choose how strongly it prioritizes privacy compared with accuracy. This is a decision with a business cost. The technique does not remove the need to understand the model or its data, but it gives teams a way to control the privacy risk during training.
Adversarial examples require training against manipulated inputs
Adversarial examples change an input so a model produces a different prediction. Sahbi applies this to spam filters, fake-news detectors, and harmful-content filters. An attacker may alter an image or text just enough to pass a filter. He says the practical defense is to anticipate the transformations an attacker may use and train with augmented examples. For images, that can include flipping or rotating inputs. For text, it can include changing characters or words. The model is then exposed to variations during training instead of encountering them for the first time in an attack.
Federated learning keeps data distributed but allows poisoned updates
Federated learning trains one model from data held in different locations without moving all the data to a central place. Sahbi notes that this is used on smartphones and can also help hospitals learn from patient data without pooling it in one location. The security problem is that a person who controls a participating device can inject arbitrary data into the training process. Medical applications make that risk especially sensitive. Sahbi mentions homomorphic encryption, which allows computation on encrypted data, and secure multiparty computation as defenses. He says these methods are still part of a changing research area and none is a perfect framework.
AI red teams and practical tools are starting to close the tooling gap
When Sahbi wrote his article, he found little practical tooling for model security. He says companies such as Microsoft and Facebook had since started AI red teams. These groups borrow the red-team and blue-team model from cybersecurity. Red teams attack an organization's systems to find weaknesses, while blue teams work on defenses. Sahbi mentions Facebook's tool for testing robustness through input augmentations and Microsoft's Counterfit, which evaluates model threats against a threat matrix. He sees this as a similar progression to MLOps: research creates methods, leading companies build tools, and wider teams can eventually apply them without needing to reproduce every research result.
Security work will expand beyond attacks on model predictions
Sahbi expects more companies to create teams focused on machine learning security as they automate model deployment. He says these teams will consider privacy and algorithmic bias alongside security and robustness. A model that scrapes comments from a website may be affected if a competitor or malicious person injects data into that source. Teams therefore need to understand the consequences of changes in external data. Sahbi says the field currently requires strong scientific skills because it remains close to research. He expects better tools to allow people with more software-focused backgrounds to work on these problems.
"Each person has to do their job, whatever that may be, and you really need to look at trying to make your piece of the puzzle as secure as possible."Demetrios Brinkmann50:57
Who should watch
Data scientists who optimize model performance first and only think about deployment or security when production work begins.
Engineers responsible for models that use scraped data, open data, public packages, or pre-trained models from external sources.
Product owners and managers who need to decide which model risks are acceptable when security measures affect accuracy or deployment.
