# Security and Privacy

Ads Dawson, Kaseya & Katharine Jarmul, Thoughtworks & David Haber, Lira | AI in Production 2024 | 34:37

Source: https://www.youtube.com/watch?v=FvHPnuwH6rg
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/security-and-privacy
Published: 2024-03-19
Tags: evals, guardrails, privacy, security

## TL;DR
- Privacy is a social, political, and legal concept that is separate from security, although the two sometimes support each other.
- LLM applications add new risks to traditional machine learning and application security, so teams need threat models, trust boundaries, and application-specific evaluations.
- Teams should aim to control and accept residual risk instead of expecting complete explainability or eliminating every possible risk.

## Summary
This panel separates privacy from security and then connects both to the practical work of deploying LLM systems. Katharine Jarmul explains that privacy has legal and social dimensions, with GDPR applying to many machine learning systems, while security can sometimes conflict with privacy. She discusses memorization in overparameterized models, copyright and private-data concerns, and privacy techniques such as differential privacy, federated learning, and encrypted computation. Ads Dawson describes how LLM applications combine traditional application and infrastructure risks with model-specific risks, including poisoning and new trust boundaries. David Haber argues that existing risk management practices are under pressure because teams are deploying generative applications quickly, often without shared templates. The panel recommends threat modeling, early evaluation criteria, vendor scrutiny, and education. Its final position is practical: AI systems do not need to be perfectly explainable, but teams need ways to control risk and accept what remains.

## Key ideas
### Privacy and security answer different questions
[03:43](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=223s)
Katharine Jarmul describes privacy as a social, political, and legal concept that is separate from security. GDPR is a privacy law rather than a cybersecurity law. The two can support each other, but they can also conflict. She uses a doorbell camera as an example of a security system that might create a serious privacy violation. In LLM systems, monitoring prompts to detect dangerous behavior can also expose or violate users' privacy. This distinction matters because teams cannot treat privacy controls as a simple extension of security controls.

### LLMs can memorize parts of their training data
[05:25](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=325s)
Jarmul says research has shown that LLMs memorize parts of their training data, especially repeated text. She explains this as a consequence of how overparameterized models generalize in sparse parts of a dataset. That creates concerns about copyright, ownership, and private data. She points to the New York Times lawsuit against OpenAI and says that retrieving full article text from a model creates a direct legal problem. Her advice is to learn how privacy technologies work and consider them during model training rather than treating privacy as a later patch.

### Agent systems change the privacy and trust problem
[08:46](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=526s)
David Haber argues that AI needs a new legal and operational paradigm as systems become more agent-driven. Machine-to-machine interactions will increase, and systems will make decisions and retain information on behalf of people. The analogy to a human learning something under an NDA becomes less useful when bits move between machines faster than people can understand the exchange. Haber says teams will need ways to trust the interactions, learning, and memorization that happen inside these systems. He expects this shift to arrive sooner than many people think.

### Privacy techniques can be built into training
[06:54](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=414s)
Jarmul names differential privacy, federated learning, and encrypted computation as ways to build privacy into model training. She refers to Google research on differential privacy with on-device text data and to federated learning used by Google and Apple for keyword prediction. She says the announced results were close to the accuracy of training without the same privacy method, although she had not yet read the paper. Her broader recommendation is to combine privacy technologies with existing machine learning practices and become familiar with the tools before deployment.

### LLM applications add security risks to existing controls
[17:43](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=1063s)
Ads Dawson says the security problem has at least two parts. Traditional machine learning still has ML security operations risks, including model and data issues. LLMs are also being embedded in applications such as chatbots, where application, infrastructure, and network controls still matter. The overlap creates new trust boundaries and risk areas. Dawson mentions data poisoning as a risk earlier in the machine learning lifecycle, while dependency vulnerabilities, API authorization, and other traditional application concerns remain relevant once a model is inside an application.

### Risk management is under pressure from fast deployment
[22:57](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=1377s)
Haber says security and risk teams are dealing with product teams that deploy faster than existing processes can handle. Organizations must decide what it means to add an OpenAI model, fine-tune a model, or build one from scratch. He says there are no universally applicable templates yet. Risk depends on the application: some companies keep generative applications internal, while others expose them publicly. He points to OWASP work as a useful source for learning about risks and says teams need guidance for securing applications before deployment and during operation.

### Threat modeling and evaluation need to happen early
[25:48](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=1548s)
Dawson recommends threat modeling to identify risks, trust boundaries, and the strengths and weaknesses of each boundary. Traditional controls can still address some LLM application issues, including cross-site scripting through content security policies and API-to-API authorization problems. Jarmul adds that teams should define evaluation criteria early and update them with beta testers. Model behavior can change after small version updates, and users will find unexpected behaviors that should become part of the evaluation set.

### The goal is to control risk and accept what remains
[31:29](https://www.youtube.com/watch?v=FvHPnuwH6rg&t=1889s)
Haber says the goal is to control risk rather than assume that every system will be fully explainable. He compares an AI system to a rocket: a system that is not fully explainable but has flown many times may be preferable to one that is completely explainable but has never flown. Jarmul and Dawson agree that teams need to identify and accept residual risk. Jarmul adds that privacy engineering is still less mature than security engineering, but it gives machine learning practitioners a way to apply mathematical privacy methods to deployment decisions.

## Notable quotes
- Katharine Jarmul: "Privacy is both a social, a political, and a legal concept that is quite separate from security." (03:49)
- David Haber: "The short answer should certainly be yes, we probably need a new paradigm for what we are going through now." (08:46)
- Katharine Jarmul: "Develop your evaluation criteria early and then update it quite often with your beta testers." (28:03)
- David Haber: "I think ultimately what we need to get to here is a world where the world of AI meets systems and safety engineering." (31:29)
- Katharine Jarmul: "Privacy engineering is another thing to look at, but we're still kind of in the infancy compared to security engineering." (32:52)

## Tools & references mentioned
- Guardrails AI
- GDPR
- The New York Times
- OpenAI
- Google Research
- Apple
- OWASP
- Gandalf
- Practical Data Privacy
- Differential privacy
- Federated learning
- Encrypted machine learning

## Who should watch
- You are adding an LLM or generative feature to an existing product and need to understand which traditional security controls still apply.
- Your risk, security, or privacy process has no clear way to assess model providers, fine-tuned models, or public-facing AI applications.
- You want practical guidance on threat modeling, evaluation criteria, memorization, and residual risk before putting an AI system into production.

## Related talks

- [Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security](https://mlopstalks.com/talks/guarding-llm-and-nlp-apis-a-trailblazing-odyssey-for-enhanced-security) (Ads Dawson, Cohere, 59:41)
- [LLM Security](https://mlopstalks.com/talks/llm-security) (Raahul Dutta, Elsevier & Uri Shamay, Null & Sankalp Gilda, DevelopYours, 1:00:17)
- [ML Security: Why should you care?](https://mlopstalks.com/talks/ml-security-why-should-you-care) (Sahbi Chaieb, SAS, 52:20)
- [Data Privacy and Security](https://mlopstalks.com/talks/data-privacy-and-security) (Diego Oppenheimer, Factory & Gevorg Karapetyan, ZERO Systems & Vin Vashishta, V Squared & Saahil Jain, U.com & Shreya Rajpal, 25:44)
- [Challenges and Opportunities in Building Data Science Solutions with LLMs](https://mlopstalks.com/talks/challenges-and-opportunities-in-building-data-science-solutions-with-llms) (Pascal Brokmeier, McKinsey and Company & Daniel Herde & Viktoriia Oliinyk, QuantumBlack, AI by McKinsey, 37:19)