Privacy is a social, political, and legal concept that is separate from security, although the two sometimes support each other.
2
LLM applications add new risks to traditional machine learning and application security, so teams need threat models, trust boundaries, and application-specific evaluations.
3
Teams should aim to control and accept residual risk instead of expecting complete explainability or eliminating every possible risk.
Summary
This panel separates privacy from security and then connects both to the practical work of deploying LLM systems. Katharine Jarmul explains that privacy has legal and social dimensions, with GDPR applying to many machine learning systems, while security can sometimes conflict with privacy. She discusses memorization in overparameterized models, copyright and private-data concerns, and privacy techniques such as differential privacy, federated learning, and encrypted computation. Ads Dawson describes how LLM applications combine traditional application and infrastructure risks with model-specific risks, including poisoning and new trust boundaries. David Haber argues that existing risk management practices are under pressure because teams are deploying generative applications quickly, often without shared templates. The panel recommends threat modeling, early evaluation criteria, vendor scrutiny, and education. Its final position is practical: AI systems do not need to be perfectly explainable, but teams need ways to control risk and accept what remains.
Katharine Jarmul describes privacy as a social, political, and legal concept that is separate from security. GDPR is a privacy law rather than a cybersecurity law. The two can support each other, but they can also conflict. She uses a doorbell camera as an example of a security system that might create a serious privacy violation. In LLM systems, monitoring prompts to detect dangerous behavior can also expose or violate users' privacy. This distinction matters because teams cannot treat privacy controls as a simple extension of security controls.
Jarmul says research has shown that LLMs memorize parts of their training data, especially repeated text. She explains this as a consequence of how overparameterized models generalize in sparse parts of a dataset. That creates concerns about copyright, ownership, and private data. She points to the New York Times lawsuit against OpenAI and says that retrieving full article text from a model creates a direct legal problem. Her advice is to learn how privacy technologies work and consider them during model training rather than treating privacy as a later patch.
Agent systems change the privacy and trust problem
David Haber argues that AI needs a new legal and operational paradigm as systems become more agent-driven. Machine-to-machine interactions will increase, and systems will make decisions and retain information on behalf of people. The analogy to a human learning something under an NDA becomes less useful when bits move between machines faster than people can understand the exchange. Haber says teams will need ways to trust the interactions, learning, and memorization that happen inside these systems. He expects this shift to arrive sooner than many people think.
Jarmul names differential privacy, federated learning, and encrypted computation as ways to build privacy into model training. She refers to Google research on differential privacy with on-device text data and to federated learning used by Google and Apple for keyword prediction. She says the announced results were close to the accuracy of training without the same privacy method, although she had not yet read the paper. Her broader recommendation is to combine privacy technologies with existing machine learning practices and become familiar with the tools before deployment.
LLM applications add security risks to existing controls
Ads Dawson says the security problem has at least two parts. Traditional machine learning still has ML security operations risks, including model and data issues. LLMs are also being embedded in applications such as chatbots, where application, infrastructure, and network controls still matter. The overlap creates new trust boundaries and risk areas. Dawson mentions data poisoning as a risk earlier in the machine learning lifecycle, while dependency vulnerabilities, API authorization, and other traditional application concerns remain relevant once a model is inside an application.
Risk management is under pressure from fast deployment
Haber says security and risk teams are dealing with product teams that deploy faster than existing processes can handle. Organizations must decide what it means to add an OpenAI model, fine-tune a model, or build one from scratch. He says there are no universally applicable templates yet. Risk depends on the application: some companies keep generative applications internal, while others expose them publicly. He points to OWASP work as a useful source for learning about risks and says teams need guidance for securing applications before deployment and during operation.
Threat modeling and evaluation need to happen early
Dawson recommends threat modeling to identify risks, trust boundaries, and the strengths and weaknesses of each boundary. Traditional controls can still address some LLM application issues, including cross-site scripting through content security policies and API-to-API authorization problems. Jarmul adds that teams should define evaluation criteria early and update them with beta testers. Model behavior can change after small version updates, and users will find unexpected behaviors that should become part of the evaluation set.
The goal is to control risk and accept what remains
Haber says the goal is to control risk rather than assume that every system will be fully explainable. He compares an AI system to a rocket: a system that is not fully explainable but has flown many times may be preferable to one that is completely explainable but has never flown. Jarmul and Dawson agree that teams need to identify and accept residual risk. Jarmul adds that privacy engineering is still less mature than security engineering, but it gives machine learning practitioners a way to apply mathematical privacy methods to deployment decisions.
