# Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security

Ads Dawson, Cohere | MLOps Podcast | Episode 190 | 59:41
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=u-VCfL3l_lw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/guarding-llm-and-nlp-apis-a-trailblazing-odyssey-for-enhanced-security
Published: 2023-11-14
Tags: guardrails, security, testing

## TL;DR
- Ads Dawson says LLM applications expand the attack surface because they combine ordinary application vulnerabilities with an intelligent model that may have access to data, plugins, and private systems.
- Security teams should treat the boundary between clients and models as unsafe in both directions, then use threat modeling, least privilege, logging, red teaming, and layered controls to reduce risk.
- Data poisoning can create hidden vulnerabilities when attackers place content on domains likely to enter public training datasets, allowing later prompts to trigger that content.

## Summary
Ads Dawson explains how security work changes when an application includes an LLM. Traditional issues such as supply-chain attacks, rate limiting, and private-data access still apply, but the model adds prompt injection, excessive agency, toxic citations, hallucinations, and model-specific attack paths. Ads recommends threat modeling the full system, checking data lineage and permissions, limiting model access, placing controls such as safe proxies around models, and logging both inputs and outputs. He describes OWASP's Top 10 for LLM Applications as a practical starting point for developers and security teams. The discussion also covers red teaming, including the difference between testing a model's outputs and attacking the surrounding application. Ads gives examples of malicious packages recommended by models and hidden instructions in web pages. Near the end, he describes data poisoning through cheap, attacker-controlled domains that public crawlers may collect for future training.

## Key ideas
### LLM applications combine ordinary security risks with model-specific attack paths
[06:30](https://www.youtube.com/watch?v=u-VCfL3l_lw&t=390s)
Ads Dawson says LLM security begins with the same concerns found in other software, including supply-chain vulnerabilities, static analysis, and dependency checks. The difference is that an intelligent model sits on top of the application and may have access hooks, plugins, private databases, or other tools. That combination spreads the attack surface. A model can also have poor quality or be vulnerable to prompt injection, while the application around it has separate weaknesses. Ads treats the model and its surrounding application as two related security problems that need to be assessed together.

### Threat modeling should follow data, permissions, and trust boundaries
[12:05](https://www.youtube.com/watch?v=u-VCfL3l_lw&t=725s)
Ads describes threat modeling as assessing risk, mapping the attack vector, finding gaps, and applying mitigations where the risk is highest. For a private beta trained on Slack data, he would check the lineage of that data and the permissions held by the access token. A token that can read private channels creates a path for prompt injection to expose information that should remain private. He calls the boundary between a client and a model bidirectionally unsafe. Teams need to log and analyze what enters and leaves the model, then compare releases when unsafe behavior appears.

### Least privilege limits the damage caused by prompt injection
[19:35](https://www.youtube.com/watch?v=u-VCfL3l_lw&t=1175s)
Ads gives a concrete attack path in which a model summarizes a website containing hidden Markdown instructions. If the model is vulnerable to indirect prompt injection and has a GitHub plugin with administrative access, the hidden instructions could make it send a request to delete repository secrets. His basic recommendation is to give the model only the permissions it needs and to review those permissions. A safe proxy can add another layer by controlling what the model is allowed to access. Rate limiting and other controls around the API also reduce the effect of excessive automated requests.

### Red teaming LLM applications is still highly tailored to each system
[17:15](https://www.youtube.com/watch?v=u-VCfL3l_lw&t=1035s)
Ads distinguishes testing a model for jailbreaks or toxic output from traditional security red teaming of the application and its environment. LLM application security has few standardized automated workflows, and the right tests depend on the model, its data, its plugins, whether it can access the internet, and whether it uses retrieval. A system without plugins does not need plugin-focused tests. A system that retrieves web citations does. Ads says this customization makes it difficult to put red teaming into a standard CI/CD pipeline, even though regular exercises can reveal attacks before release.

### OWASP's Top 10 for LLM Applications provides a starting checklist
[31:40](https://www.youtube.com/watch?v=u-VCfL3l_lw&t=1900s)
Ads explains that OWASP's Top 10 for LLM Applications applies traditional application-security thinking to systems with an LLM layer. It is a community-built reference rather than a fixed rulebook. Each entry describes the vulnerability, gives examples and attack scenarios, recommends defenses, and links to relevant research or public exploits. The list is meant to become a starting point similar to OWASP's web-application Top 10. Ads says teams should not stop at the list, but it gives developers and security engineers a shared taxonomy and a place to begin assessing risk.

### Model outputs can create supply-chain attacks through invented packages
[23:40](https://www.youtube.com/watch?v=u-VCfL3l_lw&t=1420s)
Ads describes a scenario where one client causes a model to encounter a nonexistent JavaScript package. An attacker then creates that package and adds a backdoor. When another client later asks the model for help, the model may recommend the attacker-controlled package, and the client may install it. Ads connects this to overreliance on model outputs. Developers may not know every package well enough to spot an invented recommendation, so model responses need checking. He also says application developers have a responsibility to scrub and sanitize sensitive information before it enters a model.

### Training-data poisoning can hide an attack until a later prompt activates it
[52:06](https://www.youtube.com/watch?v=u-VCfL3l_lw&t=3126s)
Ads describes a data-poisoning technique associated with Will and a Defcon talk called "Split View." The attacker monitors known public training-data sources, buys expired domains that Common Crawl may collect, and adds chosen content to them. Ads says Will spent 13 dollars and gained control of a thousand domains. The content could be harmless-looking cat memes or something intended to affect a model's later behavior. The vulnerability can remain latent until someone knows what to ask. Ads recommends tracking lineage and running internal red-team exercises across the pipeline so teams can compare model releases and investigate when the behavior first appeared.

## Notable quotes
- Ads Dawson: "The trust boundary between a client and a model is bidirectionally unsafe." (14:19)
- Ads Dawson: "You don't want your model to become available from lack of rate limiting, but also the lack of rate limiting on a model could also relate to things like shadow modeling." (36:30)
- Ads Dawson: "You want to be careful if you're allowing free tier people to send like 50 prompts a second." (48:28)
- Ads Dawson: "If you surround and you secure the environment and the ecosystem around it, then you reduce so much risk and you mitigate so much threats just by doing that itself." (50:05)
- Ads Dawson: "Attacker uses domain monitoring for known Common Crawl and public training data sources, purchases a bunch of expired bogus domains." (52:47)

## Tools & references mentioned
- OWASP
- OWASP Top 10 for LLM Applications
- Cohere
- MLOps Community
- Common Crawl
- Defcon31
- AI Village
- Hack the Future
- Embrace The Red
- Gandalf
- Lira
- GitHub
- Slack
- Python
- JavaScript
- Open Crawler
- Falcon
- Diego Oppenheimer
- Algorithmia
- DataRobot
- Philip Carter
- Honeycomb

## Who should watch
- You are building an LLM application that can read private data, call plugins, retrieve web content, or access other systems, and you need a way to map its attack surface.
- Your security team knows web and API security but needs concrete examples of prompt injection, excessive agency, toxic citations, and model-specific red teaming.
- You want to contribute to OWASP's Top 10 for LLM Applications or build a safe sandbox where engineers can practice attacking LLM applications.

## Related talks

- [LLM Security](https://mlopstalks.com/talks/llm-security) (Raahul Dutta, Elsevier & Uri Shamay, Null & Sankalp Gilda, DevelopYours, 1:00:17)
- [Harnessing AI APIs for Safer, Accurate, & Reliable Applications](https://mlopstalks.com/talks/harnessing-ai-apis-for-safer-accurate-reliable-applications) (Ron Heichman, SentinelOne, 1:08:14)
- [Evaluating LLMs for AI Risk](https://mlopstalks.com/talks/evaluating-llms-for-ai-risk) (Finn Howell, Robust Intelligence, 10:38)
- [Impact of LLMs on the Tech Stack and Product Development](https://mlopstalks.com/talks/impact-of-llms-on-the-tech-stack-and-product-development) (Anand Das, Bito, 55:31)
- [Transforming AI Safety & Security](https://mlopstalks.com/talks/transforming-ai-safety-security) (Manojkumar Parmar, AIShield, a Corporate Startup of Bosch, 23:33)