# Harnessing AI APIs for Safer, Accurate, & Reliable Applications

Ron Heichman, SentinelOne | MLOps Podcast | Episode 252 | 1:08:14
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=Yee8JNkWfcM
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/harnessing-ai-apis-for-safer-accurate-reliable-applications
Published: 2024-08-06
Tags: evals, guardrails, prompt-engineering, security

## TL;DR
- Ron Heichman argues that production AI security starts with inspecting the text assembled in an LLM context window, rather than relying only on an agent framework's architecture.
- Multi-turn jailbreaks build agreement gradually, so input classifiers, output detectors, monitoring, rate limits, and cooldowns need to work together.
- Teams can turn natural-language safety preferences into classifiers by generating examples, measuring their diversity, and keeping a human involved for narrow concerns such as brand damage.

## Summary
Ron Heichman explains why reliable LLM applications depend on understanding and testing the context sent to a model. Agent frameworks may include retrieval, vector stores, tools, and prompt templates, but the model ultimately receives text. Developers should inspect that text directly and ask whether it would equip a person to complete the task. The conversation then focuses on jailbreaks and prompt injection. Ron describes how attackers build agreement across multiple turns, manipulate role formatting, exploit function-call representations, or place malicious instructions in retrieved content. He recommends combining input checks, output classifiers, monitoring, and controls that slow repeated probing. For custom safety concerns, such as protecting a company's reputation, he describes a black-box version of constitutional AI. A product manager can state a policy in natural language, generate positive and negative examples, measure whether the examples cover diverse cases, and use human review where the concern is subjective. The discussion is candid about latency and usability costs.

## Key ideas
### LLM products fit vertical workflows more readily than traditional ML services
[01:55](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=115s)
Demetrios Brinkmann contrasts LLM-based support or HR products with fraud detection and recommender systems. Ron Heichman says banks are reluctant to share transaction data or models because compliance concerns and exposed vulnerabilities can reveal a company's secret sauce. Even when customer data is not copied between clients, a service provider learns the shape of that data and how to build tools around it. That institutional knowledge is part of the value. Ron says companies often prefer to develop these systems in house because there is no one-size-fits-all fraud detector. LLM products can be easier to offer vertically when the base model already has a prior for conversational data, while proprietary business data is harder to access and less represented in pretraining.

### Prompt design works by steering the model toward a useful part of its training data
[07:18](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=438s)
Ron describes a completion model as starting with little context and predicting likely next tokens. Adding a phrase such as "be a CSR" narrows the context toward text associated with that role, although ordinary training manuals may not begin with those exact words. The practical task is to find wording that resembles the documentation or style being requested. He connects this to synthetic survey responses, where a company might ask an LLM to roleplay a demographic. That idea depends on whether the model learned enough about the group to produce representative answers. Ron is honest that this is interesting, but he questions how well such generated responses reflect a real sample.

### Every agent workflow eventually becomes text in the model's context window
[13:00](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=780s)
Ron says retrieval, chunking, vector stores, prompt templates, and agent tools all produce one underlying artifact: input text for an LLM. Architectural abstractions can hide that fact. He recommends looking at the actual prompt and asking whether a person receiving the same wall of text would have enough information to do the requested task. He mentions LM Studio as a useful inspection point because it exposes the call manifest sent to a local model. Ron imagines a text editor that would let developers rearrange prompt sections, inspect the system prompt and chat messages, replace words with synonyms, or mark sections as reusable functions before making the call. He argues that this level of editing can matter when improving quality or removing ambiguity.

### Jailbreaks exploit context, agreement, and the model's text-based interfaces
[24:00](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=1440s)
Ron explains that jailbreaks do not require deep technical knowledge. They can resemble a text-based RPG, where a user gradually discovers which wording unlocks a response. Because a chat is represented as text with roles and special markers, an attacker may try to break the expected order or make the model treat one message as another role. Function calling also reduces to text before the output is parsed. Ron describes satirical-looking constructions such as an "admin mode activated" message or a manually written function definition that asks the model to reveal a system prompt. He also points to sycophancy: after the model has repeatedly agreed with a user, that established context can increase the chance that it agrees to a later request it should reject.

### Multi-turn red teaming is more effective because it teaches an attack strategy
[31:00](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=1860s)
Ron says single-turn attacks are easier to block than conversations that build rapport. An attacker can begin with reasonable requests, receive agreement, and then increase the demands. The model sees the whole thread, so previous acceptance changes the context for the later request. Ron recommends testing these patterns with automated red teaming, drawing on work by Ethan Perez and research groups such as Anthropic. A red-team model can receive feedback about which prompts caused a failure and learn the relationship between prompts and undesirable outputs. For subjective targets such as brand damage, a human must label what counts as unacceptable. Ron calls this semi-automatic red teaming, because the generation can be automated while the judgment remains with a person.

### Safety controls need to account for bad output that is discovered only after generation
[52:25](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=3145s)
Ron says LLM systems differ from ordinary classifiers because teams may not know an output is bad until the model has generated it. A streaming response may begin appearing and then be dropped when a detector finds a problem. Teams can check inputs for jailbreaks before making a call, inspect generated output, and use classifiers or validators around the model. Monitoring can identify users who repeatedly probe for weak spots. Ron suggests adding cooldowns, warnings, temporary blocks, or other friction because the attacker's most valuable resource is feedback about what works. He also notes the tradeoff: checking every input and output adds inference cost and latency, and users may prefer a fast imperfect response to a slower system that performs extensive safety checks.

### Natural-language policies can become classifiers through generated and measured examples
[40:04](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=2404s)
Ron describes what he calls "warm start policies," where a product manager writes down a policy in ordinary language and an LLM generates examples of acceptable and unacceptable behavior. He compares the idea with Anthropic's constitutional AI, while stressing that his approach works as a black-box method without changing the underlying model. The examples need enough variety. Ron proposes measuring embedding distances and graph structure to estimate whether a dataset is clumpy or degenerate. A smaller set that describes a larger set may indicate limited diversity. The score can be fed back into an iterative generation loop, so the LLM creates examples that improve the score. The same setup can generate prompts that break a model, with a Boolean score indicating whether each attempt succeeded.

### Indirect prompt injection makes retrieved content part of the attack surface
[1:01:35](https://www.youtube.com/watch?v=Yee8JNkWfcM&t=3695s)
Ron describes a malicious instruction hidden in a web page or another retrieved document. When an LLM retrieves it through browsing or RAG, the instruction enters the model's context as material the system itself found. That can make it more persuasive than the same text supplied directly by a user. He gives a scenario where a retrieved payload asks the model to disclose sensitive database information. Demetrios Brinkmann connects this with poisoning web content that crawlers may collect. Ron extends the concern to code-generation systems: an attacker could seed public repositories or web pages with vulnerable code patterns, hoping data-hungry training pipelines absorb them. The model could then reproduce a subtle vulnerability in software used in a critical product.

## Notable quotes
- Ron Heichman: "All that this boils down to is some input to an LLM in its context window." (14:34)
- Ron Heichman: "The reality is that those are not nearly as successful as building up a context." (31:43)
- Ron Heichman: "The data about your reactions of your LLM, the data about what actually works, that essentially labeled data, that's your most important piece." (1:00:59)
- Ron Heichman: "You never know essentially what the LLM is going to say until it says it." (53:24)
- Ron Heichman: "Anything that slows it down a little bit helps a lot." (06:06)

## Tools & references mentioned
- SentinelOne
- Preamble
- Ethan Perez
- Anthropic
- LM Studio
- LangChain
- LlamaIndex
- CrewAI
- AutoGen
- Weights & Biases
- Constitutional AI
- Common Crawl
- Snapchat
- Meta
- OpenAI
- GPT-2
- GPT-3
- RAG

## Who should watch
- You are building an LLM or RAG product and need a practical way to inspect what your framework actually sends to the model.
- Your team is testing jailbreaks, prompt injection, or data poisoning and needs to understand why multi-turn and indirect attacks are hard to catch.
- You need safety policies for a narrow product concern, such as brand damage, without fine-tuning or changing the underlying model.

## Related talks

- [Transforming AI Safety & Security](https://mlopstalks.com/talks/transforming-ai-safety-security) (Manojkumar Parmar, AIShield, a Corporate Startup of Bosch, 23:33)
- [Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security](https://mlopstalks.com/talks/guarding-llm-and-nlp-apis-a-trailblazing-odyssey-for-enhanced-security) (Ads Dawson, Cohere, 59:41)
- [Guiding LLMs While Staying in the Driver's Seat](https://mlopstalks.com/talks/guiding-llms-while-staying-in-the-drivers-seat) (Jacob van Gogh, Adept AI, 10:02)
- [Fraud Detection in the AI Era](https://mlopstalks.com/talks/fraud-detection-in-the-ai-era) (Rafael Sandroni, GardionAI, 41:20)
- [Evaluating LLMs for AI Risk](https://mlopstalks.com/talks/evaluating-llms-for-ai-risk) (Finn Howell, Robust Intelligence, 10:38)