Ron Heichman argues that production AI security starts with inspecting the text assembled in an LLM context window, rather than relying only on an agent framework's architecture.
2
Multi-turn jailbreaks build agreement gradually, so input classifiers, output detectors, monitoring, rate limits, and cooldowns need to work together.
3
Teams can turn natural-language safety preferences into classifiers by generating examples, measuring their diversity, and keeping a human involved for narrow concerns such as brand damage.
Summary
Ron Heichman explains why reliable LLM applications depend on understanding and testing the context sent to a model. Agent frameworks may include retrieval, vector stores, tools, and prompt templates, but the model ultimately receives text. Developers should inspect that text directly and ask whether it would equip a person to complete the task. The conversation then focuses on jailbreaks and prompt injection. Ron describes how attackers build agreement across multiple turns, manipulate role formatting, exploit function-call representations, or place malicious instructions in retrieved content. He recommends combining input checks, output classifiers, monitoring, and controls that slow repeated probing. For custom safety concerns, such as protecting a company's reputation, he describes a black-box version of constitutional AI. A product manager can state a policy in natural language, generate positive and negative examples, measure whether the examples cover diverse cases, and use human review where the concern is subjective. The discussion is candid about latency and usability costs.
LLM products fit vertical workflows more readily than traditional ML services
Demetrios Brinkmann contrasts LLM-based support or HR products with fraud detection and recommender systems. Ron Heichman says banks are reluctant to share transaction data or models because compliance concerns and exposed vulnerabilities can reveal a company's secret sauce. Even when customer data is not copied between clients, a service provider learns the shape of that data and how to build tools around it. That institutional knowledge is part of the value. Ron says companies often prefer to develop these systems in house because there is no one-size-fits-all fraud detector. LLM products can be easier to offer vertically when the base model already has a prior for conversational data, while proprietary business data is harder to access and less represented in pretraining.
Prompt design works by steering the model toward a useful part of its training data
Ron describes a completion model as starting with little context and predicting likely next tokens. Adding a phrase such as "be a CSR" narrows the context toward text associated with that role, although ordinary training manuals may not begin with those exact words. The practical task is to find wording that resembles the documentation or style being requested. He connects this to synthetic survey responses, where a company might ask an LLM to roleplay a demographic. That idea depends on whether the model learned enough about the group to produce representative answers. Ron is honest that this is interesting, but he questions how well such generated responses reflect a real sample.
Every agent workflow eventually becomes text in the model's context window
Ron says retrieval, chunking, vector stores, prompt templates, and agent tools all produce one underlying artifact: input text for an LLM. Architectural abstractions can hide that fact. He recommends looking at the actual prompt and asking whether a person receiving the same wall of text would have enough information to do the requested task. He mentions LM Studio as a useful inspection point because it exposes the call manifest sent to a local model. Ron imagines a text editor that would let developers rearrange prompt sections, inspect the system prompt and chat messages, replace words with synonyms, or mark sections as reusable functions before making the call. He argues that this level of editing can matter when improving quality or removing ambiguity.
Jailbreaks exploit context, agreement, and the model's text-based interfaces
Ron explains that jailbreaks do not require deep technical knowledge. They can resemble a text-based RPG, where a user gradually discovers which wording unlocks a response. Because a chat is represented as text with roles and special markers, an attacker may try to break the expected order or make the model treat one message as another role. Function calling also reduces to text before the output is parsed. Ron describes satirical-looking constructions such as an "admin mode activated" message or a manually written function definition that asks the model to reveal a system prompt. He also points to sycophancy: after the model has repeatedly agreed with a user, that established context can increase the chance that it agrees to a later request it should reject.
Multi-turn red teaming is more effective because it teaches an attack strategy
Ron says single-turn attacks are easier to block than conversations that build rapport. An attacker can begin with reasonable requests, receive agreement, and then increase the demands. The model sees the whole thread, so previous acceptance changes the context for the later request. Ron recommends testing these patterns with automated red teaming, drawing on work by Ethan Perez and research groups such as Anthropic. A red-team model can receive feedback about which prompts caused a failure and learn the relationship between prompts and undesirable outputs. For subjective targets such as brand damage, a human must label what counts as unacceptable. Ron calls this semi-automatic red teaming, because the generation can be automated while the judgment remains with a person.
Safety controls need to account for bad output that is discovered only after generation
Ron says LLM systems differ from ordinary classifiers because teams may not know an output is bad until the model has generated it. A streaming response may begin appearing and then be dropped when a detector finds a problem. Teams can check inputs for jailbreaks before making a call, inspect generated output, and use classifiers or validators around the model. Monitoring can identify users who repeatedly probe for weak spots. Ron suggests adding cooldowns, warnings, temporary blocks, or other friction because the attacker's most valuable resource is feedback about what works. He also notes the tradeoff: checking every input and output adds inference cost and latency, and users may prefer a fast imperfect response to a slower system that performs extensive safety checks.
Natural-language policies can become classifiers through generated and measured examples
Ron describes what he calls "warm start policies," where a product manager writes down a policy in ordinary language and an LLM generates examples of acceptable and unacceptable behavior. He compares the idea with Anthropic's constitutional AI, while stressing that his approach works as a black-box method without changing the underlying model. The examples need enough variety. Ron proposes measuring embedding distances and graph structure to estimate whether a dataset is clumpy or degenerate. A smaller set that describes a larger set may indicate limited diversity. The score can be fed back into an iterative generation loop, so the LLM creates examples that improve the score. The same setup can generate prompts that break a model, with a Boolean score indicating whether each attempt succeeded.
Indirect prompt injection makes retrieved content part of the attack surface
Ron describes a malicious instruction hidden in a web page or another retrieved document. When an LLM retrieves it through browsing or RAG, the instruction enters the model's context as material the system itself found. That can make it more persuasive than the same text supplied directly by a user. He gives a scenario where a retrieved payload asks the model to disclose sensitive database information. Demetrios Brinkmann connects this with poisoning web content that crawlers may collect. Ron extends the concern to code-generation systems: an attacker could seed public repositories or web pages with vulnerable code patterns, hoping data-hungry training pipelines absorb them. The model could then reproduce a subtle vulnerability in software used in a critical product.
"The data about your reactions of your LLM, the data about what actually works, that essentially labeled data, that's your most important piece."Ron Heichman1:00:59
Who should watch
You are building an LLM or RAG product and need a practical way to inspect what your framework actually sends to the model.
Your team is testing jailbreaks, prompt injection, or data poisoning and needs to understand why multi-turn and indirect attacks are hard to catch.
You need safety policies for a narrow product concern, such as brand damage, without fine-tuning or changing the underlying model.
