AI text detectors have unavoidable trade-offs between catching generated text and falsely accusing human writers.
2
Recursive paraphrasing can reduce the accuracy of current watermarking systems below 50% after two rounds.
3
AI red teaming has moved from manual prompts to automated attacks that can scale jailbreaks, hallucinations, and privacy attacks.
Summary
Vinu Sankar Sadasivan explains why AI-generated text detection remains difficult even when models use watermarks. He describes four detector types, including watermarks, trained classifiers, zero-shot statistical detectors, and retrieval systems. Larger language models can imitate human writing more closely, while paraphrasing can disrupt the signals detectors rely on. Vinu's research found that recursive paraphrasing can break current watermarking schemes after two rounds, although stronger detection would increase false accusations against human writers. He then describes the development of AI red teaming, from emotional or deceptive prompts to automated gradient-based suffix attacks such as GCG. Defenses include input and output filters, classifiers, internal activation monitoring, and circuit breaking. The conversation also covers transferability between models, hallucination attacks, privacy attacks, and differences between open and closed models. Vinu is direct about the limits: jailbreak prevention depends on how the model was trained, while the definition of harmful behavior remains difficult to specify.
AI text detectors use several different signals, and none is foolproof
Vinu describes four detector families. Watermarking changes how a model selects tokens. Trained detectors classify text as AI-generated or human-written. Zero-shot detectors inspect model statistics such as loss values, since generated text may have lower loss. Retrieval systems compare a candidate against stored AI-generated text. His paper tested attacks that make AI text look more human and human text look more like AI text. The central limit is a trade-off between false positives and false negatives. Larger language models make detection harder because they can imitate a person's style when given enough instructions or examples.
Watermarking works by steering generation toward a changing green list
Vinu explains watermarking with a vocabulary split. If a model has 50,000 possible words, the watermarking algorithm divides them into a red list and a green list, then favors green-list words as it generates text. A human writer does not know these lists and should use roughly equal proportions, while watermarked output contains many more green-list words. More advanced systems change the lists after every word. The previous token seeds the next partition, which helps preserve quality and lets the detector identify a statistical pattern. This dynamic design also means that changing one word affects the partition for the following word, while broader rewriting can disrupt the sequence.
Watermarks depend on model providers adopting compatible schemes
Demetrios Brinkmann points out that watermarking must begin inside the model. Vinu agrees and says it would not help if one leading model had a watermark while another did not, because attackers could choose the non-watermarked model. Open models that can be downloaded and run locally create another gap. Vinu specifically mentions Llama 3.2 as an example of a model that can be kept on a hard drive and used for plagiarism or scams without a watermark. He also says that many different watermarking schemes would make detection harder, since detectors would need to identify which system produced the text.
Paraphrasing breaks watermark patterns while creating an accuracy trade-off
Changing a few words is a weak attack against watermarking because the watermark can tolerate small edits. Vinu says an attacker may need to change almost 50% of the words to disrupt it through direct replacement, which would damage meaning and quality. Paraphrasing changes sentence structure, word order, voice, grammar, and synonyms, so it can disturb the dynamic red-list and green-list sequence. In his experiments, recursively paraphrasing generated text twice reduced watermarking accuracy below 50%. Making the watermark harder to evade would require labeling more semantically similar passages as watermarked, which raises the chance of falsely accusing a human writer.
Detection systems cannot improve both error types at the same time
Vinu frames detection as a type one and type two error problem. If a passage has 100 semantically similar alternatives, only a small number can be labeled as watermarked without causing many human texts to be falsely flagged. A paraphraser can then move from a watermarked passage to one of the alternatives that is not labeled. Increasing the number of alternatives marked as watermarked makes evasion harder, but also increases false positives. Vinu says even the best theoretical detector has an upper limit determined by the distance between the human and AI text distributions. Tools that detect whether text was paraphrased can reduce evasion, but they also hurt the rate of correct decisions.
Red teaming has moved from personal prompt tricks to automated attacks
Vinu describes early jailbreaks as manual prompt-engineering attempts. People used sympathy, family stories, school projects, and other emotional framing to persuade a model to answer a restricted question. Word filtering is inadequate because terms such as "bomb" and "screw" can have different meanings in different contexts. The process then became iterative: an attacker refined a prompt based on the model's refusals. Automated attacks made this more scalable. The GCG method adds a random-looking suffix and uses gradients to optimize the tokens so the model produces a desired response. The suffix may look meaningless to a person while still affecting the model.
Jailbreak defenses can inspect inputs, outputs, and internal activations
Model providers respond with several layers of filtering. A model can reject inputs with poor-quality or suspicious suffixes, although later attacks make the suffixes more readable. Llama Guard uses a classifier trained to label harmful and non-harmful prompts, and the classifier can include adversarial examples from earlier attacks. Systems may inspect both inputs and outputs when compute allows it. Other methods examine transformer activations and stop generation when a pattern suggests harmful behavior. Vinu calls this circuit breaking. He warns that using AI to defend AI creates dependence: if one model in the pipeline fails, other defenses may fail with it.
Model strength depends on the attack being tested and the training it received
Vinu says red teaming includes more than jailbreaks. His work also tests attacks that increase hallucinations and attacks that improve the performance of privacy methods such as membership inference. He describes Llama as relatively resistant to jailbreak attempts, while Claude, ChatGPT, and Gemini also perform well in that area. The comparison changes for other attacks. Llama was easier to push into generating fake news and showed more vulnerability to hallucination attacks than some other models. Vinu's examples include false claims about watermelon seeds or walking into a wardrobe. He says these weaknesses depend on fine-tuning, and a model deployed without training against a particular attack can remain exposed.
"We need to define the problem first which is very ambiguous because the context changes and the scope of harmful questions changes and things like that."Vinu Sankar Sadasivan47:42
Who should watch
You are evaluating AI-generated text detectors and need to understand why watermarking, paraphrasing, and false positives are connected.
You are building safety filters for language models and want examples of automated jailbreak, hallucination, and privacy attacks.
You are deciding whether open or closed models are safer for a product and need to distinguish jailbreak resistance from other forms of robustness.
