AI assistants need integration contracts, regression tests, output controls, and guardrails before they can be trusted in regulated products.
2
LLM security should follow zero trust, with narrow permissions, restricted data access, and security testing across the whole application.
3
AI gives fraudsters more speed and creativity, so teams need red teaming, human review for risky actions, and continually updated detection data.
Summary
Rafael Sandroni discusses the overlap between AI security and fraud detection. Drawing on work with Siri, Nubank, and GardionAI, he argues that AI assistants require more than a capable language model. Teams need tested integrations, defined contracts, output controls, and guardrails for profanity, hallucinations, prompt injection, and regulatory risks. His main security principle is zero trust: restrict each system, agent, API, and data source to the permissions needed for a specific task. Rafael explains how indirect prompt injection can travel through email, documents, or other external content, and why a second LLM is a weak filter because it can be attacked in the same way. He recommends traditional machine-learning classifiers, red teaming, penetration testing, and human approval for high-risk actions. AI also gives fraudsters tools to create fake documents, clone voices, automate attacks, and test weaknesses faster. Security work therefore has to cover the whole system and keep collecting new attack data.
AI assistants need testing at every integration boundary
Rafael says the easy availability of LLM APIs means many people can build assistants, while dependable behavior remains difficult. A useful assistant needs orchestration across APIs and other systems, with tests that check those integrations. He recommends contracts between integrated components so that changes do not create regressions. At Citi, he worked on controls that checked outputs for issues such as profanity and references to competitors. Banks and other regulated companies also need control over how the assistant behaves. Rafael describes a mix of AI and human testing as part of this process. He treats testing and output guardrails as an early foundation for an assistant that creates value for users.
Traditional machine learning still matters around LLM applications
When Demetrios Brinkmann asks about the move from traditional machine learning to LLMs, Rafael says the two are connected. Teams may start with an API, then build internal models when they need more control. He suggests that guardrails can use traditional classifiers or small language models instead of relying only on a large model. The engineering problems also remain familiar. Teams still need to deploy systems, integrate components, and test how they work together. Rafael's fraud work at Nubank included automatically retraining models to detect new transaction and onboarding patterns and to improve model performance. The model type changes, but much of the surrounding software work does not.
Zero trust means limiting each AI action to its real task
Rafael applies a zero-trust approach to LLM systems. An assistant should receive only the permissions, credentials, API access, and tool access needed for a particular function. The same rule applies to data. Teams should decide which documents, tables, or database information the AI actually needs instead of exposing an entire database or every Slack channel. Demetrios Brinkmann points out the tension between restricted access and the fact that models often perform better with more context. Rafael accepts that the design is difficult, but keeps the recommendation simple: make data and permissions more restrictive. The assistant should not gain access to information that the person asking the question could not access directly.
Indirect prompt injection can enter through trusted external content
Rafael calls prompt injection the most difficult AI vulnerability he is seeing. He describes a Black Hat example involving Microsoft Copilot configured to summarize emails. An external sender hid instructions in an email using formatting that was difficult for a person to notice but readable by the AI. The injected instructions could cause the system to send phishing emails to users in the organization. This risk appears when an assistant reads content from the internet, email, documents, or other sources that the application does not own. Rafael recommends filtering external content and avoiding reliance on system prompts alone. Guardrails can reduce malicious patterns, but he is clear that they do not make the system completely safe.
Demetrios Brinkmann proposes a critic at the end of an agent workflow that reviews the prompt, retrieved information, actions, and result. Rafael agrees with the idea of a judge, then adds a limitation: the judge has the same vulnerability as the original LLM. A malicious input can affect the second model and create a cascade of prompt injection. Rafael therefore points to traditional machine-learning classifiers as a safer filter for some inputs because they do not follow instructions in the same way. He also warns against exposing another channel or model that gives the attack more room to spread. The control should match the threat instead of assuming that another LLM will understand the first model's behavior.
Banking agents need stronger controls than low-risk assistants
Rafael discusses fintech companies building banking assistants on WhatsApp. These assistants may access bank accounts, transfer money, or pay bills. He has seen scenarios where attackers tried to override a transfer value or an account balance. Additional banking infrastructure may provide validation, but Rafael still recommends zero trust and software-side protections. Demetrios Brinkmann contrasts this with a food-ordering assistant, where a mistaken recommendation is less damaging than sending the wrong amount of money to the wrong person. Rafael says banks may eventually treat an AI assistant as another banking channel, similar to a mobile application. The acceptable design depends on the customer profile, the task, and the risk the company is willing to accept.
AI gives fraudsters more speed, scale, and creativity
Rafael says AI gives both fraudsters and fraud teams new capabilities. Fraudsters can create fake documents, clone voices, automate attacks, and search for weaknesses faster. A process that once took a month can be tested in minutes. He also describes attacks against the AI system itself, such as prompt injection. This expands the fraud surface beyond traditional transaction or onboarding checks. The same dynamics still apply: attackers look for weaknesses while defenders learn new patterns and add protections. Rafael suggests that fraud operations' use of human review has a parallel in AI operations. People may need to approve or inspect actions when an automated system is handling money, sensitive data, or other high-risk work.
Security testing must cover the whole agent system
Rafael recommends red teaming and penetration testing before deploying a critical AI application. The exercise should examine the full system, including the LLM, integrations, permissions, data access, and the way information moves between agent steps. He cites an OWASP report about AI-agent vulnerabilities and explains that multiple LLMs make control harder. A prompt injection received in an early step can persist through later context and affect a final action. Rafael says access control is a difficult balance between agent autonomy and security. Each agent should have permissions restricted to its task and scope. He also recommends working with cybersecurity engineers, whose testing perspective covers the complete application rather than only the model.
Guardrail quality depends on private, continually updated attack data
Rafael says guardrails need data about the methods attackers use and may use to bypass them. Public data alone is insufficient, and it is difficult to publish useful attack datasets because the information can help attackers. Teams can collect examples through their own red teams and through collaboration with cybersecurity professionals. That data can support smaller models or classifiers for guardrails. Rafael compares this process with traditional fraud detection, where teams continually collect new patterns and retrain models. For AI applications, the update may involve changing prompts, model calls, or other parts of the application rather than fine-tuning the main LLM. His view is that guardrails require an ongoing process of learning and iteration.
"Don't use another LLM to filter malicious inputs because this another LLM can suffer and follow this malicious input as well."Rafael Sandroni20:21
Who should watch
You are building an AI assistant that can call APIs, read external content, or take actions on a user's behalf.
Your team needs to set permissions and data boundaries for an agent used in banking, fintech, healthcare, or another regulated setting.
You are responsible for fraud detection and want to understand how prompt injection, voice cloning, fake documents, and automated attacks change the threat model.
