# Transforming AI Safety & Security

Manojkumar Parmar, AIShield, a Corporate Startup of Bosch | LLMs in Production 2023 | 23:33

Source: https://www.youtube.com/watch?v=XDvvB-DkmRw
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/transforming-ai-safety-security
Published: 2023-07-28
Tags: governance, guardrails, security

## TL;DR
- AIShield.GuArdIan places a policy-enforcing layer between enterprise applications and large language models, checking both prompts and responses.
- The product blocks harmful, confidential, unauthorized, or policy-violating requests while allowing different roles, such as doctors and compliance officers, to receive different access.
- Manojkumar Parmar presented early jailbreak tests in which Guardian increased the blockage rate from 22% to 74% for Azure OpenAI and from zero to 44% for DALL-E 2.

## Summary
Manojkumar Parmar presents AIShield.GuArdIan as a control layer for enterprise use of generative AI. He describes the problem as a choice between adopting LLMs without enough protection and banning them so completely that the business misses their benefits. Guardian is placed between users or applications and an LLM provider. It checks prompts before they leave the organization, then checks generated responses before they reach the user. Policies can cover legal, ethical, role-based, usage-based, privacy, and intellectual-property concerns. Examples include blocking unsafe medical requests, restricting patient information by role, preventing proprietary code from being sent to an LLM, and stopping requests to use unapproved manufacturing parts. Parmar also explains the product architecture, including policy mapping, classification models, fine-tuned Guardian models, prompt engineering, decision aggregation, and logging. His early jailbreak results are promising but incomplete, and he openly says the system does not stop every attack.

## Key ideas
### Enterprise LLM adoption is caught between unrestricted use and prohibition
[03:08](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=188s)
Parmar describes two common responses to generative AI. Some organizations use tools such as ChatGPT while accepting the risks of intellectual-property loss, privacy problems, compliance violations, and reputational damage. Others prohibit large language models and lose possible productivity and business benefits. He argues that organizations need a middle layer that allows adoption while controlling the risks. Guardian is presented as that balancing mechanism, rather than as a reason to stop using LLMs.

### Guardian checks both incoming prompts and outgoing answers
[06:35](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=395s)
Guardian is deployed as a demilitarized zone between an application and an LLM provider. When a user submits a prompt, the system checks it against configured policies. A disallowed prompt is blocked before it reaches the model. If the prompt is allowed, the model generates a response, which Guardian checks again before returning it to the user. Parmar says this second check is needed because an acceptable question can still produce a response that violates legal, privacy, role-based, or organizational rules.

### Policies can control access according to the user's role
[12:07](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=727s)
In a medical example, Guardian blocks a doctor's request for unsafe prescribing guidance and prevents an unauthorized doctor from retrieving confidential information. A compliance officer or auditor may still receive the information needed to review prescriptions. The difference comes from role-based policy. Parmar uses this example to show that the system does not apply one blanket rule to every user. Access depends on the user's responsibilities and the context of the request.

### The product covers software, manufacturing, privacy, and jailbreak cases
[12:52](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=772s)
Parmar describes several applications for Guardian. It can block users from submitting proprietary source code to an LLM and can prevent generated code from being returned when copyright infringement is a concern. In manufacturing, it allows a question about using a cheap part in a boiler machine but blocks a request to use a non-approved part. He also shows multilingual privacy handling with a Hindi example and says Guardian identifies and blocks jailbreak attempts.

### Guardian is designed as a simple middleware integration
[15:38](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=938s)
The product is presented as middleware with a Python SDK. Parmar says developers can start using it with two lines of code and configure policies through the SDK. It supports dynamic policy enforcement and offline logging of explanations. The system can work across different LLM deployments and has partial support for large vision models for textual violations. The intended integration is a pass-through, except when a policy violation requires Guardian to stop the interaction.

### The architecture combines policy rules with several model-based checks
[17:29](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=1049s)
Under the hood, Guardian includes a policy table with role mapping, fine-tuned Guardian LLMs, domain-specific Guardian models, purpose-built foundation models, prompt-engineering components, classification models, and a decision aggregation mechanism. These components assess the context of prompts and responses, apply the configured policies, decide whether a violation occurred, and log the data. Parmar says the system uses different domain adaptations for healthcare, finance, and software engineering.

### Early jailbreak testing improved protection without eliminating attacks
[19:28](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=1168s)
Parmar reports an early benchmark using 50 crafted jailbreaks. Without Guardian, Azure OpenAI stopped 11 of the 50 attacks, a 22 percent blockage rate. With Guardian, the rate rose to 74 percent. For DALL-E 2, he reports no inherent jailbreak blockage in the tested setup, while Guardian reached 44 percent. He describes these results as early and says the system still needs more work, so the product is presented as an added defense rather than a complete guarantee.

### Parmar presents Guardian as one part of responsible LLM adoption
[21:52](https://www.youtube.com/watch?v=XDvvB-DkmRw&t=1312s)
In his conclusion, Parmar says generative AI can improve productivity and efficiency, but organizations also need to account for the risks that can delay or damage adoption. He describes training LLMs and balancing their use with organizational requirements as difficult and expensive. Guardian is offered as one way to let businesses experiment and use LLMs while applying compliance and safety controls. He explicitly says it is one answer among others, not the only answer.

## Notable quotes
- "The Guardian is the solution in a product which can ensure the security, data policy control for any kind of an LLM that you want to adopt in your organization." (06:35)
- "If it is not allowed, it will simply block and nothing happens." (08:35)
- "The output also gets blocked." (10:01)
- "Without Guardian, OpenAI APIs are able to stop 11 out of the 50 jailbreaks." (19:48)
- "Guardian is one of the answers, not the answer, to ensure that we can work in the best setting to mitigate this risk and still enjoy the benefits of generative AI." (22:33)

## Tools & references mentioned
- AIShield
- Bosch
- AIShield.GuArdIan
- ChatGPT
- GPT-J
- Azure OpenAI
- DALL-E 2

## Who should watch
- You are building an internal assistant, search tool, or coding application and need prompts and responses checked against company policy.
- Your organization is deciding whether to allow generative AI while controlling privacy, intellectual-property, medical, or compliance risks.
- You want an overview of policy enforcement, role-based access, and jailbreak filtering around LLM applications.

## Related talks

- [Harnessing AI APIs for Safer, Accurate, & Reliable Applications](https://mlopstalks.com/talks/harnessing-ai-apis-for-safer-accurate-reliable-applications) (Ron Heichman, SentinelOne, 1:08:14)
- [Guarding LLM and NLP APIs: A Trailblazing Odyssey for Enhanced Security](https://mlopstalks.com/talks/guarding-llm-and-nlp-apis-a-trailblazing-odyssey-for-enhanced-security) (Ads Dawson, Cohere, 59:41)
- [Responsible Autonomy: Building Governance Frameworks for AI That Act in the Real World via MCP](https://mlopstalks.com/talks/responsible-autonomy-building-governance-frameworks-for-ai-that-act-in-the-real) (Saurabh Mishra, Optum, 27:55)
- [LLM Security](https://mlopstalks.com/talks/llm-security) (Raahul Dutta, Elsevier & Uri Shamay, Null & Sankalp Gilda, DevelopYours, 1:00:17)
- [Guiding LLMs While Staying in the Driver's Seat](https://mlopstalks.com/talks/guiding-llms-while-staying-in-the-drivers-seat) (Jacob van Gogh, Adept AI, 10:02)