# AI For Good - Detecting Harmful Content at Scale

Matar Haller, ActiveFence | MLOps Podcast | Episode 246 | 51:28
Hosted by Demetrios Brinkmann

Source: https://www.youtube.com/watch?v=wLKlZ6yHg1k
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/ai-for-good-detecting-harmful-content-at-scale
Published: 2024-07-09
Tags: evals, inference, monitoring, multimodal

## TL;DR
- ActiveFence gives platforms risk probabilities for harmful text, images, audio, and video, then lets customers decide which items to remove or send to moderators.
- Matar Haller says harmful-content detection depends on clear policies, specialist knowledge of languages and cultures, carefully labelled data, and models that work at low latency and high volume.
- ActiveFence continually audits production models with functional tests, false-positive checks, score distributions, and targeted retraining as language, behaviour, and content change.

## Summary
Matar Haller explains how ActiveFence helps online platforms detect harmful content without making the moderation decision for them. Its API accepts text, images, audio, and video in many languages and returns a probability of risk. Customers define their policies, thresholds, and review workflows. ActiveFence combines specialist policy knowledge, intelligence analysts, data operations, and several model types to handle content at scale. Haller describes functional tests that target weak spots such as reclaimed slurs, food-related false positives, and new forms of bullying. The company also uses go or no-go tests, benign datasets, load tests, production audits, and score-distribution checks. Haller is clear that the work involves difficult tradeoffs between precision, recall, cost, and speed. She also discusses using AI earlier in the generation pipeline through partnerships with Cohere and NVIDIA, while keeping large language models out of inference because of their cost, latency, and reliability limits at the required scale.

## Key ideas
### Online harm can move into offline life
[04:03](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=243s)
Haller says trust and safety matters because online harm can lead to offline harm. She points to misinformation tracked before January 6 and notifications sent to platforms before the attack on the U.S. Capitol. Online abuse also affects ordinary users through comments, chats, videos, and images. She connects the issue to her own family, saying her nine-year-old daughter asks for a cellphone while Haller would rather wait much longer. Platforms now face more public, regulatory, and user attention because people encounter hate speech, bullying, and vile material online, while moderation often fails in an adversarial environment.

### The service scores many kinds of content for platform-specific policies
[06:19](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=379s)
ActiveFence does not perform the final moderation action. Platforms send content through an API, and ActiveFence returns a probability of risk. The inputs can include audio, video, images, comments, chats, usernames, descriptions, titles, podcasts, and short-form content. A customer can define its own policy, such as a children's platform banning nudity, swearing, and bullying. The platform can automatically remove items above a chosen threshold, ignore low-risk items, and send the middle range to expert moderators. Haller says this also reduces the amount of disturbing material moderators must view.

### Language, culture, and adversarial behaviour make detection difficult
[06:58](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=418s)
The same word or phrase can mean different things across languages and cultures. Haller gives the example of calling someone a dog, which can have different meanings depending on the language. Models also need to recognise slang, hate-group logos, known bad actors, and changing forms of abuse. Bad actors deliberately evade detection by using altered phrases, leetspeak, and obscured content. ActiveFence works with policy managers, intelligence analysts, and subject-matter experts who understand specific groups, languages, and edge cases. Haller says a clear policy must come before labelling and evaluation because it shapes every later stage.

### Risk workflows combine fast checks with heavier models
[11:50](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=710s)
ActiveFence uses different approaches depending on the task and the customer's needs. Keyword and emoji matching can be fast, inexpensive, and effective for some cases, including drug trafficking and child sexual abuse. Other workflows can trigger a heavier model after an initial signal. Image and video processing may take longer because the system breaks content into components and checks for logos, faces, victims, nudity, and other indicators. Haller says the company uses transformer models, ensemble models, and other approaches while trying to balance accuracy, cost, latency, and scale.

### Functional tests expose failures that aggregate metrics can hide
[15:57](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=957s)
Haller recommends functional tests alongside precision and recall. These tests target specific parts of a policy, such as reclaimed slurs, direct slurs, harassment, or bullying. A model may have strong overall metrics while consistently failing on one functionality, such as wishing cancer on someone. Targeted tests reveal that weakness and guide the team toward collecting data for the particular failure. ActiveFence can create data manually or with help from a large language model. Haller also says generated benign examples can help reduce bias when a dataset contains many negative examples about one ethnicity or group.

### Production monitoring catches drift and misleading correlations
[20:57](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=1257s)
Before deployment, ActiveFence backtests models on production data, examines score distributions, and reviews examples whose predictions changed. After deployment, trained analysts audit real production data, sometimes using customer moderator decisions as labels. One example involved a food-delivery company where a hate-speech model flagged phrases such as 'I hate Indian food' because it focused on terms associated with ethnicity rather than the surrounding meaning. The team used that pattern to start targeted model improvement. Haller says the company keeps auditing and retraining because new slang, new content types, and changing user behaviour can expose blind spots.

### AI-generated content requires checks at the source and on platforms
[25:36](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=1536s)
Haller says harmful content remains harmful whether a person or a generative model created it, so detecting whether something is AI-generated is less important for ActiveFence's use cases. Generative tools have made it possible to create malicious content at both high quality and high volume, increasing the need for automated detection. ActiveFence works with Cohere to help make models safer and with NVIDIA through NeMo Guardrails, where its risk-scoring models can check prompts and outputs. Haller says the company needs to work both at model-training sources and on the platforms where user content appears because open-source models and foundation models can still produce unsafe material.

### Large language models are used around inference, not for high-volume prediction
[32:46](https://www.youtube.com/watch?v=wLKlZ6yHg1k&t=1966s)
ActiveFence uses generative models for tasks such as data generation and red teaming, but Haller says it does not currently use large language models for inference. At the required scale, she considers them too costly, slow, and less predictable for the prediction path. The production pipeline instead uses purpose-built components, such as computer vision models that extract text or image elements and models that score those signals. The company runs its models on AWS using Kubernetes and builds its inference infrastructure in-house to avoid paying an overhead for every inference.

## Notable quotes
- Matar Haller: "Online harm can lead to offline harm." (04:07)
- Matar Haller: "The only way to deal with this massive amount of content at scale is with models that are able to do the work faster and more efficient and with less mental harm than humans." (11:05)
- Matar Haller: "Our data is our gold." (48:41)
- Matar Haller: "We care about not only the users' well-being but also the moderator's well-being." (09:46)

## Tools & references mentioned
- ActiveFence
- Cohere
- NVIDIA
- NeMo Guardrails
- AWS
- Kubernetes
- January 6
- Digital Safety Act

## Who should watch
- You are building moderation or trust-and-safety systems and need to understand how policy decisions become model tests and production workflows.
- Your models work across languages, cultures, or fast-changing user behaviour, and aggregate precision and recall do not explain where they fail.
- You manage data scientists and ML engineers who need a practical path from experimentation to low-latency model serving.

## Related talks

- [If There's Free Compute, There's Abuse: Fighting Fraud with Lightweight LLM Agents](https://mlopstalks.com/talks/if-theres-free-compute-theres-abuse-fighting-fraud-with-lightweight-llm-agents) (Jonas Scholz, Sliplane, 15:07)
- [Harnessing AI APIs for Safer, Accurate, & Reliable Applications](https://mlopstalks.com/talks/harnessing-ai-apis-for-safer-accurate-reliable-applications) (Ron Heichman, SentinelOne, 1:08:14)
- [Transforming AI Safety & Security](https://mlopstalks.com/talks/transforming-ai-safety-security) (Manojkumar Parmar, AIShield, a Corporate Startup of Bosch, 23:33)
- [Hundreds of Users Love Our Data Analyst AI Agent](https://mlopstalks.com/talks/hundreds-of-users-love-our-data-analyst-ai-agent) (Ioannis Zempekakis & Donné Stevenson, 29:00)
- [A Playground for AI Engineers](https://mlopstalks.com/talks/a-playground-for-ai-engineers) (Paulo Vasconcellos, Hotmart, 54:42)