Meta uses separate Perceiver, Retriever, and Reasoner agents to detect modality misalignment and stolen short-form video content.
2
Small, fine-tuned models reduce cost and improve specialization compared with sending every video to one large language model.
3
Frame merging, semantic hashing, dynamic routing, and metadata pruning let the system avoid unnecessary video processing and reasoning.
Summary
Aditya Gautam describes an application-layer multi-agent system for two short-form video problems: modality misalignment and original-content attribution. A video can contain a brief policy-breaking segment that conflicts with its surrounding content, or a creator can re-upload someone else's work with small edits. Meta's design separates the work into a Perceiver that extracts video signals, a Retriever that searches indexed content and relationships, and a Reasoner that judges the evidence. Each agent uses a model tuned to its task, with the Reasoner able to request more context. Gautam explains why one large LLM is too expensive, has modality bias, and lacks access to the wider video corpus without external tools. He also covers evaluation, logging, data drift, latency, labeling, and four cost controls. The talk is practical about the trade-offs: multi-agent systems add latency and operational complexity, so routing and caching are necessary at scale.
Short-form video changes too quickly for clean web-trained assumptions
Gautam says short-form video is dynamic because patterns change, AI-generated videos appear, and users try new ways to game systems. User-generated content is messier than the clean web data used for many LLMs and CLIP models. A creator can download a video, re-upload it, edit it, transform it, and present it as original. This creates both integrity problems and a difficult operating environment for models that need to work across many formats and behaviors.
Modality misalignment can hide inside an otherwise normal video
One problem occurs when the text describes nature or another harmless subject while the video shows an advertisement, political content, or another policy-breaking subject. Gautam sets that aside as a problem existing models can address with embedding alignment. The harder case is within-modality intrusion: an unrelated advertisement or agenda appears briefly inside an otherwise consistent video. In a long video, these short segments become a needle-in-a-haystack problem, with several separate violations potentially hidden in different clips.
Original-content attribution requires access to the wider video corpus
The second problem is attribution in what Gautam calls a copycat economy. People can take another creator's content, make small changes, and publish it as their own to gain followers or monetize viral material. A standalone LLM cannot compare the input with videos created earlier by other users unless it can access external databases and retrieval systems. The system therefore needs indexed video content, metadata, embeddings, and other context outside the current request.
The Perceiver agent turns raw video into searchable signals
The Perceiver processes a video at a granular level. It uses visual-language and discriminative models tuned for the task, divides the video into frames, identifies scene changes, and sends useful representations to a VLM. It stores embeddings in vector databases and gathers OCR, topic information, natural-language descriptions, and other signals in databases or blob storage. This first stage performs the signal acquisition and preprocessing that later agents need.
The Retriever maximizes the evidence available to the final decision
The Retriever receives the video and the signals produced by the Perceiver, then searches the offline-processed databases. It uses nearby vectors and KNN embeddings to find similar videos. It also considers topics, OCR, similarity matrices, knowledge databases, and relationships such as creator-to-creator and user-to-user interactions. Gautam says the goal is to improve recall by extracting similar content across different modalities and attributes before the Reasoner makes a decision.
The Reasoner can ask for more context when the evidence is weak
The Reasoner receives descriptions, embeddings, similarity scores, and other results from the earlier agents. It uses a larger model to assess whether a modality problem or content theft is present, produce a confidence score, and identify possible original sources from video metadata. If the evidence is insufficient, it can ask a context builder for more videos and content. It can request a wider search even when the initial similarity threshold would have returned less data.
Specialized agents allow independent models and deployments
Gautam compares the agents with software microservices. Each can have its own CI/CD process, sandbox, and fine-tuned language model. The Perceiver handles signal acquisition and multimodal understanding, the Retriever handles search, and the Reasoner handles interpretation. Gautam says models in the 4B to 11B range can improve pipeline quality while controlling cost because they focus on defined tasks. Dynamic routing can also skip the expensive Reasoner for an obvious copy.
Evaluation must cover retrieval, reasoning, hallucinations, and every tool call
The system uses precision, recall, F1 score, and confusion matrices for its primary decisions. Retrieval is evaluated separately because weak retrieval will damage the final result even if the other agents perform well. Reasoning quality is checked with an LLM judge and human review, while production samples are examined for hallucinations, such as claims about content absent from the video. Logging covers API calls, token use, latency, cost, tools, and every hop in the pipeline.
Four optimizations remove work before expensive inference begins
Gautam describes spatial and temporal frame merging to reduce repetitive visual input. A five-minute scene with roughly one frame per second can produce hundreds of similar frames, although a much smaller sample may provide the same understanding. Semantic hashing lets the system find already-indexed viral content and bypass the full pipeline. Dynamic routing assigns different reasoning budgets based on complexity. Metadata pruning uses topics, objects, creator history, and content signals to skip videos with a high initial likelihood of being safe or original.
"Decomposability is key when you are building a really complex solution to a really complex problem."24:43
Who should watch
You are building video integrity or content-understanding systems and need a concrete division of work between extraction, retrieval, and reasoning.
Your current design sends every multimodal request through one expensive model and you want examples of where caching, routing, and pruning can remove inference.
You need an evaluation plan that separates retrieval quality from final classification and tracks latency, token use, API calls, and hallucinations.
