# Structured Dissent Patterns for Agentic Production Reliability

Phil Stafford, MLOps Community | Agents in Production 2025 | 28:13

Source: https://www.youtube.com/watch?v=blOifXIJLe4
Channel: MLOps Community, now AAIF Live (https://www.youtube.com/@AAIFLive-x1r). Summarised by MLOps Talks.
Page: https://mlopstalks.com/talks/structured-dissent-patterns-for-agentic-production-reliability
Published: 2026-01-12
Tags: agents, evals, human-in-the-loop, multi-agent, security

## TL;DR
- Single-agent systems can give confident answers without expressing uncertainty or exposing what they missed.
- Structured dissent uses believers, skeptics, and neutral agents to debate a decision before producing a confidence score and report.
- Multi-agent debate is appropriate for high-stakes, ambiguous decisions, but it adds time and cost and should not be used for simple or time-critical tasks.

## Summary
Phil Stafford presents structured dissent, a multi-agent pattern for decisions where there is no simple ground truth. A default swarm uses two believer agents, two skeptics, and one neutral agent. They analyze an input in parallel, debate each other's conclusions, and produce a report with a majority opinion, confidence score, minority views, resolved points, and unresolved conflicts. Stafford demonstrates the pattern on MCP server security findings from static analysis and dependency checks. The example takes about three and a half minutes, produces a confidence score, and can route uncertain results to human review. He is honest about the tradeoffs: the system costs more, takes longer, and can lose quality when reduced from five agents or three debate phases. He also argues that evaluation must match the task. A benchmark designed for ground-truth answers judged the swarm poorly, while task-aligned evaluation of nuanced answers produced a much better result. The pattern fits security reviews, deployment approvals, change reviews, and other decisions where a wrong answer is expensive.

## Key ideas
### Single agents can fail without showing uncertainty
[00:55](https://www.youtube.com/watch?v=blOifXIJLe4&t=55s)
Stafford says a single LLM can hallucinate while sounding certain that its answer is correct. In a security review, it might miss hard-coded keys or SQL injection vulnerabilities and give no indication that it missed them. A single agent also offers only one model's perspective, does not reliably challenge its own conclusion, and does not provide an uncertainty qualification. Stafford says newer self-correction techniques may help, but the system he presents uses several agents to create disagreement before a decision is made.

### Structured dissent separates optimism, skepticism, and balance
[03:22](https://www.youtube.com/watch?v=blOifXIJLe4&t=202s)
The swarm instantiates three types of agents. Believers are solution-focused and look for opportunities. Skeptics look for failure modes and hidden problems. Neutrals bring the views together, reduce groupthink, and build a balanced conclusion. The default configuration has two believers, two skeptics, and one neutral. Stafford says the configuration is malleable, with lighter versions available. The agents receive personas with richer backstories so they approach the same evidence from different viewpoints.

### Three debate phases turn disagreement into a report
[04:42](https://www.youtube.com/watch?v=blOifXIJLe4&t=282s)
The agents first analyze the input in parallel. They then see one another's analyses and argue about flaws, assumptions, and missing details. A final synthesis phase sends the discussion to a reporting layer. That report contains the majority opinion, confidence, minority opinions, points the team resolved, and conflicts it could not resolve. Stafford describes this as a way to identify when more information, more time, or a human decision is needed rather than forcing a false consensus.

### The MCP security demo uses debate around automated findings
[06:12](https://www.youtube.com/watch?v=blOifXIJLe4&t=372s)
Stafford feeds the swarm findings from MCP server security checks, including static analysis and dependency checks. The agents disagree about whether findings are safe, whether rollout should be phased, and whether database complexity or other hidden costs have been ignored. They work in parallel and then rebut one another. The example uses an MCP server from Anthropic and produces a professional report even though the underlying logs contain deliberately elaborate persona dialogue.

### Confidence scores can route uncertain decisions to people
[12:17](https://www.youtube.com/watch?v=blOifXIJLe4&t=737s)
In the demonstration, one skeptic returns a 58% confidence score and marks human review as required. Stafford describes a 50% to 75% range where the swarm asks for a human and passes the result to the first analyst layer. A later result reaches 76% and is approved automatically. He says the system is intended to help analysts do more work, not replace them. The confidence score gives a production system a signal about how much trust to place in its recommendation.

### The pattern adds measurable operating cost
[13:30](https://www.youtube.com/watch?v=blOifXIJLe4&t=810s)
The example processes about 35,000 characters and produces roughly 10,000 characters of output, with the report tailored for executives, security teams, technical teams, or a help desk. Stafford says the presentation version took about three and a half minutes to process and that a complex case can take 15 minutes. He gives an API cost of about $15 per analysis in the example, while comparing that with the time and cost of manual security analyst review. These costs are part of deciding whether the pattern fits a task.

### Evaluation must match decisions without ground truth
[15:02](https://www.youtube.com/watch?v=blOifXIJLe4&t=902s)
Stafford reports that ordinary LLM evaluations and a task-aligned evaluation gave the swarm a 92% result for answer quality in tasks without ground truth. A benchmark designed around a different interpretation misunderstood the architecture as fictional roleplay and penalized its use of debate. His conclusion is that evaluations for nuanced, qualitative answers cannot rely only on binary right-or-wrong checks. Ground-truth tasks can still be tested deterministically, but the test must reflect what the system is meant to do.

### Use the swarm when a wrong answer is expensive and ambiguity is real
[17:14](https://www.youtube.com/watch?v=blOifXIJLe4&t=1034s)
Stafford recommends the pattern for high-stakes decisions such as security assessments, deployment approvals, change reviews, and complex trade-off analysis. He gives a rule of thumb that a wrong answer costing more than $1,000 may justify the overhead. He advises against using it for simple lookups, low-stakes tasks with ground truth, or situations that require a very fast response. A calculator is enough for two plus two, and the swarm is poorly matched to questions with a single obvious answer.

### Optimization should follow reliability measurement
[19:18](https://www.youtube.com/watch?v=blOifXIJLe4&t=1158s)
Possible optimizations include using a faster model in the first phase, using multiple models, reducing the process to two phases, or reducing the swarm from five agents to three. Stafford says these changes can save cost or time, but they also reduce adversarial checking or diversity. His recommendation is to measure reliability first and optimize only after finding where the system can tolerate a reduction. He also mentions future work on dynamic personas and other sampling approaches.

## Notable quotes
- "Single agents fail silently. They are just certain they're right even if they're wrong." (21:02)
- "This is adversarial verification in real time." (07:46)
- "This isn't going to replace anybody. It is going to help them do more." (12:44)
- "I would not optimize until you just measure how reliable it is." (20:40)
- "You want the bestest answer. That's what you're looking for." (27:24)

## Tools & references mentioned
- MCP
- Model Context Protocol
- Anthropic
- Bandit
- Semgrep
- Syft
- Trivy
- Cursor
- VS Code
- GitHub
- Simply Cyber
- HackSmarter
- Black Hills Discords

## Who should watch
- You are building an agent system that makes security, deployment, or change decisions and need a way to expose disagreement before approval.
- Your application handles ambiguous questions where there is no single ground-truth answer, and you are deciding whether extra model calls are worth the cost.
- You are evaluating multi-agent systems and need an evaluation method that matches qualitative reasoning rather than only binary correctness.

## Related talks

- [Building Reliable Agents](https://mlopstalks.com/talks/building-reliable-agents) (Eno Reyes, Factory.ai, 24:45)
- [Beyond the Gold Standard: Evaluating and Trusting Agents in the Wild](https://mlopstalks.com/talks/beyond-the-gold-standard-evaluating-and-trusting-agents-in-the-wild) (Sanjana Sharma, Prosus, 24:45)
- [Building Agentic Tools for Production](https://mlopstalks.com/talks/building-agentic-tools-for-production) (Sam Partee, Arcade AI, 23:55)
- [Sub-Agent Architectures: What You Can Leverage](https://mlopstalks.com/talks/sub-agent-architectures-what-you-can-leverage) (Sidd Seethepalli, Vellum, 14:48)
- [Evaluation of Agentic System](https://mlopstalks.com/talks/evaluation-of-agentic-system) (Aditya Gautam, Meta, 28:21)