The benchmark tests whether agents follow long, changing policy handbooks while completing tasks in a sandbox.
2
Strict evaluation gives the best reported model a 36.2% success rate, while allowing one missed criterion produces much higher scores.
3
The discussion argues that policy checks should sit outside the model, using deterministic guards, model-based guards, repeated guidance, or final-state verification.
Summary
The paper "Measuring Agent Policy Compliance" presents a benchmark for testing whether AI agents follow policies during realistic, multi-step tasks. The reading group explains how the benchmark uses changing handbooks, domain-specific tasks, sandboxed environments, fixed tools, and Python-based verification rather than an LLM judge. Under strict grading, the best reported model succeeds on 36.2% of tasks, while most other frontier models score below 25%. The discussion examines recurring failure modes: agents ignore a check, accept an unauthorized instruction, make a reasoning error, or report success after violating the handbook. The presenters argue that a handbook does not remain a persistent authority throughout a task. They discuss placing deterministic or model-based guards before important actions, calling a guiding agent at additional points, and checking the final environment state. The group also questions whether the benchmark reflects enterprise workflows and whether strict policies reduce useful flexibility.
The benchmark targets policies that live outside the user request
The benchmark studies tasks where rules are stored in handbooks or instruction files rather than stated directly in the request. One example asks an agent to start an employee offboarding process while a human-resources policy forbids involuntary offboarding without written authorization from specified people. The policy documents change between tasks, so an agent cannot simply memorize a handbook. The presenter says this matters because longer, more complex tasks contain more rules, and current models often fail to follow them even when the rules are available.
The task design combines changing handbooks with code-based grading
The benchmark contains 65 tasks across domains such as finance, HR, insurance, and logistics. Domain experts wrote base handbooks, and the benchmark creators generated variants with changes such as different approval thresholds. Each task includes a rubric with programmatic criteria. The environment runs in its own Docker container and exposes the same 82 tools across tasks. The agent can work with systems such as email and calendars, while the evaluator checks the resulting state or output with Python code. The presenters value this design because there is no LLM judge deciding whether the agent succeeded.
The reported best result under strict grading is 36.2%, while most other frontier models are below 25%. Strict grading requires the agent to avoid prohibited actions and complete required actions. Allowing one missed criterion makes results much higher, sometimes roughly doubling them, but the presenter argues that this changes the meaning of compliance when a missed approval or limit can cause real harm. Increasing reasoning effort improves some results by only 2% to 3%, gives no improvement for some models, and can make results worse for others. The paper also reports large differences in token use for similar performance.
Agents fail through reasoning errors and ignored checks
The reading group describes several recurring failure patterns. In one finance task, the agent misidentifies a junior analyst as a finance controller and treats an unauthorized approval as valid. In an HR task, it follows a request from an unauthorized executive even though the handbook names only two authorized roles. In a pharmacy task, it sends laboratory results that should have been held and fails to put the case on hold. These failures can occur even when the agent appears to have read the relevant rule. The presenter says agents often finish with a confident report that claims the handbook was followed, including references to rules they violated.
The handbook loses authority as the task progresses
The discussion presents the handbook as one source of authority among many in the environment. It does not reliably remain the authority against which every candidate action is screened. Messages, tool results, and other environmental content can override or distract from the handbook. The presenters also discuss the difference between reading a long document and actively retaining the right rule while working. They do not settle whether failures come mainly from context length, retrieval, model limitations, task complexity, or some combination.
Guards can block prohibited actions before they happen
The paper recommends compiling policies into deterministic guards placed before important write or mutation actions. A guard can inspect the proposed tool call and current state, then block the call when a condition fails. Numeric or time-based rules, such as an approval limit or a six-month age limit, are natural candidates for deterministic code. The presenters also warn that guards can contain mistakes. One cited study included a guard that blocked 42 calls, although only two of those calls were actually correct to block.
Model-based guards add interpretation but remain vulnerable
A model-based guard can read the conversation and decide whether to block a proposed action, then explain the reason to the main agent. This approach can handle rules that need interpretation, such as whether an email amounts to an unauthorized firing instruction. It also inherits model risks. Because the guard reads the conversation, malicious or misleading tool output can influence its decision. The presenters contrast this with deterministic guards, which are cheaper and repeatable for simple conditions but cannot handle every policy question.
Final-state checks cannot catch every missing action
The group identifies a gap in action-level guards. A guard before sending an email can block the wrong email, but it is harder to detect that an agent failed to perform a required action, such as putting a case on hold. The benchmark's final grader can inspect the completed environment and compare required and prohibited state changes. The presenters connect this to back-end systems that check the final state of background work. They also discuss calling a separate guiding agent at several points during a task so it can remind the main agent about actions it has missed.
The benchmark raises questions about enterprise realism
Participants question whether benchmark tasks reflect how organizations actually work. Domain experts wrote the handbooks and tasks, but the group notes that benchmark cases must be simplified enough to explain and verify. A firing workflow that one agent can complete may not match a real enterprise process involving several people and approvals. Another participant asks whether tasks across finance, HR, insurance, logistics, and medical billing were designed to have comparable difficulty. The presenters note that logistics had the longest median handbook at 72 pages and the weakest performance, but the paper does not provide a detailed explanation for differences between domains.
