Skip to content
Agentic Control Plane
Benchmark series · Part 12 of 3
AgentGovBench →

Full scorecard: seven frameworks, 48 scenarios, one open benchmark

David Crowe David Crowe · · 18 min read
benchmark governance scorecard agentgovbench comparison
Share X HN LinkedIn

Seven agent frameworks, 48 governance scenarios each, run two ways: the bare framework, then the same framework with ACP attached. No model in the loop, so the runs repeat. Everything ran against the production gateway.

These are the April 2026 numbers, as of 20 April. We have not re-measured the per-framework runners since, and the scoring changed afterwards (runner-neutral scoring, a daily run). Read this as history. ACP’s current number is on the /benchmark page and is not comparable with the table below.

Framework Pattern Native + ACP
CrewAI Decorator (Python) 13/48 40/48
LangChain / LangGraph Decorator (Python) 13/48 40/48
Cursor MCP 13/48 37/48
Claude Code Hook (CLI) 13/48 43/48
OpenAI Codex CLI Hook (CLI) 13/48 43/48
OpenAI Agents SDK Proxy 13/48 45/48
Anthropic Agent SDK TS handler-wrapper 13/48 46/48
48 governance scenarios, scoredeach cell: scenarios passed out of 6 · columns 3–10 are frameworks with ACP attachedvanillaaudit-onlyCrewAILangGraphCursorClaude CodeCodexOpenAI SDKAnthropic SDKACP directAudit completenessAudit completeness · vanilla: 1/61Audit completeness · audit-only: 5/65Audit completeness · CrewAI: 6/66Audit completeness · LangGraph: 6/66Audit completeness · Cursor: 6/66Audit completeness · Claude Code: 6/66Audit completeness · Codex: 6/66Audit completeness · OpenAI SDK: 6/66Audit completeness · Anthropic SDK: 6/66Audit completeness · ACP direct: 6/66Cross-tenant isolationCross-tenant isolation · vanilla: 4/64Cross-tenant isolation · audit-only: 4/64Cross-tenant isolation · CrewAI: 4/64Cross-tenant isolation · LangGraph: 4/64Cross-tenant isolation · Cursor: 4/64Cross-tenant isolation · Claude Code: 4/64Cross-tenant isolation · Codex: 4/64Cross-tenant isolation · OpenAI SDK: 4/64Cross-tenant isolation · Anthropic SDK: 4/64Cross-tenant isolation · ACP direct: 4/64Delegation provenanceDelegation provenance · vanilla: 0/60Delegation provenance · audit-only: 5/65Delegation provenance · CrewAI: 2/62Delegation provenance · LangGraph: 2/62Delegation provenance · Cursor: 4/64Delegation provenance · Claude Code: 6/66Delegation provenance · Codex: 6/66Delegation provenance · OpenAI SDK: 6/66Delegation provenance · Anthropic SDK: 6/66Delegation provenance · ACP direct: 6/66Fail-mode disciplineFail-mode discipline · vanilla: 3/63Fail-mode discipline · audit-only: 4/64Fail-mode discipline · CrewAI: 6/66Fail-mode discipline · LangGraph: 6/66Fail-mode discipline · Cursor: 3/63Fail-mode discipline · Claude Code: 4/64Fail-mode discipline · Codex: 4/64Fail-mode discipline · OpenAI SDK: 6/66Fail-mode discipline · Anthropic SDK: 6/66Fail-mode discipline · ACP direct: 6/66Identity propagationIdentity propagation · vanilla: 0/60Identity propagation · audit-only: 6/66Identity propagation · CrewAI: 6/66Identity propagation · LangGraph: 6/66Identity propagation · Cursor: 5/65Identity propagation · Claude Code: 6/66Identity propagation · Codex: 6/66Identity propagation · OpenAI SDK: 6/66Identity propagation · Anthropic SDK: 6/66Identity propagation · ACP direct: 6/66Per-user policy enforcementPer-user policy enforcement · vanilla: 1/61Per-user policy enforcement · audit-only: 1/61Per-user policy enforcement · CrewAI: 6/66Per-user policy enforcement · LangGraph: 6/66Per-user policy enforcement · Cursor: 5/65Per-user policy enforcement · Claude Code: 6/66Per-user policy enforcement · Codex: 6/66Per-user policy enforcement · OpenAI SDK: 6/66Per-user policy enforcement · Anthropic SDK: 6/66Per-user policy enforcement · ACP direct: 6/66Rate-limit cascadeRate-limit cascade · vanilla: 3/63Rate-limit cascade · audit-only: 3/63Rate-limit cascade · CrewAI: 6/66Rate-limit cascade · LangGraph: 6/66Rate-limit cascade · Cursor: 4/64Rate-limit cascade · Claude Code: 5/65Rate-limit cascade · Codex: 5/65Rate-limit cascade · OpenAI SDK: 5/65Rate-limit cascade · Anthropic SDK: 6/66Rate-limit cascade · ACP direct: 5/65Scope inheritanceScope inheritance · vanilla: 1/61Scope inheritance · audit-only: 1/61Scope inheritance · CrewAI: 4/64Scope inheritance · LangGraph: 4/64Scope inheritance · Cursor: 6/66Scope inheritance · Claude Code: 6/66Scope inheritance · Codex: 6/66Scope inheritance · OpenAI SDK: 6/66Scope inheritance · Anthropic SDK: 6/66Scope inheritance · ACP direct: 6/66Total /4813294040374343454645
All 48 scenarios, 8 categories of 6, as of April 2026. Every framework starts at the same bare floor (13/48). Attaching ACP lifts each to 37–46, depending on how much of the call path its integration can see. Scores from the open agentgovbench harness; we graded them ourselves.

Two results matter.

  1. Every framework scores 13/48 bare. That is the same score as the vanilla runner, which is no framework at all. Out of the box, none of them writes a structured audit record or enforces a per-user policy.
  2. The integration pattern sets the score, not the framework. Decorator about 40, MCP 37, hook 43, proxy 45, handler-wrapper 46. The same /govern/tool-use endpoint sits behind all of them.

One thing the 13/48 does not say: that these tools have no safety controls. Claude Code, Codex and Cursor all have their own permission prompts. The benchmark does not score those. It scores identity, per-user policy, delegation, rate limits, audit, tenant isolation, outage behaviour and scope, and a permission prompt does none of that on its own.

The complete table

Every cell is <passed>/<total> for that category. Bold is best in class.

Category vanilla audit-only crewai_acp langgraph_acp cursor_acp claude_code_acp codex_acp openai_agents_acp anthropic_agent_sdk_acp acp (direct)
Audit completeness 1/6 5/6 6/6 6/6 6/6 6/6 6/6 6/6 6/6 6/6
Cross-tenant isolation 4/6 4/6 4/6 4/6 4/6 4/6 4/6 4/6 4/6 4/6
Delegation provenance 0/6 5/6 2/6 2/6 4/6 6/6 6/6 6/6 6/6 6/6
Fail-mode discipline 3/6 4/6 6/6 6/6 3/6 4/6 4/6 6/6 6/6 6/6
Identity propagation 0/6 6/6 6/6 6/6 5/6 6/6 6/6 6/6 6/6 6/6
Per-user policy enforcement 1/6 1/6 6/6 6/6 5/6 6/6 6/6 6/6 6/6 6/6
Rate-limit cascade 3/6 3/6 6/6 6/6 4/6 5/6 5/6 5/6 6/6 5/6
Scope inheritance 1/6 1/6 4/6 4/6 6/6 6/6 6/6 6/6 6/6 6/6
Total 13/48 29/48 40/48 40/48 37/48 43/48 43/48 45/48 46/48 45/48

Pattern notes

Same backend, different scores. The difference is how much of the call path each integration shape can see.

Decorator (about 40/48). CrewAI and LangChain/LangGraph wrap individual tool functions. Identity, audit, per-user policy and rate limits all pass. They lose delegation_provenance (2/6) and scope_inheritance (4/6) because a decorator cannot see the framework’s orchestration: Hierarchical Process handoffs in CrewAI, StateGraph state changes in LangGraph. In April both integrations were decorator-only (acp-crewai 0.1.0, acp-langchain 0.1.0).

MCP (37/48). In April, the Cursor runner only governed tools exposed through an MCP server. Cursor’s built-in tools (Edit, Read, Bash) never passed through MCP, so nothing could check them. That is a limit of the MCP shape, not something a gateway can fix from outside. Cursor has since gained hooks, and ACP installs there as a hook today; the coverage matrix has the current row.

Hook (43/48). Claude Code and Codex CLI. Best on delegation_provenance and scope_inheritance, because the host’s hook payload carries the call chain. Each dropped one scenario, fail_open_honored, because the April hooks blocked every call when ACP was unreachable. That has changed; see the next section.

Proxy (45/48). OpenAI Agents SDK. The proxy sits where the request is serialised and sees the whole HTTP envelope. It made the same fail-closed choice as the April hooks.

TS handler-wrapper (46/48). Anthropic Agent SDK. A single-agent loop with a native dispatch boundary, and both fail modes honoured. The least framework code between the wrapper and the tool, so the least context lost.

What happens when ACP is down, today

The benchmark’s fail-mode scenarios ask one question: when the gateway cannot be reached, does the integration do what it said it would do? The April hooks always blocked. The current behaviour depends on which hook is running.

The Claude Code plugin fails open only for an interactive session, and it prints a loud UNGOVERNED notice when it does. Subagent, background and API sessions stay blocked until the gateway answers. To block interactive calls during an outage too, run echo closed > ~/.acp/failmode (or set ACP_FAIL_MODE=closed) on that machine. The shared hook the installer gives Cursor, Codex and the other hook-file harnesses reads the same setting. The safety floor runs on the machine either way.

The 13/48 floor

Seven frameworks, all at 13/48 bare. The same as no framework at all.

No framework’s tool dispatch writes structured audit data unless you wire a callback. None enforces per-user policy or rate limits without an integration. The categories vanilla “passes” (4/6 cross-tenant, 3/6 rate-limit) pass because those scenarios check that a benign call goes through. The ones it fails outright (0/6 on identity propagation and delegation provenance) need enforcement nothing provides by default.

If you have heard “framework X has audit”: that audit depends on you wiring the handler. Out of the box, it is vanilla.

Audit vs enforcement

vanilla to audit_only (13 to 29) is what a logging library buys you. Every call is recorded with who made it, where it came from and when. Nothing is denied and nothing is rate-limited.

audit_only to ACP (29 to 40–46) is enforcement: the sixteen scenarios that ask whether the bad call was stopped, not just written down.

The spread among the ACP runners (40 to 46) is integration tax, the context lost between the wrapper and the gateway. A native dispatch boundary loses the least (46). A decorator that cannot see orchestration loses the most (40).

Scenarios that did not pass

As of April, ACP itself missed a few scenarios, and every result file lists them. Three stand out.

  • cross_tenant_isolation.03_user_scope_does_not_leak and .05_admin_cannot_cross. The gateway code existed but the production deployment was not yet running in multi-tenant mode. The daily run passes both today.
  • scope_inheritance.04_task_narrowing. A sub-agent handed a narrower task should be held to that task, not to everything its parent could do. ACP still does not do this; it is counted as a failure in the current score too.

On top of that, fail_open_honored did not pass for Claude Code, Codex CLI or the OpenAI Agents SDK, because all three failed closed in April.

Run it on your stack

This is the April setup for the per-framework runners. The current ACP score is reproduced differently: one script, ./scripts/reproduce.sh, with two benchmark workspace keys from the console. The /benchmark page has that command.

git clone https://github.com/agentic-control-plane/agentgovbench
cd agentgovbench
python -m venv .venv && source .venv/bin/activate
pip install -e .

# Install the framework you want to benchmark
pip install crewai acp-crewai            # or whichever
# or for TS frameworks: npm install in a separate workspace

# Point at YOUR ACP project (Firebase service account, see README)
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/firebase-creds.json
export AGB_TENANT_ID=...
export FIREBASE_WEB_API_KEY=...

# Run
python -m benchmark.cli run --runner crewai_acp --out my-result.json

A different number than ours means your ACP version differs or you found a gap we have not. Either way, file an issue at the agentgovbench repo.


Receipts:

Running coding agents? ACP records each tool call its hook sees and makes the dangerous shell commands (a force-push, destructive SQL, curl | sh) wait for you. One command, free up to 5 agents:

macOS · Linux · WSL
curl -fsSL https://agenticcontrolplane.com/install.sh | bash
Windows · PowerShell
irm https://agenticcontrolplane.com/install.ps1 | iex

What the installer writes  ·  Getting started

Share X HN LinkedIn
Get the next data drop
What agents actually cost, new tool-surface captures, and the occasional incident post-mortem — sent when we publish something worth your inbox, not on a schedule. Unsubscribe anytime.
Share: Twitter LinkedIn
More in AgentGovBench
  1. 7. Is --dangerously-skip-permissions Safe? What the Flag Turns Off
  2. 12. Full scorecard: seven frameworks, 48 scenarios, one open benchmark · you are here
  3. 18. Seven agent frameworks, one backend, governance diverges on 9 of 48 tests
Related posts

← back to blog