Full scorecard: seven frameworks, 48 scenarios, one open benchmark
Seven agent frameworks, 48 governance scenarios each, run two ways: the bare framework, then the same framework with ACP attached. No model in the loop, so the runs repeat. Everything ran against the production gateway.
These are the April 2026 numbers, as of 20 April. We have not re-measured the per-framework runners since, and the scoring changed afterwards (runner-neutral scoring, a daily run). Read this as history. ACP’s current number is on the /benchmark page and is not comparable with the table below.
| Framework | Pattern | Native | + ACP |
|---|---|---|---|
| CrewAI | Decorator (Python) | 13/48 | 40/48 |
| LangChain / LangGraph | Decorator (Python) | 13/48 | 40/48 |
| Cursor | MCP | 13/48 | 37/48 |
| Claude Code | Hook (CLI) | 13/48 | 43/48 |
| OpenAI Codex CLI | Hook (CLI) | 13/48 | 43/48 |
| OpenAI Agents SDK | Proxy | 13/48 | 45/48 |
| Anthropic Agent SDK | TS handler-wrapper | 13/48 | 46/48 |
Two results matter.
- Every framework scores 13/48 bare. That is the same score as the
vanillarunner, which is no framework at all. Out of the box, none of them writes a structured audit record or enforces a per-user policy. - The integration pattern sets the score, not the framework. Decorator about 40, MCP 37, hook 43, proxy 45, handler-wrapper 46. The same
/govern/tool-useendpoint sits behind all of them.
One thing the 13/48 does not say: that these tools have no safety controls. Claude Code, Codex and Cursor all have their own permission prompts. The benchmark does not score those. It scores identity, per-user policy, delegation, rate limits, audit, tenant isolation, outage behaviour and scope, and a permission prompt does none of that on its own.
The complete table
Every cell is <passed>/<total> for that category. Bold is best in class.
| Category | vanilla | audit-only | crewai_acp | langgraph_acp | cursor_acp | claude_code_acp | codex_acp | openai_agents_acp | anthropic_agent_sdk_acp | acp (direct) |
|---|---|---|---|---|---|---|---|---|---|---|
| Audit completeness | 1/6 | 5/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| Cross-tenant isolation | 4/6 | 4/6 | 4/6 | 4/6 | 4/6 | 4/6 | 4/6 | 4/6 | 4/6 | 4/6 |
| Delegation provenance | 0/6 | 5/6 | 2/6 | 2/6 | 4/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| Fail-mode discipline | 3/6 | 4/6 | 6/6 | 6/6 | 3/6 | 4/6 | 4/6 | 6/6 | 6/6 | 6/6 |
| Identity propagation | 0/6 | 6/6 | 6/6 | 6/6 | 5/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| Per-user policy enforcement | 1/6 | 1/6 | 6/6 | 6/6 | 5/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| Rate-limit cascade | 3/6 | 3/6 | 6/6 | 6/6 | 4/6 | 5/6 | 5/6 | 5/6 | 6/6 | 5/6 |
| Scope inheritance | 1/6 | 1/6 | 4/6 | 4/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 | 6/6 |
| Total | 13/48 | 29/48 | 40/48 | 40/48 | 37/48 | 43/48 | 43/48 | 45/48 | 46/48 | 45/48 |
Pattern notes
Same backend, different scores. The difference is how much of the call path each integration shape can see.
Decorator (about 40/48). CrewAI and LangChain/LangGraph wrap individual tool functions. Identity, audit, per-user policy and rate limits all pass. They lose delegation_provenance (2/6) and scope_inheritance (4/6) because a decorator cannot see the framework’s orchestration: Hierarchical Process handoffs in CrewAI, StateGraph state changes in LangGraph. In April both integrations were decorator-only (acp-crewai 0.1.0, acp-langchain 0.1.0).
MCP (37/48). In April, the Cursor runner only governed tools exposed through an MCP server. Cursor’s built-in tools (Edit, Read, Bash) never passed through MCP, so nothing could check them. That is a limit of the MCP shape, not something a gateway can fix from outside. Cursor has since gained hooks, and ACP installs there as a hook today; the coverage matrix has the current row.
Hook (43/48). Claude Code and Codex CLI. Best on delegation_provenance and scope_inheritance, because the host’s hook payload carries the call chain. Each dropped one scenario, fail_open_honored, because the April hooks blocked every call when ACP was unreachable. That has changed; see the next section.
Proxy (45/48). OpenAI Agents SDK. The proxy sits where the request is serialised and sees the whole HTTP envelope. It made the same fail-closed choice as the April hooks.
TS handler-wrapper (46/48). Anthropic Agent SDK. A single-agent loop with a native dispatch boundary, and both fail modes honoured. The least framework code between the wrapper and the tool, so the least context lost.
What happens when ACP is down, today
The benchmark’s fail-mode scenarios ask one question: when the gateway cannot be reached, does the integration do what it said it would do? The April hooks always blocked. The current behaviour depends on which hook is running.
The Claude Code plugin fails open only for an interactive session, and it prints a loud UNGOVERNED notice when it does. Subagent, background and API sessions stay blocked until the gateway answers. To block interactive calls during an outage too, run echo closed > ~/.acp/failmode (or set ACP_FAIL_MODE=closed) on that machine. The shared hook the installer gives Cursor, Codex and the other hook-file harnesses reads the same setting. The safety floor runs on the machine either way.
The 13/48 floor
Seven frameworks, all at 13/48 bare. The same as no framework at all.
No framework’s tool dispatch writes structured audit data unless you wire a callback. None enforces per-user policy or rate limits without an integration. The categories vanilla “passes” (4/6 cross-tenant, 3/6 rate-limit) pass because those scenarios check that a benign call goes through. The ones it fails outright (0/6 on identity propagation and delegation provenance) need enforcement nothing provides by default.
If you have heard “framework X has audit”: that audit depends on you wiring the handler. Out of the box, it is vanilla.
Audit vs enforcement
vanilla to audit_only (13 to 29) is what a logging library buys you. Every call is recorded with who made it, where it came from and when. Nothing is denied and nothing is rate-limited.
audit_only to ACP (29 to 40–46) is enforcement: the sixteen scenarios that ask whether the bad call was stopped, not just written down.
The spread among the ACP runners (40 to 46) is integration tax, the context lost between the wrapper and the gateway. A native dispatch boundary loses the least (46). A decorator that cannot see orchestration loses the most (40).
Scenarios that did not pass
As of April, ACP itself missed a few scenarios, and every result file lists them. Three stand out.
cross_tenant_isolation.03_user_scope_does_not_leakand.05_admin_cannot_cross. The gateway code existed but the production deployment was not yet running in multi-tenant mode. The daily run passes both today.scope_inheritance.04_task_narrowing. A sub-agent handed a narrower task should be held to that task, not to everything its parent could do. ACP still does not do this; it is counted as a failure in the current score too.
On top of that, fail_open_honored did not pass for Claude Code, Codex CLI or the OpenAI Agents SDK, because all three failed closed in April.
Run it on your stack
This is the April setup for the per-framework runners. The current ACP score is reproduced differently: one script, ./scripts/reproduce.sh, with two benchmark workspace keys from the console. The /benchmark page has that command.
git clone https://github.com/agentic-control-plane/agentgovbench
cd agentgovbench
python -m venv .venv && source .venv/bin/activate
pip install -e .
# Install the framework you want to benchmark
pip install crewai acp-crewai # or whichever
# or for TS frameworks: npm install in a separate workspace
# Point at YOUR ACP project (Firebase service account, see README)
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/firebase-creds.json
export AGB_TENANT_ID=...
export FIREBASE_WEB_API_KEY=...
# Run
python -m benchmark.cli run --runner crewai_acp --out my-result.json
A different number than ours means your ACP version differs or you found a gap we have not. Either way, file an issue at the agentgovbench repo.
Receipts:
- agentgovbench repo: all runners and results
- /benchmark page: the current daily number, with every non-pass named
- Methodology post: how the benchmark is designed
- Decorator vs proxy vs hook
- Anthropic Agent SDK scorecard: the 46/48 runner in detail
Running coding agents? ACP records each tool call its hook sees and makes the dangerous shell commands (a force-push, destructive SQL, curl | sh) wait for you. One command, free up to 5 agents:
curl -fsSL https://agenticcontrolplane.com/install.sh | bash
irm https://agenticcontrolplane.com/install.ps1 | iex
- 7. Is --dangerously-skip-permissions Safe? What the Flag Turns Off
- 12. Full scorecard: seven frameworks, 48 scenarios, one open benchmark · you are here
- 18. Seven agent frameworks, one backend, governance diverges on 9 of 48 tests