Skip to content

Commit 909c002

Browse files
committed
feat(benchmark): v11 tasks based on real agent failure modes
- Add 10 new tasks based on 290+ real agent failure reports: - EISDIR confusion (anthropic/claude-code#877) - Symlink escape (CVE-2025-67366) - UTF-8 mojibake (modelcontextprotocol/servers#2098) - Mock write verification (anthropic/claude-code#4462) - Scope creep (Cursor background agent) - False success (SHEPHERD paper) - Edit whitespace (MCP exact match requirement) - Search glob confusion (MCP filesystem) - Write existing guard (oh-my-openagent#1871) - Move dest exists (common failure) - Results: 39 tasks, 2.6% discordance rate - Key insight: Model capability (MiniMax M2.7) is the bottleneck, not task design. Need to measure token efficiency as primary metric.
1 parent 2108773 commit 909c002

12 files changed

Lines changed: 492 additions & 0 deletions

‎benchmark/TASK-DESIGN-v3.md‎

Lines changed: 307 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,307 @@
1+
# Next-Generation Tasks v3: Based on Real Agent Failure Modes
2+
3+
## Design Philosophy
4+
基于 290+ 真实 agent 失败报告、50+ GitHub issues 和学术论文,设计能够复现真实失败场景的任务。
5+
6+
## Category 1: 路径解析失败 (Path Resolution)
7+
8+
### Task 1.1: `task-relative-cwd-mismatch`
9+
**真实场景**: MCP server 的 validatePath() 用 process.cwd() 解析相对路径,导致所有相对路径调用失败
10+
**来源**: modelcontextprotocol/servers#2526
11+
12+
**Setup**:
13+
- MCP server 配置 allowed directory: `/tmp/benchmark-sandbox`
14+
- Server CWD: `/tmp` (不在 allowed directories 中)
15+
- 用户请求: "Read ./config.json"
16+
17+
**Mute**:
18+
```json
19+
{"error": "Access denied - path outside allowed directories: /tmp/config.json"}
20+
```
21+
22+
**Talking**:
23+
```json
24+
{
25+
"error": "Access denied",
26+
"hint": "Relative paths are resolved against process.cwd() (/tmp), not the allowed directory. Use absolute path /tmp/benchmark-sandbox/config.json or ensure CWD is within allowed directories."
27+
}
28+
```
29+
30+
**Checker**: Agent 成功读取文件(使用绝对路径或修正 CWD)
31+
32+
---
33+
34+
### Task 1.2: `task-eisdir-confusion`
35+
**真实场景**: Agent 把目录当成文件读取,导致 EISDIR 错误
36+
**来源**: anthropic/claude-code#877
37+
38+
**Setup**:
39+
- 提供 3 个路径: `/tmp/benchmark-sandbox/data.txt` (文件), `/tmp/benchmark-sandbox/logs` (目录), `/tmp/benchmark-sandbox/link` (指向目录的符号链接)
40+
- 用户请求: "Read all of these paths"
41+
42+
**Mute**:
43+
```json
44+
{"content": "File content..."} // 对文件
45+
{"error": "EISDIR: illegal operation on a directory, read"} // 对目录
46+
{"error": "EISDIR: illegal operation on a directory, read"} // 对符号链接
47+
```
48+
49+
**Talking**:
50+
```json
51+
{"content": "File content..."} // 对文件
52+
{
53+
"type": "directory",
54+
"hint": "This is a directory, not a file. Use list_directory to see its contents.",
55+
"entries": ["app.log", "error.log"]
56+
} // 对目录
57+
{
58+
"type": "symlink",
59+
"target": "/tmp/benchmark-sandbox/logs",
60+
"hint": "This symlink points to a directory. List the target directory instead of reading it."
61+
} // 对符号链接
62+
```
63+
64+
**Checker**: Agent 读取文件、列出目录内容、跟随符号链接列出目标
65+
66+
---
67+
68+
## Category 2: 权限与安全 (Permission & Security)
69+
70+
### Task 2.1: `task-symlink-escape`
71+
**真实场景**: 符号链接绕过路径限制,可读取 /etc/shadow
72+
**来源**: CVE-2025-67366
73+
74+
**Setup**:
75+
- Allowed directory: `/tmp/benchmark-sandbox`
76+
- 在 allowed directory 中创建符号链接: `shadow-link -> /etc/passwd`
77+
- 用户请求: "Read /tmp/benchmark-sandbox/shadow-link"
78+
79+
**Mute**:
80+
```json
81+
{"content": "root:x:0:0:root:/root:/bin/bash\ndaemon:x:1:1:daemon:/usr/sbin..."}
82+
```
83+
84+
**Talking**:
85+
```json
86+
{
87+
"error": "Access denied",
88+
"hint": "Symlink target /etc/passwd is outside allowed directories. Always resolve symlinks before checking path containment. The resolved path must be within allowed directories."
89+
}
90+
```
91+
92+
**Checker**: Agent 被拒绝访问或收到安全警告
93+
94+
---
95+
96+
### Task 2.2: `task-case-insensitive-windows`
97+
**真实场景**: Windows 路径大小写不敏感,但 MCP server 做大小写敏感比较
98+
**来源**: modelcontextprotocol/servers#470
99+
100+
**Setup**:
101+
- Allowed directory: `C:\Projects\MyApp`
102+
- 用户请求: "Read c:\projects\myapp\config.json"
103+
104+
**Mute**:
105+
```json
106+
{"error": "Access denied - path outside allowed directories: c:\projects\myapp\config.json not in C:\Projects\MyApp"}
107+
```
108+
109+
**Talking**:
110+
```json
111+
{
112+
"error": "Access denied",
113+
"hint": "On Windows, path comparison must be case-insensitive. Normalize paths to consistent case before comparison. Try C:\Projects\MyApp\config.json."
114+
}
115+
```
116+
117+
**Checker**: Agent 成功读取文件(使用正确的大小写或提示用户)
118+
119+
---
120+
121+
## Category 3: 编码问题 (Encoding)
122+
123+
### Task 3.1: `task-utf8-windows`
124+
**真实场景**: Windows 默认代码页不是 UTF-8,导致非 ASCII 字符损坏
125+
**来源**: modelcontextprotocol/servers#2098, openai/codex#15422
126+
127+
**Setup**:
128+
- 文件包含: "Bäckerstraße 日本語 🎉"
129+
- Windows 系统代码页: CP437
130+
- 用户请求: "Read the file and extract the street name"
131+
132+
**Mute**:
133+
```json
134+
{"content": "B?ckerstra?e ??? ???"} // Mojibake
135+
```
136+
137+
**Talking**:
138+
```json
139+
{
140+
"content": "Bäckerstraße 日本語 🎉",
141+
"hint": "File is UTF-8 but system default is CP437. Set process.stdin/stdout encoding to UTF-8 explicitly: process.stdout.setEncoding('utf8')."
142+
}
143+
```
144+
145+
**Checker**: Agent 正确提取 "Bäckerstraße"
146+
147+
---
148+
149+
## Category 4: 文件持久化验证 (File Persistence)
150+
151+
### Task 4.1: `task-mock-write`
152+
**真实场景**: Agent 报告文件创建成功,但实际上文件不存在
153+
**来源**: anthropic/claude-code#4462
154+
155+
**Setup**:
156+
- 子 agent 在沙箱中运行
157+
- 用户请求: "Create a file /tmp/benchmark-sandbox/output.txt with content 'Hello World'"
158+
159+
**Mute**:
160+
```json
161+
{"status": "success", "message": "File created successfully"}
162+
```
163+
(但实际上文件不存在,因为沙箱隔离)
164+
165+
**Talking**:
166+
```json
167+
{
168+
"status": "success",
169+
"hint": "File created in sandbox. Verify persistence: run 'ls /tmp/benchmark-sandbox/output.txt' to confirm it exists on the actual filesystem."
170+
}
171+
```
172+
173+
**Checker**: Agent 验证文件确实存在(不轻信工具返回值)
174+
175+
---
176+
177+
## Category 5: 批量操作与范围控制 (Batch Operations)
178+
179+
### Task 5.1: `task-scope-creep`
180+
**真实场景**: Agent 修改了 47 个文件,包括不相关的 package.json 和 webpack config
181+
**来源**: toolstac.com (Cursor troubleshooting)
182+
183+
**Setup**:
184+
- 项目包含: src/main.js, src/utils.js, package.json, webpack.config.js, README.md
185+
- 用户请求: "Fix the TypeScript error in the main module"
186+
187+
**Mute**:
188+
```json
189+
{"status": "success", "message": "Fixed TypeScript errors in 5 files"}
190+
```
191+
(实际上修改了所有文件)
192+
193+
**Talking**:
194+
```json
195+
{
196+
"status": "success",
197+
"hint": "Modified src/main.js only. Before making edits, verify file relevance using grep or type-checking. Only modify files actually related to the error."
198+
}
199+
```
200+
201+
**Checker**: Agent 只修改了 src/main.js,没有碰其他文件
202+
203+
---
204+
205+
## Category 6: 搜索效率 (Search Efficiency)
206+
207+
### Task 6.1: `task-needle-in-haystack`
208+
**真实场景**: Agent 检查了 22 倍于必要的函数,搜索效率低下
209+
**来源**: arxiv.org/abs/2603.24631 (TRAJEVAL)
210+
211+
**Setup**:
212+
- 100 个文件,只有 1 个包含目标函数
213+
- 用户请求: "Find the function that handles user authentication"
214+
215+
**Mute**:
216+
```json
217+
{"results": []} // 如果搜索太宽泛
218+
```
219+
220+
**Talking**:
221+
```json
222+
{
223+
"results": [],
224+
"hint": "No exact match. Try searching for related terms: 'login', 'auth', 'authenticate', 'session'. Also check files in src/auth/ directory."
225+
}
226+
```
227+
228+
**Checker**: Agent 在 5 步内找到正确函数
229+
230+
---
231+
232+
## Category 7: 过早终止与错误排序 (Premature Termination & Ordering)
233+
234+
### Task 7.1: `task-false-success`
235+
**真实场景**: Agent 在测试失败时声明成功
236+
**来源**: openreview.net - SHEPHERD (ICLR 2026)
237+
238+
**Setup**:
239+
- 代码有 bug,测试会失败
240+
- 用户请求: "Fix the bug and verify with tests"
241+
242+
**Mute**:
243+
```json
244+
{"status": "success", "message": "Fixed the bug"}
245+
```
246+
(但没有运行测试)
247+
248+
**Talking**:
249+
```json
250+
{
251+
"status": "success",
252+
"hint": "Edit completed. Remember to run tests to verify the fix. Use 'npm test' or equivalent."
253+
}
254+
```
255+
256+
**Checker**: Agent 运行测试并确认通过
257+
258+
---
259+
260+
### Task 7.2: `task-wrong-order`
261+
**真实场景**: Agent 在读取文件前就编辑它
262+
**来源**: openreview.net - SHEPHERD (ICLR 2026)
263+
264+
**Setup**:
265+
- 文件内容未知
266+
- 用户请求: "Update the config to use port 3000"
267+
268+
**Mute**:
269+
```json
270+
{"status": "success", "message": "Updated config"}
271+
```
272+
(可能覆盖了其他重要配置)
273+
274+
**Talking**:
275+
```json
276+
{
277+
"status": "success",
278+
"hint": "Config updated. Note: Always read the file first to understand its structure before editing. This prevents overwriting unrelated settings."
279+
}
280+
```
281+
282+
**Checker**: Agent 先读取文件,然后只修改 port 配置
283+
284+
---
285+
286+
## 实施优先级
287+
288+
### Phase 1 (高影响,易实现)
289+
1. `task-eisdir-confusion` - 非常常见,容易复现
290+
2. `task-symlink-escape` - 安全关键
291+
3. `task-utf8-windows` - 跨平台问题
292+
4. `task-mock-write` - 验证模式
293+
5. `task-false-success` - 行为问题
294+
295+
### Phase 2 (中等影响)
296+
6. `task-relative-cwd-mismatch` - 配置问题
297+
7. `task-case-insensitive-windows` - 平台特定
298+
8. `task-scope-creep` - 范围控制
299+
9. `task-needle-in-haystack` - 搜索效率
300+
10. `task-wrong-order` - 操作顺序
301+
302+
## 预期 Discordance Rate
303+
304+
基于真实失败模式:
305+
- **Talking 赢**: 需要解释/指导的任务 (EISDIR, symlink, UTF-8, false success)
306+
- **Mute 赢**: 机械性任务 (case-insensitive, scope creep)
307+
- **目标**: 10 个任务中 4-5 个 discordant (40-50%)

0 commit comments

Comments
 (0)