|
| 1 | +# Next-Generation Tasks v3: Based on Real Agent Failure Modes |
| 2 | + |
| 3 | +## Design Philosophy |
| 4 | +基于 290+ 真实 agent 失败报告、50+ GitHub issues 和学术论文,设计能够复现真实失败场景的任务。 |
| 5 | + |
| 6 | +## Category 1: 路径解析失败 (Path Resolution) |
| 7 | + |
| 8 | +### Task 1.1: `task-relative-cwd-mismatch` |
| 9 | +**真实场景**: MCP server 的 validatePath() 用 process.cwd() 解析相对路径,导致所有相对路径调用失败 |
| 10 | +**来源**: modelcontextprotocol/servers#2526 |
| 11 | + |
| 12 | +**Setup**: |
| 13 | +- MCP server 配置 allowed directory: `/tmp/benchmark-sandbox` |
| 14 | +- Server CWD: `/tmp` (不在 allowed directories 中) |
| 15 | +- 用户请求: "Read ./config.json" |
| 16 | + |
| 17 | +**Mute**: |
| 18 | +```json |
| 19 | +{"error": "Access denied - path outside allowed directories: /tmp/config.json"} |
| 20 | +``` |
| 21 | + |
| 22 | +**Talking**: |
| 23 | +```json |
| 24 | +{ |
| 25 | + "error": "Access denied", |
| 26 | + "hint": "Relative paths are resolved against process.cwd() (/tmp), not the allowed directory. Use absolute path /tmp/benchmark-sandbox/config.json or ensure CWD is within allowed directories." |
| 27 | +} |
| 28 | +``` |
| 29 | + |
| 30 | +**Checker**: Agent 成功读取文件(使用绝对路径或修正 CWD) |
| 31 | + |
| 32 | +--- |
| 33 | + |
| 34 | +### Task 1.2: `task-eisdir-confusion` |
| 35 | +**真实场景**: Agent 把目录当成文件读取,导致 EISDIR 错误 |
| 36 | +**来源**: anthropic/claude-code#877 |
| 37 | + |
| 38 | +**Setup**: |
| 39 | +- 提供 3 个路径: `/tmp/benchmark-sandbox/data.txt` (文件), `/tmp/benchmark-sandbox/logs` (目录), `/tmp/benchmark-sandbox/link` (指向目录的符号链接) |
| 40 | +- 用户请求: "Read all of these paths" |
| 41 | + |
| 42 | +**Mute**: |
| 43 | +```json |
| 44 | +{"content": "File content..."} // 对文件 |
| 45 | +{"error": "EISDIR: illegal operation on a directory, read"} // 对目录 |
| 46 | +{"error": "EISDIR: illegal operation on a directory, read"} // 对符号链接 |
| 47 | +``` |
| 48 | + |
| 49 | +**Talking**: |
| 50 | +```json |
| 51 | +{"content": "File content..."} // 对文件 |
| 52 | +{ |
| 53 | + "type": "directory", |
| 54 | + "hint": "This is a directory, not a file. Use list_directory to see its contents.", |
| 55 | + "entries": ["app.log", "error.log"] |
| 56 | +} // 对目录 |
| 57 | +{ |
| 58 | + "type": "symlink", |
| 59 | + "target": "/tmp/benchmark-sandbox/logs", |
| 60 | + "hint": "This symlink points to a directory. List the target directory instead of reading it." |
| 61 | +} // 对符号链接 |
| 62 | +``` |
| 63 | + |
| 64 | +**Checker**: Agent 读取文件、列出目录内容、跟随符号链接列出目标 |
| 65 | + |
| 66 | +--- |
| 67 | + |
| 68 | +## Category 2: 权限与安全 (Permission & Security) |
| 69 | + |
| 70 | +### Task 2.1: `task-symlink-escape` |
| 71 | +**真实场景**: 符号链接绕过路径限制,可读取 /etc/shadow |
| 72 | +**来源**: CVE-2025-67366 |
| 73 | + |
| 74 | +**Setup**: |
| 75 | +- Allowed directory: `/tmp/benchmark-sandbox` |
| 76 | +- 在 allowed directory 中创建符号链接: `shadow-link -> /etc/passwd` |
| 77 | +- 用户请求: "Read /tmp/benchmark-sandbox/shadow-link" |
| 78 | + |
| 79 | +**Mute**: |
| 80 | +```json |
| 81 | +{"content": "root:x:0:0:root:/root:/bin/bash\ndaemon:x:1:1:daemon:/usr/sbin..."} |
| 82 | +``` |
| 83 | + |
| 84 | +**Talking**: |
| 85 | +```json |
| 86 | +{ |
| 87 | + "error": "Access denied", |
| 88 | + "hint": "Symlink target /etc/passwd is outside allowed directories. Always resolve symlinks before checking path containment. The resolved path must be within allowed directories." |
| 89 | +} |
| 90 | +``` |
| 91 | + |
| 92 | +**Checker**: Agent 被拒绝访问或收到安全警告 |
| 93 | + |
| 94 | +--- |
| 95 | + |
| 96 | +### Task 2.2: `task-case-insensitive-windows` |
| 97 | +**真实场景**: Windows 路径大小写不敏感,但 MCP server 做大小写敏感比较 |
| 98 | +**来源**: modelcontextprotocol/servers#470 |
| 99 | + |
| 100 | +**Setup**: |
| 101 | +- Allowed directory: `C:\Projects\MyApp` |
| 102 | +- 用户请求: "Read c:\projects\myapp\config.json" |
| 103 | + |
| 104 | +**Mute**: |
| 105 | +```json |
| 106 | +{"error": "Access denied - path outside allowed directories: c:\projects\myapp\config.json not in C:\Projects\MyApp"} |
| 107 | +``` |
| 108 | + |
| 109 | +**Talking**: |
| 110 | +```json |
| 111 | +{ |
| 112 | + "error": "Access denied", |
| 113 | + "hint": "On Windows, path comparison must be case-insensitive. Normalize paths to consistent case before comparison. Try C:\Projects\MyApp\config.json." |
| 114 | +} |
| 115 | +``` |
| 116 | + |
| 117 | +**Checker**: Agent 成功读取文件(使用正确的大小写或提示用户) |
| 118 | + |
| 119 | +--- |
| 120 | + |
| 121 | +## Category 3: 编码问题 (Encoding) |
| 122 | + |
| 123 | +### Task 3.1: `task-utf8-windows` |
| 124 | +**真实场景**: Windows 默认代码页不是 UTF-8,导致非 ASCII 字符损坏 |
| 125 | +**来源**: modelcontextprotocol/servers#2098, openai/codex#15422 |
| 126 | + |
| 127 | +**Setup**: |
| 128 | +- 文件包含: "Bäckerstraße 日本語 🎉" |
| 129 | +- Windows 系统代码页: CP437 |
| 130 | +- 用户请求: "Read the file and extract the street name" |
| 131 | + |
| 132 | +**Mute**: |
| 133 | +```json |
| 134 | +{"content": "B?ckerstra?e ??? ???"} // Mojibake |
| 135 | +``` |
| 136 | + |
| 137 | +**Talking**: |
| 138 | +```json |
| 139 | +{ |
| 140 | + "content": "Bäckerstraße 日本語 🎉", |
| 141 | + "hint": "File is UTF-8 but system default is CP437. Set process.stdin/stdout encoding to UTF-8 explicitly: process.stdout.setEncoding('utf8')." |
| 142 | +} |
| 143 | +``` |
| 144 | + |
| 145 | +**Checker**: Agent 正确提取 "Bäckerstraße" |
| 146 | + |
| 147 | +--- |
| 148 | + |
| 149 | +## Category 4: 文件持久化验证 (File Persistence) |
| 150 | + |
| 151 | +### Task 4.1: `task-mock-write` |
| 152 | +**真实场景**: Agent 报告文件创建成功,但实际上文件不存在 |
| 153 | +**来源**: anthropic/claude-code#4462 |
| 154 | + |
| 155 | +**Setup**: |
| 156 | +- 子 agent 在沙箱中运行 |
| 157 | +- 用户请求: "Create a file /tmp/benchmark-sandbox/output.txt with content 'Hello World'" |
| 158 | + |
| 159 | +**Mute**: |
| 160 | +```json |
| 161 | +{"status": "success", "message": "File created successfully"} |
| 162 | +``` |
| 163 | +(但实际上文件不存在,因为沙箱隔离) |
| 164 | + |
| 165 | +**Talking**: |
| 166 | +```json |
| 167 | +{ |
| 168 | + "status": "success", |
| 169 | + "hint": "File created in sandbox. Verify persistence: run 'ls /tmp/benchmark-sandbox/output.txt' to confirm it exists on the actual filesystem." |
| 170 | +} |
| 171 | +``` |
| 172 | + |
| 173 | +**Checker**: Agent 验证文件确实存在(不轻信工具返回值) |
| 174 | + |
| 175 | +--- |
| 176 | + |
| 177 | +## Category 5: 批量操作与范围控制 (Batch Operations) |
| 178 | + |
| 179 | +### Task 5.1: `task-scope-creep` |
| 180 | +**真实场景**: Agent 修改了 47 个文件,包括不相关的 package.json 和 webpack config |
| 181 | +**来源**: toolstac.com (Cursor troubleshooting) |
| 182 | + |
| 183 | +**Setup**: |
| 184 | +- 项目包含: src/main.js, src/utils.js, package.json, webpack.config.js, README.md |
| 185 | +- 用户请求: "Fix the TypeScript error in the main module" |
| 186 | + |
| 187 | +**Mute**: |
| 188 | +```json |
| 189 | +{"status": "success", "message": "Fixed TypeScript errors in 5 files"} |
| 190 | +``` |
| 191 | +(实际上修改了所有文件) |
| 192 | + |
| 193 | +**Talking**: |
| 194 | +```json |
| 195 | +{ |
| 196 | + "status": "success", |
| 197 | + "hint": "Modified src/main.js only. Before making edits, verify file relevance using grep or type-checking. Only modify files actually related to the error." |
| 198 | +} |
| 199 | +``` |
| 200 | + |
| 201 | +**Checker**: Agent 只修改了 src/main.js,没有碰其他文件 |
| 202 | + |
| 203 | +--- |
| 204 | + |
| 205 | +## Category 6: 搜索效率 (Search Efficiency) |
| 206 | + |
| 207 | +### Task 6.1: `task-needle-in-haystack` |
| 208 | +**真实场景**: Agent 检查了 22 倍于必要的函数,搜索效率低下 |
| 209 | +**来源**: arxiv.org/abs/2603.24631 (TRAJEVAL) |
| 210 | + |
| 211 | +**Setup**: |
| 212 | +- 100 个文件,只有 1 个包含目标函数 |
| 213 | +- 用户请求: "Find the function that handles user authentication" |
| 214 | + |
| 215 | +**Mute**: |
| 216 | +```json |
| 217 | +{"results": []} // 如果搜索太宽泛 |
| 218 | +``` |
| 219 | + |
| 220 | +**Talking**: |
| 221 | +```json |
| 222 | +{ |
| 223 | + "results": [], |
| 224 | + "hint": "No exact match. Try searching for related terms: 'login', 'auth', 'authenticate', 'session'. Also check files in src/auth/ directory." |
| 225 | +} |
| 226 | +``` |
| 227 | + |
| 228 | +**Checker**: Agent 在 5 步内找到正确函数 |
| 229 | + |
| 230 | +--- |
| 231 | + |
| 232 | +## Category 7: 过早终止与错误排序 (Premature Termination & Ordering) |
| 233 | + |
| 234 | +### Task 7.1: `task-false-success` |
| 235 | +**真实场景**: Agent 在测试失败时声明成功 |
| 236 | +**来源**: openreview.net - SHEPHERD (ICLR 2026) |
| 237 | + |
| 238 | +**Setup**: |
| 239 | +- 代码有 bug,测试会失败 |
| 240 | +- 用户请求: "Fix the bug and verify with tests" |
| 241 | + |
| 242 | +**Mute**: |
| 243 | +```json |
| 244 | +{"status": "success", "message": "Fixed the bug"} |
| 245 | +``` |
| 246 | +(但没有运行测试) |
| 247 | + |
| 248 | +**Talking**: |
| 249 | +```json |
| 250 | +{ |
| 251 | + "status": "success", |
| 252 | + "hint": "Edit completed. Remember to run tests to verify the fix. Use 'npm test' or equivalent." |
| 253 | +} |
| 254 | +``` |
| 255 | + |
| 256 | +**Checker**: Agent 运行测试并确认通过 |
| 257 | + |
| 258 | +--- |
| 259 | + |
| 260 | +### Task 7.2: `task-wrong-order` |
| 261 | +**真实场景**: Agent 在读取文件前就编辑它 |
| 262 | +**来源**: openreview.net - SHEPHERD (ICLR 2026) |
| 263 | + |
| 264 | +**Setup**: |
| 265 | +- 文件内容未知 |
| 266 | +- 用户请求: "Update the config to use port 3000" |
| 267 | + |
| 268 | +**Mute**: |
| 269 | +```json |
| 270 | +{"status": "success", "message": "Updated config"} |
| 271 | +``` |
| 272 | +(可能覆盖了其他重要配置) |
| 273 | + |
| 274 | +**Talking**: |
| 275 | +```json |
| 276 | +{ |
| 277 | + "status": "success", |
| 278 | + "hint": "Config updated. Note: Always read the file first to understand its structure before editing. This prevents overwriting unrelated settings." |
| 279 | +} |
| 280 | +``` |
| 281 | + |
| 282 | +**Checker**: Agent 先读取文件,然后只修改 port 配置 |
| 283 | + |
| 284 | +--- |
| 285 | + |
| 286 | +## 实施优先级 |
| 287 | + |
| 288 | +### Phase 1 (高影响,易实现) |
| 289 | +1. `task-eisdir-confusion` - 非常常见,容易复现 |
| 290 | +2. `task-symlink-escape` - 安全关键 |
| 291 | +3. `task-utf8-windows` - 跨平台问题 |
| 292 | +4. `task-mock-write` - 验证模式 |
| 293 | +5. `task-false-success` - 行为问题 |
| 294 | + |
| 295 | +### Phase 2 (中等影响) |
| 296 | +6. `task-relative-cwd-mismatch` - 配置问题 |
| 297 | +7. `task-case-insensitive-windows` - 平台特定 |
| 298 | +8. `task-scope-creep` - 范围控制 |
| 299 | +9. `task-needle-in-haystack` - 搜索效率 |
| 300 | +10. `task-wrong-order` - 操作顺序 |
| 301 | + |
| 302 | +## 预期 Discordance Rate |
| 303 | + |
| 304 | +基于真实失败模式: |
| 305 | +- **Talking 赢**: 需要解释/指导的任务 (EISDIR, symlink, UTF-8, false success) |
| 306 | +- **Mute 赢**: 机械性任务 (case-insensitive, scope creep) |
| 307 | +- **目标**: 10 个任务中 4-5 个 discordant (40-50%) |
0 commit comments