Repository navigation
Commit 11a8e6d
committed
bench: help-conformance case for settled scroll-to-find planning
The #1638 extension of the closed --settle grammar to scroll/back is the
feature's entire payoff — collapsing scroll-then-observe into one call — and
the closed command list is an enumerated N whose enumerator is this bench.
The regex over the help text proves the sentence exists; this case checks
whether a model plans differently because of it.
One focused case, deliberately not coached: a pinned visible-first snapshot
(rendered by formatSnapshotText, pinned by the sample-producers gate) whose
wanted row is summarized off-screen with no ref anywhere in the output. The
tempting pre-#1638 plan is `scroll` plus a separate `snapshot -i`; acceptance
is the single settled call. Scoring was verified against eight plan shapes in
both directions before recording.
Model-backed record (claude-haiku-4-5, 3 trials, current help): 0/3 — but the
decomposition is the finding. Settle eligibility GENERALIZED (3/3 trials put
--settle on scroll unprompted; the mutation-suffix framing concern did not
materialize) and the two-call habit is residual (1/3). All three trials failed
on `scroll @e3 down --settle` — the pre-existing #1366 scroll-takes-no-target
confusion, which the live CLI recovers with a dedicated hint but a single-shot
bench cannot. The recorded gap is therefore a first-30 doc gap (nothing
teaches that scroll takes no target), not a settle-eligibility gap; tuning the
case until it passes would just delete the evidence.1 parent 2c6955a commit 11a8e6d
3 files changed
Lines changed: 118 additions & 0 deletions
File tree
- scripts
- __tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
208 | 209 | | |
209 | 210 | | |
210 | 211 | | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
211 | 270 | | |
212 | 271 | | |
213 | 272 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
| 7 | + | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
| |||
377 | 378 | | |
378 | 379 | | |
379 | 380 | | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
380 | 421 | | |
381 | 422 | | |
382 | 423 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
47 | 65 | | |
48 | 66 | | |
49 | 67 | | |
| |||
0 commit comments