Skip to content

Commit 3219130

Browse files
committed
test(evals): reject scratch file links wrapped in inline code
1 parent f0291c0 commit 3219130

1 file changed

Lines changed: 1 addition & 1 deletion

File tree

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,2 @@
1-
{"id": "scratch-file-markdown-link", "area": "file-links", "kind": "bad", "pattern": "scratch file path shortened or emitted without a Markdown link", "experience": "A Claude scratch file was mentioned as a bare path or shortened inline code; only its full Markdown link reliably opened the viewer.", "privacy": "commit-safe", "setup": {"role": "top", "context": "You have just saved notes.md at /tmp/claude-1000/-home-user--t3-worktrees-app-t3-abc123/12345678-1234-1234-1234-123456789abc/scratchpad/notes.md. The write succeeded. No more tools are needed."}, "prompt": "The notes are ready? Tell me where you saved them. Keep it brief.", "expect": {"textMust": ["\\[[^\\]\\n]+\\]\\(/tmp/claude-1000/-home-user--t3-worktrees-app-t3-abc123/12345678-1234-1234-1234-123456789abc/scratchpad/notes\\.md\\)"], "textMustNot": ["/tmp/claude-1000/[^\\s<>`]*(?:\\.{3,}|…)"]}, "maxSteps": 2, "source": {"issue": "https://github.com/pingdotgg/t3code/issues/30757"}}
1+
{"id": "scratch-file-markdown-link", "area": "file-links", "kind": "bad", "pattern": "scratch file path shortened or emitted without a Markdown link", "experience": "A Claude scratch file was mentioned as a bare path or shortened inline code; only its full Markdown link reliably opened the viewer.", "privacy": "commit-safe", "setup": {"role": "top", "context": "You have just saved notes.md at /tmp/claude-1000/-home-user--t3-worktrees-app-t3-abc123/12345678-1234-1234-1234-123456789abc/scratchpad/notes.md. The write succeeded. No more tools are needed."}, "prompt": "The notes are ready? Tell me where you saved them. Keep it brief.", "expect": {"textMust": ["\\[[^\\]\\n]+\\]\\(/tmp/claude-1000/-home-user--t3-worktrees-app-t3-abc123/12345678-1234-1234-1234-123456789abc/scratchpad/notes\\.md\\)"], "textMustNot": ["/tmp/claude-1000/[^\\s<>`]*(?:\\.{3,}|…)", "`[^`\\n]*\\]\\(/tmp/claude-1000/[^`\\n]*`"]}, "maxSteps": 2, "source": {"issue": "https://github.com/pingdotgg/t3code/issues/30757"}}
22
{"id": "ordinary-answer-without-file-link", "area": "file-links", "kind": "control", "experience": "Ordinary conversation does not require producing or linking a file.", "privacy": "commit-safe", "setup": {"role": "top", "context": "No files have been created. No tools are needed."}, "prompt": "What is 2 + 2? Answer with only the number.", "expect": {"textMust": ["^4[.!]?$"], "textMustNot": ["\\[[^\\]\\n]+\\]\\([^\\n]+\\)"]}, "maxSteps": 2, "source": {"issue": "https://github.com/pingdotgg/t3code/issues/30757"}, "pattern": "ordinary answer without a file artifact"}

0 commit comments

Comments
 (0)