fix(evals): transcript assertions must grep with -a (invalid UTF-8 silently defeats matching)#1576
Merged
Aaronontheweb merged 2 commits intoJul 4, 2026
Conversation
…eating assertions
GNU grep in a UTF-8 locale silently skips lines with invalid multibyte sequences
when the -a flag is absent. Eval transcripts contain terminal control bytes and
other non-UTF-8 sequences, causing assertion helpers to randomly fail when
escape bytes landed on a matching line — the mechanism explains the historic
flakiness of memory_identity_preference_routing (0/5→3/5→5/5→3/5 swings).
Fixed all 14 grep call sites in the assertion helpers and inline transcript
assertions by adding -a:
- stdout_contains, stdout_not_contains (grep -qi → grep -qia)
- stdout_response_contains, stdout_response_not_contains (both pipes: -v → -av, -qi → -qia)
- daemon_log_contains, daemon_log_skill_loaded, daemon_log_no_skill_loaded (grep -qE → grep -qaE)
- stdout_tool_called (grep -qE → grep -qaE)
- store_metrics: grep on usage_line (grep -oP → grep -aoP, 5 invocations)
- run_prompt_resume: grep on turn_file (grep -o → grep -ao)
- assert_multi_turn_text_growth: grep on STDOUT_FILE (grep -c → grep -ac)
- assert_approval_set_working_directory_positive: grep -nE on STDOUT_FILE (2 invocations, → grep -anE)
Proven on run af0883b5: memory_identity_preference_routing cases had correct
behavior ('durable memories' literal text in transcript), but grep -ci returned 0,
while grep -cia returned the correct count. This fix eliminates the instrumentation
source of the flakiness.
…does not contain 'memory' The memory_identity_preference_routing assertion was failing intermittently because correct model responses often say 'memories' (plural), but the assertion required the literal substring 'memory' (singular). 'memories' does NOT contain 'memory' as a substring — the plural drops the y. Regression proof from run af0883b5: runs #3 and netclaw-dev#5 exhibited correct behavior ("That's already saved — ... in my durable memories", no file edits, no redundant store call) but were rejected by the old pattern. The -a flags added in the prior commit are retained as cheap defensive hardening for transcripts that may carry terminal control bytes, but they are NOT the fix for this flake. The invalid-UTF-8 justification was incorrect and is retracted.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The actual bug
The
assert_memory_identity_preference_routingassertion requires the literal substringmemory. However, correct model responses often say "memories" (plural), and the plural form drops the y — the substring "memories" does not contain "memory".Regression proof from run af0883b5:
grep -ci 'memory' $TRANSCRIPT= 0 (the bug—response is plural-only)grep -cia 'memor' $TRANSCRIPT= 1+ (fixed pattern matches)file $TRANSCRIPTconfirms clean UTF-8 (disproves the old theory)Behaviorally-correct runs (#3 and #5 from af0883b5) were rejected because they said "That's already saved — ... in my durable memories" with no file edits and no redundant store call—exactly right routing, but flagged as failures.
This flake explains the 0/5 → 3/5 → 5/5 history: the fix rate increased as model outputs naturally drifted away from singular toward plural language.
What the -a flags are
The
-aflags added in the prior commit are retained as cheap defensive hardening: captured CLI output may carry terminal control bytes, and-akeeps grep in text mode regardless.The original invalid-UTF-8 justification was incorrect and is retracted. The transcript is clean UTF-8. The flake was plain English morphology, not byte encoding.
Call sites
Assertion pattern fix:
evals/run-evals.sh:1010— changedstdout_contains 'memory'tostdout_contains 'memor'The 17
-acall sites from the prior commit are all present and remain in place as cheap hardening, but they are not the fix.Verified proof (archived run af0883b5, failing transcript run #3)