[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures #133010

Triggering signals

• KBE [ci-scan] Test failure: profiler/gc/nongcheap/nongcheap.cmd fails with missing 'PROFILER TEST PASSES' #131449 ([ci-scan], open) — MEMBER janvorli (2026-08-07): "This issue seems to be a kitchen sink for all kinds of profiler tests failures." (comment)
• KBE [ci-scan] Test failure: profiler/gc/nongcheap/nongcheap.cmd fails with missing 'PROFILER TEST PASSES' #131449 — MEMBER noahfalk (2026-08-22): "Previously the KBE matched the Helix console log which runs many tests. Whenever some other profiler test failed the full log would still contain the line saying the nongcheap.cmd test ran as well as the error message, just that the error was part of a different test's output." (comment)
• Tracker [ci-scan-feedback] KPI Tracker #128742 — MEMBER noahfalk (2026-08-22): "the KBE generation logic either ideally either needs to avoid matching error message in the console log, or we need to capture a single console output expression that both indicates an error and is test-specific." (comment)

Rubric finding: KBE #131449's array-form ErrorMessage paired Running test: profiler/gc/nongcheap/nongcheap.cmd (announcement line, present in the profiler suite's single shared Helix console log for every test) with Profiler tests are expected to contain the text 'PROFILER TEST PASSES' (a shared-harness error any profiler test can print). Both lines coexist in the same console log, so the signature matched whenever any profiler/** test failed — a catch-all.

Proposed edits

• .github/workflows/shared/create-kbe.instructions.md (array-form "Rules" list, ~L445) — new rule: do not pair a per-run announcement line with a non-test-specific error line when a leg runs many tests through one shared Helix console log; anchor every element to the failing test's own output, or set ExcludeConsoleLog: true. Ties directly to noahfalk's console-log-matching feedback.
• .github/workflows/shared/create-kbe.instructions.md ("Bad vs good signatures" table, ~L556) — new Bad/Good row using the [ci-scan] Test failure: profiler/gc/nongcheap/nongcheap.cmd fails with missing 'PROFILER TEST PASSES' #131449 signature as the concrete example.

Expected behavior change

The next scanner run will not emit array-form KBEs whose elements individually match a shared multi-test Helix console log. Signatures that can only be anchored to suite-wide output will either be tightened to the failing test's own lines or set ExcludeConsoleLog: true, avoiding the profiler-style catch-all KBE that buckets unrelated failures.

Generated by CI Outer-Loop Failure Scanner — Feedback · opus48 · 259.5 AIC · ⌖ 25.4 AIC · ⊞ 20.6K · ◷

To create a pull request with the changes:

# Download the artifact from the workflow run
gh run download 33510401456 -n agent -D /tmp/agent-33510401456

# Fetch the bundle into a temporary ref, then update the local branch
git fetch /tmp/agent-33510401456/aw-ci-scan-feedback-kbe-shared-helix-log-catchall.bundle refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall:refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git update-ref refs/heads/ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697
git checkout ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b
# Ensure the working tree matches the updated branch
git reset --hard
# Remove the temporary bundle ref
git update-ref -d refs/bundles/create-pr-ci-scan-feedback-kbe-shared-helix-log-catchall-77d6bd95fa74a85b-75f93697

# Push the branch to origin
git push origin ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b

# Create the pull request
gh pr create --title '[ci-scan-feedback] Prevent shared-Helix-log catch-all KBE signatures' --base main --head ci-scan-feedback/kbe-shared-helix-log-catchall-77d6bd95fa74a85b --repo dotnet/runtime