fix(matchers): preserve reserved rag grader vars#9994
Merged
mldangelo merged 2 commits intoJul 11, 2026
Conversation
![promptfoo-scanner[bot]](https://avatars.githubusercontent.com/in/2172945?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
👍 All Clear
Reviewed matcher changes in src/matchers/rag.ts and related tests. The PR reorders variable merging so reserved grading variables take precedence over user-supplied vars when constructing prompts and call contexts. No new tools, sinks, or sensitive data handling changes were introduced. No LLM security vulnerabilities were identified.
Minimum severity threshold: 🟡 Medium | To re-scan after changes, comment @promptfoo-scanner
Learn more
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #9994 +/- ##
==========================================
+ Coverage 79.50% 79.51% +0.01%
==========================================
Files 925 925
Lines 74314 74315 +1
Branches 23917 23917
==========================================
+ Hits 59083 59092 +9
+ Misses 15231 15223 -8
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
mldangelo
pushed a commit
to Gujiassh/promptfoo
that referenced
this pull request
Jul 13, 2026
…d matchers Extends promptfoo#9994's spread-ordering fix from the rag matchers to llm-rubric, factuality, model-graded-closedqa, select-best, search-rubric, and agent-rubric, so user vars named after reserved grader keys (output, rubric, input, ideal, completion, criteria, outputs) can no longer replace the matcher-built grading values in the rendered prompt or provider-call vars.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For the default
context-recallandcontext-faithfulnesspaths, user vars likecontext,groundTruth, andstatementscould replace the matcher-built grader values becausevarsrendered last. This keeps those reserved matcher values authoritative at the matcher render and provider-call sites, while leaving #9939's opt-ingraderVarsallowlist path untouched. I added focused recall and faithfulness tests that fail when user vars clobber the rendered grader prompt and provider vars, then pass with the reserved values preserved. I checked the focused matcher suite,npm test,npm run lint,npm run tsc, andnpm run format:check. Fixes #9977