Repository navigation
Teach the validator the StructuredJson protocol (3.7) - #192
Merged
Merged
Conversation
The first StructuredJson run was rejected on every question in both arms. One root cause: the validator reconciled AgentEval's own verdict against our re-parse of the free-text explanation, and the same failed parse triggered the diagnostic judge retry once per question, driving base judge calls to zero and tripping the call-accounting bound. Both symptoms are now fixed by plumbing the active protocol into LongMemEvalRunValidator and LongMemEvalPostRunDiagnostics, neither of which previously knew which judge had run. Under StructuredJson the free-text reconciliation is skipped and the diagnostic retry is suppressed. That is not a relaxed guard. The reconciliation exists to catch AgentEval's FREE-TEXT parser mis-scoring -- "yes, there is no discrepancy" read as a no -- which is exactly the failure StructuredJson eliminates at the source. Re-parsing prose to second-guess a structured verdict checks something the protocol removed, and gets it wrong. QuestionResult exposes no structured verdict property, so AgentEval's Correct is the verdict. The call bounds were deliberately NOT widened. With the retry suppressed they pass untouched, and bounds loose enough to admit both judge shapes are loose enough to admit real anomalies. Only the retry is suppressed, not the whole diagnostics pass -- the oracle loop is separate and still runs, so no diagnostic coverage is lost. Verified from both sides in unit tests: FreeText still catches a verdict/correctness disagreement and still rejects an unusable verdict; StructuredJson does neither; and the default is still FreeText, because every sealed base here was scored under it. Verified end to end by re-running the exact command that failed: longmemeval-prepared-20260813T011536Z, accepted, zero validation issues, both arms accepted, fingerprint.judgeVerdictProtocol = StructuredJson. 4,370 unit and 430 LongMemEval tests green (+5). Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE
There was a problem hiding this comment.
Pull request overview
This PR updates the LongMemEval validator/diagnostics pipeline to understand which judge verdict protocol (FreeText vs StructuredJson) was used, preventing StructuredJson runs from being incorrectly rejected and from triggering free-text repair retries that distort call accounting.
Changes:
- Thread
JudgeVerdictProtocolintoLongMemEvalRunValidator.Validate(...)and skip free-text explanation re-parsing when the protocol isStructuredJson. - Thread
JudgeVerdictProtocolintoLongMemEvalPostRunDiagnosticsto suppress the free-text judge-retry loop underStructuredJson. - Add unit tests pinning protocol-specific validator behavior and update the review doc with the verified end-to-end rerun outcome.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/AgentMemory.LongMemEval/LongMemEvalRunValidator.cs | Adds verdictProtocol and skips the free-text reconciliation step under StructuredJson. |
| tools/AgentMemory.LongMemEval/LongMemEvalPreparedPairProgram.cs | Plumbs the selected judge protocol into diagnostics and validator for prepared-pair runs. |
| tools/AgentMemory.LongMemEval/LongMemEvalPostRunDiagnostics.cs | Adds verdictProtocol and suppresses free-text retry behavior under StructuredJson. |
| tests/AgentMemory.Tests.Unit.LongMemEval/StructuredJsonValidatorTests.cs | New tests asserting protocol-specific reconciliation behavior and default protocol stability. |
| docs/reviews/structured-json-judge-blocker.md | Updates the blocker write-up to reflect the verified fix and rerun command/result. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+286
to
+287
| if (verdictProtocol == JudgeVerdictProtocol.StructuredJson) | ||
| continue; |
Comment on lines
+112
to
116
| // judge-call count and tripped the bound. Only the RETRY is suppressed; the oracle pass below | ||
| // is a separate loop and still runs, so no diagnostic coverage is lost. | ||
| foreach (var question in questionResults.Where(q => | ||
| verdictProtocol != JudgeVerdictProtocol.StructuredJson && NeedsJudgeRetry(q))) | ||
| { |
Comment on lines
+14
to
+16
| /// The validator compares AgentEval's own <c>Correct</c> against <b>our re-parse of the free-text | ||
| /// explanation</c>. That cross-check exists to catch AgentEval's free-text parser mis-scoring — "yes | ||
| /// — there is no discrepancy" read as a no — which is a real, systematic failure. |
This was referenced Aug 27, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first StructuredJson run was rejected on every question in both arms. One root cause: the validator reconciled AgentEval's own verdict against our re-parse of the free-text explanation, and that same failed parse triggered the diagnostic judge retry once per question — driving base judge calls to zero and tripping the call-accounting bound.
Fixed by plumbing the active protocol into
LongMemEvalRunValidatorandLongMemEvalPostRunDiagnostics, neither of which knew which judge had run.This is not a relaxed guard. The reconciliation exists to catch AgentEval's free-text parser mis-scoring — "yes, there is no discrepancy" read as a no — which is exactly what StructuredJson eliminates at the source. Re-parsing prose to second-guess a structured verdict checks something the protocol removed, and gets it wrong.
QuestionResultexposes no structured verdict property, so AgentEval'sCorrectis the verdict.The call bounds were deliberately not widened. With the retry suppressed they pass untouched, and bounds loose enough to admit both judge shapes are loose enough to admit real anomalies. Only the retry is suppressed, not the whole diagnostics pass — the oracle loop is separate and still runs.
Pinned from both sides in unit tests: FreeText still catches a verdict/correctness disagreement and still rejects an unusable verdict; StructuredJson does neither; the default is still FreeText, because every sealed base was scored under it.
Verified end to end by re-running the exact command that failed:
longmemeval-prepared-20260813T011536Z— accepted, zero validation issues, both arms accepted,fingerprint.judgeVerdictProtocol = StructuredJson.4,370 unit and 430 LongMemEval green (+5).
🤖 Generated with Claude Code
https://claude.ai/code/session_01PgDgctPpbTziBNE2RT8VdE