Repository navigation
[pull] main from cdeust:main - #129
Merged
Merged
Conversation
…nal, tokens, energy) (#647) * feat(benchmarks): prepare the benchmark refresh (pins, per-query journal, tokens, energy) Adds what the overnight re-scoring campaign needs, without running any scored benchmark and without changing retrieval or any score: - LongMemEval-S cleaned release (98d7416c, sha256-pinned) as the opt-in --only longmemeval-cleaned leg, with a byte/record/key preflight; its artifact stem is never judged against the original file's floors. - BEAM loads Mohammadta/BEAM at a pinned revision whose data equals the only data upload; every leg writes manifest.dataset, gathered into the run MANIFEST under datasets. - Per-query journal beside --results-out (queries.jsonl, phases.jsonl) for LongMemEval, LoCoMo and BEAM, written after recall returns. - Offline token counting (count_tokens, one pinned model) of retrieved texts and the bench-equivalent recall payload, plus full-history counts. - Energy per scored query from the benchmark's own phase timeline under the existing sensor lifecycle; energy results may now be committed with their boundary stated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_012S34RdbUqS6UE48NLo1H8m * fix(bench): default count_tokens model id and legacy LME hash only when that leg ran count_tokens defaulted to claude-opus-5, which the token-counting docs never use; the default is now claude-opus-5-5, with its source. The run MANIFEST wrote the original LongMemEval-S hash for a results dir that only scored the cleaned file; the legacy field is now null unless longmemeval-s.json is there, and the per-leg datasets map stays the source of truth. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_012S34RdbUqS6UE48NLo1H8m Signed-off-by: cdeust <[email protected]> --------- Signed-off-by: cdeust <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )