Skip to content

[pull] main from cdeust:main - #129

Merged
pull[bot] merged 1 commit into
asleekgeek:mainfrom
cdeust:main
Sep 29, 2026
Merged

pull[bot] merged 1 commit into
asleekgeek:mainfrom
cdeust:main

Conversation

@pull

@pull pull Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

See Commits and Changes for more details.


Created by pull[bot] (v2.0.0-alpha.4)

Can you help keep this open source service alive? 💖 Please sponsor : )

…nal, tokens, energy) (#647)

* feat(benchmarks): prepare the benchmark refresh (pins, per-query journal, tokens, energy)

Adds what the overnight re-scoring campaign needs, without running any
scored benchmark and without changing retrieval or any score:

- LongMemEval-S cleaned release (98d7416c, sha256-pinned) as the opt-in
  --only longmemeval-cleaned leg, with a byte/record/key preflight; its
  artifact stem is never judged against the original file's floors.
- BEAM loads Mohammadta/BEAM at a pinned revision whose data equals the
  only data upload; every leg writes manifest.dataset, gathered into the
  run MANIFEST under datasets.
- Per-query journal beside --results-out (queries.jsonl, phases.jsonl) for
  LongMemEval, LoCoMo and BEAM, written after recall returns.
- Offline token counting (count_tokens, one pinned model) of retrieved
  texts and the bench-equivalent recall payload, plus full-history counts.
- Energy per scored query from the benchmark's own phase timeline under
  the existing sensor lifecycle; energy results may now be committed with
  their boundary stated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012S34RdbUqS6UE48NLo1H8m

* fix(bench): default count_tokens model id and legacy LME hash only when that leg ran

count_tokens defaulted to claude-opus-5, which the token-counting docs never
use; the default is now claude-opus-5-5, with its source. The run MANIFEST
wrote the original LongMemEval-S hash for a results dir that only scored the
cleaned file; the legacy field is now null unless longmemeval-s.json is there,
and the per-leg datasets map stays the source of truth.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_012S34RdbUqS6UE48NLo1H8m
Signed-off-by: cdeust <[email protected]>

---------

Signed-off-by: cdeust <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
@pull pull Bot locked and limited conversation to collaborators Sep 29, 2026
@pull pull Bot added the ⤵️ pull label Sep 29, 2026
@pull
pull Bot merged commit 3d1cae5 into asleekgeek:main Sep 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant