perf(runtime): JSON-aware token estimation for tool paths#6281
Merged
Conversation
estimate_token_count weighted every non-CJK character at a flat 0.25,
which systematically undercounts JSON-heavy tool steps: serialized
tool-call inputs, tool results, and tool-definition schemas are dense in
structural punctuation ({}[]":,) and escapes that real tokenizers split
into their own tokens.
The token-estimation accuracy benchmark (librefang#6269) measured this as the
cross-provider signal: the tool_json bucket is undercounted by both
committed baselines and the sign agrees (-14% GLM, -46% o200k), unlike
the tokenizer-specific CJK error.
Add estimate_json_str_tokens, which counts JSON structural punctuation at
JSON_STRUCT_TOKEN_WEIGHT (0.5) while keeping CJK/alphanumeric runs at
their normal weights, and route the three tool paths through it. Prose
estimation is unchanged.
0.5 is calibrated against both baselines as the value that improves or
holds each without regressing any other bucket:
tool_json before (signed / MAE tok) after
GLM -14% / 21.0 -3% / 20.2
o200k -46% / 103.8 -39% / 88.5
cjk/english_prose/mixed_cjk_latin are byte-identical before/after. A
single global weight is a compromise because GLM tokenizes JSON structure
more efficiently than o200k; the residual o200k undercount is the known
limit of a tokenizer-independent heuristic.
Verification:
- cargo clippy -p librefang-runtime --all-targets -- -D warnings (clean)
- cargo test -p librefang-runtime --lib compactor::tests (46 passed,
incl. 2 new: structural-weight and tool-call-vs-prose)
- cargo test -p librefang-runtime --test token_estimation_accuracy
--nocapture (after-table above)
maoxin1234
force-pushed
the
feat/token-est-json-weight
branch
from
June 23, 2026 02:01
71f75a8 to
d03ec7d
Compare
github-actions
Bot
added
area/docs
houko
added a commit
that referenced
this pull request
Jun 23, 2026
…top (#6285) CHANGELOG.md had two `## [Unreleased]` sections: one buried between released version sections (it predates #6281, which then added a duplicate at the top). The release tooling reads only the first [Unreleased] and silently drops the rest, so ~150 stranded entries were never reaching release notes; the duplicate header also tripped the `## [Unreleased]` guard, blocking every CHANGELOG-touching commit (e.g. #5988). Consolidate both blocks into a single [Unreleased] at the top, merging same-named `###` subsections (collapsed 10 redundant subsection headers). Every bullet is preserved verbatim — verified that the multiset of non-header content lines is byte-identical before/after. Drop the two entries already published in released sections: #6272 (in [2026.6.22]) and #6171 (in [2026.6.17]). Co-authored-by: Evan <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
compactor::estimate_token_countweighted every non-CJK character at a flat 0.25.Serialized JSON on the tool paths — tool-call inputs, tool results, and tool-definition schemas — is dense in structural punctuation (
{}[]":,) and escapes that real tokenizers split into their own tokens, so the flat weight systematically undercounts JSON-heavy tool steps.This PR adds
estimate_json_str_tokens, which counts JSON structural characters atJSON_STRUCT_TOKEN_WEIGHT = 0.5while leaving CJK and alphanumeric runs (the string values inside the JSON) at their normal weights, and routes the three tool paths through it.Prose estimation is untouched.
Why this change, and why 0.5
The token-estimation accuracy benchmark added in #6269 measured the
tool_jsonundercount as the cross-provider signal: both committed baselines undercount it and the sign agrees (−14% GLM, −46% o200k), unlike the tokenizer-specific CJK error (which #6269 explicitly warns not to tune). This PR acts on exactly that signal.0.5is calibrated against both baselines — the value that improves or holds eachtool_jsonerror without regressing any other bucket:cjk,english_prose, andmixed_cjk_latinare byte-identical before/after — they never reach the JSON path.GLM tokenizes JSON structure more efficiently than o200k, so a single global weight is a deliberate compromise: 0.5 removes GLM's systematic bias and roughly halves the gap on o200k. The residual o200k undercount is the known limit of a tokenizer-independent heuristic — closing it fully would over-count GLM.
Changes
crates/librefang-runtime/src/compactor.rs— addJSON_STRUCT_TOKEN_WEIGHT+estimate_json_str_tokens; routeToolUseinput,ToolResultcontent, and tool-definition schemas through it; 2 new unit tests.crates/librefang-runtime/tests/fixtures/token_estimation/README.md— record the tuning and the before/after table.CHANGELOG.md—[Unreleased]entry.Verification
cargo clippy -p librefang-runtime --all-targets -- -D warnings— clean.cargo test -p librefang-runtime --lib compactor::tests— 46 passed (incl. 2 new:test_estimate_json_str_tokens_weights_structure_above_prose,test_tool_use_estimated_above_equivalent_prose).cargo test -p librefang-runtime --test token_estimation_accuracy -- --nocapture— both baseline tables reproduce the after-numbers above; the offline invariant test still passes.