Problem

compactor::estimate_token_count drives compaction triggers, the context-usage report, and cost estimation, but it is a character-weighted heuristic (CJK ~1.5 tokens, other chars/4) with no ground truth behind it.
There is no way to know how far off it is, so any tuning would be a guess and any regression would be silent.

Change

Adds a token-estimation accuracy benchmark — an offline harness, a live capture tool, and two measured baselines with different tokenizers — without changing the estimator or adding a dependency.

• corpus.json — 16 message samples, 4 per content bucket (english_prose, cjk, mixed_cjk_latin, tool_json).
  Stored as plain content, not serialized Message objects, so it stays readable and decoupled from message-struct serde changes.
• token_estimation_accuracy.rs — offline harness.
  Builds Messages from the corpus, asserts the estimator is positive and deterministic over every sample, and reports per-bucket signed error, MAE%, and MAE tokens against each captured baseline.
  It auto-discovers every tokens_truth*.json file and reports them separately, so multiple provider/tokenizer baselines are first-class.
  A regression ceiling is opt-in via LIBREFANG_TOKEN_EST_MAX_MAE_PCT (applied per baseline), so the benchmark is green offline and CI stays deterministic.
• examples/capture_token_truth.rs — the live, human-run half.
  Sends each sample once with max_tokens=1 and prompt caching disabled, records usage.input_tokens.
  Mirrors the harness's message builder so the bytes scored match the bytes sent.
  Retries with backoff and paces requests so free-tier rate limits do not abort a run.
  --label records the real provider when an OpenAI-compatible backend (Zhipu/GLM, OpenRouter, Groq, Moonshot) is driven via --provider openai.
  CI never invokes it.
• tokens_truth.json — Zhipu GLM glm-4-flash (efficient-CJK tokenizer).
• tokens_truth_gptoss.json — openai/gpt-oss-120b via OpenRouter (o200k tokenizer family, a close stand-in for GPT-4o-class OpenAI models).

Baseline findings (16 samples, 4 per bucket)

Mean signed error of estimate_token_count vs real input_tokens (positive = overestimate):

Reading the two tokenizers together is the point — it separates tokenizer-specific error from cross-provider error:

• The CJK error is strongly tokenizer-specific: +126% on GLM but only +18% on the OpenAI-style tokenizer.
  GLM tokenizes Han text very efficiently (large Chinese vocabulary), so the 1.5-tokens-per-CJK-char weight overshoots for GLM while being roughly reasonable for o200k.
  This PR therefore does not change the CJK weight — doing so on single-provider evidence would help GLM and hurt OpenAI.
• The tool_json under-estimate is the cross-provider signal: both tokenizers undercount JSON-heavy tool steps (-14% GLM, -46% o200k).
  JSON structure and escaping in tool calls are weighted at the flat 0.25/char and undercount.
  Because the sign agrees across tokenizers, that JSON/escaping path is the safe, language-independent candidate for a later tuning change.

Scope

Infrastructure plus a measured finding.
It deliberately does not touch estimate_token_count; a tuning PR can use this harness to prove an improvement rather than assert one.

Verification

• cargo test -p librefang-runtime --test token_estimation_accuracy (2 passed)
• cargo check -p librefang-llm-drivers --example capture_token_truth (clean)
• cargo clippy -p librefang-runtime --test token_estimation_accuracy -- -D warnings (clean)
• cargo clippy -p librefang-llm-drivers --example capture_token_truth -- -D warnings (clean)

Out-of-scope follow-ups

• Add OpenAI / Anthropic ground-truth files (the harness already picks up any tokens_truth*.json).
• Tune the tool_json (JSON/escaping) estimation path, gated by a committed per-baseline MAE ceiling.