Skip to content

Token report: retain selected upstream usage, status and cache reports beside each tool's row - #415

Merged
seathatflowsinourveins merged 11 commits into
mainfrom
claude/token-report-upstream-reports-20260927
Sep 29, 2026
Merged

Token report: retain selected upstream usage, status and cache reports beside each tool's row#415
seathatflowsinourveins merged 11 commits into
mainfrom
claude/token-report-upstream-reports-20260927

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Part of the token-saving practice manifest: each tool row in the private token manifest should carry that tool's own upstream report, not only the five tools with a savings counter.

Change

  • report_sources (private config) selects upstream report commands. At refresh each one is captured like the existing counters (complete stdout and stderr, exit code, hashes), then parsed and recorded on its component's coverage_matrix row.
  • A report never carries a savings value. saved is always null, and kind is one of usage report, status report, cache report or savings report. savings report covers an upstream estimate, such as OmniRoute's compression tokensSaved, which is kept as evidence and never counted.
  • Entries are validated when the configuration loads: allowed keys, nonempty name/tool/boundary, a known kind, an argv list, format json|text, timeout 1-600, unique names, no NUL in argv, name/tool/boundary/argv encodable as UTF-8 (no lone surrogate), a folded capture-folder label of at most 100 characters, and no counter_scopes value or context_roots name starting with Report / , the scope prefix reserved for reports.
  • A nonzero exit, a JSON report that fails to parse, or a report that cannot be captured at all (for example an overlong folder name) is a failed report; the refresh continues, and the last good report stays separate, as with the counters. A tool that is not a component id in manifests/stack.json is still captured, but adds an issue, so every refresh exits nonzero until it is corrected; a stack manifest that is missing, unreadable or lists no component skips that check.
  • Both HTML templates render a report row as its kind, never as an "Unavailable" or "estimated tokens" savings counter, and a report sorting before a counter no longer blanks the RTK or Headroom card.
  • The README documents the entries for this workstation's tools, each with its boundary: one ccusage source per report (ccusage claude daily and ccusage codex daily, because the bare daily mixes every detected agent), OmniRoute's retention and login boundaries, why argv is retained verbatim and how to pass a credential through curl's header-file form instead, the runtime a refresh may take within the unit's TimeoutStartSec=900, and the load-time rejections. It warns against OmniRoute's per-account routes, which carry account emails.

Why

The 2026-09-27 read-only survey found token_manifest.py ran only rtk gain, headroom savings, jCodeMunch get_session_stats and Context Mode stats files. ccusage, ai-memory, qmd, agentsview and OmniRoute have their own report commands but never reached the manifest. token-efficiency-evidence-cards assigned this gap to this session; its in-flight branches do not touch tools/token-report/. The #381 invoke metrics stay in PR-A's child-usage.mjs/skill_usage.py; this change covers only native reports.

Evidence

Measured 2026-09-29 by the coordinator at head 357d9076 (main b1be50c8 merged in by the hot-file protocol). These are local integration checks and repository unit tests, not upstream tests.

  • python3 -B -m unittest discover -s tools/token-report -p 'test_*.py': 72 tests OK (main has 56). node --test tools/token-report/test_returned_results.cjs: 3 pass, 0 fail.
  • python3 -B scripts/validate.py: {"components": 69, "hashed_files": 7994, "profiles": 4, "receipts": 168, "status": "passed"}; scripts/evidence_manifest.py --check files 7994 passed; scripts/component_matrix.py --check rows 32; scripts/new_host_grand_list.py --check passed, 32 layers; scripts/build_ecosystem.py --check passed; python3 -B -m unittest tests.test_ecosystem_manifest tests.test_native_dashboard_data tests.test_omniroute_gateway_unit tests.test_token_e2e_preregistration: 148 tests OK.
  • Compare parity: a local scratch script (not committed) ran token_manifest.py compare over the 25 retained ledger pairs that have both files present, once with main's copy of the script and once with this branch's, and diffed the payloads: 25 byte-identical after timestamps and paths are scrubbed, 0 differing, 0 failures in either run.
  • Red first: the 4 tests in the original PR failed before the change (9 failures, 2 errors). In the repair rounds an independent verifier re-ran the two round-2 tests against the pre-fix code and both failed (exit 1); the tool and boundary cases added on this head failed 2 subtests before the fix and pass after it.
  • Not done: no host config was changed. The token-report-refresh.timer runs from the live clone, so enabling report_sources on the workstation is a follow-up after merge.

Review rounds (2026-09-29, native-agent-stack-10)

Round 1, head 78f61121. GPT-6 (gpt-6-astra, effort max, codex exec -s read-only; the model is the -m request, not an observed resolution): needs_changes (2 major, 2 minor): a counter_scopes value starting with Report / could replace a counter's last good value; an argv NUL aborted the whole refresh; an unknown tool was stored silently; the hot-file edit sat in two commits (moot under a squash merge). Claude security-reviewer (Opus, max): needs_changes (4 minor, 3 nit): one entry that raises aborts the refresh, unbounded output and stdin, no secret-safe credential path, double storage of each snapshot, the scope rule held only by convention, a relative argv[0]. Claude evidence-reviewer (Opus, max): needs_changes (1 major, 7 minor, 2 nit): the README's bare ccusage daily mixes every detected agent; OmniRoute's cache boundary and login requirement; both HTML templates rendered reports as savings counters; timeout budget; the body's test counts. Round 1 repair (d04f8c76) fixed isolation, NUL and label length, the reserved scope prefix, the unknown-tool issue and the README boundaries (69 tests); the second builder round (a470293e, 72 tests) fixed the ccusage examples, the OmniRoute retention and login paragraphs, both templates, UTF-8 rejection, the empty component list and the timing budget. Each round had an independent Opus verifier that re-ran the acceptance commands.

Round 2 verifier (ready_to_push true, 3 minor and 5 nit): fixed on this head: a lone surrogate in tool or boundary passed load_config and aborted the refresh at the sqlite bind or the UTF-8 manifest write (now rejected with the name and argv check); the failed-refresh line on a report card said the value above was an observation (it now says the kind is configured, not observed); the README named an OmniRoute build the stack does not pin (it names the 3.8.50 pin; the verifier read every cited source line at upstream v3.8.50 too) and listed two credentials where the management policy accepts more; the README wrap. Left as residuals below.

Residuals

  • ccusage claude daily --help (20.0.26) does not list --no-cost, which the subcommand applies (measured exit 0, cost fields omitted); the README example is valid and the help text is an upstream gap.
  • A successful report card shows its kind twice, as the headline and in the small line (cosmetic, disclosed by the builder).
  • Not exercised: a login-protected OmniRoute answering 401 through curl exit 22 to a failed report and a nonzero refresh (source- and manual-derived); the roughly 35-second steady-state refresh with a 106,000-row RTK history, measured once on one host.
  • The hot-file edit to manifests/evidence.json sits in earlier branch commits than the last one (moot under a squash merge).

SOTA sources

Each optional report source runs an upstream tool's own documented report command. This PR only captures their output.

🤖 Generated with Claude Code

Scout and others added 2 commits September 27, 2026 12:16
Scout @claude
Token report: retain selected upstream usage, status and cache report…
431eaaf 
…s beside each tool's row

report_sources runs each selected upstream report command at refresh (ccusage daily,
ai-memory status, qmd status, agentsview usage, OmniRoute cache and compression
analytics), retains its complete output with the other captures, and records the
parsed report on that component's coverage row. A report never carries a savings
value: saved is always null, so nothing is added to a counter. Entries are validated
when the configuration loads; a failed or unparsable report stays separate from the
last good one. Four tests failed before the change and pass after it (56 -> 60).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Scout @claude
Re-register the changed token-report files in manifests/evidence.json
ec77696 
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 27, 2026
Scout @claude
Token report: repair round from the GPT-6 review of report_sources
0aafd9c 
- Names that fold to the same capture folder are rejected when the config loads
  (report_label), instead of aborting the refresh with FileExistsError.
- Reports are stored under the scope "Report / <name>", which never equals a
  counter's scope, so a report cannot replace a counter's last good value.
- The report's tool goes through native_tool_identity, so the documented
  component id (jcodemunch-mcp) joins its coverage row.
- README curl examples use --fail-with-body, so an HTTP error exits nonzero.
- New tests: last good report kept on the same ledger after a real
  TimeoutExpired (partial output retained, nondefault timeout forwarded);
  report and counter scopes stay separate; the component id joins its row.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins

seathatflowsinourveins commented Sep 27, 2026

Copy link
Copy Markdown
Collaborator Author

Cross-family review, one repair round.

Review. GPT-6 (gpt-6-astra, effort max) ran through the packaged landscape-sweep lane (codex_call.sh, read-only sandbox) against head ec77696b. Verdict: request_changes, with 4 medium findings and 1 low. GPT-6 reproduced each finding in isolation: in-memory SQLite and virtualized writes, because the sandbox has no writable tmp.

  1. Medium: capture-folder collision. Report names that differ only in punctuation, such as "usage daily" and "usage-daily", passed the uniqueness check but mapped to the same capture folder, so capture() raised FileExistsError and aborted the refresh.
  2. Medium: report and counter identity collision. A report with tool: rtk and the name of a counter's scope shared the counter's (tool, scope) ledger key.
  3. Medium: component id did not join. The documented component id jcodemunch-mcp did not reach its coverage row, because the report path skipped native_tool_identity.
  4. Medium: HTTP errors looked successful. The README's curl examples had no --fail-with-body, so an HTTP 401 or 500 with a JSON body exited 0 and was taken as a good report.
  5. Low: test gaps. The tests did not cover keeping the last good report on the same ledger, a real TimeoutExpired, or forwarding of a nondefault timeout.

Repair (0aafd9c9) resolves all five:

  • report_label() uniqueness is checked when the config loads;
  • reports use the Report / <name> scope, which never equals a counter's scope;
  • the report's tool goes through native_tool_identity;
  • the README examples use curl --fail-with-body;
  • 3 new tests.

Checks: 63 tests OK, validate.py passed, evidence_manifest.py --check passed.

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 27, 2026
Merged
Scout and others added 8 commits September 28, 2026 22:33
Scout @claude
Merge origin/main into PR #415 branch for fresh CI (no conflicts)
78f6112 
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Scout @claude
report_sources repair round: one failing capture no longer aborts ref…
d875b31 
…resh; reserved report scope prefix; unknown-tool issue; argv/stdin/memory notes

F1: capture_report_sources records a failed snapshot (saved null, kind,
boundary, error; evidence holds argv, cwd and the error, never output)
and an issue of the existing failed-report shape when capture() raises
OSError or ValueError, then continues with the next entry. Python 3.13
docs: subprocess raises ValueError for invalid Popen arguments (an
embedded NUL); os functions raise OSError for names the OS rejects
(ENAMETOOLONG). validate_report_sources now rejects a NUL in any argv
element and a name whose folded report_label exceeds 100 characters.

F2: REPORT_SCOPE_PREFIX ("Report / ") is a module constant; counter_scopes
values and context_roots names that start with it are rejected in
load_config and at the start of refresh(). Default scopes are unchanged.

F3: a report entry whose tool is not a component id in the stack
manifest adds an issue naming the entry and the tool; the report is
still captured and recorded. A missing or unreadable manifest skips
the check.

F4: README states that argv is stored literally, that commands inherit
the reporter's environment, stdin and the project working directory,
and that a successful JSON report is stored twice per snapshot.

Tests added red first (6 tests; 7 failures and 1 error before the fix).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Evidence registry: re-register the token-report files changed by the …
d04f8c7 
…report_sources repair round

Hot-file protocol (docs/lanes.md): manifests/evidence.json in the last
commit only. Re-registered tools/token-report/token_manifest.py,
test_token_manifest.py and README.md; component_matrix.py --write and
new_host_grand_list.py --write left their reports unchanged;
scripts/validate.py passed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
report_sources round 2: report rows render their kind; UTF-8 name/arg…
bb19896 
…v check; empty component list skips the tool check

A4: both HTML templates rendered a report row (scope "Report / <name>", saved
null) as a savings counter with the headline "Unavailable". The portable
card and the full view's native_reports item now show the report's kind
(usage/status/cache/savings report) and its scope, never "Unavailable" or
"estimated tokens"; the full view's good() lookup skips report rows, so a
report scope that sorts before a counter no longer blanks the RTK or
Headroom card. A failed report keeps the failed-counter state. Counter
rows render byte-identically to the previous templates (scratch A/B on
counter-only data). The new test renders a real refresh with both
templates and runs each template's own script in node against a small
fake DOM, like test_returned_results.cjs; it is skipped only without node.

A6: validate_report_sources rejects a name or argv element that UTF-8
cannot encode (a lone surrogate such as "\ud800" or "\udcff" from JSON),
which otherwise aborted the whole refresh at the UTF-8 manifest write.

A7: stack_component_ids returns None for an empty components list, like a
missing or unreadable manifest, so the caller skips the unknown-tool check
instead of flagging every report tool.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Token report README: one ccusage source per report, OmniRoute retenti…
23523d6 
…on and login, timeout budget, load-time rejections

A1: the ccusage example ran bare `ccusage daily`, which covers every
detected source (ccusage 20.0.26 README L85, L115-116), under a Claude Code
boundary. It is now `ccusage claude daily` (README L66) with a Claude-only
boundary, plus a `ccusage codex daily` entry (L67) with a Codex-only
boundary. Both focused forms accepted --offline --no-cost --json on empty
scratch data (exit 0, cost fields omitted; a bogus flag exits 2).

A2: OmniRoute 3.8.51 computes /api/cache promptCache totals from
usage_history (src/lib/db/settings/cacheMetrics.ts), which cleanup prunes
at retention.usageHistory (src/lib/db/cleanup.ts L100-135; default 30,
src/types/databaseSettings.ts L161), and the response also carries
upstream savings estimates. since=all on the compression route drops the
time filter over compression_analytics, pruned at 30 days by default
(databaseSettings.ts L152, cleanup.ts L149-171). Both boundaries now say so.

A3: both routes answer without a credential only while requireLogin is
false (upstream default true, src/lib/db/settings.ts L176) or during
first-run setup from loopback (src/shared/utils/apiAuth.ts isAuthRequired);
otherwise a dashboard session or a manage-scope Bearer key in a header.
The README states the 401 / curl exit 22 / nonzero refresh consequence and
curl's -H @file form (curl --manual: added in 7.55.0), with the file kept
0600 outside every worktree, referring back to the existing argv paragraph.

A5: the scheduled unit's TimeoutStartSec=900 (token-report-refresh.service
L135, Type=oneshot) bounds the sequential captures; one steady-state
refresh measured about 35 seconds on one host (coordinator, 2026-09-29
UTC). The load-time rejections from the current code (NUL, UTF-8, folded
label over 100, reserved Report / prefix) and the unknown-tool issue are
documented in one paragraph.

A8: the OmniRoute examples give curl as /usr/bin/curl, to adjust per host.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Evidence registry: re-register the token-report files changed by the …
a470293 
…second report_sources repair round

docs/lanes.md hot-file protocol: the five changed token-report files
(README.md, test_token_manifest.py, token_manifest.py and both HTML
templates) re-registered with host_receipts.register_file; component_matrix
--write and new_host_grand_list --write regenerated their reports with no
change; scripts/validate.py passed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Scout @claude
Token report: reject lone surrogates in tool and boundary; failed rep…
64a2fc0 
…ort card says its kind is configured; README names the pin

Round-2 verifier minors on the report_sources repair: a lone surrogate in a
report's tool or boundary passed load_config and aborted the whole refresh at
the sqlite bind or the UTF-8 manifest write; the failed-refresh line on a
report card said the value above was an observation when it is a configured
kind; the README named an OmniRoute build the stack does not pin and listed
only two of the credentials the management policy accepts. The README wrap is
repaired and its load-time rejection list now names tool and boundary.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Scout @claude
Merge origin/main b1be50c (hot-file protocol: main's evidence manifes…
357d907 
…t, branch files re-registered)

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 29, 2026
Closed
@seathatflowsinourveins

seathatflowsinourveins commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator Author

Review record, 2026-09-29 (native-agent-stack-10). Merge candidate: head 357d9076, which is 78f61121 plus two repair rounds, a small third fix and a merge of main at b1be50c8 by the hot-file protocol.

  • Three reviews of 78f61121, read-only, two model families: GPT-6 (gpt-6-astra, effort max, codex exec -s read-only; the model is the -m request, not an observed resolution): needs_changes (2 major, 2 minor). Claude security-reviewer (Opus, max): needs_changes (4 minor, 3 nit). Claude evidence-reviewer (Opus, max): needs_changes (1 major, 7 minor, 2 nit). Nothing was disputed; the details and the fixes are in the body's "Review rounds" section.
  • Repair: round 1 (d04f8c76, an Opus builder, then an independent Opus verifier): failure isolation for one report, NUL and label-length rejection, the reserved Report / scope prefix, the unknown-tool issue. Round 2 (a470293e, a second Opus builder, then a second verifier that returned ready_to_push with 3 minors and 5 nits): the ccusage example split by agent, OmniRoute retention and login boundaries, report rows rendered as their kind in both templates, UTF-8 rejection, the empty component list, the timing budget. The coordinator fixed the verifier's three minors on this head (a lone surrogate in tool or boundary, the failed-report card wording, the README naming an unpinned OmniRoute build); that last change has a red-first test but no second independent review.
  • Checks on the merge candidate (local integration checks, not upstream tests): python3 -B -m unittest discover -s tools/token-report -p 'test_*.py' 72 OK (main has 56); node --test tools/token-report/test_returned_results.cjs 3 pass; scripts/validate.py, scripts/evidence_manifest.py --check, scripts/component_matrix.py --check, scripts/new_host_grand_list.py --check and scripts/build_ecosystem.py --check exit 0; tests.test_ecosystem_manifest tests.test_native_dashboard_data tests.test_omniroute_gateway_unit tests.test_token_e2e_preregistration 148 OK. token_manifest.py compare over the 25 retained ledger pairs gives byte-identical payloads for main's script and this branch's (0 differing). Hosted required checks are read with gh pr checks 415 --required immediately before merging.
  • Residuals, in the body: ccusage claude daily --help omits --no-cost (upstream help gap); a successful report card shows its kind twice; a login-protected OmniRoute answering 401 and the roughly 35-second steady-state refresh are not exercised here.

@seathatflowsinourveins
seathatflowsinourveins merged commit 3ce6f53 into main Sep 29, 2026
25 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/token-report-upstream-reports-20260927 branch September 29, 2026 05:07
@seathatflowsinourveins seathatflowsinourveins mentioned this pull request Sep 29, 2026
Merged
4 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@seathatflowsinourveins