Skip to content

fix(core): parse MiMo Code and fx on the local report path - #1300

Merged
junhoyeo merged 2 commits into
junhoyeo:mainfrom
MaxFreedomPollard:fix/local-parse-micode-fx
Sep 6, 2026
Merged

junhoyeo merged 2 commits into
junhoyeo:mainfrom
MaxFreedomPollard:fix/local-parse-micode-fx

Conversation

@MaxFreedomPollard

@MaxFreedomPollard MaxFreedomPollard commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

MiMo Code and fx usage never shows up in tokscale report, and tokscale clients prints a zero message count for both, even though both clients scan and submit fine.

Cause

MiMoCode and Fx both declare parse_local: true in define_clients! (crates/tokscale-core/src/clients.rs), and the submit path parses both: MiMo Code from scan_result.micode_dbs at crates/tokscale-core/src/lib.rs:2047, fx from scan_result.get(ClientId::Fx) at crates/tokscale-core/src/lib.rs:3133. parse_local_clients (same file, from line 5134) has a block for every other locally parsed client and none for these two, so it returns no rows and sets no count for either.

Everything built on parse_local_clients therefore sees nothing. tokscale report builds its whole table from the wiki DB, and populate_wiki_from_sessions (crates/tokscale-cli/src/commands/report.rs:114) fills that DB from parse_local_clients alone, so a MiMo Code or fx session is never recorded. run_clients_command (crates/tokscale-cli/src/main.rs:4266) asks for every parse_local() client and then prints parsed.counts.get(client), so both rows read 0 no matter how much local data exists.

This is the mirror of #986, where opencodereview was parsed locally but missing from the submit path.

Fix

MiMo Code goes in after the OpenCode block and reads scan_result.micode_dbs, deduplicating on the payload message id through the existing should_keep_deduped_message helper. That part matters: MiMo writes channel-suffixed databases, so one session can appear in both mimocode.db and mimocode-<channel>.db, and the submit lane already collapses them the same way.

fx goes in after the Mux block and reads scan_result.get(ClientId::Fx). Session titles are deliberately not applied here, because ParsedMessage has no title field for the shared sessions/index.json lookup to write to.

Both count through summed_parsed_message_count rather than the row count, so an fx row contributes its request_count.

Verification

The new test writes a ~/.local/share/mimocode/mimocode.db with one assistant row and a ~/.fx/sessions/sess-1/usage-v2.json snapshot, then asserts both reach parse_local_clients with their token buckets and client counts.

With the test applied to an otherwise unmodified tree:

$ cargo test --locked -p tokscale-core --all-features --lib -- test_parse_local_clients_includes_micode_and_fx
assertion `left == right` failed: MiMo Code assistant row must reach the local parse, got []
  left: 0
 right: 1
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2110 filtered out

With the fix:

$ cargo fmt --all -- --check
(clean)

$ cargo clippy --locked -p tokscale-core --all-features --all-targets -- -D warnings
(no warnings)

$ cargo test --locked -p tokscale-core --all-features --no-fail-fast
test result: ok. 2109 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out
(plus the eight integration binaries, all ok)

The same suite on main before the change was 2108 passed, 0 failed, so the only difference is the new test. Ran on macOS 15 with rustc 1.95.0. The test uses only tempdirs and fixed literals, with no clock, locale or timezone dependence, so it behaves the same on the Windows leg.


Summary by cubic

Fixes the local report path so MiMo Code and fx usage now appears in tokscale report and tokscale clients instead of always showing zero, and preserves provider-reported dates and costs so the figures are correct.

  • parse_local_clients now parses MiMo Code from scan_result.micode_dbs, deduplicating by payload message id since one session can appear in both mimocode.db and mimocode-<channel>.db, and fx from per-session usage snapshots.
  • fx rows now derive their date before conversion so year/since/until filters work on the local lane.
  • ParsedMessage now carries cost and cost_source; compute_msg_cost uses a provider-reported cost as-is, so a token-free fx session records its snapshot cost instead of $0.00.
  • Both clients count through summed message counts, so an fx row contributes its request_count.

Written for commit 8891eb3. Summary will update on new commits.

Review in cubic

MiMoCode and Fx both declare `parse_local: true` in `define_clients!`, and `parse_all_messages_with_pricing_with_env_strategy` parses both, so their usage reaches submit and the TUI. `parse_local_clients` in `crates/tokscale-core/src/lib.rs` had no block for either client, so everything built on that function saw nothing. `tokscale report` fills its wiki DB from `populate_wiki_from_sessions`, which reads only `parse_local_clients`, so no MiMo Code or fx session ever reached the report table, and `tokscale clients` printed a zero message count for both even though it explicitly asks for every `parse_local()` client.

This is the mirror of #986, where opencodereview was parsed locally but missing from the submit path.

Read `scan_result.micode_dbs` after the OpenCode block, deduplicating on the payload message id the way the submit lane does, because MiMo writes channel-suffixed databases and one session can appear in both `mimocode.db` and `mimocode-<channel>.db`. Read `scan_result.get(ClientId::Fx)` after the Mux block. Both count through `summed_parsed_message_count`, so an fx row contributes its `request_count` rather than one. fx session titles are not applied here: `ParsedMessage` has no title field for the shared `sessions/index.json` lookup to write to.

`test_parse_local_clients_includes_micode_and_fx` fails on main with zero rows for both clients.
@vercel

vercel Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
tokscale Ignored Ignored Preview Sep 6, 2026 10:05pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 1 file

Re-trigger cubic

…al lane

PR #1300 added MiMo Code and fx to parse_local_clients, mirroring the
submit lane. Two things the streaming loader does for fx did not make
it across the ParsedMessage boundary.

Date. parse_fx_file leaves `date` empty and relies on the loader's
refresh_derived_fields to derive it from the timestamp. The local block
converted straight to ParsedMessage, so every fx row reached
filter_parsed_messages with date == "": `year` and `since` dropped all
of them and `until` alone admitted rows after the cutoff. Default
scanner settings pin no zone, so rebucket_parsed_days could not repair
the key either. No shipped command reaches this today: `tokscale
report` (report.rs:112) and `tokscale clients` (main.rs:4266) pass
since/until/year as None, and `tokscale wrapped`, the one caller that
filters (by year, wrapped.rs:228), feeds the result only to
build_top_agents, which skips every row whose client is not opencode;
fx is not in wrapped's default client list and its rows carry
agent: None regardless. Everything else wrapped renders comes from
generate_graph, whose streaming lane already parses fx and refreshes
the date. The defect is on the public parse_local_clients API and any
future caller that passes a date filter. The block now refreshes the
derived date before conversion; the pinned-zone rebucket pass still
runs afterwards. Reproduced by
test_parse_local_clients_fx_rows_carry_a_derived_date, which failed
with left: "" / right: "2026-05-29" before the refresh. fx is the only
parser that leaves `date` empty; MiMo Code derives it in the parser, so
its block did not share this.

Cost. ParsedMessage carried tokens only, so `tokscale report` priced
every row from its tokens. fx reports a positive total_cost for a
snapshot with empty `models` and zero tokens (the synthetic fx-unknown
row); pricing that from tokens recorded $0.00 in the wiki, and the wiki
never rewrites a session it has recorded. ParsedMessage now carries
`cost` and `cost_source`, and report's compute_msg_cost uses a
ProviderReported figure as-is, the way the submit lane never reprices
an authoritative cost. Reproduced by
wiki_keeps_the_reported_cost_of_a_token_free_fx_session: with the
precedence removed it fails with "wiki must keep the provider-reported
cost, got 0", and compute_msg_cost_keeps_a_provider_reported_cost fails
with 0.009 instead of 0.42. MiMo Code's embedded per-message cost now
reaches the lane with the same provenance
(test_parse_local_clients_keeps_provider_reported_cost).

Constraint: parse_local_clients applies no pricing, so a non-authoritative row still has to be priced by the consumer from its tokens
Rejected: deriving the date inside parse_fx_file | the streaming loader and cache already refresh it; the local block was the only caller that skipped that step
Rejected: leaving the local lane token-only | an fx-unknown row has no tokens to price, so the wiki figure was wrong rather than approximate
Directive: every parser that stamps CostSource::ProviderReported (cline, claudecode, commandcode, cursor, fx, gjc, junie, lmstudio, micode, opencode, unsloth) now drives report figures with that cost, authoritative zeros included; do not reprice a ProviderReported row
Confidence: high
Scope-risk: moderate
Not-tested: wiki sessions recorded at $0 before this change keep that figure; `--rebuild` resets summaries only
Not-tested: the local MiMo Code block keeps the first row per message id and does not upgrade its cost from a later channel db the way the submit lane does
Not-tested: no CLI command exercises a date filter against the fx local block; the fix is covered at the parse_local_clients boundary only

@junhoyeo junhoyeo left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your diagnosis is right: MiMoCode and Fx both declare parse_local: true, the submit path parses both, and parse_local_clients had a block for every other locally parsed client and none for these two — so tokscale report and tokscale clients showed nothing for either. The two new blocks are the correct fix.

An adversarial Codex pass found two integration defects in the new fx block, both reproduced and fixed in maintainer commit 8891eb3c on top of yours:

  • Reported cost was dropped. The fx conversion discarded cost and cost_source, so an fx snapshot with a positive total_cost, empty models and zero tokens — a shape fx explicitly supports — reached the report as a provider-priced fx-unknown row at $0, and the wiki then kept that figure on later scans. The conversion now carries the ProviderReported provenance through, matching the streaming path; wiki_keeps_the_reported_cost_of_a_token_free_fx_session and test_parse_local_clients_keeps_provider_reported_cost pin it.
  • fx rows carried an empty date. parse_fx_file returns date as an empty string (the only parser under sessions/ that does), and the direct local path skipped the derived-field refresh the streaming loader performs, so year and since filters would drop every fx row. The block now refreshes the derived date before conversion; test_parse_local_clients_fx_rows_carry_a_derived_date pins it. To be precise about impact: no shipped command exercises a date filter against this block today (report.rs and main.rs pass since/until/year as None, and wrapped builds its graph from the streaming lane), so this hardens the public parse_local_clients API for future filtered callers rather than fixing a visible symptom.

The MiMoCode block was checked for both and has neither: micode.rs derives its own date, and its embedded per-message cost reaches the lane with ProviderReported provenance.

One nit, not blocking: parsed_to_unified, the public reverse of unified_to_parsed, still stamps CostSource::Unknown, so a ParsedMessage carrying a provider-reported cost loses its provenance on the round trip. Nothing shipped calls it beyond the round-trip test.

Gates on my machine: fmt, clippy with and without --all-targets, full workspace suite.

@junhoyeo
junhoyeo merged commit dff9949 into junhoyeo:main Sep 6, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants