Native-data snapshot: read token reports up to their documented embedded size - #373

Summary

The native-data snapshot (G2, #365) rejected the workstation's token report as unavailable_or_invalid_report. All four report-derived savings rows therefore stayed unknown, and u2's host apply refused to deploy ("ERROR: a preflight row is unknown").

Root cause.

• The token report embeds each returned-results attachment twice: as content_base64 and as a JSON-escaped text copy (tools/token-report/token_manifest.py:386-392).
• Its documented attachment limits are 2 MiB per attachment and 16 MiB in total (tools/token-report/README.md; RETURNED_RESULTS_TOTAL_LIMIT). Its other sections are not bounded.
• The workstation's report was 40,532,680 bytes, 34 MB of it additional_evidence.returned_results (650 records). The snapshot read at most 16 MiB of the whole report.

Fix.

• REPORT_LIMIT becomes 128 MiB. This is an explicit consumer ceiling, not a producer maximum: it sits above the attachments' worst embedded size, about 117 MiB (base64 at 4/3 plus escaped text at up to 6 bytes per control byte, RFC 8259 §7).
• A larger report stays unavailable and its rows stay unknown, as before.
• The README's report-bound line is corrected.

Verification

• Tests. tests.test_native_dashboard_data runs 41 tests OK.
  • test_report_limit_covers_the_token_reports_embedded_attachment_total pins the ceiling above the attachments' worst case, computed from token_manifest constants.
  • test_report_beyond_the_attachment_total_still_projects_its_rows runs collect() on a report larger than 16 MiB.
  • Both fail on the old 16 MiB bound.
  • test_report_ceiling_is_inclusive_and_a_larger_report_stays_unknown patches the ceiling: a report of exactly the ceiling is read; one byte more gives unavailable_or_invalid_report and unknown rows.
• Related modules. host_receipts, component_matrix, new_host_grand_list, evidence_manifest, validate and adoption_docs_consistency run 398 tests OK. scripts/validate.py FAILS=0.
• Host preview. A paired local-integration run on the workstation, 2026-09-26 22:50Z. Both runs read the same report: 40,532,680 bytes, sha256 prefix ff4f69dcb5a37ca4, unchanged across both runs.
  • Before: u2-p2/apply.sh --dry-run on main 4a32e646 (the live clone) exits 1.
  • After: --dry-run --repository <this branch> exits 0.
  • Home paths are replaced with ~, and the scratch path with <scratch>:

== G2 native-data deployment (dry-run) 2026-09-26T22:50:54Z
-- preflight snapshot: the repository's snapshot.py without --publish, in a clean environment that emulates the unit's
     savings  rtk-global                   ok       value=42359134 source=2026-09-26T22:50:55.318364+00:00
     savings  rtk-project                  ok       value=26529389 source=2026-09-26T22:50:55.412413+00:00
     memory   ai-memory                    ok       value=71 source=2026-09-26T22:50:55.459888+00:00
     memory   qmd                          ok       value=100 source=2026-09-26T22:50:55.720453+00:00
     memory   qdrant                       ok       value=85656 source=2026-09-26T22:50:55.743382+00:00
     savings  context-mode-native-codex    unknown  value=None source=None
     savings  context-mode-native-claude   unknown  value=None source=None
     savings  headroom                     unknown  value=None source=None
     savings  jcodemunch                   unknown  value=None source=None
     marker rows=9 unknown=4 stale=0
ERROR: a preflight row is unknown: fix the config or source before deploying

== G2 native-data deployment (dry-run) 2026-09-26T22:50:55Z
-- preflight snapshot: the repository's snapshot.py without --publish, in a clean environment that emulates the unit's
     savings  rtk-global                   ok       value=42359134 source=2026-09-26T22:50:57.013984+00:00
     savings  rtk-project                  ok       value=26529389 source=2026-09-26T22:50:57.126727+00:00
     memory   ai-memory                    ok       value=71 source=2026-09-26T22:50:57.163765+00:00
     memory   qmd                          ok       value=100 source=2026-09-26T22:50:57.403129+00:00
     memory   qdrant                       ok       value=85656 source=2026-09-26T22:50:57.426288+00:00
     savings  context-mode-native-codex    ok       value=6089984 source=2026-09-26T21:33:59.924000+00:00
     savings  context-mode-native-claude   ok       value=3076864 source=2026-09-26T21:34:13.651000+00:00
     savings  headroom                     ok       value=579411 source=2026-09-26T21:34:17.134984+00:00
     savings  jcodemunch                   ok       value=23643 source=2026-09-26T21:34:18.257500+00:00
     marker rows=77 unknown=0 stale=0
ok   no preflight row is unknown
ok   systemd-analyze --user verify accepts both rendered units
== done (dry-run). Verify with ~/.local/state/native-agent-stack/native-data-apply/prove.sh; undo with ~/.local/state/native-agent-stack/native-data-apply/rollback.sh --apply.

• Parse cost. Measured on the same 40.5 MB report with /usr/bin/time -v python3 -c 'json.loads(...)': 0.43 s wall and 260 MiB maximum RSS.
  • The unit runs every 2 minutes (native-data.timer.example: OnUnitActiveSec=2min), with TimeoutStartSec=90 and no memory limit.
  • At the 128 MiB ceiling the cost scales to roughly 1.4 s and about 0.8 GiB. That is an extrapolation, not a measurement.

Scope and residuals

• Scope. Consumer side only. The token report's limits and embedding format are unchanged.
• Residual. A compact companion file from the reporter, holding native and coverage_matrix only, would avoid parsing the attachments every 2 minutes. That is a producer change, not done here.
• Review. GPT-6 (gpt-6-astra, max effort, live search) reviewed 2994a725 and found 2 mediums and 2 lows. All four hold, and 9d88126e resolves them:
  • the ceiling is now framed as a consumer ceiling and boundary-tested;
  • the before/after output is retained above;
  • the README line is corrected;
  • the interval is corrected to 2 min, with measured parse cost.

SOTA sources

• tools/token-report/README.md ("Limits are 2 MiB per attachment and 16 MiB total") and tools/token-report/token_manifest.py:323-324 and :386-392 at 4a32e646: the producer's documented embedding.
• RFC 4648 §4: base64 encodes 3 bytes as 4 characters.
• RFC 8259 §7: control characters are escaped as \u00XX.
• Python json documentation (docs.python.org/3/library/json.html): ensure_ascii=False still escapes control characters.

🤖 Generated with Claude Code