Collector: keep Claude tool_use_id and cost_usd through Loki, drop Codex tool arguments, whole-run query guidance - #408

Summary

Claude tool events lost tool_use_id and API requests lost cost_usd in the
Collector privacy allowlist. Preserve both fields so the existing run/session
identities can support transcript joins and per-arm usage reconciliation in
Loki. The repair also makes Codex argument removal explicit in the privacy
transform while retaining measurement metadata, and corrects the deployment,
query and historical gate documentation. These fields remain structured
metadata; only service.name is indexed.

Changes

• observability/collector/collector.yaml: retain tool_use_id and cost_usd;
  explicitly delete Codex arguments with native OTTL; retain
  output_truncated and supplied arguments_length, output_length and
  output_line_count. Preserve tool/namespace, guarded MCP-server names,
  duration, success and existing run/session fields. Fix the Claude source anchor.
• tests/test_observability_run_correlation.py: exercise actual native
  Collector-to-Loki ingestion, two Workflow arms, matching resource/event run
  attributes, whole-run prefix/single-arm queries, session/tool joins,
  string/numeric/zero costs, absent values and indexed-series cardinality.
  Add Codex content-removal and metadata-retention fixtures, including a
  context-mode call without MCP-server metadata and optional size counters.
  Two configuration regressions run without optional dependencies; native
  tests require the documented pins and PyYAML.
• observability/collector/README.md: correct Claude custom-attribute emission,
  whole-run filters, historical bases and the host installation path. Explain
  the Codex output-preview/content boundary and the trace-only size fields.
• manifests/evidence.json: the coordinator re-registers this branch's three
  changed files in the last commit (docs/lanes.md hot-file protocol).

Evidence

• Documentation/source review: live official pages and pinned source fetched
  on 2026-09-27; installed Codex 0.157.1, Claude Code 2.1.283, Collector 0.161.0
  and Loki 3.7.8 (revision 09e6ce2f) checked. Matching release notes were read
  through gh api. These are capability/version checks, not model runs.
• Local integration with synthetic fixtures: installed native Collector and
  Loki on isolated loopback ports; independently queried Loki and inspected the
  file exporter. Both copies of Claude's run key survive and native whole-run
  queries include suffixed arms while excluding an adjacent run. Codex arguments
  and output are absent; tool/namespace, optional guarded server, timing,
  success, correlation and supplied sizes survive without new index labels.
• Local regression checks: final covering run passed 83 tests, zero skips
  using /usr/bin/python3 with installed PyYAML 6.0.1. No dependencies installed.
• Native configuration acceptance: the complete profile passes the upstream
  Collector validate command with its queue/receipt-directory prerequisites.
• Live provider execution / unchanged upstream tests / local cost measurement:
  none performed. The fixture checks do not establish these evidence classes.

Original build, historical returned excerpts (2026-09-27): failing first
against the original configuration, then the same module after retaining the
Claude fields. These excerpts preserve the earlier result, rather than claiming
a new execution of that earlier configuration:

/usr/bin/python3 -m unittest tests.test_observability_run_correlation -v
exit: 1
FAILED (failures=6)
AssertionError: None != '0.0125' : cost_usd

/usr/bin/python3 -m unittest tests.test_observability_run_correlation -v
exit: 0
Ran 3 tests in 23.597s
OK

/usr/bin/python3 -m unittest tests.test_observability_run_correlation tests.test_observability_tool_names tests.test_observability_writer_identity tests.test_observability_writer_identity_host -v
exit: 0
Successful test lines: 81
Skipped test lines: 0
Ran 81 tests in 72.914s
OK

The original complete-profile check first failed on its missing queue directory;
after creating the queue and SDK receipt directories from the install recipe,
the same native command passed. Returned excerpts, with the workspace sanitized:

Native Collector 0.161.0 validate --config=observability/collector/collector.yaml
exit: 1
Error: extensions::file_storage: directory must exist: stat <worktree>/.runtime/collector-validation/collector/queue: no such file or directory. You can enable the create_directory option to automatically create it

Native Collector 0.161.0 validate --config=observability/collector/collector.yaml
exit: 0
(no output)

Repair, returned excerpts (2026-09-27): TMPDIR was explicitly exported for
all repair unittests. Native shell creation of that directory was refused by the
execution sandbox; the context-mode test runner created it successfully. An
initial fixture draft assumed a third metadata element in default Loki query
rows and errored in three subtests. Corrected it to inspect the file exporter
and use Loki metadata filters. Before changing the configuration, the corrected
C1 tests then failed on four absent size/truncation keys and three missing
output_truncated values. Argument removal already passed on this branch;
these failures are metadata retention failures, not proof of a repository leak.

export TMPDIR=/var/tmp/claude-collector
rtk /usr/bin/python3 -m unittest tests.test_observability_run_correlation.CollectorProfileTests.test_codex_measurement_metadata_survives_without_content tests.test_observability_run_correlation.NativeRunCorrelationTests.test_codex_arguments_are_removed_and_measurement_metadata_reaches_loki -v
exit: 1
AssertionError: None != True : output_truncated
AssertionError: None != False : output_truncated
Ran 2 tests in 22.421s
FAILED (failures=7)

export TMPDIR=/var/tmp/claude-collector
rtk /usr/bin/python3 -m unittest tests.test_observability_run_correlation -v
exit: 0
Ran 5 tests in 45.127s
OK

export TMPDIR=/var/tmp/claude-collector
rtk /usr/bin/python3 -m unittest tests.test_observability_run_correlation tests.test_observability_tool_names tests.test_observability_writer_identity tests.test_observability_writer_identity_host -v
exit: 0
Successful test lines: 83
Skipped test lines: 0
Ran 83 tests in 111.662s
OK

Native Collector 0.161.0 validate --config=observability/collector/collector.yaml
exit: 0
(no output)
git diff --check
exit: 0
(no output)

SOTA sources

• Anthropic: Claude Code monitoring,
  checked 2026-09-27, and anthropics/claude-code v2.1.283.
  workflow.run_id is documented on Workflow-agent API/tool events, including
  descendants, since 2.1.202. Non-Workflow events rely on the caller-supplied
  ecosystem.task.id custom key for this run contract, copied to resources,
  events and datapoints per Multi-team organization support.
• openai/codex rust-v0.157.1:
  tool_result.rs:55-79
  and preview truncation at line 94,
  events/shared.rs:59-60
  and call_trace.rs:38-54.
• open-telemetry/opentelemetry-collector-contrib v0.161.0:
  Transform Processor
  and keep_keys implementation,
  plus the existing native delete_key operation.
• grafana/loki v3.7.8, installed revision 09e6ce2f:
  release,
  native OTLP ingestion
  and query/series HTTP APIs,
  with regex label filters.
• open-telemetry/opentelemetry-proto v1.9.0:
  OTLP specification.
  Test lifecycle and transport reuse the repository's
  tests/test_observability_tool_names.py pattern and committed backend template.
• Repository history: base ec27a300; Telemetry writer identity (G1): one writer per token series, integrity alerts and a committed host recipe #364 c71d66d0; Tool invoke rates: tool, MCP server, skill and subagent names in Loki and Grafana (stacked on #364) #366 a464d288.
  The writer-identity host/merge_collector.py:116-118 explains why the
  current logs change needs reviewed-profile installation instead of apply.sh.

Review dispositions

Five reviewer findings are repaired. The receipt part of finding 4 is disputed with source:

1. Correct the missing Claude anchor to #multi-team-organization-support.
2. State the documented resource/event/datapoint copies and metric removal;
   exercise both log copies against native Loki instead of a resource-only fixture.
3. Publish the actual red/green/validate excerpts above, without scratchpad links.
4. The collector README part is repaired: it names base ec27a300, Telemetry writer identity (G1): one writer per token series, integrity alerts and a committed host recipe #364 and Tool invoke rates: tool, MCP server, skill and subagent names in Loki and Grafana (stacked on #364) #366. The receipt edits are
   withdrawn and disputed with source. RUNBOOK.md is hash-sealed: the E2E README seal list, checked by
   tests/test_token_e2e_preregistration.test_sealing_hashes_and_amendment_rule. The E2E README changes
   only through its append-only amendment rule. Their correction goes to a dated PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381 amendment.
5. State that apply.sh leaves log_statements unchanged (merge_collector.py:116-118), and document the
   install, native validate and user-service restart path.
6. Document and test prefix matching for a whole run and exact single-arm matching.

C1: implemented explicit argument deletion and safe metadata retention.
The original Codex source calls the server field mcp_server; the existing
guarded mcp_server.name mapping supplies Loki's mcp_server_name. Namespace
retention covers absent-server calls. At the pinned source, size counters are
in the trace branch, while output_truncated is common to logs/traces. The
fixture tests supplied counters without claiming a new native-log capability.
The peer's live-host leak is a deployment finding; this branch already excluded
arguments via its content deletion and allowlist. No host deployment was made.

Residuals and not done

• The active host was not deployed or restarted and no provider calls were
  made. The token-adoption E2E still needs actual tagged-run delivery/flush and
  transcript reconciliation. A merge or synthetic pass is not that acceptance.
• cost_usd remains an estimate; it does not establish billing or replace the
  frozen dated-price method. Missing cost or identity stays absent.
• Non-Workflow arm grouping needs a distinct resource value per arm/task/attempt
  or a separately retained session mapping. The shared run key alone cannot
  assign arms. Existing Workflow ID validation still bounds accepted values.
• Codex env/call_id residuals, new correlation fields, dashboards and metrics
  remain outside this repair. Codex arguments and safe tool metadata are included
  under C1. Missing size counters stay absent; traces remain disabled.
• Native cost_usd type has not been confirmed through live provider execution;
  the fixtures cover both string and numeric values, including zero.
• The sealed PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381 receipts still describe the pre-Tool invoke rates: tool, MCP server, skill and subagent names in Loki and Grafana (stacked on #364) #366 field gap and Telemetry writer identity (G1): one writer per token series, integrity alerts and a committed host recipe #364 as open (RUNBOOK.md:36-42 and
  :429-435, README.md:542), and the RUNBOOK's whole-run query at :457 uses exact equality. Their correction
  waits for PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381's dated amendment. Until then, this collector README section supersedes their field-gap
  statements.

Recommended lane label: lane:foundation.

Coordinator note on this repair round

The repair round also edited two #381 receipts, evidence/artifacts/token-adoption-e2e-20260926/RUNBOOK.md and its README.md. The coordinator withdrew both edits: RUNBOOK.md is hash-sealed (tests.test_token_e2e_preregistration.test_sealing_hashes_and_amendment_rule failed on the change), and the E2E README changes only through its append-only amendment rule. #381's owner carries these facts in its pre-freeze readiness amendment:

• tool_use_id and cost_usd are kept in transform/privacy keep_keys;
• workflow.run_id has been kept since Tool invoke rates: tool, MCP server, skill and subagent names in Loki and Grafana (stacked on #364) #366 (a464d28);
• Codex tool_result arguments is dropped;
• whole-run views over suffixed values need ecosystem_task_id=~"<run>(\\..+)?";
• host delivery/flush proof is still required.

After the repair round the coordinator also corrected two collector README statements that the independent repair verification found unsupported. One claimed the sealed receipts "now carry a dated update". The other cited #366's deployment as byte-identical, which its receipt does not show.

Acceptance after the repair (local integration, this host): validate.sh FAILS=0. The full suite ran 6598 tests. It had two known environment failures: test_pre_commit_gate, which fails on the unchanged base, and test_host_profile_copy_is_verbatim, which fails on old-base branches since this host installed #402's guard.

🤖 Generated with Claude Code