Skip to content

Gateway status dimensions in collector span metrics; agentsview full session content - #848

Merged
seathatflowsinourveins merged 1 commit into
mainfrom
cc/otel-gateway-dims-20261008
Oct 8, 2026
Merged

seathatflowsinourveins merged 1 commit into
mainfrom
cc/otel-gateway-dims-20261008

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Summary

Monitoring additions applied on NativeStack2604 at 08:12-08:17Z, recorded in the install plan so a fresh host renders the same config:

  • Collector span metrics carry gateway status, outcome and provider. connectors.span_metrics.dimensions gains omniroute.routing.status, omniroute.routing.outcome and gen_ai.provider.name. OmniRoute's request spans set these attributes (open-sse/services/routing/otel.ts), so gateway errors reach Prometheus through traces_span_metrics_calls_total once the gateway's OTLP trace sink is on. The host turns that on with a 30-otel.conf drop-in at the next lane pause.
  • agentsview keeps full session content. archive_content changes from usage to full in the plan's launcher and config, so the command center can read live lane and session transcripts and tool calls locally. The valid values full, transcripts and usage are defined in kenn-io/agentsview@413a87f7 internal/config/config.go:670-692.
  • Four changed files are re-registered in manifests/evidence.json: 8 lines, no reformatting.

Host evidence

  • Published with the repository's own observability_config.py otel (which runs otelcol-contrib validate), rc 0.
  • ns2604-otelcol-contrib restarted: active, /health/status 200.
  • agentsview sync --full: 4,364 sessions in 2m44s. 232,934 of 232,934 message rows now carry content.

SOTA sources

  • open-telemetry/opentelemetry-collector-contrib v0.162.0, spanmetrics connector dimensions (the pinned collector release in the install plan).
  • diegosouzapw/OmniRoute: open-sse/services/routing/otel.ts, the span attributes omniroute.routing.status, omniroute.routing.outcome and gen_ai.provider.name (read in the 3.8.52 source tree; the running 3.8.51 drain build ships the same code).
  • kenn-io/agentsview@413a87f internal/config/config.go:670-692 (ArchiveContent full/transcripts/usage).

Checks

  • python3 scripts/validate.py: passed (10,820 hashed files).

🤖 Generated with Claude Code

… full session content

Collector span_metrics gains omniroute.routing.status, omniroute.routing.outcome and
gen_ai.provider.name so gateway errors reach Prometheus once OmniRoute's OTLP sink is on.
agentsview archive_content usage -> full for live lane review (values per agentsview
internal/config/config.go:670-692 at 413a87f7). Four files re-registered.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 8, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 8, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-08T08:27:17.690584Z a438443 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@seathatflowsinourveins
seathatflowsinourveins merged commit e450e37 into main Oct 8, 2026
33 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the cc/otel-gateway-dims-20261008 branch October 8, 2026 09:03
seathatflowsinourveins added a commit that referenced this pull request Oct 8, 2026
## Scope

Add per-lane tool/API rates, separate estimated Claude/Codex costs, and gateway provider/model/outcome/HTTP-status panels to the existing cc-lanes dashboard. #848 already supplies the five collector dimensions on main; this PR has no collector YAML delta after its landing rebase. Host publication remains the CC's operation.

- Base commit: `e450e378c6bf1b21dd0bb565812f84afe028bad5` (CC091807Z landing rebase).
- Lane: `lane:foundation`.
- Owned paths: `observability/lanes_dashboard.py`; generated plan `config/grafana-lanes.json`; the existing writer-identity test fixture; dated decision; own evidence registry entries.

## Pipeline exactness

Dropped the old #851 `traces.span.metrics.*` statements from `transform/privacy`. They matched only direct OTLP metrics named `traces.span.metrics.calls/duration` entering the `metrics` pipeline. Gateway/Harbor connector output instead uses `metrics/spans` and `transform/metric_export_privacy`; it never passes through those statements. No reviewed shipped producer is bound to sending connector-named metrics through that direct OTLP metrics route.

Main's collector YAML is retained byte-for-byte, including #848's five-dimension flow-form line and comment. Its existing export guard retains provider/model/status/outcome. The synthetic test follows that actual connector pipeline and proves dimension export, not causality from the dropped exceptions.

## SOTA sources

- [open-telemetry/[email protected], spanmetricsconnector README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.162.0/connector/spanmetricsconnector/README.md): additional dimensions and default `traces.span.metrics.calls/duration` names. [Main's processor/pipeline source](https://github.com/seathatflowsinourveins/native-agent-stack/blob/e450e378c6bf1b21dd0bb565812f84afe028bad5/evidence/artifacts/new-wsl-install-plan-20261002/config/otel.yaml#L86) establishes the exact routing and retained dimensions.
- [open-telemetry/semantic-conventions-genai@06ec68e, client-inference.md](https://github.com/open-telemetry/semantic-conventions-genai/blob/06ec68e722c45a7218e23ea1bc1339fe4e21ecae/docs/gen-ai/client-inference.md): `gen_ai.provider.name`; current conventions are Development.
- [cheaper-inference/OmniRoute@c1e30b7, routing/otel.ts:193](https://github.com/diegosouzapw/OmniRoute/blob/c1e30b7676975feb298b49eff6ff58923c04b89e/open-sse/services/routing/otel.ts#L193): vendor `omniroute.routing.status/outcome`, provider/model fields and absent lane/session parentage. Routing status is distinct from OTel `status.code`.
- [openai/codex@9790114, cost emission](https://github.com/openai/codex/blob/979011409de0a60b52f179721948e65531d26144/codex-rs/otel/src/events/session_telemetry.rs#L368), [Claude Code monitoring](https://code.claude.com/docs/en/monitoring-usage#cost-counter), and [Prometheus rate/increase](https://prometheus.io/docs/prometheus/latest/querying/functions/#increase): reset-aware counter queries, micro-USD conversion, separate cost sources and interval-extrapolation limits.
- Existing Grafana13.2.3/Loki3.7.8 native renderer and provisioning formats are reused.

## Evidence-class table

| Claim | Evidence class | Command / receipt |
| --- | --- | --- |
| Dimension names, native cost contracts and processor ownership | source_review | Versioned sources above, immutable main/old-head inspection and dated correction below |
| Generated dashboard matches source; existing publisher contracts remain passing | local_integration | Renderer check and 17 dashboard contract tests |
| Native connector retains provider/status/outcome and merges equal groups | synthetic | Existing fixture sends four synthetic spans through main's connector pipeline; 3 exported groups have counts 2/1/1 |
| Source publisher produces an exact copy; collector accepts it | local_integration | Repository `observability_config.py otel` into a temporary destination, then native `otelcol-contrib validate` |
| Native query parsers accept the eight new targets | local_integration | Read-only Prometheus/Loki query APIs returned success; missing Codex-cost/gateway series remain unknown |
| Deployed collector/gateway export and complete cost coverage | unverified | CC host publication/read-back; no host restart or configuration apply in this PR |

## Local commands run

```text
python3 observability/ns2604_dashboards.py
python3 observability/ns2604_dashboards.py --check
  rc=0
python3 -m unittest tests.test_ns2604_dashboards tests.test_j775_panel_contracts tests.test_observability_writer_identity -v
  rc=0; 43 collected, 39 passed, 4 optional pinned-Prometheus/promtool checks skipped
NS2604_OBSERVABILITY_DATA=<temporary>/data python3 evidence/artifacts/new-wsl-install-plan-20261002/config/observability_config.py otel --source-root <checkout>/evidence/artifacts/new-wsl-install-plan-20261002/config --config-root <temporary>/config
  rc=0; published YAML matches source and named main byte-for-byte
NS2604_OBSERVABILITY_DATA=<temporary>/data otelcol-contrib validate --config=<temporary>/config/otel.yaml
  rc=0; installed collector 0.162.0
python3 scripts/component_matrix.py --write
python3 scripts/new_host_grand_list.py --write
  rc=0; no derived-report changes
python3 scripts/validate.py
  rc=0
git diff --check
  rc=0
```

## Decision record

`docs/decisions/2026-10-08-cc-lanes-rate-cost-gateway.md` records the corrected processor assumption.

Costs are estimates with source-specific coverage. Codex provider enrichment is conditional; `increase()` is not an exact total or guaranteed lower bound. No zero fill is used for missing costs or gateway routing data. Gateway events lack lane/session parentage and are not assigned to client lanes.

## Host evidence

No files under `evidence/hosts/` change, and no deployment acceptance is claimed. The existing native collector integration uses temporary configuration and loopback ports with synthetic input. The runtime lock, broker keys, live Grafana, services and gateway are untouched.

## Checklist

- [x] No GitHub Actions changes.
- [x] No secrets, new credentials, paid hosting or billing surface.
- [x] Peer worktrees preserved; #834 retains its separate landing cue.
- [x] Main registry taken; only own changed files re-registered last.
- [x] Source-only scope respects the paper freeze and quiet window.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant