Codex lane identity in the OTel pipeline: ecosystem.lane kept and promoted to a metric label, lane alerts, a Grafana row and a separate ntfy topic for lane warnings - #671

Scope

• What this PR changes, in one or two sentences: the existing OTel pipeline now names each Codex lane. The Collector keeps the launch-time resource attribute ecosystem.lane (shape-checked) for logs and metrics, and the Prometheus exporter promotes it to the ecosystem_lane label. A new ecosystem-lanes rule group, a "Codex lanes" Grafana row and a quiet ntfy route use the label. This is fix A5 of the 2026-10-03 Codex lane review (section 3A item 5; section 4 "Lane identity", "Signals, thresholds and where each is shown", "Rule mechanics").
• Base commit: 1f5a791b02a230aced670c88bab3d3d0ebcf401a (origin/main at push time)
• Lane: lane:foundation
• Owned paths touched:
  • observability/collector/collector.yaml
  • observability/collector/codex-identity-launcher.sh.example (comment only)
  • observability/backends/templates/ecosystem-prometheus-rules.yml.example
  • observability/backends/templates/ecosystem-dashboard.json.example
  • observability/backends/templates/ecosystem-alertmanager.yml.example
  • tests/test_observability_backends_alerts.py, tests/test_observability_tool_names.py, tests/test_observability_writer_identity.py
  • manifests/evidence.json (hot file; registry edit is the last commit, per the hot-file protocol in docs/lanes.md)

Commits: e1b4decc8 (content), 86b9bd270 (registry, last).

SOTA sources

• OpenTelemetry Collector Contrib v0.161.0 (https://github.com/open-telemetry/opentelemetry-collector-contrib, tag v0.161.0); the binary is the installed otelcol-contrib 0.161.0 that observability/collector/README.md pins.
  • exporter/prometheusexporter/README.md L30-35: resource_constant_labels (included patterns), and resource_to_telemetry_conversion marked deprecated ("use resource_constant_labels instead"). L91-100: other resource attributes stay on target_info.
  • exporter/prometheusexporter/config.go L45-46 (the field), L76-77 (configuring both options is a validation error).
  • exporter/prometheusexporter/factory.go L66-73: the exporter is wrapped with resource_constant_labels when that option is set.
  • pkg/resourcetotelemetry/resource_to_telemetry.go L49-51 (IsEmpty: the removed enabled: false block was empty, i.e. the default) and L159-175 (WrapMetricsExporter).
  • OTTL IsMatch and delete_key (pkg/ottl/ottlfuncs/README.md), following the existing env shape check in transform/privacy.
• OpenTelemetry Rust SDK v0.31.0, sdk/src/resource/env.rs: Codex reads OTEL_RESOURCE_ATTRIBUTES (already cited by the launcher).
• openai/codex rust-v0.160.0 (commit a956835; source for the metric choices):
  • codex-rs/otel/src/metrics/names.rs L3 (codex.tool.call), L35 (codex.turn.e2e_duration_ms), L53-54 (codex.goal.usage_limited, codex.goal.blocked).
  • codex-rs/core/src/tasks/mod.rs L71 (codex.task.compact), L404-406 (the turn timer starts for each task; the panel counts its records).
  • codex-rs/core/src/session/handlers.rs L342-350: codex.conversation.turn.count is written once, at session shutdown, as the whole session's turn count, so it is not used as a turn rate.
• Prometheus v3.15.0 (the pinned server and promtool):
  • docs/configuration/alerting_rules.md L42-45: keep_firing_for.
  • docs/configuration/unit_testing_rules.md L98-100: stale input values.
  • docs/feature_flags.md L64-86: created-timestamp-zero-ingestion injects the 0 sample that lets increase() see a new event series.
• Alertmanager v0.34.1 (the pinned amtool): docs/configuration.md route match/continue (L222-229), and amtool config routes test.
• ntfy v2.28.0 (https://github.com/binwiederhier/ntfy, tag v2.28.0):
  • server/server.go L1200 and L1210: template and priority are read through readParam.
  • server/util.go L56-62: readParam falls back to query parameters.
  • Correction after the Codex root's read (608 thread, CODEX-ROOT-PR671-86B9BD2-SOURCE-FINDINGS): with template=alertmanager, ntfy 2.28.0 (10cb6506) renders in file-template mode without the query priority (server/server.go L1210, server/types.go L117, server/server_template.go L58 and L103; the built-in templates/alertmanager.yml has no priority field). The lane route therefore sets no priority: lane warnings go to their own ecosystem-lanes topic, which the user can mute separately, and the test asserts no priority= is promised.
• Thresholds and rule mechanics: the 2026-10-03 Codex lane review synthesis, section 4 table and "Rule mechanics". These are starting values calibrated on 2026-10-03 data; re-tune them after 7 days in a dated decision.

Evidence-class table

Local commands run

$ ECOSYSTEM_OBSERVABILITY_DATA=<scratch>/data otelcol-contrib validate --config=<scratch>/config/collector.yaml   # 0.161.0
exit 0 (no output)
$ promtool check rules <scratch>/b/config/ecosystem-prometheus-rules.yml   # 3.15.0, configure.py render
exit 0: SUCCESS: 21 rules found
$ amtool check-config <scratch>/b/config/ecosystem-alertmanager.yml   # 0.34.1
exit 0: SUCCESS; global config, route, 0 inhibit rules, 2 receivers, 0 templates
$ jq -e . observability/backends/templates/ecosystem-dashboard.json.example
exit 0 (and json.dumps(indent=2) round trip byte-identical)
$ python3 -m unittest tests.test_observability tests.test_observability_backends_configure tests.test_observability_tool_names tests.test_observability_writer_identity tests.test_observability_run_correlation tests.test_observability_writer_identity_host tests.test_observability_backends_alerts tests.test_sdk_usage_scope tests.test_freeze_snapshot
exit 0: Ran 275 tests, OK (skipped=61; system python3 has no PyYAML)
$ uv run --no-project --with pyyaml python3 -m unittest <same modules>   # PyYAML 6.0.3, Python 3.14.7
exit 0: Ran 275 tests, OK (0 skipped)
$ python3 scripts/validate.py
exit 0: {"components": 69, "hashed_files": 9541, "profiles": 4, "receipts": 193, "status": "passed"}

Decision record

None. This PR implements fix A5 as specified. The thresholds are starting values; the 7-day re-tune is due as a dated decision record.

Residuals and live apply (not done here)

• Not applied live. No running service was touched. A live apply needs to:
  • patch the host collector.yaml with the log and metric resource statements and the exporter block (the dated evidence/artifacts/telemetry-writer-identity-20260926/host/merge_collector.py carries only groupbyattrs/session and the metric statements, not the log statements or the exporter), validate it, and restart ecosystem-otelcol;
  • render and install the rules and the Alertmanager config, then reload Prometheus and Alertmanager;
  • re-provision the dashboard;
  • relaunch each Codex lane with OTEL_RESOURCE_ATTRIBUTES=ecosystem.lane=<lane> through the identity launcher, then confirm that one parent series and one child series carry ecosystem_lane.
  • Until that relaunch, every series groups under unattributed, and the alerts carry an empty ecosystem_lane.
• Usage-limited signal unexercised. ecosystem_codex_goal_usage_limited_total is a known name here, but it had no series in the last 24 h, so CodexLaneUsageLimited has only synthetic coverage.
• NativeStack2604 exports nothing. The synthesis recorded no Codex series from that distro in 16 h; the token lane's alerts need its [otel] exporters pointed at a reachable collector.
• Zero-sample dependency. The event rules rely on Prometheus created-timestamp-zero-ingestion (the host apply already requires it). Re-check them if the host moves to st-storage only.
• Deprecated match. Alertmanager 0.34.1 marks route match deprecated in favour of matchers. The new routes follow the existing route's form; migrating every route together is a separate change.

Host evidence

Not applicable: no files under evidence/hosts/ changed.

• python3 scripts/host_receipts.py validate passes for every new/changed receipt. (n/a)
• Independent review is recorded (scripts/host_receipts.py review) or explicitly requested in this PR. (n/a)
• No platform_status change is made from a host receipt alone; a platform status flip still needs the matrix rule in docs/component-evidence-matrix.md. (n/a)

Checklist

• New/changed GitHub Actions are pinned to a full commit SHA with a
  version comment (no floating tags). (none changed)
• New/changed workflows declare top-level permissions: contents: read
  (or a narrower, explicitly justified addition). (none changed)
• No secrets are printed, logged or committed; no new required secret was
  added without a documented owner.
• No new paid hosting, subscription or billing surface was introduced.
• Peer-owned untracked files and worktrees were preserved (not deleted,
  moved or overwritten).

🤖 Generated with Claude Code