Skip to content

Fix unit alert lifetime and native invoke attribution - #825

Draft
seathatflowsinourveins wants to merge 12 commits into
mainfrom
codex/ns2604-alert-f09-20261007
Draft

seathatflowsinourveins wants to merge 12 commits into
mainfrom
codex/ns2604-alert-f09-20261007

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

Keep registered user-unit alerts firing through failed, missing or stale state.
Option C sends human firing/recovery notifications only for the STATE family;
the unchanged v3 EVENT senders go to a receiver without notification integrations.
Dagu retains its separate event-only notification route. This supplies reliable
foundation observation for the north-star research and simulation work.

Review fixes

P1 uses original observation timestamps as recording-rule values. Active recovery
must be strictly newer than every fresh contradictory nonactive observation;
ties are conservative and fresh failed evidence wins. The native receiver zeroes
every noncurrent state on successful scrapes; overlap fixtures defend against
cache/transport retention rather than asserting normal zero omission.

P2 records unit kind, armed state, recovery and completion contracts explicitly.
Continuous services require active recovery. The CC-named W2 drill alone uses
failure-cleared for reset-failed. Reused one-shot/timer entries stay unarmed with
unknown recovery/completion; inactive completion never manufactures a resolution.
This separates the review's active-service contract from the later W2 ruling.
The source proposal goes through the named CC ACK; no production contract is armed.

P3 asserts persistent firing at seven and twelve minutes. The lowercase W2 unit
also produces the actual DRILL alert annotation. No live failure was run.

P2b restores all fourteen original continuous-service regressions under active
recovery, including healthy→inactive newly firing at90s,6m and12m, with no
inactive recovery. Every failure-cleared fixture now uses only the permitted
named W2 drill. A renderer integration check validates every fixture roster's
actual contract, preventing a passing fixture from bypassing the source policy.

P2c preserves those active cases and adds six separate W2 failure-cleared
negatives: unknown/empty/foreign alone and cached inactive overlapped by newer
invalid evidence. No recovery while fresh and firing through12m are asserted.
Both requested mutations fail native promtool in disposable copies: broadening
only this branch's whitelist and removing only its timestamp veto. Production
rules are unchanged; no mutated file is committed.

SOTA sources

Evidence-class table

Claim Evidence class Evidence
Declared freshness/recovery and unknown cases synthetic Unchanged native promtool executes33 locally authored groups/154 assertions; both targeted scratch mutations fail
Explicit roster, contract guards and native C/A notification configuration local_integration Nine renderer and eight notifier checks; native amtool accepts C and fallbackA
Native receiver overlay/configuration parses native_proven, config-only Contrib0.162 validate on read-only #775 base plus overlay, exit0; user bus not started
Registry/source consistency local_integration validate69 components/10409 files/4 profiles/223 receipts, exit0
Real firing,15-minute hold and correct recovery delivery pending One CC-only W2 attempt specified in WINDOW.md; no live drill or host apply

Local commands run

All substantive checks use nice -n19 ionice -c3.

  • promtool test rules observability/alert-lifecycle/user-unit-alerts.test.json: exit0.
  • promtool check rules observability/alert-lifecycle/rules.example.json: exit0,8 rules.
  • python3 -m unittest tests.test_user_unit_alert_renderer tests.test_user_unit_alert_notifier_config: exit0,17 tests.
  • python3 observability/alert-lifecycle/render.py --check: exit0;
    configured3, armed1, held completion2.
  • amtool check-config on C and fallbackA: exit0; C has0 inhibitors/3 receivers.
  • otelcol-contrib validate --config=<ACKed775 source> --config=<receiver.example.json>: exit0.
  • python3 scripts/validate.py: exit0. git diff --check: exit0.

IR-1 / IR-2 follower commits

The alert-fix push was d24c7bd. A separate source commit adds a supported native
Collector overlay, leaving every ACKed #775 byte and alert-fix file unchanged.
#775 already preserves the native hook dimensions; this overlay adds event
as an alias of validated lifecycle hook_name. It does not recover script
or repository identity. The panel must retain that native attribution gap.

Claude skill/hook events already retain session.id in a later bounded native
sample; query Loki metadata session_id. Derived actor is explicitly ambiguous
main_or_subagent, or workflow only with a real retained workflow.run_id. No
native main/subagent field or six-hour organic count is invented. Unlabelled
exec/SDK jobs require CC-assigned native resource environment at their real
launch, including the explicit SDK environment map; no launcher is changed here.

The complete #775 base plus F09 and identity overlays passes native Contrib
validate. Repository validate now passes69/10411/4/223. Source critic checked
pipeline/SDK/field coverage; its activation/history note is included: changed
series have no backfill, partial24h windows stay incomplete and7d p95 stays
disabled until sufficient baseline. Native runtime labels and organic counts
remain CC read-back gates. The registry is the final commit.

Decision record

docs/decisions/2026-10-07-user-unit-alert-lifecycle.md keeps its historical
8,054-byte prefix unchanged and appends dated design-C/review sections.
Source tests do not replace host read-back, delivery or organic evidence.

Host evidence

No host receipt is rewritten. CC/5f owns the unchanged v3 senders, SHA pins,
complete roster, user bus, allocated21890 listener, provisioning/reload and W2.
STATE is the sole unit human notifier; a short unobserved failure may leave only
an EVENT record. Whole evaluator/sender loss still has finite protocol leases.
Pipeline scrape/rules are included; Watchdog/deadman remain disabled examples
until independent owner custody is established. No general completion/cadence
or silent-stop coverage is claimed. Other guidance edits stay in their owned
post-#775 chain.

Checklist

  • No GitHub Actions change, credential value or paid service.
  • Peer-owned files, recorded observations and decided text preserved.
  • Registry last; one review-fix push; draft and named review/ACK/landing path.

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Oct 7, 2026
@seathatflowsinourveins seathatflowsinourveins changed the title fix: keep user-unit alerts active until confirmed recovery Fix unit alert lifetime and native invoke attribution Oct 7, 2026
seathatflowsinourveins added a commit that referenced this pull request Oct 7, 2026
### Scope

Provision the NativeStack2604 Lanes dashboard and retain the native dimensions needed to distinguish tool, hook, skill, service and token activity. Monitoring steps 6–8 include the collector route, supported file provisioning and matching install-plan acceptance. A13 makes main's rendered Prometheus unit the sole startup carrier; the old drop-in is a dated, inert, do-not-apply record. Activation and start-timestamp/new-turn qualification remain the command center's later host step.

- Landing base commit: `475127d43c1cce70dba5cd31238af4e078fa17eb` (the command center's relayed #645 squash merge).
- Lane: `lane:foundation`.
- Owned paths: observability collector/dashboard/readers, focused checks, generated NativeStack2604 Grafana assets, monitoring install-plan mirrors and dated decisions.
- Shared install-plan changes preserve landed #723/A27 qualification, custody and the authoritative rendered Prometheus unit. The repair was replayed once onto #645's relayed squash merge. The Grafana notes retain main's exact W1b selection/qualification statement and re-apply the reviewed dashboard/provisioning note. Main's registry is restored and owned rows are regenerated last; no shared registry lines are hand-merged.

The Lanes UID is `cc-lanes`, provisioned under the existing Token efficiency provider. Withdrawal is file based: restore the prior `lanes.json`, or remove only it, temporarily allow provider deletion, restart in a permitted slot, verify the anonymous dashboard endpoint returns 404, and restore `disableDeletion: true`. Keep token-layer, the provider and private registry.

Every provisioned dashboard using `codex_turn_token_usage_sum` carries lower-bound titles and descriptions until deployed start timestamps and a newly born single-turn reconciliation pass. The standalone template-copy publisher and maintained renderer share that policy. Latest unknown status values retain nulls; known RTK zero share requires an observed classified-shell denominator.

One clock panel retains the producer's 50 ms error-bound warning, 100 ms absolute-offset critical threshold and 1,200 s reference-age warning. Its separate three-minute UI freshness preserves missing data as No data. Collector serialization and native configuration validation pass; live clock-panel acceptance remains pending.

## SOTA sources

- [Codex rust-v0.160.1 / immutable source](https://github.com/openai/codex/tree/d27764b82f7118f674371e6d6e76271d9d606edb/codex-rs/otel): `events/session_telemetry.rs` and config schema for native named tool events. Native `item_completed` and fork-history metadata supply the bounded readers.
- [OpenTelemetry Collector Contrib v0.162.0](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.162.0): transformprocessor, signaltometricsconnector, spanmetricsconnector and otlpjsonfilereceiver for bounded dimensions, sanitization and supported routes.
- [Grafana v13.2.3 file provisioning](https://github.com/grafana/grafana/blob/v13.2.3/docs/sources/administration/provisioning/index.md): provider deletion policy and JSON dashboard format. [Loki v3.7.8 metric queries](https://github.com/grafana/loki/blob/v3.7.8/docs/sources/query/metric_queries.md): range aggregation and observed-denominator zero filling.
- [Prometheus v3.15.0](https://github.com/prometheus/prometheus/tree/v3.15.0): `created-timestamp-zero-ingestion` feature flag; deployed single-turn reconciliation remains required.
- [OmniRoute v3.8.51](https://github.com/diegosouzapw/OmniRoute/tree/v3.8.51): `open-sse/services/routing/otel.ts` optional upstream GenAI OTLP sink. Its environment application and native identity qualification remain with the command center.

### Evidence-class table

| Claim | Evidence class | Evidence |
| --- | --- | --- |
| Named nested context-mode and RTK shell results reach Loki | native_proven | Retained one-shot per-run Codex OTel probe, 2026-10-06; no repeated model probe |
| Native BEFORE readers exclude inherited history | local_integration | Bounded 90-minute/246-minute receipts; method differences remain recorded |
| Fork/window/dedup, null-preserving status and provisioning behavior | synthetic / local_integration | Focused fixtures and 12 dashboard tests; not unchanged upstream acceptance |
| Both Collector configurations are valid | native_proven | Installed Collector validation, exit 0; configuration validation is not host application |
| Dashboard routes and file-based withdrawal follow supported formats | source_review / local_integration | Pinned provisioning sources, actual publisher coverage and passing install-plan checker; no deployed browser acceptance |

### Local commands run

```text
nice -n 19 python3 -m unittest tests.test_ns2604_dashboards
exit 0; 12 tests pass, including every provisioned Codex-token dashboard and standalone portability

nice -n 19 python3 evidence/artifacts/new-wsl-install-plan-20261002/check_plan.py
exit 0; 84 rows, 186 commands and 113 acceptance entries agree

nice -n 19 otelcol-contrib validate --config=evidence/artifacts/new-wsl-install-plan-20261002/config/otel.yaml
nice -n 19 otelcol-contrib validate --config=observability/collector/collector.yaml
both exit 0

nice -n 19 python3 scripts/validate.py
exit 0; 69 components, 10,379 hashed files, 4 profiles, 214 receipts

git diff --check
exit 0
```

The initial plan-check failure from import-generated configuration bytecode is retained in the private check receipt. The loader now suppresses bytecode during dynamic imports, restores the prior setting and passes the rerun. Generated dashboard JSON is source evidence; it is not visual acceptance.

### Decision record

- `docs/decisions/2026-10-06-native-lane-invocation-observability.md`.
- `docs/decisions/2026-10-06-ns2604-native-dashboard-countability.md`.

The README classifies all six retained 13000 references without rewriting frozen observations and requires a compatible emitter checkout while its timer is enabled. Storage COUNT/AGE read-back, unit/GPU export, client limit headroom and the GitHub snapshot feed retain named owners and explicit remaining gates.

### Host evidence

No host receipt or client configuration is applied by this PR. Host apply waits for the GPT delta read, command-center ACK and the permitted post-Claude-window slot. Native Loki/Prometheus read-backs and an anonymous browser capture remain required.

Organic BEFORE/AFTER counts are not yet measured under the all-prompts, first-turn and actual steer-visibility rules. Effective role grants and referenced co-op blocks remain coverage gaps. No exclusion, readiness or token-row closure is claimed. No local A/B or trial gates today's read-back.

### Earlier rebase

Commit 1 (`f19626d6` → `35bfa72f`) preserves main's Grafana
`--show-error --output "$GRAFANA_HEALTH_FILE"` flags and uses the same flags
in the NativeStack2604 port annotation. Commit 4 (`ceaca5ee` → `a752852c`)
changed only the surrounding anti-pattern-log context: both zero-context stable
patch IDs are `947dc92f4ad5ded7b6258d65eb1b290de6c294f2`, so its added/deleted
content is unchanged.

### Checklist

- [x] GitHub Actions changes: not applicable.
- [x] No secrets printed, logged or committed; no new required secret.
- [x] No new paid hosting, subscription or billing surface.
- [x] Peer-owned worktrees and unrelated files preserved.
- [x] Registry commit is last; PR remains draft for cross-family review.



## Landing review follow-up

The local follow-up preserves landed metric privacy and records the configured source/query for every invoke column. Validated native names remain on Loki; bounded native metric keys remain permitted where the provider supplies them, and hook identity is qualified as lifecycle rather than repository identity. It restores main's run-correlation test byte-for-byte and removes run correlation from every exported metric route, while retaining log correlation. The clock serialization note moves verbatim into a dated appended amendment, and renderer source references use named exporter sections. F6/F7 remove the unnecessary agentsview version literal from the health comment. F8 preserves render-only Grafana checking and requests installed-unit byte verification explicitly in the three acceptance calls and their plan mirrors. The next exact-head read covers all eight findings. No host activation or client change is included.

## Landing validation and path

The authorized landing replay is on475127d4 (#645). It preserves main's complete Grafana W1b notes and re-applies the reviewed dashboard note. All unowned main registry entries remain byte-identical, and all53 owned non-registry files have current registry hashes. Replay checks pass:223 affected tests (7 skips),721 tests in the eleven PyYAML-gated modules (47 skips), and six focused merger controls (no skips). Main's unchanged sources from475127d4 pass150 tests (7 skips), including the unchanged host-merger, tool-name and run-correlation modules. The install-plan and generated-dashboard checks pass; scripts/validate.py exits0. Native Node6/6 is reused from the completed phase because its inputs did not change.

The earlier complete Python phase on base003804f2 ran11,561 tests in the private, activated Python3.12 environment with checksum-locked CI dependencies and an initially empty HOME. It returned12 failures,1 setup error and1,211 skips. Each of the13 failing method/class selectors reproduces on clean main003804f2 in the same environment; no test was edited. These are declared baseline failures under the co-op's instructions, not a claim that the full phase passed. The setup-error selector is the runnable LinuxPlatformDependencyTests class. CI requirements SHA256:0b37433fb0fb165c893737e8e887581a07a6469683fbdf67ada1a08288758605; original pip-freeze SHA256:efd7c60ca69afe49918441138c3f3461b9c6e513628f562d5503362f8225c105. C1 exposed that this environment lacked the workflow's second pinned requirements file, so its YAML-gated tests had skipped. The replay adds unchanged requirements-validation.txt (SHA256:458d8cb1fef26d0d0a9a6a76707c0bad04fe84558550db750d50af206fecba81) and runs all eleven gated modules, as the command center directed instead of a second full phase. The new freeze SHA256 is e7d19cac4e91ceebe0d41b55581eb7a6dd8e7ebc021ce9a8b03dfe16056789d0. Original baseline logs and runnable IDs remain unchanged in durable lane state.

Landing path: the co-op's content read of27c8a4de closes F1–F8; the next head carries C1/V1–V4 and only the authorized #645 replay delta. The command center ACKs that head;5f alone lands it with green hosted CI. The relayed queue position is after#645;#825 remains dependent on LANDED775. The public hosted validate result remains the landing gate. Shared install-plan neighbour#723 is already landed and preserved.

## Contract changes

- `tests.test_observability_writer_identity.NativeCollectorTests`: kind5 runtime binding/eligibility adaptation, accepted by name under CC item171356Z. Maintained inputs are `manifests/stack.json:1102` (Collector pin) and `evidence/artifacts/new-wsl-install-plan-20261002/install-plan.json:2239` (native executable destination). The dated7October existing-native-writer-contracts amendment records the choice; it does not supply either maintained input.
- `NativeCollectorTests.setUp` and `.post`: kind2a synthetic fixture extensions for the new isolated guard/scope-attribute test; existing methods retain the prior pipeline and default payload.
- `NativeCollectorTests.test_sessions_and_processes_are_separate_writers_and_collapsed_streams_add_up`: kind1 assertion inventory, retained by name under CC item172216Z. All five prior main assertions remain AST-identical (Claude60/14, Codex2000/4000, phase4/4, no conversion errors, incoming session-group telemetry); exactly-two-point/no-agent-label checks add constraints. Main had no three-point assertion.

The npm setup-class failure is environment-dependent: this lane resolves npm to a mise shim invalid under empty HOME, while the co-op launcher places real node/npm directories first on PATH and all28 tests pass there. The lane's exact main-run failure remains in baseline-final-summary-failures.log and its class selector remains in the declared baseline file.

## C1 and V1–V4 closures

- C1: the maintained host merger explicitly copies the reviewed export guard and permits its one insertion immediately before conversion. Other host settings/order and refusal checks remain. Main's host module is byte-identical and passes; six new controls cover acceptance, idempotence, unknown processors, misplaced/duplicate guards, missing definitions and conflicting host settings.
- V1: the four existing-member declarations above are recorded in the dated native-writer-contracts and disposition amendments. The command center accepts the maintained runtime binding as kind5 and the session assertion inventory as kind1 by name.
- V2: the environment-dependent npm setup-class failure stays declared with its same main-run log; no test contract is weakened.
- V3: the disk-headroom decision gains only an appended dated amendment naming processors.batch.timeout; every earlier byte and its observed one-second value remain unchanged.
- V4: the current NativeStack2604 README explicitly requires the21300 manual Loki push URL.

The replay's extra adaptations are the dated M1/disposition record, preserved645 Grafana notes, the C1 merger guard/order support and new controls, the appended historical documentation locator and manual Loki endpoint instructions. The original fifteen disclosed adaptations remain. No host activation, client configuration or new feature selection is included.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant