Skip to content

fix(gateway): emit stale exec approval followup diagnostics#98293

Merged
steipete merged 4 commits into
openclaw:mainfrom
BsnizND:codex/structured-stale-approval-event
Jul 1, 2026
Merged

fix(gateway): emit stale exec approval followup diagnostics#98293
steipete merged 4 commits into
openclaw:mainfrom
BsnizND:codex/structured-stale-approval-event

Conversation

@BsnizND

@BsnizND BsnizND commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Related: #98279
Alternative to #98292 with the gateway preflight suppression point and public/stability diagnostics covered.

What Problem This Solves

Fixes an issue where operators could not consume a machine-readable signal when OpenClaw intentionally suppressed a stale exec approval follow-up after the target session was rebound by /new or /reset.

Why This Change Was Made

This adds a structured exec.approval.followup_suppressed diagnostic event at the existing stale-follow-up drop points without changing delivery behavior. The event includes the approval id, a stable session_rebound reason, and the suppression phase while avoiding raw session keys, route ids, command text, or other high-cardinality routing details.

Follow-up repairs also register the new event with the diagnostics OTEL exhaustive switch as an intentional no-op, update the plugin SDK surface budgets for the newly public diagnostic event type, and preserve approvalId when reading persisted stability bundles.

User Impact

Operator status and inbox tooling can now distinguish an intentional stale approval follow-up suppression from missing channel delivery or a stuck approval path. Existing Slack/direct/gateway suppression behavior remains unchanged; this is additive observability only.

AI-assisted with Codex.

Evidence

  • git diff --check
  • node scripts/plugin-sdk-surface-report.mjs --check
  • node scripts/run-vitest.mjs test/scripts/plugin-sdk-surface-report.test.ts -> 1 file, 8 tests passed.
  • node scripts/run-vitest.mjs src/logging/diagnostic-stability-bundle.test.ts -> 1 file, 10 tests passed.
  • node scripts/run-vitest.mjs src/agents/bash-tools.exec-approval-followup.test.ts src/gateway/server-methods/agent.test.ts src/infra/diagnostic-events.test.ts src/logging/diagnostic-stability.test.ts src/logging/diagnostic-stability-bundle.test.ts test/scripts/plugin-sdk-surface-report.test.ts -> 5 Vitest shards passed: tooling 8 tests, gateway 344 tests, infra 30 tests, logging 22 tests, agents 27 tests.
  • GitHub CI on 3be1325f passed the prior failing checks: check-lint, check-additional-extension-bundled, and checks-node-compact-small-whole-2.

Coverage added:

  • Direct stale denied follow-up suppression emits exec.approval.followup_suppressed.
  • Gateway preflight stale follow-up suppression emits the same event before touching the rebound session.
  • Public diagnostic subscribers receive the event.
  • Diagnostic stability records approval id, reason, and phase for operator tooling.
  • Persisted stability bundle reads preserve approval id, reason, and phase while stripping unrelated raw fields.
  • Plugin SDK surface budget check covers the new public diagnostic event type.

Local proof notes:

  • node scripts/run-bundled-extension-oxlint.mjs could not be completed in this checkout because node_modules/.bin/tsgo and node_modules/.bin/oxlint were unavailable after pnpm install --frozen-lockfile timed out on registry downloads for native/package tarballs.
  • OPENCLAW_STATE_DIR=$(mktemp -d /tmp/openclaw-pr98293-state.XXXXXX) node openclaw.mjs gateway stability --bundle latest --json was attempted as an isolated persisted-bundle CLI proof, but this source checkout has no dist/entry.(m)js build output. The command failed with openclaw: missing dist/entry.(m)js (build output).

Real behavior proof

Crabbox proof added 2026-07-01 UTC.

  • Crabbox v0.33.0, provider local-container, lease cbx_14a7f978e2bd, slug golden-crab, target ubuntu_26.04.
  • Checkout db2a5de505; command exit 0.
  • Scenario seeded agent:main:main as rebound from session-original to session-after-reset, called the production sendExecApprovalFollowup direct stale guard, wrote a stability bundle, then read it through corepack pnpm openclaw gateway stability --bundle latest --json --type exec.approval.followup_suppressed.
  • CLI readback showed exec.approval.followup_suppressed with approvalId: req-crabbox-pr98293-rebound, reason: session_rebound, phase: direct_delivery, and summary { "exec.approval.followup_suppressed": 1 }.
  • Full redacted proof is in fix(gateway): emit stale exec approval followup diagnostics #98293 (comment).

Hosted Crabbox provider attempts were blocked by local auth/tool prerequisites: Azure missing subscription/CLI login, Blacksmith CLI auth timed out, AWS requires Crabbox broker login, and GCP auth returned invalid_grant.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime agents Agent runtime and tooling size: S labels Jun 30, 2026
@clawsweeper

clawsweeper Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 1, 2026, 12:16 AM ET / 04:16 UTC.

Summary
The branch adds exec.approval.followup_suppressed diagnostics for stale exec approval follow-up suppression, records bounded stability metadata, updates diagnostics-otel/SDK budget handling, documents the event, and adds focused tests.

PR surface: Source +38, Tests +125, Docs +4, Other 0. Total +167 across 14 files.

Reproducibility: yes. at source level: current main drops stale direct and gateway preflight follow-ups with log/ok-only behavior, and public diagnostic subscribers filter out log.record. I did not run a live external-channel approval timeout in this read-only review.

Review metrics: 1 noteworthy metric.

  • Public Diagnostic Contract: 1 event type added. The new diagnostic event enters operator/plugin-facing surfaces, so its name and payload shape matter before merge.

Stored data model
Persistent data-model change detected: vector/embedding metadata: docs/cli/gateway.md, vector/embedding metadata: src/logging/diagnostic-stability.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #98279
Summary: This PR is a candidate fix for the open structured stale approval follow-up suppression event issue; sibling PRs target the same request with different event vocabulary or narrower coverage.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Choose the public event vocabulary for the canonical issue before landing.
  • Use final exact-head merge gates after the maintainer-selected contract is settled.

Risk before merge

  • [P1] The event name, session_rebound reason, direct_delivery/gateway_preflight phases, and persisted approvalId become operator/plugin-facing diagnostic contract once merged.
  • [P1] Related open PRs propose different public event vocabulary for the same canonical issue, so maintainers should select one stable contract before landing competing shapes.

Maintainer options:

  1. Accept This Event Contract (recommended)
    Maintain the current exec.approval.followup_suppressed shape and land this PR once exact-head merge gates are final.
  2. Revise Vocabulary Before Merge
    If maintainers prefer the timeout-suppressed naming or different field taxonomy, update this branch before the SDK-visible event type lands.
  3. Consolidate Into A Sibling Candidate
    If another open PR becomes the chosen contract, keep the canonical issue open and close this PR only after the replacement path is viable.

Next step before merge

  • [P2] Maintainers need to choose the public event vocabulary and canonical PR; no narrow automated code repair remains.

Security
Cleared: No concrete security or supply-chain concern found; the diff adds bounded diagnostic metadata and does not change permissions, dependencies, secrets handling, or command/session text exposure.

Review details

Best possible solution:

Land one clean canonical implementation for #98279 after maintainers choose the public diagnostic vocabulary, then close or supersede the competing candidate PRs.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main drops stale direct and gateway preflight follow-ups with log/ok-only behavior, and public diagnostic subscribers filter out log.record. I did not run a live external-channel approval timeout in this read-only review.

Is this the best way to solve the issue?

Conditionally yes: emitting a bounded event at the existing stale-drop points is the narrow maintainable shape and avoids a sidecar log. The remaining decision is the stable public event vocabulary across sibling candidates.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 5a5913a98b03.

Label changes

Label justifications:

  • P2: The PR addresses a focused operator-observability gap for stale approval follow-up suppression with limited runtime blast radius.
  • merge-risk: 🚨 compatibility: The PR adds a public diagnostic event and persisted stability metadata that downstream operator/plugin tooling may rely on after merge.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR includes after-fix Crabbox live output showing production-path stale suppression, stability bundle write, and CLI readback of the new diagnostic event.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR includes after-fix Crabbox live output showing production-path stale suppression, stability bundle write, and CLI readback of the new diagnostic event.
Evidence reviewed

PR surface:

Source +38, Tests +125, Docs +4, Other 0. Total +167 across 14 files.

View PR surface stats
Area Files Added Removed Net
Source 6 38 0 +38
Tests 5 126 1 +125
Docs 2 5 1 +4
Config 0 0 0 0
Generated 0 0 0 0
Other 1 3 3 0
Total 14 172 5 +167

What I checked:

  • Repository policy read and applied: Full root AGENTS.md plus scoped guides for docs, extensions, scripts, agents, gateway, and gateway server methods were read; the public diagnostic contract and compatibility guidance affected this review. (AGENTS.md:24, 5a5913a98b03)
  • Current main direct suppression is log-only: The direct denied and fallback stale branches log the drop and return false without emitting a structured diagnostic event. (src/agents/bash-tools.exec-approval-followup.ts:397, 5a5913a98b03)
  • Current main gateway preflight suppression is log/ok-only: The gateway preflight guard logs, writes an ok dedupe payload, responds ok, and returns before touching the rebound session, but it has no structured diagnostic emission. (src/gateway/server-methods/agent.ts:1519, 5a5913a98b03)
  • Current public diagnostic stream does not expose log records: onDiagnosticEvent filters log.record, so the existing free-form log line is not the operator/plugin-facing structured stream requested by the issue. (src/infra/diagnostic-events.ts:1305, 5a5913a98b03)
  • PR adds the public diagnostic contract and emission points: The diff adds a DiagnosticExecApprovalFollowupSuppressedEvent, registers it for async dispatch, emits it at direct and gateway preflight stale-follow-up guards, and adds stability/docs/tests coverage. (src/infra/diagnostic-events.ts:517, 937e08c698f7)
  • Real behavior proof is positive: The PR body and follow-up proof comment report Crabbox local-container live output that seeded a rebound session, called the production sendExecApprovalFollowup direct stale guard, wrote a stability bundle, and read the new event back through openclaw gateway stability --bundle latest --json --type exec.approval.followup_suppressed. (937e08c698f7)

Likely related people:

  • openperf: Authored the merged stale exec approval follow-up suppression behavior that this PR instruments in both agent and gateway paths. (role: introduced behavior; confidence: high; commits: 2ffbea20d2f6; files: src/agents/bash-tools.exec-approval-followup.ts, src/gateway/server-methods/agent.ts, src/agents/bash-tools.exec-approval-followup.test.ts)
  • gumadeiras: Introduced the gateway diagnostic stability and persisted bundle surfaces that this PR extends with approval suppression metadata. (role: diagnostic stability surface contributor; confidence: high; commits: 28818f914074; files: src/logging/diagnostic-stability.ts, src/logging/diagnostic-stability-bundle.ts, src/gateway/server-methods/diagnostics.ts)
  • steipete: Introduced the diagnostics event and diagnostics-otel exporter surfaces that this PR extends with a new event type. (role: diagnostics event introducer; confidence: medium; commits: 5c4079f66cc0; files: src/infra/diagnostic-events.ts, extensions/diagnostics-otel/src/service.ts)
  • vincentkoc: Recent diagnostics-otel migration work touched the central event contract and exporter switch that must tolerate the new event type. (role: recent diagnostics/OTEL contributor; confidence: medium; commits: de656e319471; files: src/infra/diagnostic-events.ts, extensions/diagnostics-otel/src/service.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. labels Jun 30, 2026
@openclaw-barnacle openclaw-barnacle Bot added extensions: diagnostics-otel Extension: diagnostics-otel scripts Repository scripts labels Jun 30, 2026
@BsnizND

BsnizND commented Jun 30, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Addressed the stability bundle finding in db2a5de5: persisted bundle reads now preserve approvalId for exec.approval.followup_suppressed, with a round-trip sanitizer test covering approval id, reason, and phase while stripping unrelated raw fields. Latest CI is green, including check-lint, check-additional-extension-bundled, checks-node-compact-small-whole-2, Real behavior proof, and QA Smoke CI.

@clawsweeper

clawsweeper Bot commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@BsnizND

BsnizND commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Real behavior proof update for the stale exec approval followup diagnostic.

Crabbox run:

  • Crabbox v0.33.0
  • Provider: local-container
  • Lease: cbx_14a7f978e2bd, slug golden-crab
  • Target: Linux ubuntu_26.04
  • Checkout: db2a5de505
  • Exit: 0

Scenario run inside the disposable Crabbox container:

  1. Seeded the runtime session store so agent:main:main resolved to session-after-reset while the approval carried expectedSessionId: "session-original".
  2. Called the production sendExecApprovalFollowup path with direct: true and a denied exec result.
  3. Confirmed the followup was suppressed and the stability recorder captured the structured diagnostic event.
  4. Wrote a persisted stability bundle, then read it back through the actual CLI surface:
    corepack pnpm openclaw gateway stability --bundle latest --json --type exec.approval.followup_suppressed

Redacted proof output:

{
  "delivered": false,
  "event": {
    "type": "exec.approval.followup_suppressed",
    "approvalId": "req-crabbox-pr98293-rebound",
    "reason": "session_rebound",
    "phase": "direct_delivery"
  },
  "bundlePath": "$OPENCLAW_STATE_DIR/logs/stability/openclaw-stability-2026-07-01T00-04-11-696Z-3947-pr98293.stale_exec_approval_followup_proof.json"
}

CLI stability readback:

{
  "cliPathRedacted": "$OPENCLAW_STATE_DIR/logs/stability/openclaw-stability-2026-07-01T00-04-11-696Z-3947-pr98293.stale_exec_approval_followup_proof.json",
  "cliEvent": {
    "type": "exec.approval.followup_suppressed",
    "approvalId": "req-crabbox-pr98293-rebound",
    "reason": "session_rebound",
    "phase": "direct_delivery"
  },
  "summary": {
    "exec.approval.followup_suppressed": 1
  }
}

Hosted Crabbox providers were attempted first but blocked by local auth/tooling prerequisites:

  • azure: missing AZURE_SUBSCRIPTION_ID / az CLI login.
  • blacksmith-testbox: required local blacksmith CLI auth; CLI installed, browser auth timed out for org openclaw.
  • aws: requires crabbox login --url https://crabbox.openclaw.ai --provider aws.
  • gcp: active project found, but provider auth failed with invalid_grant.

@clawsweeper re-review

@BsnizND

BsnizND commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Proof is now in the PR body under Real behavior proof, with the Crabbox local-container run and CLI stability readback linked/details included.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 1, 2026
@steipete
steipete requested a review from a team as a code owner July 1, 2026 03:46
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation channel: discord Channel integration: discord channel: googlechat Channel integration: googlechat channel: line Channel integration: line channel: matrix Channel integration: matrix channel: mattermost Channel integration: mattermost channel: msteams Channel integration: msteams labels Jul 1, 2026
@openclaw-barnacle openclaw-barnacle Bot added extensions: copilot plugin: workboard channel: sms Channel integration: sms channel: raft Channel integration: Raft labels Jul 1, 2026
@github-actions github-actions Bot added the dependencies-changed PR changes dependency-related files label Jul 1, 2026
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Dependency graph guard cleared

This PR no longer has blocked dependency graph changes. A future dependency graph change requires a fresh /allow-dependencies-change comment after the guard blocks that new head SHA.

  • Current SHA: 937e08c698f72a13778655dad51a112d207456fe

@steipete

steipete commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

/allow-dependencies-change false-positive ancestry after maintainer prepare: the prepared tree has no dependency-file delta versus current main, and the repo-native merge wrapper will verify the prepared tree and mainline base before squash.

@steipete

steipete commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Land-ready maintainer proof for 0e4f16aee4991ae1ec3a72a1b170c8f8da80c6dd:

  • Focused behavior tests: node scripts/run-vitest.mjs src/agents/bash-tools.exec-approval-followup.test.ts src/infra/diagnostic-events.test.ts src/logging/diagnostic-stability.test.ts src/logging/diagnostic-stability-bundle.test.ts test/scripts/plugin-sdk-surface-report.test.ts — 87 passed.
  • Gateway preflight regression: node scripts/run-vitest.mjs src/gateway/server-methods/agent.test.ts -t "drops a stale exec approval followup at preflight" — 2 passed, 344 skipped.
  • SDK/docs checks: node scripts/plugin-sdk-surface-report.mjs --check, node scripts/check-docs-mdx.mjs docs/gateway/opentelemetry.md docs/cli/gateway.md, pnpm format:docs:check, pnpm docs:check-i18n-glossary, and git diff --check — all passed.
  • Fresh autoreview against origin/main — no accepted/actionable findings (confidence 0.94).
  • Changed-surface remote gate: Blacksmith Testbox tbx_01kwdvy7e9yhvfex86rvc50347 / Actions run https://github.com/openclaw/openclaw/actions/runs/28491692386 — passed.
  • Exact-head release-gate CI: https://github.com/openclaw/openclaw/actions/runs/28492227982 — 110 jobs passed, 0 failed.
  • Repo-native scripts/pr prepare-run 98293 accepted the exact-head hosted gates and verified the prepared and PR heads match.

The contributor also supplied real production-path Crabbox proof (cbx_14a7f978e2bd) for the direct stale guard and persisted stability CLI readback. Known gap: no external Slack workspace interaction was exercised; the changed contract is the channel-independent diagnostic emission path, with both direct-delivery and gateway-preflight guards covered by production-function regression tests.

@steipete
steipete force-pushed the codex/structured-stale-approval-event branch from 0e4f16a to 937e08c Compare July 1, 2026 04:06
@openclaw-barnacle openclaw-barnacle Bot removed channel: discord Channel integration: discord channel: googlechat Channel integration: googlechat channel: line Channel integration: line channel: matrix Channel integration: matrix channel: mattermost Channel integration: mattermost channel: msteams Channel integration: msteams channel: nextcloud-talk Channel integration: nextcloud-talk channel: nostr Channel integration: nostr channel: slack Channel integration: slack channel: telegram Channel integration: telegram labels Jul 1, 2026
@steipete

steipete commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Final-head addendum: the ancestry-only sync produced 937e08c698f72a13778655dad51a112d207456fe with the same reviewed change and a clean merge base. Exact-head release-gate CI passed all 110 jobs with zero failures: https://github.com/openclaw/openclaw/actions/runs/28492777136. OPENCLAW_TESTBOX=1 scripts/pr prepare-run 98293 then accepted the hosted gates and verified the prepared and remote heads match.

@steipete

steipete commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Merged via squash.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling docs Improvements or additions to documentation extensions: diagnostics-otel Extension: diagnostics-otel gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. scripts Repository scripts size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants