Skip to content

fix(agents): cron and agent runs mark successful exec commands with semantic nonzero exits as failed#111574

Open
yetval wants to merge 1 commit into
openclaw:mainfrom
yetval:fix/exec-completed-nonzero-exit-not-error
Open

fix(agents): cron and agent runs mark successful exec commands with semantic nonzero exits as failed#111574
yetval wants to merge 1 commit into
openclaw:mainfrom
yetval:fix/exec-completed-nonzero-exit-not-error

Conversation

@yetval

@yetval yetval commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Fixes #97318
Related: #107278 and #110153 (same root cause), #94846 (downstream cron delivery handling), #110244 (adjacent open PR for zero-exit false positives; it does not change nonzero-exit classification), #107283 (prior attempt with the same classifier change, closed by its author after review noted missing live behavior proof)

What Problem This Solves

Fixes an issue where agent and isolated cron runs treat a successfully completed exec command with a semantic nonzero exit code (for example grep exiting 1 on no match, or an audit script exiting 1 to mean "alert needed") as a tool failure. The exec runtime already returns these results as status completed with stdout preserved, but the shared tool-result classifier still flags any finite nonzero exitCode as an error. That error state flows into embedded-agent isError envelopes, "Exec failed" warnings on chat and TUI progress surfaces, false failure diagnostics persisted on cron run records, and cron outcome resolution, which only rescues such runs through fragile text-prefix matching on the warning payloads.

Why This Change Was Made

Root cause is the final fallback in isToolResultError:

// before (src/agents/tool-result-error.ts)
const exitCode = details ? readToolErrorField(details, "exitCode") : undefined;
return typeof exitCode === "number" && Number.isFinite(exitCode) && exitCode !== 0;

This exit-code fallback was introduced in #93228 and predates the exec runtime's completed/failed split. The exec runtime is the owner of exit semantics: buildExecExitOutcome (src/agents/bash-tools.exec-runtime.ts) marks normal process exits as completed (except shell 126/127) and marks spawn errors, timeouts, signals, denials, and 126/127 as failed with a reason. The classifier was second-guessing that classification.

// after
if (normalized === "completed") {
  return false;
}
const exitCode = details ? readToolErrorField(details, "exitCode") : undefined;
return typeof exitCode === "number" && Number.isFinite(exitCode) && exitCode !== 0;

The guard sits after every failure check, so nothing else weakens: explicit ok/success false, all failure statuses (failed, timeout, denied, killed, and the rest), timedOut, and error fields still classify as errors first, and statusless results with a nonzero exitCode keep the old behavior. resolveToolResultFailureKind inherits the same boundary. This is the repair ClawSweeper recommended on #97318 ("Make normal exec nonzero nonfatal" at the shared classifier), left unimplemented pending proof after #107283 was closed without it.

The issue carries clawsweeper:no-new-fix-pr pending a maintainer product decision between three options. This PR implements the recommended option so the decision has a reviewable, proven candidate; happy to close if maintainers pick a different boundary.

User Impact

Cron jobs and agent turns that use exit codes semantically no longer record false "Exec failed" diagnostics or error-flagged tool payloads, and no longer depend on warning-text prefix heuristics to avoid being marked as failed runs. True failures (command not found, permission denied, timeouts, signals, spawn errors) still classify as failures, including shell 126/127.

Evidence

Live before/after on a real Gateway, driven end to end through the production CLI: isolated OPENCLAW_STATE_DIR, real gateway process on port 19917, a real isolated cron agentTurn job with toolsAllow ["exec"], and the exec tool spawning a real grep subprocess that exits 1 (no match). The only mocked seam is the model provider HTTP endpoint (a local OpenAI Responses server that first returns an exec tool call for grep CRITICAL service.log, then replies NO_REPLY). The gateway, cron scheduler, embedded agent runner, exec runtime, SQLite state, and cron CLI are all real, and both runs used identical job settings, command, provider stub, and inputs.

BEFORE (pristine main 3c0f55b): the run record persists failure diagnostics for the successful no-match scan:

$ openclaw cron runs --id bd6a7bf0-6205-4659-b302-5367948158f2 ...
{
  "entries": [
    {
      "action": "finished",
      "status": "ok",
      "summary": "NO_REPLY",
      "diagnostics": {
        "summary": "⚠️ 🛠️ Exec failed: `search \"CRITICAL\" in .../service.log` (exit 1)",
        "entries": [
          {
            "source": "tool",
            "severity": "warn",
            "message": "⚠️ 🛠️ Exec failed: `search \"CRITICAL\" in .../service.log` (exit 1)"
          }
        ]
      },
      ...

AFTER (this patch, identical inputs): the same run is clean, with no failure diagnostics at all:

$ openclaw cron runs --id 5afc9d0e-6fea-4ac8-88f9-651c69bcbcb0 ...
{
  "entries": [
    {
      "action": "finished",
      "status": "ok",
      "summary": "NO_REPLY",
      "runAtMs": 1784505044721,
      "durationMs": 2490,
      ...

The provider stub logged the tool output the model received in both runs, byte-identical: "\n\n(Command exited with code 1)". This confirms the lower exec layer already preserves output on main and the remaining defect was classification only. The BEFORE run only avoided status "error" because resolveCronPayloadOutcome (src/cron/isolated-agent/helpers.ts) special-cases payload text starting with the literal prefix "⚠️ 🛠️ "; the false error payload was real and any tool result not matching that prefix would have failed the run, which is the failure mode reported in #97318 and #94846.

Verification

  • node scripts/run-vitest.mjs src/agents/tool-result-error.test.ts src/agents/embedded-agent-subscribe.tools.test.ts src/agents/embedded-agent-runner/extensions.test.ts src/agents/embedded-agent-runner/cli-backend-dispatch.test.ts src/agents/harness/tool-result-middleware.test.ts src/agents/embedded-agent-message-tool-source-reply.test.ts src/cron/run-diagnostics.test.ts src/agents/bash-tools.test.ts: all pass.
  • node scripts/run-oxlint.mjs and oxfmt --check on the changed files: clean.
  • tsgo core and core-test lanes: clean.
  • The new tool-result-error.test.ts assertions fail on pristine main (completed exitCode 1 classified as error) and pass with the patch.
  • gateway-codex-harness command-evidence helpers intentionally keep their own stricter exit-0 requirement for proof evidence and are unaffected (no isToolResultError usage).

…tool failures

The exec runtime marks normal process exits as completed and preserves
stdout, but isToolResultError still flagged any finite nonzero exitCode
as an error. Completed exec results with semantic exit codes such as
grep exiting 1 on no match were pushed into isError envelopes, false
Exec failed warnings, and cron run failure diagnostics. The classifier
now defers to an explicit completed status. Failure statuses, ok or
success false, error fields, timedOut, and statusless nonzero exits
still classify as errors.
@yetval
yetval marked this pull request as ready for review July 19, 2026 23:56
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS labels Jul 19, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 20, 2026
@clawsweeper

clawsweeper Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 19, 2026, 9:55 PM ET / July 20, 2026, 01:55 UTC.

Summary
Changes the shared tool-result error classifier so completed results with semantic nonzero exit codes are nonfatal, with regression coverage for completed, failed, and statusless outcomes.

PR surface: Source +3, Tests +34. Total +37 across 3 files.

Reproducibility: yes. the supplied controlled Gateway cron run gives a concrete before/after path, and the changed classifier is directly source-reproducible. The review did not independently rerun that scenario against the current main SHA.

Review metrics: none identified.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Obtain maintainer confirmation that completed is authoritative for structured tool results beyond the exec runtime.

Risk before merge

  • [P1] This shared classifier feeds agent error envelopes and cron outcome handling; merging changes warning, persisted diagnostic, and delivery behavior for every structured result marked completed with a nonzero exit code, so maintainers should confirm that completed is the intended cross-producer success contract rather than an exec-only convention.

Maintainer options:

  1. Confirm the shared terminal-state contract (recommended)
    Before merge, confirm that all structured producers may rely on completed to suppress nonzero-exit failure handling and retain cross-boundary regression coverage.
  2. Keep nonzero exits globally fatal
    If completed status is not authoritative outside exec, pause or close this generic classifier change and choose an exec-owned boundary instead.

Next step before merge

  • [P2] A maintainer must choose the shared terminal-state contract before this otherwise narrow PR can be safely merged.

Maintainer decision needed

  • Question: Should the shared tool-result contract treat status: "completed" as authoritative over a nonzero exitCode for all structured tool producers, as this PR does for exec-derived results?
  • Rationale: The linked canonical reports explicitly retain a maintainer product-decision gate, and this generic classifier affects more than the exec runtime demonstrated by the supplied proof.
  • Likely owner: steipete — The relevant shared classifier appears to originate in the merged structured-terminal-state work.
  • Options:
    • Confirm completed is authoritative (recommended): Approve the generic contract, retain the current failure-precedence tests, and merge this focused repair.
    • Limit the exception to exec: Keep the generic fallback and instead introduce an exec-specific representation or call-site boundary if maintainers do not want completed status to define success globally.
    • Preserve current behavior: Close this PR if nonzero exit codes must remain warning-worthy regardless of a completed structured status.

Security
Cleared: The three-file TypeScript and test-only diff adds no dependency, workflow, permission, secret, artifact-download, or supply-chain surface.

Review details

Best possible solution:

Confirm that structured status: "completed" is the canonical nonfatal contract across tool producers, then land this narrow classifier fix with the retained failure-precedence regressions.

Do we have a high-confidence way to reproduce the issue?

Yes, the supplied controlled Gateway cron run gives a concrete before/after path, and the changed classifier is directly source-reproducible. The review did not independently rerun that scenario against the current main SHA.

Is this the best way to solve the issue?

Unclear pending maintainer intent: the patch is the narrowest repair if completed status is the shared success authority, but an exec-owned boundary is safer if that status is not a generic contract.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against 27f05c8993fb.

Label changes

Label justifications:

  • P2: The patch targets a real agent and cron false-failure path with bounded blast radius, but it is not an emergency availability or security regression.
  • merge-risk: 🚨 message-delivery: Cron outcome classification can determine whether a completed isolated run proceeds through delivery handling.
  • merge-risk: 🚨 session-state: The shared result classifier controls isError envelopes and persisted run diagnostics across agent execution paths.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body supplies a controlled real Gateway and isolated cron before/after transcript with an actual nonzero-exit subprocess; it directly demonstrates the claimed diagnostic change.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies a controlled real Gateway and isolated cron before/after transcript with an actual nonzero-exit subprocess; it directly demonstrates the claimed diagnostic change.
Evidence reviewed

PR surface:

Source +3, Tests +34. Total +37 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 1 3 0 +3
Tests 2 35 1 +34
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 38 1 +37

What I checked:

Likely related people:

  • steipete: The merged structured-terminal-classifier change is the recorded origin of the shared exit-code fallback that this PR adjusts. (role: introduced shared classifier; confidence: medium; commits: 0314819f918a; files: src/agents/tool-result-error.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: XS status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: exec tool treats subprocess nonzero exit codes as tool failures, masking stdout and exit-code semantics

1 participant