Skip to content

Commit 1319632

Browse files
5H3LL5H3LL
5H3LL
authored and
5H3LL
committed
feat(slack): distinguish benign no-result from tool failure in progress rendering
Implements U1 + U2 + U3 for #85478 requirements. U1: Pure exec outcome classifier (bash-tools.exec-outcome-classification.ts) - classifyExecOutcome(): narrow detection of rg exit 1 and xargs-rg exit 123 - execOutcomeStatusLabel(): human-readable labels per classification - Only two narrow patterns are benign: direct rg no-match (exit 1) and xargs-wrapped rg no-match (exit 123), both requiring clean empty output U2: Command-output event wiring - pi-embedded-subscribe.handlers.tools.ts: compute classification on exec end - infra/agent-events.ts: add outcomeClassification + statusLabel fields - auto-reply/get-reply-options.types.ts: new fields in onCommandOutput callback - agent-runner-execution.ts: pass classification through reply pipeline U3: Slack progress rendering - channel-streaming.ts: render benign_no_result as statusLabel or 'No matches found' - dispatch.ts: pass outcomeClassification + statusLabel to progress renderer Test coverage: - bash-tools.exec-outcome-classification.test.ts: 8 cases - pi-embedded-subscribe.handlers.tools.test.ts: new benign rg classification test - dispatch.preview-fallback.test.ts: 40 tests including benign Slack rendering - channel-streaming.test.ts: 22 tests - agent-runner-execution.test.ts: 113 tests
1 parent d756e1c commit 1319632

14 files changed

Lines changed: 711 additions & 8 deletions

‎docs/brainstorms/2026-05-22-slack-benign-tool-failure-ux-requirements.md‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
---
2+
date: 2026-05-22
3+
topic: slack-benign-tool-failure-ux
4+
---
5+
6+
# Slack Benign Tool Failure UX
7+
8+
## Summary
9+
10+
Slack-facing OpenClaw responses should distinguish a broken tool from a
11+
successful optional lookup that found nothing. Benign search misses, such as
12+
markdown `rg` returning no matches through `xargs`, should become a concise
13+
notice instead of a scary failed tool step.
14+
15+
---
16+
17+
## Problem Frame
18+
19+
In the Erum Jiva demo brief, the agent produced useful business context, but
20+
Slack appended a warning that a markdown search in the `niemand-b2b` workspace
21+
failed. The workspace existed and contained markdown files. The likely command
22+
simply found no matching text.
23+
24+
That distinction matters in Slack. Operators scanning a brief should not have to
25+
debug shell semantics to decide whether the answer is trustworthy. Real failures
26+
still need to be visible, but optional context probes should report their
27+
outcome in user language.
28+
29+
---
30+
31+
## Actors
32+
33+
- A1. Slack user: Reads operational answers and needs to know whether
34+
supporting context was found or whether something actually broke.
35+
- A2. OpenClaw agent: Runs optional context searches while preparing responses.
36+
- A3. OpenClaw runtime or presentation layer: Converts tool outcomes into
37+
channel-visible status.
38+
- A4. Maintainer: Needs real failures preserved in logs for debugging.
39+
40+
---
41+
42+
## Key Flows
43+
44+
- F1. Optional search finds no results
45+
- **Trigger:** An agent searches workspace markdown while preparing a Slack
46+
response.
47+
- **Actors:** A1, A2, A3
48+
- **Steps:** The search command exits with a benign no-match status. OpenClaw
49+
classifies the outcome as no context found. Slack shows either a short
50+
notice or nothing, depending on whether the missing context affects the
51+
answer.
52+
- **Outcome:** The user sees the answer without a misleading failure banner.
53+
- **Covered by:** R1, R2, R3, R5
54+
- F2. Optional search actually fails
55+
- **Trigger:** A search command cannot run because of a missing directory,
56+
permission error, unavailable command, timeout, malformed invocation, or a
57+
similar real failure.
58+
- **Actors:** A1, A3, A4
59+
- **Steps:** OpenClaw classifies the outcome as an actual failure. Slack output
60+
remains visible and actionable. Logs retain the raw command and error
61+
details.
62+
- **Outcome:** Real tool problems stay loud enough to fix.
63+
- **Covered by:** R1, R4, R6
64+
65+
---
66+
67+
## Requirements
68+
69+
### User-Facing Classification
70+
71+
- R1. OpenClaw must distinguish at least three Slack-visible classes for tool
72+
outcomes: successful result, benign no-result, and actual failure.
73+
- R2. A no-result search must not be presented with failure language such as
74+
"failed" when the command completed normally but found no matching context.
75+
- R3. Optional context searches should show a concise operational notice, such
76+
as "No local markdown context found," or be omitted when the missing context
77+
does not affect the answer.
78+
- R4. Actual failures must remain visible and actionable when the tool could not
79+
run correctly, could not access its target, timed out, or returned a real
80+
error.
81+
82+
### Initial Benign Cases
83+
84+
- R5. The first supported benign cases must include ripgrep no-match exits and
85+
the common `find ... | xargs rg ...` pattern where no matches surface as an
86+
`xargs` non-zero status.
87+
- R6. Missing workspace, missing file path, permission denied, command not
88+
found, syntax errors, and timeouts must not be downgraded to benign no-result
89+
notices.
90+
91+
### Observability
92+
93+
- R7. Raw command, exit status, stdout, stderr, and classification must remain
94+
available in logs or diagnostics even when Slack hides or softens the
95+
user-facing message.
96+
- R8. The classification must be narrow enough that maintainers can add new
97+
benign patterns deliberately instead of suppressing broad failure classes.
98+
99+
### Channel Behavior
100+
101+
- R9. Slack output must prioritize the main answer and show tool-status notices
102+
only when they change how the user should interpret the answer.
103+
- R10. When a no-result notice is shown, it must name the missing context in
104+
plain terms, not shell implementation details.
105+
106+
---
107+
108+
## Acceptance Examples
109+
110+
- AE1. **Covers R1, R2, R3, R5.** Given a Slack-triggered agent searches
111+
markdown files for a prospect name and `rg` finds no matches, when the
112+
response is sent, Slack does not show a failed tool warning and may show
113+
"No local markdown context found."
114+
- AE2. **Covers R1, R2, R5.** Given the agent uses a
115+
`find ... | xargs rg ...` shape and the wrapped command exits non-zero only
116+
because there were no matches, when the result is classified, it is treated as
117+
benign no-result rather than an actual failure.
118+
- AE3. **Covers R4, R6.** Given the workspace path does not exist, when the same
119+
search is attempted, Slack shows an actual actionable failure rather than
120+
"no local context found."
121+
- AE4. **Covers R7.** Given a no-result search is softened in Slack, when a
122+
maintainer inspects diagnostics, they can still see the raw command, exit
123+
status, stdout, stderr, and no-result classification.
124+
- AE5. **Covers R9, R10.** Given the answer is complete without the optional
125+
context, when the search finds nothing, Slack either omits the notice or states
126+
the missing context plainly without exposing shell details.
127+
128+
---
129+
130+
## Success Criteria
131+
132+
- Slack users no longer interpret optional no-match searches as system breakage.
133+
- Real tool failures remain visible enough that operators and maintainers can
134+
fix them quickly.
135+
- Downstream planning can implement the behavior without inventing which
136+
outcomes are benign, which remain failures, or what Slack should show.
137+
138+
---
139+
140+
## Scope Boundaries
141+
142+
- This does not create a runtime-wide taxonomy for every possible command or
143+
tool.
144+
- This does not rewrite `niemand-b2b` agent instructions or `TOOLS.md`.
145+
- This does not hide raw failures from logs or diagnostics.
146+
- This does not require changing the answer-generation behavior for the demo
147+
brief itself.
148+
- This does not require agents to stop using shell commands, though later policy
149+
improvements may still encourage safer search patterns.
150+
151+
---
152+
153+
## Key Decisions
154+
155+
- Optimize the Slack user experience first: the immediate pain is misleading
156+
user-facing failure language, not missing search capability.
157+
- Keep the first classifier narrow: `rg` no-match and `xargs`-wrapped no-match
158+
are worth handling now because they are common and easy to explain.
159+
- Preserve observability: softening Slack output must not make debugging harder.
160+
161+
---
162+
163+
## Dependencies / Assumptions
164+
165+
- OpenClaw has a presentation or result-normalization point where tool outcomes
166+
can be classified before Slack renders them.
167+
- Optional context searches can be identified either from tool metadata, command
168+
shape, or a conservative classifier without inspecting arbitrary private
169+
content.
170+
- The raw tool result remains stored somewhere accessible to maintainers.
171+
172+
---
173+
174+
## Outstanding Questions
175+
176+
### Resolve Before Planning
177+
178+
- Affects R3, R9. Product: Should benign no-result notices be shown by default,
179+
or only when the agent relied on that search as part of its answer?
180+
181+
### Deferred to Planning
182+
183+
- Affects R5. Technical: Where is the narrowest safe normalization point for
184+
classifying `rg` no-match and `xargs`-wrapped no-match outcomes?
185+
- Affects R7. Technical: Which existing diagnostic surface should expose the raw
186+
command plus user-facing classification?

‎docs/plans/2026-05-22-slack-benign-tool-failure-ux.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# Slack Benign Tool Failure UX
2+
3+
## Status
4+
5+
active
6+
7+
## Origin
8+
9+
Requirements source: `docs/brainstorms/2026-05-22-slack-benign-tool-failure-ux-requirements.md`.
10+
11+
## Problem Frame
12+
13+
Slack progress/status rendering can make optional local context searches look like scary tool failures when the shell command simply found no matches. The first target cases are `rg` no-match exits and `find ... | xargs rg ...` no-match exits. Actual command failures must stay visible.
14+
15+
Plan assumption: benign no-result notices should be suppressed in Slack by default and shown only when the interpretation changes the visible answer or progress state. This follows the origin document's Slack priority requirement instead of adding a noisy default notice.
16+
17+
## Scope Boundaries
18+
19+
- Fix only the narrow exec/search outcome classification and Slack-facing progress wording needed for the requirements.
20+
- Do not add a broad runtime-wide outcome taxonomy.
21+
- Do not alter `niemand-b2b` prompts, TOOLS docs, or model answer generation.
22+
- Do not hide missing paths, permission errors, command-not-found, syntax errors, timeouts, or other real failures.
23+
- Preserve raw command output, exit status, stdout/stderr-derived aggregate output, and diagnostic details for logs and tool/result consumers.
24+
25+
## Requirements Trace
26+
27+
1. Distinguish success, benign no-result, and actual failure for exec search progress.
28+
2. Do not show `failed` wording for no-result `rg` searches.
29+
3. Keep Slack progress concise; suppress benign no-result progress unless it is the useful interpretation.
30+
4. Keep actual failures visible and actionable.
31+
5. Cover direct `rg` exit `1` no-match and `find ... | xargs rg ...` exit `123` no-match.
32+
6. Do not downgrade missing paths, permission errors, command-not-found, syntax errors, or timeouts.
33+
7. Preserve raw exit/output data and expose the narrow classification in diagnostics/events.
34+
8. Keep the classifier conservative and command-pattern-specific.
35+
36+
## Implementation Units
37+
38+
### U1: Narrow exec no-result classifier
39+
40+
Files:
41+
42+
- Create `src/agents/bash-tools.exec-outcome-classification.ts`
43+
- Create `src/agents/bash-tools.exec-outcome-classification.test.ts`
44+
- Modify `src/infra/agent-events.ts`
45+
46+
Approach:
47+
48+
- Add a small pure classifier for completed exec outcomes, keyed by command text, exit code, timeout flag, and aggregate output.
49+
- Return one of `success`, `benign_no_result`, or `failure`.
50+
- Classify direct `rg` exit `1` with no error-looking output as `benign_no_result`.
51+
- Classify `xargs`-wrapped `rg` exit `123` with no error-looking output as `benign_no_result`.
52+
- Treat exit `0` as `success`.
53+
- Treat timeout, shell failures, stderr/error-looking aggregate output, and unsupported commands as `failure` or unclassified.
54+
- Extend command output event data with an optional `outcomeClassification` field and optional human status label.
55+
56+
Verification:
57+
58+
- Direct `rg` no-match exits classify as `benign_no_result`.
59+
- `find ... | xargs rg ...` no-match exits classify as `benign_no_result`.
60+
- Missing path, permission denied, command not found, syntax error, timeout, and unrelated exit `123` do not classify as benign.
61+
62+
### U2: Emit classification through command-output events
63+
64+
Files:
65+
66+
- Modify `src/agents/pi-embedded-subscribe.handlers.tools.ts`
67+
- Modify `src/agents/pi-embedded-subscribe.handlers.tools.test.ts`
68+
- Modify `src/auto-reply/get-reply-options.types.ts`
69+
- Modify `src/auto-reply/reply/agent-runner-execution.ts`
70+
71+
Approach:
72+
73+
- Read the original exec command from the saved tool-start args when an exec tool ends.
74+
- Compute the classifier using sanitized exec details so diagnostics stay redacted where they already are.
75+
- Attach the classification and a concise `No matches found` status label to `command_output` events only for benign no-result cases.
76+
- Keep `exitCode` and output fields unchanged.
77+
- Forward the new fields through reply option callbacks so channel renderers can use them without re-parsing shell output.
78+
79+
Verification:
80+
81+
- Command-output events include `outcomeClassification: "benign_no_result"` for no-match cases while preserving `exitCode`.
82+
- Secret redaction tests still pass.
83+
- Actual failed exec events retain failed status/error behavior.
84+
85+
### U3: Slack progress wording
86+
87+
Files:
88+
89+
- Modify `src/plugin-sdk/channel-streaming.ts`
90+
- Modify `src/plugin-sdk/channel-streaming.test.ts`
91+
- Modify `extensions/slack/src/monitor/message-handler/dispatch.ts`
92+
- Modify `extensions/slack/src/monitor/message-handler/dispatch.preview-fallback.test.ts`
93+
94+
Approach:
95+
96+
- Extend command-output progress input with the optional outcome classification and status label.
97+
- Render `benign_no_result` as a completed/search-no-result line such as `No matches found` rather than `exit 1` or `exit 123`.
98+
- Keep Slack's main answer prioritized; do not add a separate warning card.
99+
- Continue rendering actual nonzero, timeout, and failed statuses as actionable progress/failure lines.
100+
101+
Verification:
102+
103+
- Slack preview/progress tests show benign no-result progress without failure wording.
104+
- Existing error-final and streaming fallback tests remain unchanged.
105+
- Progress rendering for a real nonzero non-benign command still includes the exit code.
106+
107+
## Test Plan
108+
109+
Run:
110+
111+
```bash
112+
pnpm test src/agents/bash-tools.exec-outcome-classification.test.ts
113+
pnpm test src/agents/pi-embedded-subscribe.handlers.tools.test.ts -t "command_output"
114+
pnpm test src/plugin-sdk/channel-streaming.test.ts
115+
pnpm test extensions/slack/src/monitor/message-handler/dispatch.preview-fallback.test.ts
116+
pnpm check:changed
117+
```
118+
119+
If `src/channels` or Slack hot entrypoints are broadened beyond the files above, also run:
120+
121+
```bash
122+
pnpm build
123+
```
124+
125+
## Risks
126+
127+
- Shell commands are arbitrary strings; the classifier must remain conservative to avoid downgrading real failures.
128+
- GNU `xargs` exit `123` can mean child failure, not just no matches. Only classify it when the command visibly wraps `rg` and aggregate output is clean.
129+
- Some useful no-result searches may still show an exit code if they do not match the narrow first-pass patterns. That is acceptable for this scope.

0 commit comments

Comments
 (0)