Skip to content

Commit 7ff5372

Browse files
committed
fix: ask tool schema validation and bash carriage return handling
- Add permissive input parsing with preprocessing to ask tool (anomalyco#136) - Normalize string options to {value, label} objects - Accept message aliases (text, prompt, question) - Normalize type variations (multiselect, multi_select -> multi-select) - Wrap single string defaultValue in array for multi-select - Parse string timeout to number - Auto-generate question IDs if missing - Add formatValidationError for actionable model guidance - Fix bash tool carriage return sanitization - Add processCarriageReturns() fallback for when ptyToText doesn't process terminal sequences (e.g., native module unavailable) - Add comprehensive unit tests for ask tool normalization (23 tests)
1 parent bd84bea commit 7ff5372

4 files changed

Lines changed: 882 additions & 5 deletions

File tree

Lines changed: 277 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
# Plan: Fix Ask Tool Schema Validation Error (Broad/Permissive)
2+
3+
**Issue:** [#136 - [BUG] Ask Tool](https://github.com/Latitudes-Dev/shuvcode/issues/136)
4+
**Created:** 2025-12-17
5+
**Status:** Implementation Complete (Milestones 2, 3.1-3.2, 5.1 done)
6+
7+
## Goals & Scope
8+
9+
### Primary Goal
10+
11+
Make the `ask` tool work reliably across models/providers by preventing schema-validation failures from blocking the TUI modal.
12+
13+
### Specifically, we must fix
14+
15+
- `ask` tool calls where select/multi-select `options` are provided as strings instead of `{ value, label }` objects.
16+
- `ask` tool calls where `message` (required question text) is missing but may exist under common alias keys.
17+
18+
### Non-goals
19+
20+
- Do not redesign the Ask dialog UX (already addressed elsewhere).
21+
- Do not change user-facing TUI behavior beyond making `ask` reliably open.
22+
- Do not introduce provider-specific logic unless broad, provider-agnostic fixes are insufficient.
23+
24+
## Planning Context Capture (Repo Reality)
25+
26+
### Where the failure occurs
27+
28+
Tool argument validation happens before the tool’s `execute` body:
29+
30+
- `packages/opencode/src/tool/tool.ts:48-59` validates `toolInfo.parameters.parse(args)`.
31+
32+
If this validation fails, `AskTool.execute()` never runs, and the modal never appears.
33+
34+
### Canonical “question” contract is strict and used by the TUI
35+
36+
The TUI expects canonical `Question.Item` shapes:
37+
38+
- `packages/opencode/src/question/index.ts:51-52` defines `Question.Item` as a `z.discriminatedUnion("type", ...)`.
39+
- The dialog components assume `select`/`multi-select` options are objects with `value` and `label`:
40+
- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-select.tsx:126-142`
41+
- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-multi-select.tsx:165-190`
42+
43+
### AskTool currently publishes to the TUI with a cast (no secondary validation)
44+
45+
- `packages/opencode/src/tool/ask.ts:125-133` publishes `questions: params.questions as Question.Item[]`.
46+
47+
This makes it critical that any “permissive input” is normalized back into canonical `Question.Item[]` before publishing.
48+
49+
### Tool schema sent to models
50+
51+
The schema passed to models is generated in-session:
52+
53+
- `packages/opencode/src/session/prompt.ts:68-75` uses `z.toJSONSchema(item.parameters)` and `ProviderTransform.schema(...)`.
54+
55+
Note: the server endpoint that lists tools uses a different converter:
56+
57+
- `packages/opencode/src/server/server.ts:491-500` uses `zodToJsonSchema(...)`.
58+
59+
Do not assume the server endpoint schema exactly matches what the model receives at runtime.
60+
61+
### Existing Ask tool operational context (must be considered)
62+
63+
`ASK_TOOL_UPDATES.md` documents prior Ask integration and UX work and is relevant context for maintaining compatibility:
64+
65+
- `ASK_TOOL_UPDATES.md:5-60`
66+
67+
## Problem Summary
68+
69+
### Observed error
70+
71+
When a model calls `ask`, the TUI modal does not appear; instead the user sees a Zod error like:
72+
73+
- missing required `questions[n].message`
74+
- `questions[n].options[m]` is a string, but schema expects an object
75+
76+
### Key insight
77+
78+
This is not primarily a TUI rendering issue. It’s a tool input validation mismatch that prevents the question request event from ever being published.
79+
80+
## Failure Mode Taxonomy (So We Fix The Right Thing)
81+
82+
### A) Invalid tool name / invalid JSON tool call
83+
84+
Handled in the model layer via tool call repair fallback:
85+
86+
- `packages/opencode/src/session/llm.ts:120-139` (`experimental_repairToolCall` routes unrepaired calls to the `invalid` tool).
87+
88+
### B) Valid JSON tool call, but schema-invalid args (this bug)
89+
90+
- Validation happens inside `Tool.define` before tool execute.
91+
- Current plugin hooks like `tool.execute.before` won’t run if parsing fails.
92+
93+
### C) Schema-valid args, but non-canonical shape breaks TUI
94+
95+
This can happen if we loosen the Zod schema but fail to normalize. The TUI will then break because it assumes canonical `Question.Item`.
96+
97+
## Design Principles
98+
99+
1. **Model-facing schema should remain simple and strict** to guide correct output.
100+
2. **Runtime parsing should be permissive and normalize** common deviations into canonical internal types.
101+
3. **Always publish only canonical `Question.Item[]`** to the TUI.
102+
4. **Be provider-agnostic by default**; avoid provider-specific hacks unless proven necessary.
103+
5. **Add observability at the actual failure point** (tool validation), with privacy safeguards.
104+
105+
## Proposed Approach (Preferred: Permissive Input + Canonical Normalization)
106+
107+
### Summary
108+
109+
Implement _permissive parsing via preprocessing_ (so models can still see a strict JSON schema), then _normalize_ into canonical `Question.Item[]` before publishing.
110+
111+
In practice, this means:
112+
113+
- Keep the model-visible JSON schema close to the current expected structure (objects for `options`, required `message`).
114+
- Add Zod `preprocess` / transformation layers so that when a model sends slightly wrong shapes (like string options), the tool still accepts them.
115+
- After preprocessing, validate against the canonical `Question.Item` schema and only then publish to TUI.
116+
117+
### Why this fits the codebase
118+
119+
- Prevents schema-validation failures from blocking the modal (`packages/opencode/src/tool/tool.ts:48-59`).
120+
- Preserves canonical invariants expected by the dialog UI (`packages/opencode/src/cli/cmd/tui/component/dialog-question/*`).
121+
- Avoids adding new complexity to provider schema transforms (`packages/opencode/src/provider/transform.ts:383-441`) unless needed.
122+
123+
## Normalization Specification
124+
125+
### Supported input variations (compatibility targets)
126+
127+
These are common model/tool-call deviations we will accept and normalize.
128+
129+
| Field | Canonical | Accept also | Normalization rule |
130+
| --------------------------- | ------------------------------------------- | -------------------------------------- | ------------------------------------------------------------------ |
131+
| `question.message` | `string` (required) | `text`, `prompt`, `question` (strings) | Map first present alias to `message` |
132+
| `question.type` | `select`, `multi-select`, `confirm`, `text` | `multiselect`, `multi_select`, `multi` | Map to `multi-select` if unambiguous |
133+
| `select.options[]` | `{ value, label, hint? }[]` | `string[]` | Convert `"A"` -> `{ value: "A", label: "A" }` |
134+
| `select.options[]` | `{ value, label }` | `{ label }` or `{ value }` | Fill missing field with the other (`value=label` or `label=value`) |
135+
| `multi-select.defaultValue` | `string[]` | `string` | Wrap into `[string]` |
136+
| `timeout` | `number` | `string` containing digits | Parse to number if safe |
137+
138+
### Canonical validation
139+
140+
After normalization, enforce canonical types by parsing with:
141+
142+
- `Question.Item` / `Question.Request` schemas (`packages/opencode/src/question/index.ts`).
143+
144+
If canonical validation fails, the tool must:
145+
146+
- Not publish the request
147+
- Return an error message designed to trigger a correct retry
148+
149+
## Tool Error Handling & Observability
150+
151+
### Validation errors must be actionable for models
152+
153+
Implement `formatValidationError` (supported by `Tool.define`) to provide:
154+
155+
- a brief explanation of what’s wrong
156+
- a minimal example tool call input
157+
- explicit guidance: “`options` must be objects with `value` and `label`”
158+
159+
Relevant hook point:
160+
161+
- `packages/opencode/src/tool/tool.ts:52-58`
162+
163+
### Capture invalid args at the correct point
164+
165+
Because validation fails before `tool.execute.before` runs (`packages/opencode/src/session/prompt.ts:75-87`), logging must be done inside tool validation failure handling.
166+
167+
**Privacy requirement:** never log full user text/options by default.
168+
169+
Proposed logging policy:
170+
171+
- log only:
172+
- tool name (`ask`)
173+
- providerID/modelID from `ctx.extra.model` (if present)
174+
- Zod issue paths and codes
175+
- counts/shape metadata (e.g., `questions.length`, option item types), not raw strings
176+
- optionally enable verbose logging only behind an explicit debug flag
177+
178+
## Implementation Plan (Sequenced by Dependencies)
179+
180+
### Milestone 1 — Reproduce, Measure, and Confirm Failure Class
181+
182+
- [ ] 1.1 Reproduce issue locally using the exact provider/model configuration reported in the issue (record `providerID` and `model.api.id` values from runtime logs)
183+
- [ ] 1.2 Confirm whether failure is case (B) schema-invalid args vs (A) invalid tool call
184+
- [ ] 1.3 Capture the tool-call args that failed validation (redacted) at `Tool.define` validation boundary (`packages/opencode/src/tool/tool.ts:48-59`)
185+
- [ ] 1.4 Confirm the exact tool schema sent to the model is coming from `packages/opencode/src/session/prompt.ts:68-75` (not `/experimental/tool`)
186+
187+
### Milestone 2 — Add Permissive Parsing Without Weakening Model Schema
188+
189+
- [x] 2.1 Update `packages/opencode/src/tool/ask.ts` to accept common input deviations using preprocessing (string options, alias keys for `message`, etc.)
190+
- [x] 2.2 Add explicit normalization functions that convert inputs into canonical `Question.Item[]`
191+
- [x] 2.3 Validate normalized output using `Question.Item` / `Question.Request` schemas (`packages/opencode/src/question/index.ts`)
192+
- [x] 2.4 Ensure `AskTool.execute` publishes only canonical `Question.Request` fields to `TuiEvent.QuestionRequest` (`packages/opencode/src/cli/cmd/tui/event.ts:40-42`)
193+
194+
### Milestone 3 — Improve Retry Behavior for Unfixable Inputs
195+
196+
Some invalid inputs cannot be safely inferred (e.g., truly missing question text).
197+
198+
- [x] 3.1 Implement `formatValidationError` for AskTool to produce a minimal, copy-pastable example payload
199+
- [x] 3.2 Include specific guidance for the observed failure (strings in `options`, missing `message`)
200+
- [ ] 3.3 Optionally add a narrow "ask tool repair" path in `packages/opencode/src/session/llm.ts:120-139`:
201+
- [ ] 3.3.1 If tool name is `ask` and error indicates string options, attempt to wrap them into `{value,label}` and retry once
202+
- [ ] 3.3.2 Never invent missing `message`; instead rely on `formatValidationError` to trigger a corrected retry
203+
204+
### Milestone 4 — Provider-Specific Handling (Only If Still Needed)
205+
206+
This is explicitly a fallback path.
207+
208+
- [ ] 4.1 If failures persist only for certain `providerID`/`model.api.id` pairs, document those identifiers
209+
- [ ] 4.2 Consider provider schema sanitization in `packages/opencode/src/provider/transform.ts:383-441` only if we can prove schema incompatibility is the root cause
210+
- [ ] 4.3 Add a regression test demonstrating the provider-specific need
211+
212+
### Milestone 5 — Tests, Validation, and Documentation
213+
214+
- [x] 5.1 Add unit tests covering normalization rules (string options, alias message keys, etc.)
215+
- [ ] 5.2 Add tests that assert published `Question.Request` always matches `Question.Request` schema
216+
- [ ] 5.3 Add/update a short internal note describing accepted "compatibility inputs" and why
217+
- [ ] 5.4 Manual QA checklist with at least:
218+
- [ ] select with object options
219+
- [ ] select with string options
220+
- [ ] multi-select with string defaultValue vs array
221+
- [ ] confirm/text question types
222+
- [ ] multi-question dialog
223+
224+
## Validation Criteria
225+
226+
### Functional acceptance
227+
228+
- [ ] Ask modal appears and is usable for valid canonical inputs
229+
- [ ] Ask modal appears for string-options inputs after normalization
230+
- [ ] If truly missing question text, the error message clearly explains required format and prompts model retry
231+
232+
### Compatibility
233+
234+
- [ ] Works across multiple providers/models without provider-specific logic
235+
- [ ] Does not break existing Ask dialog behavior (UX/components)
236+
237+
### Safety & privacy
238+
239+
- [ ] No sensitive question content is logged by default when validation fails
240+
241+
## Risk Assessment (Updated)
242+
243+
| Risk | Likelihood | Impact | Mitigation |
244+
| ----------------------------------------------------------- | ---------- | ------ | ---------------------------------------------------------------------------------- |
245+
| Loosening schema causes non-canonical payloads to reach TUI | Medium | High | Mandatory post-normalization validation against `Question.Item`/`Question.Request` |
246+
| Overly permissive parsing causes more “bad” model calls | Medium | Medium | Keep model-visible schema strict; normalize only known-safe deviations |
247+
| Logging leaks user content | Medium | High | Redaction + debug-only verbose logs |
248+
| Provider-specific hacks proliferate | Medium | Medium | Make provider-specific a gated fallback milestone |
249+
250+
## Code References
251+
252+
### Internal
253+
254+
- `packages/opencode/src/tool/ask.ts`
255+
- `packages/opencode/src/tool/tool.ts`
256+
- `packages/opencode/src/question/index.ts`
257+
- `packages/opencode/src/session/prompt.ts`
258+
- `packages/opencode/src/session/llm.ts`
259+
- `packages/opencode/src/provider/transform.ts`
260+
- `packages/opencode/src/cli/cmd/tui/event.ts`
261+
- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question.tsx`
262+
- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-select.tsx`
263+
- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-multi-select.tsx`
264+
- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-text.tsx`
265+
- `ASK_TOOL_UPDATES.md`
266+
267+
### External
268+
269+
- https://v4.zod.dev/json-schema
270+
- https://ai-sdk.dev/docs/reference/ai-sdk-core/zod-schema
271+
- https://github.com/vercel/ai/issues/9351 (thinking mode + structured/tool constraints context)
272+
273+
## Open Questions (Answer During Milestone 1)
274+
275+
- [ ] What are the exact runtime `providerID` and `model.api.id` values for the failing setup?
276+
- [ ] Does the failing model send `message` under an alias key (e.g., `text`/`prompt`), or is it truly missing?
277+
- [ ] Is the tool schema actually being delivered to the model as expected (verify via runtime introspection / logging)?

0 commit comments

Comments
 (0)