|
| 1 | +# Plan: Fix Ask Tool Schema Validation Error (Broad/Permissive) |
| 2 | + |
| 3 | +**Issue:** [#136 - [BUG] Ask Tool](https://github.com/Latitudes-Dev/shuvcode/issues/136) |
| 4 | +**Created:** 2025-12-17 |
| 5 | +**Status:** Implementation Complete (Milestones 2, 3.1-3.2, 5.1 done) |
| 6 | + |
| 7 | +## Goals & Scope |
| 8 | + |
| 9 | +### Primary Goal |
| 10 | + |
| 11 | +Make the `ask` tool work reliably across models/providers by preventing schema-validation failures from blocking the TUI modal. |
| 12 | + |
| 13 | +### Specifically, we must fix |
| 14 | + |
| 15 | +- `ask` tool calls where select/multi-select `options` are provided as strings instead of `{ value, label }` objects. |
| 16 | +- `ask` tool calls where `message` (required question text) is missing but may exist under common alias keys. |
| 17 | + |
| 18 | +### Non-goals |
| 19 | + |
| 20 | +- Do not redesign the Ask dialog UX (already addressed elsewhere). |
| 21 | +- Do not change user-facing TUI behavior beyond making `ask` reliably open. |
| 22 | +- Do not introduce provider-specific logic unless broad, provider-agnostic fixes are insufficient. |
| 23 | + |
| 24 | +## Planning Context Capture (Repo Reality) |
| 25 | + |
| 26 | +### Where the failure occurs |
| 27 | + |
| 28 | +Tool argument validation happens before the tool’s `execute` body: |
| 29 | + |
| 30 | +- `packages/opencode/src/tool/tool.ts:48-59` validates `toolInfo.parameters.parse(args)`. |
| 31 | + |
| 32 | +If this validation fails, `AskTool.execute()` never runs, and the modal never appears. |
| 33 | + |
| 34 | +### Canonical “question” contract is strict and used by the TUI |
| 35 | + |
| 36 | +The TUI expects canonical `Question.Item` shapes: |
| 37 | + |
| 38 | +- `packages/opencode/src/question/index.ts:51-52` defines `Question.Item` as a `z.discriminatedUnion("type", ...)`. |
| 39 | +- The dialog components assume `select`/`multi-select` options are objects with `value` and `label`: |
| 40 | + - `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-select.tsx:126-142` |
| 41 | + - `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-multi-select.tsx:165-190` |
| 42 | + |
| 43 | +### AskTool currently publishes to the TUI with a cast (no secondary validation) |
| 44 | + |
| 45 | +- `packages/opencode/src/tool/ask.ts:125-133` publishes `questions: params.questions as Question.Item[]`. |
| 46 | + |
| 47 | +This makes it critical that any “permissive input” is normalized back into canonical `Question.Item[]` before publishing. |
| 48 | + |
| 49 | +### Tool schema sent to models |
| 50 | + |
| 51 | +The schema passed to models is generated in-session: |
| 52 | + |
| 53 | +- `packages/opencode/src/session/prompt.ts:68-75` uses `z.toJSONSchema(item.parameters)` and `ProviderTransform.schema(...)`. |
| 54 | + |
| 55 | +Note: the server endpoint that lists tools uses a different converter: |
| 56 | + |
| 57 | +- `packages/opencode/src/server/server.ts:491-500` uses `zodToJsonSchema(...)`. |
| 58 | + |
| 59 | +Do not assume the server endpoint schema exactly matches what the model receives at runtime. |
| 60 | + |
| 61 | +### Existing Ask tool operational context (must be considered) |
| 62 | + |
| 63 | +`ASK_TOOL_UPDATES.md` documents prior Ask integration and UX work and is relevant context for maintaining compatibility: |
| 64 | + |
| 65 | +- `ASK_TOOL_UPDATES.md:5-60` |
| 66 | + |
| 67 | +## Problem Summary |
| 68 | + |
| 69 | +### Observed error |
| 70 | + |
| 71 | +When a model calls `ask`, the TUI modal does not appear; instead the user sees a Zod error like: |
| 72 | + |
| 73 | +- missing required `questions[n].message` |
| 74 | +- `questions[n].options[m]` is a string, but schema expects an object |
| 75 | + |
| 76 | +### Key insight |
| 77 | + |
| 78 | +This is not primarily a TUI rendering issue. It’s a tool input validation mismatch that prevents the question request event from ever being published. |
| 79 | + |
| 80 | +## Failure Mode Taxonomy (So We Fix The Right Thing) |
| 81 | + |
| 82 | +### A) Invalid tool name / invalid JSON tool call |
| 83 | + |
| 84 | +Handled in the model layer via tool call repair fallback: |
| 85 | + |
| 86 | +- `packages/opencode/src/session/llm.ts:120-139` (`experimental_repairToolCall` routes unrepaired calls to the `invalid` tool). |
| 87 | + |
| 88 | +### B) Valid JSON tool call, but schema-invalid args (this bug) |
| 89 | + |
| 90 | +- Validation happens inside `Tool.define` before tool execute. |
| 91 | +- Current plugin hooks like `tool.execute.before` won’t run if parsing fails. |
| 92 | + |
| 93 | +### C) Schema-valid args, but non-canonical shape breaks TUI |
| 94 | + |
| 95 | +This can happen if we loosen the Zod schema but fail to normalize. The TUI will then break because it assumes canonical `Question.Item`. |
| 96 | + |
| 97 | +## Design Principles |
| 98 | + |
| 99 | +1. **Model-facing schema should remain simple and strict** to guide correct output. |
| 100 | +2. **Runtime parsing should be permissive and normalize** common deviations into canonical internal types. |
| 101 | +3. **Always publish only canonical `Question.Item[]`** to the TUI. |
| 102 | +4. **Be provider-agnostic by default**; avoid provider-specific hacks unless proven necessary. |
| 103 | +5. **Add observability at the actual failure point** (tool validation), with privacy safeguards. |
| 104 | + |
| 105 | +## Proposed Approach (Preferred: Permissive Input + Canonical Normalization) |
| 106 | + |
| 107 | +### Summary |
| 108 | + |
| 109 | +Implement _permissive parsing via preprocessing_ (so models can still see a strict JSON schema), then _normalize_ into canonical `Question.Item[]` before publishing. |
| 110 | + |
| 111 | +In practice, this means: |
| 112 | + |
| 113 | +- Keep the model-visible JSON schema close to the current expected structure (objects for `options`, required `message`). |
| 114 | +- Add Zod `preprocess` / transformation layers so that when a model sends slightly wrong shapes (like string options), the tool still accepts them. |
| 115 | +- After preprocessing, validate against the canonical `Question.Item` schema and only then publish to TUI. |
| 116 | + |
| 117 | +### Why this fits the codebase |
| 118 | + |
| 119 | +- Prevents schema-validation failures from blocking the modal (`packages/opencode/src/tool/tool.ts:48-59`). |
| 120 | +- Preserves canonical invariants expected by the dialog UI (`packages/opencode/src/cli/cmd/tui/component/dialog-question/*`). |
| 121 | +- Avoids adding new complexity to provider schema transforms (`packages/opencode/src/provider/transform.ts:383-441`) unless needed. |
| 122 | + |
| 123 | +## Normalization Specification |
| 124 | + |
| 125 | +### Supported input variations (compatibility targets) |
| 126 | + |
| 127 | +These are common model/tool-call deviations we will accept and normalize. |
| 128 | + |
| 129 | +| Field | Canonical | Accept also | Normalization rule | |
| 130 | +| --------------------------- | ------------------------------------------- | -------------------------------------- | ------------------------------------------------------------------ | |
| 131 | +| `question.message` | `string` (required) | `text`, `prompt`, `question` (strings) | Map first present alias to `message` | |
| 132 | +| `question.type` | `select`, `multi-select`, `confirm`, `text` | `multiselect`, `multi_select`, `multi` | Map to `multi-select` if unambiguous | |
| 133 | +| `select.options[]` | `{ value, label, hint? }[]` | `string[]` | Convert `"A"` -> `{ value: "A", label: "A" }` | |
| 134 | +| `select.options[]` | `{ value, label }` | `{ label }` or `{ value }` | Fill missing field with the other (`value=label` or `label=value`) | |
| 135 | +| `multi-select.defaultValue` | `string[]` | `string` | Wrap into `[string]` | |
| 136 | +| `timeout` | `number` | `string` containing digits | Parse to number if safe | |
| 137 | + |
| 138 | +### Canonical validation |
| 139 | + |
| 140 | +After normalization, enforce canonical types by parsing with: |
| 141 | + |
| 142 | +- `Question.Item` / `Question.Request` schemas (`packages/opencode/src/question/index.ts`). |
| 143 | + |
| 144 | +If canonical validation fails, the tool must: |
| 145 | + |
| 146 | +- Not publish the request |
| 147 | +- Return an error message designed to trigger a correct retry |
| 148 | + |
| 149 | +## Tool Error Handling & Observability |
| 150 | + |
| 151 | +### Validation errors must be actionable for models |
| 152 | + |
| 153 | +Implement `formatValidationError` (supported by `Tool.define`) to provide: |
| 154 | + |
| 155 | +- a brief explanation of what’s wrong |
| 156 | +- a minimal example tool call input |
| 157 | +- explicit guidance: “`options` must be objects with `value` and `label`” |
| 158 | + |
| 159 | +Relevant hook point: |
| 160 | + |
| 161 | +- `packages/opencode/src/tool/tool.ts:52-58` |
| 162 | + |
| 163 | +### Capture invalid args at the correct point |
| 164 | + |
| 165 | +Because validation fails before `tool.execute.before` runs (`packages/opencode/src/session/prompt.ts:75-87`), logging must be done inside tool validation failure handling. |
| 166 | + |
| 167 | +**Privacy requirement:** never log full user text/options by default. |
| 168 | + |
| 169 | +Proposed logging policy: |
| 170 | + |
| 171 | +- log only: |
| 172 | + - tool name (`ask`) |
| 173 | + - providerID/modelID from `ctx.extra.model` (if present) |
| 174 | + - Zod issue paths and codes |
| 175 | + - counts/shape metadata (e.g., `questions.length`, option item types), not raw strings |
| 176 | +- optionally enable verbose logging only behind an explicit debug flag |
| 177 | + |
| 178 | +## Implementation Plan (Sequenced by Dependencies) |
| 179 | + |
| 180 | +### Milestone 1 — Reproduce, Measure, and Confirm Failure Class |
| 181 | + |
| 182 | +- [ ] 1.1 Reproduce issue locally using the exact provider/model configuration reported in the issue (record `providerID` and `model.api.id` values from runtime logs) |
| 183 | +- [ ] 1.2 Confirm whether failure is case (B) schema-invalid args vs (A) invalid tool call |
| 184 | +- [ ] 1.3 Capture the tool-call args that failed validation (redacted) at `Tool.define` validation boundary (`packages/opencode/src/tool/tool.ts:48-59`) |
| 185 | +- [ ] 1.4 Confirm the exact tool schema sent to the model is coming from `packages/opencode/src/session/prompt.ts:68-75` (not `/experimental/tool`) |
| 186 | + |
| 187 | +### Milestone 2 — Add Permissive Parsing Without Weakening Model Schema |
| 188 | + |
| 189 | +- [x] 2.1 Update `packages/opencode/src/tool/ask.ts` to accept common input deviations using preprocessing (string options, alias keys for `message`, etc.) |
| 190 | +- [x] 2.2 Add explicit normalization functions that convert inputs into canonical `Question.Item[]` |
| 191 | +- [x] 2.3 Validate normalized output using `Question.Item` / `Question.Request` schemas (`packages/opencode/src/question/index.ts`) |
| 192 | +- [x] 2.4 Ensure `AskTool.execute` publishes only canonical `Question.Request` fields to `TuiEvent.QuestionRequest` (`packages/opencode/src/cli/cmd/tui/event.ts:40-42`) |
| 193 | + |
| 194 | +### Milestone 3 — Improve Retry Behavior for Unfixable Inputs |
| 195 | + |
| 196 | +Some invalid inputs cannot be safely inferred (e.g., truly missing question text). |
| 197 | + |
| 198 | +- [x] 3.1 Implement `formatValidationError` for AskTool to produce a minimal, copy-pastable example payload |
| 199 | +- [x] 3.2 Include specific guidance for the observed failure (strings in `options`, missing `message`) |
| 200 | +- [ ] 3.3 Optionally add a narrow "ask tool repair" path in `packages/opencode/src/session/llm.ts:120-139`: |
| 201 | + - [ ] 3.3.1 If tool name is `ask` and error indicates string options, attempt to wrap them into `{value,label}` and retry once |
| 202 | + - [ ] 3.3.2 Never invent missing `message`; instead rely on `formatValidationError` to trigger a corrected retry |
| 203 | + |
| 204 | +### Milestone 4 — Provider-Specific Handling (Only If Still Needed) |
| 205 | + |
| 206 | +This is explicitly a fallback path. |
| 207 | + |
| 208 | +- [ ] 4.1 If failures persist only for certain `providerID`/`model.api.id` pairs, document those identifiers |
| 209 | +- [ ] 4.2 Consider provider schema sanitization in `packages/opencode/src/provider/transform.ts:383-441` only if we can prove schema incompatibility is the root cause |
| 210 | +- [ ] 4.3 Add a regression test demonstrating the provider-specific need |
| 211 | + |
| 212 | +### Milestone 5 — Tests, Validation, and Documentation |
| 213 | + |
| 214 | +- [x] 5.1 Add unit tests covering normalization rules (string options, alias message keys, etc.) |
| 215 | +- [ ] 5.2 Add tests that assert published `Question.Request` always matches `Question.Request` schema |
| 216 | +- [ ] 5.3 Add/update a short internal note describing accepted "compatibility inputs" and why |
| 217 | +- [ ] 5.4 Manual QA checklist with at least: |
| 218 | + - [ ] select with object options |
| 219 | + - [ ] select with string options |
| 220 | + - [ ] multi-select with string defaultValue vs array |
| 221 | + - [ ] confirm/text question types |
| 222 | + - [ ] multi-question dialog |
| 223 | + |
| 224 | +## Validation Criteria |
| 225 | + |
| 226 | +### Functional acceptance |
| 227 | + |
| 228 | +- [ ] Ask modal appears and is usable for valid canonical inputs |
| 229 | +- [ ] Ask modal appears for string-options inputs after normalization |
| 230 | +- [ ] If truly missing question text, the error message clearly explains required format and prompts model retry |
| 231 | + |
| 232 | +### Compatibility |
| 233 | + |
| 234 | +- [ ] Works across multiple providers/models without provider-specific logic |
| 235 | +- [ ] Does not break existing Ask dialog behavior (UX/components) |
| 236 | + |
| 237 | +### Safety & privacy |
| 238 | + |
| 239 | +- [ ] No sensitive question content is logged by default when validation fails |
| 240 | + |
| 241 | +## Risk Assessment (Updated) |
| 242 | + |
| 243 | +| Risk | Likelihood | Impact | Mitigation | |
| 244 | +| ----------------------------------------------------------- | ---------- | ------ | ---------------------------------------------------------------------------------- | |
| 245 | +| Loosening schema causes non-canonical payloads to reach TUI | Medium | High | Mandatory post-normalization validation against `Question.Item`/`Question.Request` | |
| 246 | +| Overly permissive parsing causes more “bad” model calls | Medium | Medium | Keep model-visible schema strict; normalize only known-safe deviations | |
| 247 | +| Logging leaks user content | Medium | High | Redaction + debug-only verbose logs | |
| 248 | +| Provider-specific hacks proliferate | Medium | Medium | Make provider-specific a gated fallback milestone | |
| 249 | + |
| 250 | +## Code References |
| 251 | + |
| 252 | +### Internal |
| 253 | + |
| 254 | +- `packages/opencode/src/tool/ask.ts` |
| 255 | +- `packages/opencode/src/tool/tool.ts` |
| 256 | +- `packages/opencode/src/question/index.ts` |
| 257 | +- `packages/opencode/src/session/prompt.ts` |
| 258 | +- `packages/opencode/src/session/llm.ts` |
| 259 | +- `packages/opencode/src/provider/transform.ts` |
| 260 | +- `packages/opencode/src/cli/cmd/tui/event.ts` |
| 261 | +- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question.tsx` |
| 262 | +- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-select.tsx` |
| 263 | +- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-multi-select.tsx` |
| 264 | +- `packages/opencode/src/cli/cmd/tui/component/dialog-question/dialog-question-text.tsx` |
| 265 | +- `ASK_TOOL_UPDATES.md` |
| 266 | + |
| 267 | +### External |
| 268 | + |
| 269 | +- https://v4.zod.dev/json-schema |
| 270 | +- https://ai-sdk.dev/docs/reference/ai-sdk-core/zod-schema |
| 271 | +- https://github.com/vercel/ai/issues/9351 (thinking mode + structured/tool constraints context) |
| 272 | + |
| 273 | +## Open Questions (Answer During Milestone 1) |
| 274 | + |
| 275 | +- [ ] What are the exact runtime `providerID` and `model.api.id` values for the failing setup? |
| 276 | +- [ ] Does the failing model send `message` under an alias key (e.g., `text`/`prompt`), or is it truly missing? |
| 277 | +- [ ] Is the tool schema actually being delivered to the model as expected (verify via runtime introspection / logging)? |
0 commit comments