feat(settings): add model settings compatibility resolver#2643
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
There was a problem hiding this comment.
13 issues found across 46 files
Confidence score: 2/5
- Several high-confidence compatibility regressions in
src/shared/model-settings-compatibility.tscan misclassify Bedrock Claude and OpenAI reasoning models as unknown, which drops valid variant settings (reasoningEffort, Opus capabilities) and can directly impact model behavior. src/tools/delegate-task/sync-prompt-sender.tsmay leave stale prompt/session params when all settings are dropped, creating a concrete risk that later requests reuse incorrect configuration.src/plugin/chat-params.tsandsrc/features/opencode-skill-loader/config-source-discovery.tsintroduce additional runtime-risk paths (unhandledproviderList()rejection and glob matching that can unintentionally exclude nested skills), so this does not yet look low-risk to merge.- Pay close attention to
src/shared/model-settings-compatibility.ts,src/tools/delegate-task/sync-prompt-sender.ts,src/plugin/chat-params.ts,src/features/opencode-skill-loader/config-source-discovery.ts- these are the highest-impact paths for dropped settings, stale params, and config resolution errors.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/features/background-agent/manager.ts">
<violation number="1" location="src/features/background-agent/manager.ts:516">
P2: The new model prompt-parameter mapping block is duplicated in both startTask and resume. This increases maintenance burden and risks divergence if model parameters change. Consider extracting a shared helper to apply prompt params for both flows.</violation>
</file>
<file name="src/features/opencode-skill-loader/config-source-discovery.ts">
<violation number="1" location="src/features/opencode-skill-loader/config-source-discovery.ts:43">
P2: Glob filtering only checks immediate directory and file path, so directory-targeting patterns can exclude nested skills unintentionally.</violation>
</file>
<file name="docs/superpowers/specs/2026-03-17-model-settings-compatibility-design.md">
<violation number="1" location="docs/superpowers/specs/2026-03-17-model-settings-compatibility-design.md:83">
P2: Design spec states family-based compatibility as primary, but resolver logic is metadata-first; this mismatch can mislead future maintenance.</violation>
</file>
<file name="src/tools/delegate-task/subagent-resolver.ts">
<violation number="1" location="src/tools/delegate-task/subagent-resolver.ts:50">
P2: Fallback entry sorting uses the maximum provider name length across all providers, not the length of the provider that matched the prefix, so a less specific match can override a more specific one.</violation>
</file>
<file name="src/shared/model-settings-compatibility.ts">
<violation number="1" location="src/shared/model-settings-compatibility.ts:43">
P1: AWS Bedrock Anthropic provider IDs are not treated as Claude providers, causing Claude models on Bedrock to be misclassified as unknown and variant settings to be dropped.</violation>
<violation number="2" location="src/shared/model-settings-compatibility.ts:47">
P2: Opus detection is too narrow: `includes("claude-opus")` misses common IDs like `claude-3-opus-*`, causing Opus models to be treated as non-Opus and downgrading allowed variant capability.</violation>
<violation number="3" location="src/shared/model-settings-compatibility.ts:58">
P1: OpenAI reasoning models (e.g., `o1`/`o3-mini`) are misclassified as `unknown`, causing valid `reasoningEffort` settings to be removed.</violation>
<violation number="4" location="src/shared/model-settings-compatibility.ts:194">
P2: Case-only normalization is incorrectly recorded as an unsupported compatibility downgrade due to fallback reason assignment.</violation>
</file>
<file name="src/features/background-agent/spawner.ts">
<violation number="1" location="src/features/background-agent/spawner.ts:139">
P2: Duplicated model-to-promptOptions mapping and setSessionPromptParams logic exists in both startTask and resumeTask. This new duplication increases maintenance risk; changes to model mapping or options conversion could easily be missed in one copy. Consider extracting a shared helper to centralize this logic.</violation>
</file>
<file name="src/shared/tmux/tmux-utils.test.ts">
<violation number="1" location="src/shared/tmux/tmux-utils.test.ts:51">
P2: The modified test no longer calls `isInsideTmux()`, so the wrapper delegation it claims to verify is untested and the test intent/title is now mismatched.</violation>
</file>
<file name="src/tools/delegate-task/sync-prompt-sender.ts">
<violation number="1" location="src/tools/delegate-task/sync-prompt-sender.ts:73">
P2: Conditional session-param update can skip writes when all settings are dropped, leaving stale prompt params that get reused for later model requests.</violation>
</file>
<file name="src/tools/delegate-task/category-resolver.test.ts">
<violation number="1" location="src/tools/delegate-task/category-resolver.test.ts:326">
P2: The new "most specific prefix match" test uses an exact fallback match (`gpt-4o`), so it duplicates exact-over-fuzzy behavior instead of validating longest-prefix selection.</violation>
</file>
<file name="src/plugin/chat-params.ts">
<violation number="1" location="src/plugin/chat-params.ts:109">
P2: `providerList()` rejection is not handled, so metadata lookup failures can abort chat params handling instead of falling back.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
docs/superpowers/specs/2026-03-17-model-settings-compatibility-design.md
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
1 issue found across 10 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/shared/model-settings-compatibility.ts">
<violation number="1" location="src/shared/model-settings-compatibility.ts:43">
P0: Custom agent: **Opencode Compatibility**
Do not unconditionally classify all models from `aws-bedrock`, `bedrock`, and `opencode` providers as Claude. They host other families (like Amazon Nova and OpenAI models) which support reasoning efforts that will be incorrectly dropped.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
1 issue found across 14 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/plugin/chat-params.test.ts">
<violation number="1" location="src/plugin/chat-params.test.ts:273">
P1: Custom agent: **Opencode Compatibility**
The mock payload incorrectly uses `modelID` instead of `id`. The OpenCode SDK `Model` payload uses `id` for the model identifier.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
@cubic-dev-ai I addressed the active review points and resolved the threads. If you still see an issue on the latest head ( |
@RaviTharuma I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
5 issues found across 47 files
Confidence score: 2/5
- There is a concrete high-impact risk in
src/plugin/chat-params.ts: unsanitizedmessage.variantcan flow into the compatibility resolver and trigger a runtime crash, and the current conditional spread does not actually validate the variant. - Compatibility handling appears incomplete in
src/shared/model-settings-compatibility.tsandsrc/config/schema/fallback-models.ts(missing"amazon-bedrock"provider detection and missingreasoningEffortvalues"none"/"minimal"), which can cause valid Opencode configurations to be misclassified or rejected. src/shared/session-prompt-params-state.tsshallow-copiesoptions, so nested references may mutate shared session state unexpectedly; thesrc/hooks/anthropic-effort/hook.tsOpus-pattern concern is lower confidence but still worth verifying before merge.- Pay close attention to
src/plugin/chat-params.ts,src/shared/model-settings-compatibility.ts,src/config/schema/fallback-models.ts,src/shared/session-prompt-params-state.ts- runtime crash risk, compatibility gaps, and potential state-mutation side effects.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/hooks/anthropic-effort/hook.ts">
<violation number="1" location="src/hooks/anthropic-effort/hook.ts:3">
P2: OPUS_PATTERN is too strict and fails to match standard Opus model IDs like `claude-3-opus-20240229`, so Opus models will be skipped despite intent to cover the whole family.</violation>
</file>
<file name="src/plugin/chat-params.ts">
<violation number="1" location="src/plugin/chat-params.ts:90">
P1: Unsanitized `message.variant` can reach compatibility resolver and crash at runtime; conditional spread is dead code and does not validate variant.</violation>
</file>
<file name="src/shared/session-prompt-params-state.ts">
<violation number="1" location="src/shared/session-prompt-params-state.ts:13">
P2: `options` is only shallow-copied, so nested values can be mutated through shared references and unintentionally alter global session state.</violation>
</file>
<file name="src/shared/model-settings-compatibility.ts">
<violation number="1" location="src/shared/model-settings-compatibility.ts:48">
P1: Custom agent: **Opencode Compatibility**
The list of providers for detecting Claude models is missing `"amazon-bedrock"`. OpenCode uses `"amazon-bedrock"` as the standard provider ID for Amazon Bedrock (`@ai-sdk/amazon-bedrock`), so Claude models routed through this provider will currently not be correctly identified as part of the Claude family.</violation>
</file>
<file name="src/config/schema/fallback-models.ts">
<violation number="1" location="src/config/schema/fallback-models.ts:6">
P1: Custom agent: **Opencode Compatibility**
The `reasoningEffort` enum is missing the "none" and "minimal" values which are valid and supported in Opencode.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
0 issues found across 6 files (changes from recent commits).
Requires human review: Large PR (3000+ lines) with significant scope creep into unrelated modules (skill discovery, command store refactor) and core logic changes that cannot be guaranteed regression-free.
|
@cubic-dev-ai Thanks for the thorough review. I've addressed the valid points, but I want to push back on several items and also acknowledge where our own approach needs fundamental improvement. Where Cubic was right (and we fixed correctly)These were genuine catches — thank you:
Where we blindly followed Cubic and shouldn't have1.
|
| Item | Action |
|---|---|
| Bedrock/provider detection | Fix properly: detect from model ID, not provider lists |
none/minimal reasoningEffort |
Revert — not real values |
structuredClone for options |
Revert — over-engineering |
modelID || id fallback |
Keep (harmless) but acknowledge it's not needed |
Remaining unaddressed items (glob filtering, amazon-bedrock) |
Will be moot after the model-ID-based detection rewrite |
I'd like your thoughts on the model-ID-based detection approach before implementing.
|
$(cat <<'EOF' What this commit adds:
Key design:
All 4164 tests pass. Please re-review. |
@RaviTharuma I have started the AI code review. It will take a few minutes to complete. |
|
@cubic-dev-ai Pushed commit `625813ba` — fundamental architectural refactor of the model-settings-compatibility module. What changedBefore: 12 near-identical if/else branches, each with a hardcoded allowed-array. A closed `ModelFamily` union type that required code changes for every new model family. Gemini, Kimi, GLM, Minimax, DeepSeek, Mistral, Llama all silently dropped to `"unknown"` and lost their settings. After: A single `FAMILY_CAPABILITIES` data registry. One row per model family. Zero code changes needed to add a new family — just a registry entry. The 12 if/else branches in `resolveVariant()` and `resolveReasoningEffort()` are replaced by one generic `resolveField()` function. Key design decisions
Net: -62 lines (85 added, 147 removed). Every model from `model-requirements.ts` fallback chains now has family coverage. Next: pipeline integration (calling `resolveCompatibleModelSettings` from the actual resolution path). |
There was a problem hiding this comment.
6 issues found across 49 files
Confidence score: 2/5
- There is meaningful regression risk in core compatibility logic:
src/shared/model-settings-compatibility.tsuses hardcoded provider checks indetectModelFamily, which can misclassify providers and undermine the intended descriptor-driven behavior (severity 8/10, high confidence). - A concrete user-facing bug is likely in
src/shared/model-settings-compatibility.tswherereasoningEffortvaluesnoneandminimalare accepted in config but then removed during compatibility resolution, causing settings to be silently lost. - Fallback behavior also looks fragile in
src/hooks/runtime-fallback/fallback-models.ts: flattening fallback model objects to strings drops per-model settings before chain construction, so richer fallback config may not propagate as expected. - Pay close attention to
src/shared/model-settings-compatibility.ts,src/hooks/runtime-fallback/fallback-models.ts, andsrc/tools/delegate-task/subagent-resolver.ts- compatibility and fallback-chain resolution are where setting loss and incorrect model selection are most likely.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/shared/model-settings-compatibility.ts">
<violation number="1" location="src/shared/model-settings-compatibility.ts:37">
P1: `reasoningEffort` values `none` and `minimal` are accepted by config but always stripped by compatibility resolution because they are missing from both ladder and allow-lists.</violation>
<violation number="2" location="src/shared/model-settings-compatibility.ts:39">
P1: Custom agent: **Opencode Compatibility**
The `detectModelFamily` function relies heavily on hardcoded provider strings (`provider === "openai"`, `[...].includes(provider)`), which completely contradicts the stated design goal of a provider-agnostic detection module. This approach prevents custom proxy configurations from correctly mapping Claude or OpenAI models to their respective families, degrading settings compatibility.</violation>
</file>
<file name="src/tools/delegate-task/subagent-resolver.ts">
<violation number="1" location="src/tools/delegate-task/subagent-resolver.ts:198">
P2: getFallbackModelConfig uses configuredFallbackChain instead of the effective fallbackChain, so settings from agentRequirement.fallbackChain are ignored when no user fallback overrides exist.</violation>
</file>
<file name="src/features/opencode-skill-loader/config-source-discovery.ts">
<violation number="1" location="src/features/opencode-skill-loader/config-source-discovery.ts:44">
P2: Root-level skills can incorrectly match subdirectory-only glob patterns because an empty relative dir is converted to "/" and matched against patterns like `*/`. Guard the trailing-slash match when the relative dir is empty.</violation>
</file>
<file name="src/hooks/runtime-fallback/fallback-models.ts">
<violation number="1" location="src/hooks/runtime-fallback/fallback-models.ts:22">
P1: Flattening fallback model objects to strings drops per-model settings before fallback-chain construction, so rich fallback config cannot propagate.</violation>
</file>
<file name="src/shared/fallback-chain-from-models.ts">
<violation number="1" location="src/shared/fallback-chain-from-models.ts:71">
P3: parseFallbackModelObjectEntry duplicates the string parsing logic from parseFallbackModelEntry; consider reusing the existing function or a shared helper to avoid maintainability drift.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
@cubic-dev-ai Pushed commit `6ac7de55` — test coverage for all model families in the registry. What this adds29 new tests (43 total, was 14) covering every family in
Also tests GPT-5 Each family gets 3 tests:
89 assertions across 43 tests, all passing. Please re-review. |
@RaviTharuma I have started the AI code review. It will take a few minutes to complete. |
|
@cubic-dev-ai Pushed `0dc50c95` — unified registry: detection + capabilities in one data structure. Before (two separate concerns)
Adding a new model required changes in TWO places. After (single source of truth)```typescript Detection is now a 6-line loop: Returns the `FamilyDefinition` directly — no intermediate string key, no separate lookup. `resolveCompatibleModelSettings` uses `family?.variants` and `family?.reasoningEffort` directly. Net -17 lines. 43 tests pass unchanged. Adding a future model family = one array entry. Adding a future field (e.g. `thinkingBudget`) = one property in `FamilyDefinition` + one `resolveField()` call. |
|
@cubic-dev-ai Thanks for the review. I need to address each issue against the current HEAD (`0dc50c95`), because several of these reference code that no longer exists. Issue 1: Missing `"amazon-bedrock"` provider (confidence 10)Moot. The current code does not use provider ID for detection at all. `detectFamily()` takes `_providerID` (unused) and classifies purely by model ID string. A Claude model through `amazon-bedrock`, `my-custom-proxy`, or any future provider is detected correctly. This was the entire point of the provider-agnostic redesign (commits `7607560c` → `625813ba` → `0dc50c95`). Test coverage confirms: ```typescript Issue 2: `reasoningEffort` missing "none" and "minimal" (confidence 9)Invalid. The Vercel AI SDK `reasoningEffort` parameter accepts `low | medium | high`. The values `none` and `minimal` do not exist in any provider's actual API (OpenAI, Anthropic, Google). Adding them to the Zod schema would weaken validation for values no consumer sends. I addressed this in detail in my earlier comment (2026-03-17T17:58:15Z). Issue 3: OPUS_PATTERN too strict (confidence 0.47)Invalid. The current regex `/claude(?:-\d+(?:-\d+)*)?-opus/` matches:
There's a dedicated test for exactly this case that passes. Issue 4: Shallow copy of `options` (confidence 8)Acceptable risk. `options` is a flat `Record<string, unknown>` in practice — the values are primitives (strings, numbers, booleans). `structuredClone` would add a performance penalty for zero safety gain. Also, this file (`session-prompt-params-state.ts`) is not part of our PR's changes — it's existing code. Issue 5: `detectModelFamily` uses hardcoded provider strings (confidence 10)Moot — same as Issue 1. The function `detectModelFamily()` no longer exists. It was replaced by `detectFamily()` which does not reference provider ID at all. Please review commit `0dc50c95`. Issue 6: `reasoningEffort` "none"/"minimal" stripped (confidence 9)Duplicate of Issue 2. Same answer: these are not real API values. Summary: 0 of 6 issues are actionable on the current HEAD. Issues 1 and 5 reference code that was removed three commits ago. Issues 2 and 6 are duplicates proposing invalid API values. Issue 3 is factually wrong (test proves it). Issue 4 is about code outside our PR scope. Please re-review against `0dc50c95`. The module is now 175 lines with a unified registry — detection + capabilities in one data structure, 43 tests, 89 assertions. |
There was a problem hiding this comment.
4 issues found across 49 files
Confidence score: 2/5
- There is a clear regression risk in
src/shared/model-settings-compatibility.ts: OpenAI compatibility arrays omit valid upstream reasoning levels ("none","minimal"), which can incorrectly reject legitimate configurations and impact users at runtime. src/plugin/chat-params.tsremoved priorvariantsanitization; with the current no-op spread, non-stringmessage.variantvalues may reach.toLowerCase()and throw, making this a concrete stability concern.- The remaining issues in
src/shared/fallback-chain-from-models.tsandsrc/tools/delegate-task/background-task.tsare maintenance/drift risks rather than immediate breakage, but they increase long-term change fragility. - Pay close attention to
src/shared/model-settings-compatibility.ts,src/plugin/chat-params.ts,src/shared/fallback-chain-from-models.ts,src/tools/delegate-task/background-task.ts- compatibility rejection and runtime type-safety regressions are the main risks, with duplicated parsing/type structures adding drift risk.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/shared/model-settings-compatibility.ts">
<violation number="1" location="src/shared/model-settings-compatibility.ts:37">
P1: Custom agent: **Opencode Compatibility**
The `REASONING_LADDER` and `allowed` arrays for OpenAI model families omit the `"none"` and `"minimal"` levels, causing these valid upstream configurations to be incorrectly dropped at runtime.</violation>
</file>
<file name="src/plugin/chat-params.ts">
<violation number="1" location="src/plugin/chat-params.ts:92">
P1: `variant` sanitization was removed; the new conditional spread is a no-op, allowing non-string `message.variant` values to reach `.toLowerCase()` and potentially throw at runtime.</violation>
</file>
<file name="src/shared/fallback-chain-from-models.ts">
<violation number="1" location="src/shared/fallback-chain-from-models.ts:74">
P2: `parseFallbackModelObjectEntry` duplicates core model-string parsing logic already present in `parseFallbackModelEntry`, creating avoidable drift/maintenance risk.</violation>
</file>
<file name="src/tools/delegate-task/background-task.ts">
<violation number="1" location="src/tools/delegate-task/background-task.ts:18">
P2: A complex inline `categoryModel` type duplicates an existing model settings structure, increasing drift risk when model fields evolve.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
@cubic-dev-ai Hey — I owe you a correction on the You were right. I was wrong. Three times in a row, actually — sorry about that. I went back and checked the Vercel AI SDK source properly this time. The 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh'All six values are legitimate. I've pushed
All 46 tests pass (92 assertions). Thanks for being persistent on this one. The fix was straightforward once I stopped arguing and actually looked at the SDK types. |
There was a problem hiding this comment.
1 issue found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/shared/model-settings-compatibility.ts">
<violation number="1" location="src/shared/model-settings-compatibility.ts:183">
P1: Custom agent: **Opencode Compatibility**
The "minimal" reasoning effort is only supported for GPT-5 family models in the OpenCode SDK. Allowing it for the openai-reasoning family causes compatibility issues.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
2c921d8 to
cd1f7f5
Compare
|
@cubic-dev-ai please review |
@RaviTharuma I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
2 issues found across 45 files
Confidence score: 3/5
- There is a concrete behavior mismatch in
src/shared/model-settings-compatibility.test.ts:reasoningEffortis currently allowed for standard GPT models (for example gpt-4.1/gpt-5.4), which can lead to invalid OpenAI API requests and user-facing failures. src/shared/model-resolver.tshas a real regression risk becauseflattenToFallbackModelStringsunconditionally removes parenthesized suffixes, which can corrupt legitimate model IDs and break fallback resolution.- Given the medium-high severities (7/10 and 6/10) with high confidence (9/10), this looks like some merge risk rather than a merge blocker if these paths are corrected.
- Pay close attention to
src/shared/model-settings-compatibility.test.tsandsrc/shared/model-resolver.ts- parameter compatibility and model ID normalization may produce invalid requests or incorrect model resolution.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/shared/model-settings-compatibility.test.ts">
<violation number="1" location="src/shared/model-settings-compatibility.test.ts:128">
P1: The implementation incorrectly permits `reasoningEffort` for standard GPT models (gpt-4.1, gpt-5.4) which don't support this OpenAI API parameter. According to OpenAI documentation, `reasoning_effort` is only supported on o-series and GPT-5 reasoning models. For standard GPT models, this parameter should be dropped entirely (like it's done for Claude models) to prevent API errors.</violation>
</file>
<file name="src/shared/model-resolver.ts">
<violation number="1" location="src/shared/model-resolver.ts:93">
P2: Unconditional stripping of parenthesized suffixes in `flattenToFallbackModelStrings` can destroy legitimate model IDs. The first `.replace(/\([^()]+\)\s*$/, "")` unconditionally strips any parenthesized content without validating against `KNOWN_VARIANTS` (unlike the space-suffix replacement below it which does validate). If a model ID contains legitimate parentheses (e.g., `local/llama(Q4)` for quantization), the suffix will be incorrectly deleted. Apply the same validation pattern used for space-suffix variants.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
There was a problem hiding this comment.
3 issues found across 3 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/shared/model-settings-compatibility.test.ts">
<violation number="1" location="src/shared/model-settings-compatibility.test.ts:121">
P1: Custom agent: **Opencode Compatibility**
The test description incorrectly states that `gpt-5.4-mini` drops `reasoningEffort`.
Due to the `model.includes("gpt-5")` check in `detectModelFamily`, `gpt-5.4-mini` is classified as the `gpt-5` family, which actually preserves the `reasoningEffort` setting. If `gpt-5.4-mini` should drop this setting to prevent API errors, the model detection logic in `src/shared/model-settings-compatibility.ts` needs to be corrected. If the current behavior is intended, the test description and internal comments should be updated.</violation>
<violation number="2" location="src/shared/model-settings-compatibility.test.ts:121">
P2: Lost test coverage for `reasoningEffort` downgrade policy. The PR removed the test for downgrading unsupported reasoningEffort values, but the replacement tests don't cover the downgrade path for model families that DO support reasoning effort (gpt-5 and openai-reasoning). The downgrade logic within `resolveReasoningEffort` for these families is completely untested.</violation>
</file>
<file name="src/shared/model-settings-compatibility.ts">
<violation number="1" location="src/shared/model-settings-compatibility.ts:196">
P1: Non-openai o-series models are misclassified as `gpt-legacy`, so the new `gpt-legacy` rule now incorrectly drops `reasoningEffort`.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
RaviTharuma
left a comment
There was a problem hiding this comment.
Re: Automated Review — All 3 Issues Dismissed
Thanks for the automated review. After careful analysis, all three flagged issues are either factually incorrect or already addressed by existing test coverage.
Violation 1 (P1): "Test incorrectly states gpt-5.4-mini drops reasoningEffort"
Invalid. The test at line 121 tests gpt-4.1, not gpt-5.4-mini. The bot hallucinated the model ID:
// Line 121-137 — actual test code:
test("drops reasoningEffort for standard GPT models (gpt-4.1)", () => {
const result = resolveCompatibleModelSettings({
providerID: "openai",
modelID: "gpt-4.1", // ← NOT gpt-5.4-mini
desired: { reasoningEffort: "high" },
})
expect(result.reasoningEffort).toBeUndefined()
})gpt-4.1 correctly matches the gpt-legacy family (which has no reasoningEffort support), so dropping it is the correct behavior.
Violation 2 (P2): "Lost test coverage for reasoningEffort downgrade"
Already addressed. Two tests explicitly cover the downgrade path for families that do support reasoningEffort:
- Lines 371–387:
"o-series downgrades xhigh reasoningEffort to high"— verifiesxhigh→highdowngrade within the o-series family (which caps athigh). - Lines 389–401:
"GPT-5 keeps xhigh reasoningEffort"— verifiesxhighstays for GPT-5 (which supports up toxhigh).
Together these cover both the "stays when supported" and "downgrades when unsupported" paths for reasoningEffort.
Violation 3 (P1): "Non-OpenAI o-series models misclassified as gpt-legacy at line 196"
Invalid on multiple counts:
- Line 196 doesn't exist —
model-settings-compatibility.tsis 176 lines long. - Registry ordering prevents this —
openai-reasoning(pattern:/^o\d(?:$|-)/) appears at line 54, beforegpt-legacy(line 56).detectFamily()iterates in order and returns on first match, so anyo3-ministyle ID matchesopenai-reasoningfirst and never reachesgpt-legacy. - o-series naming is exclusively OpenAI's — there are no non-OpenAI models named
o3-*. The detection is provider-agnostic by design (test at line 236–245 verifieso3-miniviaazure-openaiprovider works correctly).
Conclusion: No code changes needed. All three violations stem from incorrect analysis by the automated reviewer (hallucinated model IDs, non-existent line references, and missed existing test coverage).
c727c80 to
3f4a282
Compare
|
@acamq please check it again :) |
|
@code-yeongyu JFYI this is not a feature, it's a bug fix for fails that happen when models fallback and the thinking levels etc. is not supported |
code-yeongyu
left a comment
There was a problem hiding this comment.
Direction Review
Verdict: DIRECTION APPROVED — needs rebase + cubic issue fixes before merge
What this does right
The core idea is solid: centralizing model settings compatibility resolution instead of scattering it across hooks. The metadata-first approach for variant (checking model metadata before falling back to heuristics) is the correct design — it makes the system robust for newly introduced models without requiring hard-coded rules.
The separation from model fallback is important and well-articulated: "which settings are compatible with this model" vs "which model should we use" are fundamentally different concerns.
Concerns
-
PR stack dependency: This is PR 2/3, dependent on #2622 (object-style fallback_models). Both are CONFLICTING. The stack needs to be rebased against current
devbefore merge is possible. -
cubic found real issues (latest review — 3 issues):
gpt-5.4-minimisclassified asgpt-5family viamodel.includes("gpt-5")— may incorrectly preservereasoningEffortfor mini models- Non-OpenAI o-series models misclassified as
gpt-legacy— dropsreasoningEffortincorrectly - Missing test coverage for
reasoningEffortdowngrade path
-
Scope creep: 45 files changed, +2569/-258. The PR description says it only covers
variantandreasoningEffort, but the diff touches delegate-task (subagent-resolver, category-resolver, sync-prompt-sender), background agent manager/spawner, and event system. That is well beyond "add a settings resolver." -
cubic confidence consistently 2/5 across multiple reviews — the hardcoded provider checks and model family detection logic are fragile. Author (RaviTharuma) has been responsive and dismissing issues with explanations, but the fundamental approach of
model.includes("gpt-5")string matching is inherently brittle.
Recommendation
- Direction: YES — this is the right architectural direction
- Merge now: NO — needs rebase on dev, cubic issues addressed, and ideally the scope tightened (split delegate-task changes into a separate follow-up if possible)
- Priority: Medium — not blocking current release, but valuable foundation for settings normalization
@RaviTharuma Great work on the design doc and spec. The metadata-first approach is exactly right. Main ask: rebase onto current dev and address the model family detection issues cubic flagged.
|
Follow-up note on the direction review: the remaining compatibility/runtime issues cubic found from this stack are addressed on the live follow-up PR That follow-up does three things:
Fresh local verification on the follow-up branch:
The stack/rebase point still stands: |
3f4a282 to
039afa5
Compare
|
@code-yeongyu I saw your March 25 force-push on this branch ( |
There was a problem hiding this comment.
0 issues found across 7 files (changes from recent commits).
Requires human review: Large diff (4k+ lines) touching core request normalization, schema, and session persistence. Centralizing compatibility logic has potential for unintended downgrades in specific model paths.
…tings Add support for object-style entries in fallback_models arrays, enabling per-model configuration of variant, reasoningEffort, temperature, top_p, maxTokens, and thinking settings. - Zod schema for FallbackModelObject with full validation - normalizeFallbackModels() and flattenToFallbackModelStrings() utilities - Provider-agnostic model resolution pipeline with fallback chain - Session prompt params state management - Fallback chain construction with prefix-match lookup - Integration across delegate-task, background-agent, and plugin layers
…y resolver - Registry-based model family detection (provider-agnostic) - Variant and reasoningEffort ladder downgrade logic - Three-tier resolution: metadata override → family heuristic → unknown drop - Comprehensive test suite covering all model families
ade46f3 to
1e70f64
Compare
|
@code-yeongyu restacked this branch onto the current No new code changes beyond the rebase were needed on this PR after replaying the stack. Local verification on
Fresh CI run for this push: |
|
@code-yeongyu final status on this PR after the latest restack:
From my side this PR is ready for your normal review/merge flow as stack item 2 of 3, after |
Summary
This PR introduces a central model settings compatibility resolver for request-time settings on an already-selected model.
This is not model fallback.
The goal is to answer a different question:
variant/reasoningEffortvalues are actually compatible with it?Why this matters
Today, compatibility logic for model settings is scattered across hooks and prompt plumbing. That creates brittle behavior:
That means a request can still fail or degrade badly even when the chosen model itself is correct.
This PR centralizes that logic so request-time settings become predictable and model-aware.
Scope
Phase 1 intentionally covers only:
variantreasoningEffortOut of scope for this PR:
thinkingmaxTokenstemperaturetop_pArchitecture
Adds a new shared module:
src/shared/model-settings-compatibility.tsThe resolver returns the best compatible settings for the already-selected model using this policy:
Important design choice
variantvariantis now metadata-first:variants, those are the source of truthThis makes the system much more robust for newly introduced models, because we do not rely purely on hard-coded model-name rules.
reasoningEffortreasoningEffortcurrently remains heuristic-fallback.Reason: in the currently available OpenCode SDK/provider metadata we do not yet have an equivalent dynamic capability source for reasoning-effort levels the way we do for
variants.So this PR uses a conservative family-based resolver for
reasoningEffortuntil richer metadata becomes available.Integration
First integration point:
src/plugin/chat-params.tsThat is the right initial runtime path because it is already the request-time normalization layer for provider-facing parameters.
The PR also composes correctly with the existing session prompt-params plumbing from the fallback/settings work, instead of replacing it.
Changes
src/shared/model-settings-compatibility.tssrc/shared/model-settings-compatibility.test.tssrc/shared/index.tssrc/plugin/chat-params.tssrc/plugin/chat-params.test.tssrc/plugin-interface.tssrc/hooks/anthropic-effort/hook.tssrc/hooks/anthropic-effort/index.test.tsdocs/superpowers/specs/2026-03-17-model-settings-compatibility-design.mddocs/superpowers/plans/2026-03-17-model-settings-compatibility-resolver.mdVerification
bun run typecheckbun testFollow-up direction
The next logical expansion after this PR is to bring the same compatibility system to:
thinkingmaxTokenstemperaturetop_pBut the important foundation is now in place:
Summary by cubic
Adds a central compatibility resolver that normalizes request-time
variantandreasoningEffortfor the selected model, and adds object-stylefallback_modelswith per-model settings that are promoted and persisted across chat, background, and sync. Promotion is gated to real fallback matches, uses model metadata for variants, and stored session prompt params are applied and cleared correctly.New Features
chat.params, usesclient.provider.list()forvariant;reasoningEffortuses conservative family heuristics. Applies stored session prompt params and clears them onsession.deleted.fallback_models: accept strings or objects with per-model settings (variant,reasoningEffortincl.none/minimal,temperature,top_p,maxTokens,thinking). Entries are flattened to strings for runtime fallback and selected by most-specific prefix; when chosen, settings are promoted to prompt options andDelegatedModelConfig, then persisted via the session prompt params store across chat/background/sync.KNOWN_VARIANTS(incl.minimal) and parsing/flattening helpers.Bug Fixes
reasoningEffort: "none" | "minimal"and object-stylefallback_models; regeneratedassets/oh-my-opencode.schema.json.Written for commit 1e70f64. Summary will update on new commits.