Repository navigation
Portable Claude profile: no silent fallback to an older model; StructuredOutput fields at the top level - #272
Merged
seathatflowsinourveins merged 8 commits intoSep 25, 2026
Conversation
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
seathatflowsinourveins
force-pushed
the
claude/model-fallback-guard-20260925
branch
5 times, most recently
from
September 25, 2026 19:19
cc4e925 to
aa0adfd
Compare
seathatflowsinourveins
enabled auto-merge (squash)
September 25, 2026 19:19
…edOutput fields The settings template now sets CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1 and switchModelsOnFlag false. When a request is flagged, it ends in a refusal instead of silently re-running on Opus 4.8, in the main thread and in every subagent. switchModelsOnFlag alone governs only the main thread in Claude Code 2.1.282. The example user CLAUDE.md asks workers to put StructuredOutput fields at the top level of the call arguments: in an interleaved A/B, schema errors fell from 5/30 to 0/30 (p = 0.026). The dated record in docs/decisions/2026-09-25-model-fallback-guard.md names the evidence, the alternatives and the overturn conditions. The fallback variable is undocumented, so that part stays keep-but-compare. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Scope: the ultracode recipe's settings file stays unguarded, and the omitClaudeMd agents (source-scout, blind-*) and built-in Explore do not read the user file. - Biology flags fall back to Opus 5 as well as cyber flags to Opus 4.8. - The client gating is stated precisely: WM() gates target selection in aHn(), and a silent-retry experiment lane retries only on the refusing model. - availableModels is recorded as documented and subagent-wide. It is rejected because excluding versions needs pins that also exclude the next Opus release. - Record template precedence, the hidden /config toggle and the opt-out. - Add a manual watch command. The catalog has no automated one yet. - Cite the host-local evidence archive and the result's version scope. - The example CLAUDE.md says what to do when a worker returns null after a safety flag. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Opt-out: set switchModelsOnFlag true and the variable "0". Removing the keys does not undo an earlier merge. The client parses the variable with M.bool. - switchModelsOnFlag cannot stop a subagent's fallback. Its reads in wpo(), apo() and Spo() are listed. - The silent-retry lane is armed by the server (model configuration convolute_arcades or header x-cc-tender-quilt), not by a client experiment. - The built-in agents that omit CLAUDE.md include Plan. - Workers returning null: check the transcript for a refusal before retrying. - Correct the population and cutoff for the source-scout figure. - Mark the client-wiring report as a proposal in PR #264. Rebased onto dcd6f0c. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- The client-wiring check is merged (#264) and does not check the variable yet. - Drop the count of switchModelsOnFlag reads and list them. - The built-in Explore and Plan bodies cannot be edited; the rule reaches them only through a same-name custom agent. - /config omits the toggle row while the variable is set (verified in the client's row builder). - Rebased onto 5abd17e. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…tput rule Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ructuredOutput rule Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
force-pushed
the
claude/model-fallback-guard-20260925
branch
from
September 25, 2026 20:04
aa0adfd to
9d77007
Compare
seathatflowsinourveins
deleted the
claude/model-fallback-guard-20260925
branch
September 25, 2026 20:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
docs/decisions/2026-09-25-model-fallback-guard.md.5abd17e4(origin/main, which includes Token-efficiency coverage check: adoption profile plus value-free --client-wiring for every host #264 and Hugging Face sign-in as a value-free native credential: status rows, guard, deny rules, per-host token policy #268)lane:foundationadoption/templates/claude.settings.template.json: envCLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1and"switchModelsOnFlag": false, in hunks separate from Hugging Face sign-in as a value-free native credential: status rows, guard, deny rules, per-host token policy #268's deny-rule hunksexamples/claude-native/CLAUDE.md: two sentences. One puts StructuredOutput fields at the top level. The other says what to do when a worker returns null: check the transcript for a safety refusal, then edit the brief and retry on the current model, and never accept an older model's answer.docs/decisions/2026-09-25-model-fallback-guard.md(new)manifests/evidence.json, re-registering the two changed hashes in the last commitEvidence-class table
switchModelsOnFlaggoverns only the main thread; subagents skip itsource_reviewUQt,WM); agent-labdocs/tasks/2026-09-25-quality-optimization.md§H1CLAUDE_CODE_DISABLE_REFUSAL_FALLBACKstops a flagged request re-running on an older model, in every thread (cyber → Opus 4.8 and bio → Opus 5 per the live docs)source_review(not probed; undocumented)WM()gatesaHn()target selection. The only remaining lane is a server-armed silent retry on the refusing model itself (Vpe/Pbr). Deliberately tripping a classifier is not an acceptable testswitchModelsOnFlagas "When off, your session will pause instead"source_reviewopusnative_proven(transcripts)child-usage.mjsbilled-model tablesnative_proven(A/B on the authoring laptop)wf_134a2bb7-819: 5/30 → 0/30, one-sided Fisher p = 0.026;wf_7d5f1165-75b: 0/30local_integrationapply_claude_settings.py --dry-runon a sample target: flag becomesfalse, variable1, existing env keptLocal commands run
Independent review
An
evidence-revieweragent (Opus 5.5 at max) reviewed six rounds. It checked each fix against the 2.1.282 binary, the live docs, the catalog and GitHub.CLAUDE.mdand how the rule can still reach them (agent body,SubagentStarthook, brief);source-scoutfigure;/configrow;source_review.Decision record
docs/decisions/2026-09-25-model-fallback-guard.md. It names the evidence, the alternatives (keep the switch; the flag alone; anavailableModelsallowlist; agent bodies only) and the overturn conditions (a documented subagent-wide control, a client that stops reading the variable, which agent-lab's release-watch probe reports with exit 3, or a same-version A/B showing no StructuredOutput gain). The fallback variable is undocumented, so that part stays keep-but-compare.Checklist
adoption/agents/claudeStructuredOutput line stays on the rollout's contract track.🤖 Generated with Claude Code