Skip to content

Portable Claude profile: no silent fallback to an older model; StructuredOutput fields at the top level - #272

Merged
seathatflowsinourveins merged 8 commits into
mainfrom
claude/model-fallback-guard-20260925
Sep 25, 2026
Merged

seathatflowsinourveins merged 8 commits into
mainfrom
claude/model-fallback-guard-20260925

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 25, 2026 •

Copy link
Copy Markdown
Collaborator

Scope

Evidence-class table

Claim Evidence class Command / receipt
In Claude Code 2.1.282, switchModelsOnFlag governs only the main thread; subagents skip it source_review Bundled client code (UQt, WM); agent-lab docs/tasks/2026-09-25-quality-optimization.md §H1
CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK stops a flagged request re-running on an older model, in every thread (cyber → Opus 4.8 and bio → Opus 5 per the live docs) source_review (not probed; undocumented) WM() gates aHn() target selection. The only remaining lane is a server-armed silent retry on the refusing model itself (Vpe/Pbr). Deliberately tripping a classifier is not an acceptable test
The client describes switchModelsOnFlag as "When off, your session will pause instead" source_review Settings-schema description in the installed 2.1.282 binary
The six 2026-09-24 incidents ran on Opus 4.8 after requesting opus native_proven (transcripts) agent-lab §H1; child-usage.mjs billed-model tables
The top-level StructuredOutput sentence cuts schema errors native_proven (A/B on the authoring laptop) wf_134a2bb7-819: 5/30 → 0/30, one-sided Fisher p = 0.026; wf_7d5f1165-75b: 0/30
The template merges as intended local_integration apply_claude_settings.py --dry-run on a sample target: flag becomes false, variable 1, existing env kept

Local commands run

$ python3 -m unittest tests.test_credential_status tests.test_install_claude_profile tests.test_render_config tests.test_apply_claude_settings tests.test_adjudicate tests.test_claude_lane tests.test_credential_tools tests.test_landscape tests.test_verdict_lane_vendoring tests.test_record_verdicts
Ran 486 tests ... OK
$ python3 scripts/validate.py
{"components": 69, "hashed_files": 5601, "profiles": 4, "receipts": 147, "status": "passed"}
$ python3 scripts/component_matrix.py --check; python3 scripts/new_host_grand_list.py --check
checked; passed
$ gitleaks git --log-opts="origin/main..HEAD" --redact .   (guarded ecosystem wrapper)
2 commits scanned; no leaks found

Independent review

An evidence-reviewer agent (Opus 5.5 at max) reviewed six rounds. It checked each fix against the 2.1.282 binary, the live docs, the catalog and GitHub.

  • Rounds 1–5: needs_changes. All findings are resolved. They included:
  • Round 6: agree. What stays unproven, by design: the refusal behaviour itself, which is source_review.

Decision record

docs/decisions/2026-09-25-model-fallback-guard.md. It names the evidence, the alternatives (keep the switch; the flag alone; an availableModels allowlist; agent bodies only) and the overturn conditions (a documented subagent-wide control, a client that stops reading the variable, which agent-lab's release-watch probe reports with exit 3, or a same-version A/B showing no StructuredOutput gain). The fallback variable is undocumented, so that part stays keep-but-compare.

Checklist

  • No workflows changed.
  • No secrets printed, logged or committed.
  • No paid hosting or billing surface.
  • Peer-owned untracked files and worktrees were preserved. The adoption/agents/claude StructuredOutput line stays on the rollout's contract track.

🤖 Generated with Claude Code

@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 25, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/model-fallback-guard-20260925 branch 5 times, most recently from cc4e925 to aa0adfd Compare September 25, 2026 19:19
@seathatflowsinourveins
seathatflowsinourveins enabled auto-merge (squash) September 25, 2026 19:19
…edOutput fields

The settings template now sets CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1 and
switchModelsOnFlag false. When a request is flagged, it ends in a refusal
instead of silently re-running on Opus 4.8, in the main thread and in
every subagent. switchModelsOnFlag alone governs only the main thread in
Claude Code 2.1.282. The example user CLAUDE.md asks workers to put
StructuredOutput fields at the top level of the call arguments: in an
interleaved A/B, schema errors fell from 5/30 to 0/30 (p = 0.026). The
dated record in docs/decisions/2026-09-25-model-fallback-guard.md names
the evidence, the alternatives and the overturn conditions. The fallback
variable is undocumented, so that part stays keep-but-compare.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Scope: the ultracode recipe's settings file stays unguarded, and the
  omitClaudeMd agents (source-scout, blind-*) and built-in Explore do not
  read the user file.
- Biology flags fall back to Opus 5 as well as cyber flags to Opus 4.8.
- The client gating is stated precisely: WM() gates target selection in
  aHn(), and a silent-retry experiment lane retries only on the refusing
  model.
- availableModels is recorded as documented and subagent-wide. It is
  rejected because excluding versions needs pins that also exclude the
  next Opus release.
- Record template precedence, the hidden /config toggle and the opt-out.
- Add a manual watch command. The catalog has no automated one yet.
- Cite the host-local evidence archive and the result's version scope.
- The example CLAUDE.md says what to do when a worker returns null
  after a safety flag.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- Opt-out: set switchModelsOnFlag true and the variable "0". Removing the
  keys does not undo an earlier merge. The client parses the variable
  with M.bool.
- switchModelsOnFlag cannot stop a subagent's fallback. Its reads in
  wpo(), apo() and Spo() are listed.
- The silent-retry lane is armed by the server (model configuration
  convolute_arcades or header x-cc-tender-quilt), not by a client
  experiment.
- The built-in agents that omit CLAUDE.md include Plan.
- Workers returning null: check the transcript for a refusal before
  retrying.
- Correct the population and cutoff for the source-scout figure.
- Mark the client-wiring report as a proposal in PR #264. Rebased
  onto dcd6f0c.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
- The client-wiring check is merged (#264) and does not check the
  variable yet.
- Drop the count of switchModelsOnFlag reads and list them.
- The built-in Explore and Plan bodies cannot be edited; the rule
  reaches them only through a same-name custom agent.
- /config omits the toggle row while the variable is set (verified in
  the client's row builder).
- Rebased onto 5abd17e.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…tput rule

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ructuredOutput rule

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins force-pushed the claude/model-fallback-guard-20260925 branch from aa0adfd to 9d77007 Compare September 25, 2026 20:04
@seathatflowsinourveins
seathatflowsinourveins merged commit f5812d3 into main Sep 25, 2026
23 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/model-fallback-guard-20260925 branch September 25, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant