Skip to content

Model currency 2026-09-28: supersede #434 with the Sonnet 5.5 launch addendum (fallback guards, template advisor, catalog label) - #470

Merged
seathatflowsinourveins merged 11 commits into
mainfrom
claude/model-currency-20260928
Sep 29, 2026
Merged

seathatflowsinourveins merged 11 commits into
mainfrom
claude/model-currency-20260928

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Scope

  • What this PR changes, in one or two sentences: supersedes Model currency 2026-09-27: per-lane decision record, model-fallback guards, advisor template #434 (the 2026-09-27 model-currency record, orphaned and conflicting only in manifests/evidence.json). It carries Model currency 2026-09-27: per-lane decision record, model-fallback guards, advisor template #434's seven reviewed non-hot-file commits rebased onto main and adds a 2026-09-28 addendum: Claude Sonnet 5.5's launch, the measured sonnet alias resolution on Claude Code 2.1.284, the content-based fallback map with the Vals AI fallback counts, and one repair round after a cross-family review. The hot-file registration is re-derived in the last commit.
  • Base commit: f0219263 (origin/main at the last rebase)
  • Lane: lane:foundation. No shared hot file changes beyond hash registration.
  • Owned paths touched: .claude/settings.json, adoption/templates/claude.settings.template.json, catalogs/foundation/decisions.json, docs/decisions/2026-09-25-workstation-sota-refresh.md, docs/decisions/2026-09-27-model-currency.md, examples/claude-native/ultracode.settings.json, recipes/claude-native-ultracode.md, tests/test_foundation_catalog.py, tests/test_install_claude_profile.py, and manifests/evidence.json (hash registration only, last commit).

SOTA sources

Evidence-class table

Claim Evidence class Command / receipt
sonnet alias ran claude-sonnet-5-5 at effort max on this host unretained host observation of one stack-researcher child (82 of 82 messages; its transcript stays on the host); every other sonnet stage is expected from the alias table, not observed scoped in the addendum; no receipt claimed
Sonnet 5.5 launch, alias move, lifecycle status, fallback routes source_review the vendor pages and CHANGELOG cited above
Vals fallback counts (50.51% and 53.54% with fallback-assisted attempts counted as failures) source_review the two Vals model pages, read 2026-09-28
The two fallback guards stop content-based fallback source_review of the 2.1.283 client; 2.1.284 checked by the watch command's count (1) and the gate's code shape only grep -a -c '!.\{1,4\}\.CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK&&' on the real 2.1.284 binary; the Sonnet route in 2.1.284 is unverified, as the record says
Tests added by #434 and the tightened advisor test local_integration python3 -m unittest tests.test_install_claude_profile tests.test_foundation_catalog tests.test_adoption_docs_consistency

Local commands run

$ python3 scripts/validate.py
exit 0  {"components": 69, "hashed_files": 7985, "profiles": 4, "receipts": 165, "status": "passed"}
$ python3 -m unittest tests.test_install_claude_profile tests.test_foundation_catalog tests.test_adoption_docs_consistency
exit 0  Ran 117 tests, OK (skipped=4)
$ python3 -m unittest        # this branch and a clean origin/main worktree, same host
both: FAILED (failures=172, errors=148); identical set of 302 failing test ids, none only on the branch

The full-suite failures are pre-existing on this workstation (a worktree under /tmp, host tooling); CI's validate job is the authority.

Decision record

docs/decisions/2026-09-27-model-currency.md, section "Addendum 2026-09-28". Overturn conditions are in its "Overturn" section. Reviews: #434's GPT-6 review round is recorded in its thread. This PR's addendum had one read-only GPT-6 Astra review at max effort (six findings, all supported, all repaired), one re-check (five fixed, one partial, two new text findings), and those last text findings were applied without a third review; nothing else is left open from the reviews.

For the Gate A owner

A sealed #381 run started in this repository inherits the two project guards (switchModelsOnFlag: false and CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1). The preregistration does not mention the settings files (grep of evidence/artifacts/token-adoption-e2e-20260926/, 0 hits), and Gate A has no live owner today, so whoever seals the run should decide whether the frozen environment includes them.

Host evidence

Not applicable: no files under evidence/hosts/ change.

Checklist

  • No new or changed GitHub Actions.
  • No new or changed workflows.
  • No secrets are printed, logged or committed; no new required secret was added.
  • No new paid hosting, subscription or billing surface was introduced.
  • Peer-owned untracked files and worktrees were preserved.

🤖 Generated with Claude Code

Scout and others added 11 commits September 28, 2026 19:33
…he routes to an older model

Record the 2026-09-27 model-currency check the user asked for ("only the latest
SOTA models, even the latest ones released in the past weeks"): three Opus
research units (Claude, OpenAI/Codex with OmniRoute, open-weight workstation
models), one adversarial refuter each and one synthesis.

- Decision table per lane and role: current, latest available with release
  date, action. No row switches; four hold behind upstream gates:
  jina-reranker-v3.5 for qmd (stock llama.cpp rank-pooling, #26286 open),
  Nemotron-3-Embed-1B for ai-memory (2.5.0 with #859), Sonnet 5.5 and Haiku
  5.5 (announced, no model ID), and Opus 5.5 on OmniRoute (the gateway lists
  no claude-opus-5-5, so Opus-role stages stay on native Claude Code).
- The committed-settings fallback guards, the template advisor and the
  isolated-builder label are recorded with their evidence classes.
- llama.cpp: v0.5.0 is b11146 (commit 7fe450e1); the 2026-09-25 re-pin gate
  (paired acceptance against a fresh b11057 baseline) has not run, so the
  b11057 pin stays and the divergence and the qualification are recorded.

Sources (re-fetched 2026-09-27): platform.claude.com release notes, models
overview and model deprecations; anthropic.com/claude-opus-5-5;
code.claude.com settings-reference, model-config, advisor, env-vars and
settings; anthropics/claude-code CHANGELOG at 7779afb1 (tag v2.1.283);
developers.openai.com API changelog; registry.npmjs.org
@anthropic-ai/claude-code (latest 2.1.283); gh api for ggml-org/llama.cpp
tags b11057, b11146, v0.5.0 and releases/latest, akitaonrails/ai-memory
releases and #859, vllm-project/vllm and tobi/qmd releases; a read-only
GET /v1/models on the loopback gateway (582 IDs, 55 Claude, no opus-5-5).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
… settings

.claude/settings.json and examples/claude-native/ultracode.settings.json now
carry the template's two guards in the template's form: "switchModelsOnFlag":
false and env CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK="1". The recipe's embedded
copy of the portable file matches it. Before this, a session that loaded only
this repository's files, or an adopter running claude --settings with the
portable file, could re-run a flagged Opus 5.5 or Fable request on Opus 4.8
(cybersecurity) or Opus 5 (biology). This closes the "ultracode recipe alone"
gap of docs/decisions/2026-09-25-model-fallback-guard.md.

Sources (fetched 2026-09-27):
- code.claude.com/docs/en/settings-reference#switchmodelsonflag: "Scope: Any
  file"; false pauses an interactive session and ends a -p request as an
  error. The env entry: "Scope: Any file"; project values apply after
  workspace trust or at -p startup; the documented project ignore list does
  not name either key.
- code.claude.com/docs/en/model-config#automatic-model-fallback: the targets.
- CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK is not found in env-vars, model-config,
  settings-reference or the CHANGELOG at anthropics/claude-code 7779afb1
  (tag v2.1.283). Source review of the installed 2.1.283 client: jD() is
  !a.CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK&&!Ote(); the fallback handler runs
  only on jD() or a server-armed same-model silent retry (IAr returns the
  refusing model); orn() returns "subagent" before it reads
  switchModelsOnFlag; a's getters (Rhn) read process.env live; the project
  ignore set Qzn does not contain the variable. Not probed.

Evidence: CommittedSettingsFallbackGuardTests failed at ec8a489 for both
files (None != '1'); after the settings edit the recipe-copy test failed
until the recipe matched; both pass now. node test-envelope.mjs 254/254 and
node test-contract-mutations.mjs 74/74 before and after.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…assigned advisor

The user's rule pairs the Opus 5.5 main model with Fable 5.1 as the native
advisor for truly complex calls, and this host carries advisorModel "fable"
in its user settings, but the adoption template had no advisorModel key, so a
new host that applied it got no advisor.

Sources (fetched 2026-09-27):
- code.claude.com/docs/en/settings-reference#advisormodel: "Scope: Any file";
  "fable", "opus", "sonnet" or a full model ID; unset leaves the advisor off;
  /advisor saves the pick to ~/.claude/settings.json, the file the template
  renders into.
- code.claude.com/docs/en/advisor, "Choose an advisor model": an Opus 5.5 main
  model accepts "Fable, and Opus 5 or later" (the template's model is
  opus[1m]); Sonnet 5 accepts Fable; subagents inherit the advisor under the
  same pairing check (documented, not observed). Without Fable access or
  the usage-credits consent, requests go without the advisor and an
  interactive session shows a notification pointing to /model fable.
- code.claude.com/docs/en/env-vars, "Features that need feature-flag
  fetching": the template sets none of the variables that turn fetching,
  and so the advisor, off, nor CLAUDE_CODE_DISABLE_ADVISOR_TOOL.

apply_claude_settings.py merges scalars with the template winning, so a
re-apply resets a host that chose opus or no advisor; the record says how
such a host keeps its choice.

Evidence: test_the_advisor_is_fable_and_accepted_for_the_main_model failed
at ec8a489 (None != 'fable') and passes now.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ld-routing

The lean-workflow-child-routing activation in catalogs/foundation/decisions.json
still described isolated-builder as "Sonnet/max". All three copies of its
definition (.claude/agents, adoption/agents/claude,
examples/claude-native/agents) declare model: opus and effort: max since the
2026-09-27 harness-settings change (docs/decisions/2026-09-27-claude-harness-settings.md,
item 1), and the token-e2e runner binds it to opus (Amendment 2, #413). This
closes follow-up 5 of that record's independent verification.

A new test, CheckedInAgentLabelTests, compares every "`agent` (Model/effort)"
label in the foundation decisions with the agent's frontmatter, read with a
regex so it needs no PyYAML.

Evidence: the test failed at ec8a489 (('sonnet', 'max') != ('opus', 'max'))
and passes now; tests.test_foundation_catalog and scripts/validate_foundation.py
through test_checked_in_catalog_passes stay green.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…e, bin/vllm follow-up closed

Review repair (one round):
- The three 2026-09-26 llama.cpp host receipts carry independent_session
  reviews: agree for install-2 and use, needs_changes for the superseded
  first install receipt. The serving-switch receipt has no reviews field,
  and its limitations[4] "only self reviews" sentence predates them. The
  b11057 pin stands: the paired acceptance against a fresh b11057 baseline
  is the remaining gate (eval_arm.py pins every li26 arm to b11146).
- Alternative 4: the Claude Code 2.1.283 deniedModels setting, not adopted
  because it is managed-only; the guards' Overturn now names it.
- The workstation refresh record's bin/vllm follow-up is closed on this
  host, from the coordinator's host observation (one dated line).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…the repository

The coordinator's host observation has no receipt in this repository
(no evidence file changed between ec8a489 and origin/main); the row no
longer implies that none exists anywhere.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ander, li26 scope, coordinator effort)

The cross-family review found five defects, each verified against source:
- qmd 2.8.3 formats Qwen3-Embedding as well as EmbeddingGemma (src/llm.ts L86-119) and takes other models through QMD_EMBED_MODEL (L281-282, L303-304); the embedding row no longer says a 2026 embedder needs a qmd change, and cites the Gemma releases page for the 2025-09-04 date.
- tobil/qmd-query-expansion-0.6B-gguf (2026-04-24, edddacb0) is newer than the 1.7B default (L285) and loadable through QMD_GENERATE_MODEL (L307-308); it is listed as an evaluation candidate, not a switch.
- The llama.cpp row limits its claim to li26's arms and proxy workload.
- The main-session row gives the coordinator's xhigh (template effortLevel and modelSettings) and max for agents and workflow children.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…measured, fallback map)

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…eview (guard scope, Vals fallback counts, alias observation scope, lifecycle scope, advisor test)

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…servation dropped, project-file reach, Vals page dates)

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant