Repository navigation
Haiku 5.5 takes cheap exact extraction: record the 2026-10-07 repeat trial (land after 2026-10-08T17:18Z) - #840
Open
seathatflowsinourveins wants to merge 2 commits into
Conversation
…trial The 2026-10-07 repeat trial (workflow wf_aa532ddc-f32, Claude Code 2.1.293, one Mac) met the overturn condition recorded for the "Cheap exact extraction" row: on two distinct extraction packets Claude Haiku 5.5 returned 8 of 8 runs exact with no unanchored citation, as Sonnet 5.5 did, and a blind Opus reviewer found no defect in any of the 16 extraction runs. The session's Fable advisor answered 7 of 8 Sonnet and 2 of 8 Haiku runs; the 6 unassisted Haiku runs were exact too. Haiku's executor cost per child was under a tenth of Sonnet's; its verbatim command relay was byte-exact in 5 of 6 runs. - Routing: a source-scout stage may pass a per-call model 'haiku' at effort max for inventories and line-cited extraction that relay no command output; command running and output relay keep source-scout's Sonnet default, and judgment stays on Opus (workflows README section, recipe row and dated paragraph, portable CLAUDE.md and its generated new-WSL copy, profile recipe, task-model routing record, decision lean-workflow-child-routing). No claude-haiku-5-5 settings pin, on purpose (noted in the 2026-09-29 dispatch record); the claude-code minimum stays 2.1.284 for now. - child-usage.mjs: haiku alias row (Haiku 5.5 from 2.1.293, model-config version history) so a haiku child that resolves an older Haiku is flagged; tests for the boundary and the fallback; SHA256SUMS regenerated. - Model currency: the background-haiku row moves to claude-haiku-5-5 (release-note entry of 2026-10-07) and records the extraction route. - Receipt evidence/receipts/haiku55-extraction-trial-20261007.json: derived counts, timings, prices, advisor use and file hashes only; the oracle re-run on the packets' source commit b16cb8c reproduced the retained scores. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WhN8fzZs47tp6KunqRQ9gs
…ation manifests/stack.json: the Haiku model row moves from claude-haiku-4-5-20251001 to claude-haiku-5-5 (released 2026-10-07; Claude Code 2.1.293 default Haiku on the Anthropic API), records the cheap exact extraction route and links its qualifying receipt. manifests/evidence.json: receipts[] gains haiku55-extraction-trial-20261007, and files[] re-registers the files this branch changed that main already lists, plus the new receipt (docs/lanes.md hot-file protocol). The component matrix and new-host grand list regenerate unchanged. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01WhN8fzZs47tp6KunqRQ9gs
5 tasks done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scope
model: 'haiku'onsource-scoutat effortmax. Command running and verbatim output relay stay onsource-scout's Sonnet default and judgment stays on Opus. Thebackground-haikucurrency row moves toclaude-haiku-5-5, andchild-usage.mjsgains ahaikualias row.36654a8174b5cad1380bd13ed1fffc0f135aac2c(origin/mainwhen the branch was cut).lane:foundation. This PR also changes whatmanifests/stack.jsonsays (the Haiku model row moves fromclaude-haiku-4-5-20251001toclaude-haiku-5-5) and adds areceipts[]entry tomanifests/evidence.json. Enforce dated model currency and propose daily source reviews #826 changed the same rows underlane:foundation, so the label follows it; relabel tolane:sharedif docs/lanes.md's "changes what a shared hot file says" is read to cover this.examples/claude-native/workflows/{README.md,child-usage.mjs,test-child-usage.mjs,SHA256SUMS},examples/claude-native/CLAUDE.mdand its generated copyadoption/new-wsl/claude-user-instructions.md(regenerated withtools/adoption/new_wsl_client_config.py --write-blocks),recipes/claude-native-ultracode.md,recipes/claude-native-profile.md,catalogs/foundation/{decisions.json,model-currency.json},docs/model-currency.md,docs/decisions/2026-09-30-task-model-routing.md(one row, one overturn clause),docs/decisions/2026-09-29-sonnet-5-5-dispatch.mdanddocs/decisions/2026-10-07-model-currency-enforcement.md(one dated note each),evidence/receipts/haiku55-extraction-trial-20261007.json(new). The hot filesmanifests/stack.jsonandmanifests/evidence.jsonare in the last commit, per the hot-file protocol.SOTA sources
Claude Code 2.1.293 (
anthropics/claude-code; npm@anthropic-ai/[email protected], published 2026-10-07T17:18:04.464Z,dist.shasumd653d8b46b676742dfd6dea808bfcc82f2fc0fd8, integritysha512-PfPSvsd0zH1R1NhLKLebYdEXg/KcGCXc7LBntE186EWuZtBXa7sMnBqs0/uW/BvzkBEfHKqww/WbK/kMAk0TXg==): CHANGELOG.md line 5 at79babc372d64101f981bd2b52c3dbe588596dc56reads "Added Claude Haiku 5.5 (claude-haiku-5-5), now the default Haiku model on the Anthropic API — 1M context, $0.10/$0.50 per Mtok ($0.50/$2.50 for prompts over 100K)". That repository publishes no CI run at the tag.Claude Code model configuration (version history, read 2026-10-07T23:58Z; the page carries no version):
haikuresolves to Haiku 5.5 on the Anthropic API";haikuresolves to Haiku 4.5;haikuto Haiku 4.5;These are the source of the
child-usage.mjsrow, the README's version check and the Anthropic-API-only scope.Claude Code sub-agents (choose a model): the per-invocation
modelparameter comes first in the model order, andhaikuis an accepted alias. This is why the route is a per-call override, not a new agent type.Claude Code advisor (advisor): subagents inherit the configured advisor, which is the source of the trial's advisor disclosure.
Claude Platform: the model-currency and
stack.jsonrows rest on#october-7-2026section is an unrelated Sonnet 5.5 cache-price change);claude-haiku-5-5, no dated snapshot;Effort (effort doc): Haiku 5.5 supports all five levels,
maxincluded, and defaults tomedium. Anthropic advisesxhighormaxonly where evals show a gain. This is the basis for the recipe's effort sentence, the "effort is the stage rule" note, and leaving out aclaude-haiku-5-5settings pin.Trial design: the overturn condition recorded in
docs/ultracode-token-routing-20260921.mdand the workflows README: "a repeat trial in which Haiku returns no unanchored citation on two distinct extraction packets". It ran on the native Claude Code Workflow tool with this repository'ssource-scoutandevidence-revieweragents andchild-usage.mjs.Prices: LiteLLM
model_prices_and_context_window.json(BerriAI/litellmmain, ETagW/"35442afac19b8c20555fa17a4f487c7998c99b658444f04820628f1010f40a85", the copy ccusage caches). This is a third-party table and is labelled as one; its Haiku 5.5 base rate equals the CHANGELOG's $0.10/$0.50.Evidence-class table
native_provenhaiku55-extraction-trial-20261007(workflowwf_aa532ddc-f32, Claude Code 2.1.293, one Mac)evidence-reviewerfound no defect in any of the 16 extraction runs (152/152 rows)native_proven(round 1 read from the run journal; round 2 also in the retained output)data.runs[].blind_opus_verdict,data.blind_verdict_totalsnative_proven(transcript tool results andchild-usage.mjsadvisor iterations)data.advisor,data.runs[].advisor_callsnative_provendata.c1_haiku_miss,data.summaryclaude-haiku-5-5, recorded effortmaxand ran client 2.1.293native_proven(the client's own per-message record)data.identity_and_effort,data.runs[]local_integration(a deterministic re-run on retained outputs)python3 -I oracle.py <checkout at b16cb8cf> all-rounds.json <out>matchedscores-all-rounds.json; re-run by the builder and by an independent verifier (cmp: identical)local_integration(arithmetic on recorded usage times a third-party price table; not billing)data.summary,data.prices_usd_per_token,provenance.cost_methodclaude -p --model haiku,--model claude-haiku-5-5andclaude --bg --model haikuresolvedclaude-haiku-5-5with exact repliesnative_proven(reported by the trial's probe children; not re-run here)data.native_probes_reported_by_the_trialhaikuresolves to Haiku 5.5 from 2.1.293, and the background slot uses the default Haikusource_review(no read-back of the background slot)79babc37L5; model-config version historychild-usage.mjsflags ahaikuchild that resolved an older Haiku under 2.1.293 or latersynthetic(fixture rows; with the row removed, 2 tests fail)node test-child-usage.mjsLocal commands run
All commands ran on the final head
e72e070funless noted.The trial ran on 2026-10-07 from 21:31Z to 23:48Z (5:31 PM to 7:48 PM EDT). An independent three-stage Opus review covered the diff, the receipt against the private evidence, and a re-run of the checks. Its findings are folded in: the advisor disclosure and split costs, both rounds' verdicts, the portable instructions' stale line, the settings-pin and client-minimum notes, and the wording and number fixes. A second reviewer confirmed all 11 fixes and raised two items left open here: (A)
catalogs/foundation/upstream-surface-dispositions.json:73-74and:313-314citechild-usage.mjs:67and:1925, which the new comment line moved to:68and:1926; (B) the receipt's P2 Haiku executor mean ($0.0180) and P1 Haiku advisor-inclusive mean ($0.2407) are means of unrounded per-child costs, so the rounded per-run values give $0.0181 and $0.2408, andprovenance.cost_methodshould say so. A wording nit: README:1084 says "relays no command output verbatim" where CLAUDE.md says "relay none".Decision record
The routing decision is recorded in these places:
examples/claude-native/workflows/README.md: evidence, boundaries, advisor, cost basis, settings pin, client minimum and overturn.docs/decisions/2026-09-30-task-model-routing.md, the task-to-model routing record: one row and one overturn clause.lean-workflow-child-routingincatalogs/foundation/decisions.json.docs/decisions/2026-09-29-sonnet-5-5-dispatch.md(the pin is left out on purpose) anddocs/decisions/2026-10-07-model-currency-enforcement.md(the currency row).Alternatives considered:
claude-haiku-5-5settings pin. Deferred: it would lift every Haiku session that names no effort without an eval.Overturn: send extraction back to Sonnet when a routed Haiku stage returns an unanchored or misattributed citation that a later check finds, or when a
haikuchild resolves a model older than Haiku 5.5.Host evidence
Not applicable: no file under
evidence/hosts/changed.Merge interactions, follow-ups and limits
Align code graph MCP server key with native client registrations #834 also edits
examples/claude-native/workflows/child-usage.mjsandtest-child-usage.mjs. Whichever lands second rebases and regenerates that directory'sSHA256SUMSwithsha256sum -- *.mjs *.js *.json.Make standing Ultracode opt-in and record task dispatch scope #829 and Instruction core: trim the always-loaded text of both clients; trading rules load only where they apply #836 edit the workflows README, the Ultracode recipe or the portable instructions; Instruction core: trim the always-loaded text of both clients; trading rules load only where they apply #836 trims
examples/claude-native/CLAUDE.md, which this PR touches in one clause. Document retrieval-first local-model choices and supersede foundation sources #810, Pin Grafana MCP 2.0.1 and AgentsView 0.44.0 for monitoring #807 and Qualify CLI transfers and retain applied client-tools evidence #776 editmanifests/stack.json. Rebase through the hot-file protocol (docs/lanes.md). After a CLAUDE.md change, reruntools/adoption/new_wsl_client_config.py --write-blocks.Follow-up: once its 24-hour gate passes, raise the
claude-codeminimum from 2.1.284 to 2.1.293 together inmanifests/stack.jsonand both pin files, as a separate currency change. Until then the README'sclaude --versioncheck is the route's only guard.Follow-up (optional): rerun the comparison with
CLAUDE_CODE_DISABLE_ADVISOR_TOOL=1to hold the advisor equal across arms. The route does not rest on it, because the 6 unassisted Haiku runs were exact.Limits:
The receipt's limitations list each one.
Checklist
permissions: {}and grant each job only what it needs. Not applicable: no workflow file changed.🤖 Generated with Claude Code
https://claude.ai/code/session_01WhN8fzZs47tp6KunqRQ9gs