test(llm-drivers): pin claude_code resolved-model parsing (#6318)#6331
Merged
Conversation
The #6318 auto-detect mechanism for the claude_code driver was wired but untested: stream() captures the model from the stream-json system/init event, and complete() reads it from the result JSON's model field (forward-compat, since `claude -p --output-format json` does not emit it yet) — both stamping CompletionResponse.actual_model so kernel metering attributes the real model rather than the requested id. Add two regression tests pinning the deserialization contract for both shapes so a refactor that drops or renames the serde `model` field fails loudly instead of silently turning resolved-model detection back into a no-op.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #6318 (the "auto-detect the model an agent CLI actually ran" half). The detection itself already exists across the agent-CLI drivers; this PR closes the one remaining gap — it was untested for
claude_code.State of the auto-detect mechanism (verified):
parse_model_from_bannerreads themodel:line from codex's stderr startup banner and stamps it intoCompletionResponse.actual_model(landed in codex cli 需要获取实际使用的模型 #6134). Already covered bytest_parse_model_from_banner_*.stream()captures the resolved model from the stream-jsonsystem/initevent;complete()reads it from the result JSON'smodelfield (forward-compat —claude -p --output-format jsondoes not emit it yet, so that path staysNoneand the kernel falls back to the requested id). Both paths were wired but had no test.model: result.actual_model.clone().unwrap_or_else(|| model.clone())(messaging.rs,agent_execution.rs,tools_and_skills.rs), so a driver that resolves its own model attributes the real one.Change
Test-only. Two regression tests in
crates/librefang-llm-drivers/src/drivers/claude_code.rs:stream_json_init_event_carries_resolved_model— deserializes a representative stream-jsonsystem/initline and assertsClaudeStreamEvent.modelis parsed (the streaming agent path's detection source).json_output_surfaces_model_when_present_else_none— pins both shapes of the result JSON: absentmodel→None(today's--output-format jsonbehaviour, kernel falls back to requested id), presentmodel→ surfaced (forward-compat for a future CLI version).These lock the deserialization contract so a refactor that drops or renames the serde
modelfield fails loudly instead of silently turning resolved-model detection back into a no-op.Not changed (and why)
actual_model: None. Gemini CLI emits no model banner/JSON field and runs exactly the--modelit is given (no divergence), so there is nothing to detect;Nonecorrectly falls back to the requested id. Its CLI calls report zero tokens (billing is on the user's Google account), so the recorded model string is display-only and not a cost-attribution input — stamping the prefix-stripped id would only drop thegemini-cli/provenance for no benefit.Verification
In the worktree:
cargo test -p librefang-llm-drivers --lib model— 24 passed, 0 failed (includes the two new tests)cargo clippy -p librefang-llm-drivers --all-targets -- -D warnings— cleancargo fmt -p librefang-llm-drivers -- --check— cleanRefs #6318. The manual-management half landed in #6327.