feat(llm-drivers,kernel): report the model codex actually used (#6134)#6157
Merged
Conversation
The codex-cli driver echoed the requested model id; the model the CLI actually resolved (which can differ from the predefined one) was never surfaced, so metering and the dashboard showed the nominated model rather than reality. The fix threads the real model through a dedicated channel rather than reusing `actual_provider` — that field is the billing *provider* slot (`UsageRecord.provider`), so putting a model there would corrupt provider attribution. - New `CompletionResponse.actual_model: Option<String>` — "the model the provider actually ran, when it differs from the requested id; None means use the requested model". Added `actual_model: None` at every existing construction site across the workspace (drivers, runtime, kernel, tests). - Threaded `actual_model` through the agent loop exactly like `actual_provider`: captured per-LLM-call into `last_actual_model`, carried via `EndTurn` into `AgentLoopResult.actual_model`, in both the streaming and non-streaming paths. - The kernel's `UsageRecord` construction now records `actual_model` when present (the two `messaging.rs` agent-turn sites and the `tools_and_skills.rs` aux-review site), falling back to the nominated model otherwise — mirroring how `actual_provider` is already honoured next to it. Cost basis is unchanged (codex usage is 0/0, so the model-name override never alters billed cost). - codex-cli driver: parse the model from codex `exec`'s stderr startup banner. Verified against codex's own source — `print_config_summary` writes `model: <value>` to stderr via `eprintln!`, unconditionally, in human-output mode (the driver's mode; it does not pass `--json`, and the `--json` event stream carries no model field per `exec_events.rs`). The parser strips ANSI styling (codex bolds banner keys), is case-insensitive and position-independent, and degrades safely to the requested model id when the banner can't be parsed so the call never breaks. Verification: - cargo check --workspace --all-targets — clean (all construction sites updated). - cargo clippy --workspace --all-targets — zero warnings. - cargo test -p librefang-llm-drivers --lib — 562 pass, incl. 3 new codex banner-parser tests (plain banner, ANSI-styled banner, absent/empty → None fallback). - cargo test -p librefang-kernel --lib — 1238 pass. - Runtime threading: extended the existing actual_provider survival tests (non-streaming + streaming) to also assert actual_model threads from CompletionResponse to AgentLoopResult.
…nt billing path (#6134) The non-streaming primary agent path (`execute_llm_agent`) built its `UsageRecord` with `model: model.clone()` (the manifest-nominated id) while honouring `actual_provider` right beside it. Its streaming twin in `messaging.rs` and the ephemeral `/btw` path were updated to honour `result.actual_model`, but this site — the one the streaming path's comment says it "mirrors" — was missed, so a codex-cli call that resolved its own model still recorded the requested id on the main message path. Use `result.actual_model.clone().unwrap_or_else(|| model.clone())` to mirror the other billing sites.
This was referenced Jun 17, 2026
Closed
houko
pushed a commit
that referenced
this pull request
Jun 18, 2026
…nt() Coding-agent CLI drivers (Claude Code, Codex, Gemini CLI, Qwen Code) delegate model selection to an external process that may resolve a model different from the requested id, so they — not raw provider APIs — are the drivers that can meaningfully report CompletionResponse.actual_model (generalizing #6157, which did this for codex-cli). This axis is orthogonal to LlmFamily (wire format): claude-code and the Anthropic HTTP API are both LlmFamily::Anthropic, yet only the former is a coding agent. The new trait method defaults to false (additive — no existing construction site or HTTP provider changes); the four wired coding-agent drivers override it to true. aider.rs is unwired dead code and is left untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The codex-cli driver echoed the requested model id; the model the CLI actually resolved (which can differ from the predefined one) was never surfaced, so metering and the dashboard showed the nominated model rather than reality.
The fix threads the real model through a dedicated channel rather than reusing
actual_provider— that field is the billing provider slot (UsageRecord.provider), so putting a model there would corrupt provider attribution.CompletionResponse.actual_model: Option<String>— "the model the provider actually ran, when it differs from the requested id; None means use the requested model". Addedactual_model: Noneat every existing construction site across the workspace (drivers, runtime, kernel, tests).actual_modelthrough the agent loop exactly likeactual_provider: captured per-LLM-call intolast_actual_model, carried viaEndTurnintoAgentLoopResult.actual_model, in both the streaming and non-streaming paths.UsageRecordconstruction now recordsactual_modelwhen present (the twomessaging.rsagent-turn sites and thetools_and_skills.rsaux-review site), falling back to the nominated model otherwise — mirroring howactual_provideris already honoured next to it. Cost basis is unchanged (codex usage is 0/0, so the model-name override never alters billed cost).exec's stderr startup banner. Verified against codex's own source —print_config_summarywritesmodel: <value>to stderr viaeprintln!, unconditionally, in human-output mode (the driver's mode; it does not pass--json, and the--jsonevent stream carries no model field perexec_events.rs). The parser strips ANSI styling (codex bolds banner keys), is case-insensitive and position-independent, and degrades safely to the requested model id when the banner can't be parsed so the call never breaks.Verification: