Skip to content

feat(llm-drivers,kernel): report the model codex actually used (#6134)#6157

Merged
houko merged 2 commits into
mainfrom
feat-codex-actual-model
Jun 17, 2026
Merged

feat(llm-drivers,kernel): report the model codex actually used (#6134)#6157
houko merged 2 commits into
mainfrom
feat-codex-actual-model

Conversation

@houko

@houko houko commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

The codex-cli driver echoed the requested model id; the model the CLI actually resolved (which can differ from the predefined one) was never surfaced, so metering and the dashboard showed the nominated model rather than reality.

The fix threads the real model through a dedicated channel rather than reusing actual_provider — that field is the billing provider slot (UsageRecord.provider), so putting a model there would corrupt provider attribution.

  • New CompletionResponse.actual_model: Option<String> — "the model the provider actually ran, when it differs from the requested id; None means use the requested model". Added actual_model: None at every existing construction site across the workspace (drivers, runtime, kernel, tests).
  • Threaded actual_model through the agent loop exactly like actual_provider: captured per-LLM-call into last_actual_model, carried via EndTurn into AgentLoopResult.actual_model, in both the streaming and non-streaming paths.
  • The kernel's UsageRecord construction now records actual_model when present (the two messaging.rs agent-turn sites and the tools_and_skills.rs aux-review site), falling back to the nominated model otherwise — mirroring how actual_provider is already honoured next to it. Cost basis is unchanged (codex usage is 0/0, so the model-name override never alters billed cost).
  • codex-cli driver: parse the model from codex exec's stderr startup banner. Verified against codex's own source — print_config_summary writes model: <value> to stderr via eprintln!, unconditionally, in human-output mode (the driver's mode; it does not pass --json, and the --json event stream carries no model field per exec_events.rs). The parser strips ANSI styling (codex bolds banner keys), is case-insensitive and position-independent, and degrades safely to the requested model id when the banner can't be parsed so the call never breaks.

Verification:

  • cargo check --workspace --all-targets — clean (all construction sites updated).
  • cargo clippy --workspace --all-targets — zero warnings.
  • cargo test -p librefang-llm-drivers --lib — 562 pass, incl. 3 new codex banner-parser tests (plain banner, ANSI-styled banner, absent/empty → None fallback).
  • cargo test -p librefang-kernel --lib — 1238 pass.
  • Runtime threading: extended the existing actual_provider survival tests (non-streaming + streaming) to also assert actual_model threads from CompletionResponse to AgentLoopResult.

The codex-cli driver echoed the requested model id; the model the CLI actually resolved (which can differ from the predefined one) was never surfaced, so metering and the dashboard showed the nominated model rather than reality.

The fix threads the real model through a dedicated channel rather than reusing `actual_provider` — that field is the billing *provider* slot (`UsageRecord.provider`), so putting a model there would corrupt provider attribution.

- New `CompletionResponse.actual_model: Option<String>` — "the model the provider actually ran, when it differs from the requested id; None means use the requested model". Added `actual_model: None` at every existing construction site across the workspace (drivers, runtime, kernel, tests).
- Threaded `actual_model` through the agent loop exactly like `actual_provider`: captured per-LLM-call into `last_actual_model`, carried via `EndTurn` into `AgentLoopResult.actual_model`, in both the streaming and non-streaming paths.
- The kernel's `UsageRecord` construction now records `actual_model` when present (the two `messaging.rs` agent-turn sites and the `tools_and_skills.rs` aux-review site), falling back to the nominated model otherwise — mirroring how `actual_provider` is already honoured next to it. Cost basis is unchanged (codex usage is 0/0, so the model-name override never alters billed cost).
- codex-cli driver: parse the model from codex `exec`'s stderr startup banner. Verified against codex's own source — `print_config_summary` writes `model: <value>` to stderr via `eprintln!`, unconditionally, in human-output mode (the driver's mode; it does not pass `--json`, and the `--json` event stream carries no model field per `exec_events.rs`). The parser strips ANSI styling (codex bolds banner keys), is case-insensitive and position-independent, and degrades safely to the requested model id when the banner can't be parsed so the call never breaks.

Verification:
- cargo check --workspace --all-targets — clean (all construction sites updated).
- cargo clippy --workspace --all-targets — zero warnings.
- cargo test -p librefang-llm-drivers --lib — 562 pass, incl. 3 new codex banner-parser tests (plain banner, ANSI-styled banner, absent/empty → None fallback).
- cargo test -p librefang-kernel --lib — 1238 pass.
- Runtime threading: extended the existing actual_provider survival tests (non-streaming + streaming) to also assert actual_model threads from CompletionResponse to AgentLoopResult.
@github-actions github-actions Bot added size/L 250-999 lines changed area/runtime Agent loop, LLM drivers, WASM sandbox area/kernel Core kernel (scheduling, RBAC, workflows) labels Jun 17, 2026
…nt billing path (#6134)

The non-streaming primary agent path (`execute_llm_agent`) built its `UsageRecord` with `model: model.clone()` (the manifest-nominated id) while honouring `actual_provider` right beside it. Its streaming twin in `messaging.rs` and the ephemeral `/btw` path were updated to honour `result.actual_model`, but this site — the one the streaming path's comment says it "mirrors" — was missed, so a codex-cli call that resolved its own model still recorded the requested id on the main message path.

Use `result.actual_model.clone().unwrap_or_else(|| model.clone())` to mirror the other billing sites.
@houko
houko merged commit b8421a7 into main Jun 17, 2026
32 checks passed
@houko
houko deleted the feat-codex-actual-model branch June 17, 2026 08:46
houko pushed a commit that referenced this pull request Jun 18, 2026
…nt()

Coding-agent CLI drivers (Claude Code, Codex, Gemini CLI, Qwen Code) delegate model selection to an external process that may resolve a model different from the requested id, so they — not raw provider APIs — are the drivers that can meaningfully report CompletionResponse.actual_model (generalizing #6157, which did this for codex-cli).

This axis is orthogonal to LlmFamily (wire format): claude-code and the Anthropic HTTP API are both LlmFamily::Anthropic, yet only the former is a coding agent. The new trait method defaults to false (additive — no existing construction site or HTTP provider changes); the four wired coding-agent drivers override it to true. aider.rs is unwired dead code and is left untouched.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kernel Core kernel (scheduling, RBAC, workflows) area/runtime Agent loop, LLM drivers, WASM sandbox size/L 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant