fix(metering): gate provider hourly token budget pre-call so exhaustion flags the fallback chain (#5980)#5988
Conversation
7718c38 to
affdb2c
Compare
houko
left a comment
There was a problem hiding this comment.
Thanks for tackling this — the diagnosis in #5980 is solid and a pre-dispatch gate is the right shape. But as written the fix flags a store that the per-agent execution driver never reads, so it's a no-op for the reported scenario. Requesting changes.
The flag is written, but nothing on the agent path consumes it
flag_provider_budget_if_exhausted → check_provider_budget → store.mark_exhausted(provider, BudgetExceeded, …). That store is only honored by FallbackDriver/FallbackChain, and only when (a) a store is attached via with_exhaustion_store and (b) the slot has a non-empty provider_name. The driver the agent loop actually runs comes from resolve_driver:
- With fallbacks configured,
crates/librefang-kernel/src/kernel/llm_drivers.rs:326-328buildsFallbackDriver::with_models(chain)— that constructor setsexhaustion_store: None(fallback.rs:120) and leaves everyprovider_nameempty. Bothis_slot_exhaustedandrecord_exhaustionearly-return on missing store / empty name (fallback.rs:269-292), so the flag is ignored. - With no fallbacks,
llm_drivers.rs:331returns the bareprimarydriver, which has no store awareness at all — exactly #5980's case (a single Moonshot agent).
The new comment ("resolve_driver reads the same store and skips the exhausted slot") describes the intended behavior, but the diff doesn't change resolve_driver to attach the store or set provider names, so it isn't true yet. To close the loop, resolve_driver needs to use with_models_and_providers (or set_provider_names) so slots carry provider names, and attach the shared ProviderExhaustionStore via with_exhaustion_store — including a single-slot wrapper for the no-fallback case so a solo over-cap provider surfaces a clean AllProvidersExhausted instead of overspending.
Issue's proposed-fix item #2 not addressed
crates/librefang-kernel/src/kernel/agent_execution.rs:1303 still does let _ = self.metering.engine.record(&usage_record) unconditionally after a post-call QuotaExceeded, so the rolling counter keeps climbing (the 4.8M-over-1.2M symptom). Either drop the unconditional record on breach, or rely on the pre-call gate — but the gate has to be wired into a driver that reads it first.
Test coverage
No test added. The metering unit tests only prove the flag is set, not that the execution-path driver honors it — which is the whole bug. Per CLAUDE.md / #3721 this needs a test showing: provider at/over cap → next dispatch is skipped/failed-over before the HTTP send, and the counter doesn't grow unbounded across consecutive ticks.
Happy to re-review once resolve_driver carries the store + provider names through.
0ffc07e to
04f70f0
Compare
|
The pre-dispatch budget gate looks like a no-op on the normal configured-fallback path. The metering side flags exhaustion correctly, but the Where the flag is set vs. where the driver is built
The boot-time Scenario: operator sets Suggested fix: in Reviewed at 04f70f0. |
04f70f0 to
d14bad0
Compare
|
Rebased onto current |
|
Thanks for picking this up — the direction is right (restore the #4807 design and call 1. The flag is written to a store nothing reads when there's no fallback chain. 2. Agent-level The only path where the flag currently does anything is: default provider + a global Suggested fix: to stop the bleed even with no fallback, the flag-only approach isn't sufficient. When the gate hits and the resolved driver isn't a store-aware Tests (required, per the repo's integration-testing policy and #5980's acceptance items): a kernel→driver e2e test, not just the metering-layer unit coverage, asserting (a) a single-provider agent over the hourly cap has its next dispatch blocked and the counter stops; (b) a multi-provider agent over the cap falls back to a healthy provider; (c) no budget configured → behaviour unchanged. |
ce66d21 to
4b1d83f
Compare
|
Thanks @houko — addressed both defects from your review. Pushed 1. Multi-provider path now store-aware. 2. Single-provider path now hard-returns. As you noted, the flag-only approach can't help when there is no slot to skip to. 3. Kernel→driver e2e tests ( One thing I want to flag for a possible follow-up (pre-existing, not introduced here): the gate keys on the raw |
|
Re-reviewed the two follow-up commits — this resolves my earlier CHANGES_REQUESTED. The flag is now actually consumed on the agent path, both directions:
The split is correct: block only when exhausted and there's no viable fallback; otherwise let the now-store-aware LGTM from my side — clearing my changes-requested. Nice fix. |
houko
left a comment
There was a problem hiding this comment.
Two issues found during daily review pass:
1. Misplaced doc comment and #[allow] attribute in agent_execution.rs
The new functions are inserted after the existing context lines /// Execute the default LLM-based agent loop. and #[allow(clippy::too_many_arguments)], so in the resulting file those two items are syntactically attached to flag_provider_budget_if_exhausted (not execute_llm_agent):
/// Execute the default LLM-based agent loop. ← wrong: was execute_llm_agent's doc
#[allow(clippy::too_many_arguments)] ← wrong: was execute_llm_agent's allow
/// Pre-dispatch per-provider budget gate (#5980…) ← flag_provider_budget_if_exhausted's doc
pub(crate) fn flag_provider_budget_if_exhausted(…) { … }
pub(crate) fn provider_exhausted_blocks_call(…) { … }
#[allow(clippy::too_many_arguments)] ← correctly precedes execute_llm_agent
#[instrument(…)]
pub(super) fn execute_llm_agent(…) { … } ← no doc commentexecute_llm_agent loses its /// Execute the default LLM-based agent loop. doc. flag_provider_budget_if_exhausted gains it (merged with the correct doc as a two-paragraph doc comment) plus an unnecessary #[allow].
Fix: move the two new methods to before the /// Execute the default LLM-based agent loop. line (i.e. earlier in the impl block), or extract them to a dedicated impl block so the existing doc + attribute stay cleanly above execute_llm_agent.
2. Duplicate CHANGELOG entry for #5980
The diff adds two bullets that both reference (#5980): a detailed one near the top of the changes in the hunk and a short one-liner further down. Remove the shorter duplicate.
Generated by Claude Code
4b1d83f to
22b538a
Compare
22b538a to
4b91fa5
Compare
houko
left a comment
There was a problem hiding this comment.
One CHANGELOG issue to fix before merge: duplicate entry for #5980 (see inline comment on CHANGELOG.md).
Generated by Claude Code
There was a problem hiding this comment.
There are two [Unreleased] entries for the same fix (#5980):
- The detailed one near the top of the hunk —
**metering(budget): the pre-dispatch per-provider budget gate now actually enforces the hourly cap…**— describes the full root cause, two fixes, and the three regression test names. - A shorter duplicate further down —
**metering(budget): enforce per-provider hourly token/cost budgets pre-dispatch…**— covers the same change in one line.
The shorter duplicate should be removed; the detailed entry is the correct one to keep. Having two entries for the same issue will also trip the pre-commit duplicate-[Unreleased] guard.
Generated by Claude Code
f2d50e2 to
ed39bcd
Compare
8df3895 to
22643cd
Compare
22643cd to
ee49a0d
Compare
ee49a0d to
300edbb
Compare
…on flags the fallback chain (librefang#5980)
…block single-provider over-cap (librefang#5980) The librefang#5980 pre-dispatch budget gate flagged an over-cap provider in the shared ProviderExhaustionStore but never had its flag consulted in the reported scenario. The agent-fallback path built its FallbackDriver via with_models(chain), which leaves the exhaustion store unset and every slot's provider name empty, so is_slot_exhausted short-circuited to false and the over-cap provider was dispatched again. A single-provider agent had no fallback at all, so the flag landed in a store nothing read and the call still went out. resolve_driver now builds the multi-provider chain via with_models_and_providers(...) carrying each slot's provider name and attaches the same exhaustion-store handle the gate flags (from MeteringEngine::exhaustion_store()), mirroring the boot-time path, so a budget-exhausted slot is pre-skipped and the call fails over to a healthy provider. flag_provider_budget_if_exhausted now returns whether the provider is over cap, and the three dispatch call sites hard-return QuotaExceeded when the agent has no viable fallback chain instead of dispatching past the cap. Regression tests in librefang-kernel cover all three cases: a single-provider agent over the hourly cap is blocked, a multi-provider agent defers to the store-aware FallbackDriver which skips the exhausted slot and serves from a healthy one, and an agent with no configured budget dispatches unchanged.
…t gate The 'Execute the default LLM-based agent loop' doc comment and its #[allow(clippy::too_many_arguments)] were duplicated above the new flag_provider_budget_if_exhausted — they already sit above execute_llm_agent where they belong.
… exhaustion is visible to FallbackDriver (librefang#5980) The primary slot's provider_name was String::new() (empty), which made is_slot_exhausted always return false for the first slot — the budget gate flagged exhaustion correctly but the FallbackDriver never skipped the exhausted primary because it couldn't match the empty name against the store entry.
300edbb to
a5bd7ea
Compare
houko
left a comment
There was a problem hiding this comment.
The metering logic looks good and the earlier review concerns are addressed — one small thing still blocks merge.
CHANGELOG.md: duplicate [Unreleased] entry for #5980. There are two bullets for this change:
- the detailed one (
the pre-dispatch per-provider budget gate now actually enforces…) — keep this one - a redundant one-liner further down (
enforce per-provider hourly token/cost budgets pre-dispatch…) — please remove this duplicate
This is the item from the prior review that's still outstanding.
Minor, while you're in there: execute_llm_agent lost its /// Execute the default LLM-based agent loop. doc comment (it's present on main). Please restore it.
Once the duplicate CHANGELOG bullet is removed, this is good to merge.
…ent doc Removes the redundant one-line [Unreleased] bullet for librefang#5980 (the detailed entry above is the keeper) and restores the `/// Execute the default LLM-based agent loop.` doc comment that was dropped from execute_llm_agent. Addresses the review feedback on this PR.
The duplicate [Unreleased] CHANGELOG bullet is removed and the execute_llm_agent doc comment restored (pushed to the branch); budget logic was already verified. Clearing for merge.
…top (#6285) CHANGELOG.md had two `## [Unreleased]` sections: one buried between released version sections (it predates #6281, which then added a duplicate at the top). The release tooling reads only the first [Unreleased] and silently drops the rest, so ~150 stranded entries were never reaching release notes; the duplicate header also tripped the `## [Unreleased]` guard, blocking every CHANGELOG-touching commit (e.g. #5988). Consolidate both blocks into a single [Unreleased] at the top, merging same-named `###` subsections (collapsed 10 redundant subsection headers). Every bullet is preserved verbatim — verified that the multiset of non-header content lines is byte-identical before/after. Drop the two entries already published in released sections: #6272 (in [2026.6.22]) and #6171 (in [2026.6.17]). Co-authored-by: Evan <[email protected]>
# Conflicts: # .secrets.baseline
Summary
Restores the #4807 pre-dispatch per-provider budget design and makes the flag load-bearing, closing the no-op that @houko flagged in CHANGES_REQUESTED.
The original commit flagged an over-cap provider in the shared
ProviderExhaustionStorebut the flag was never consulted in the reported single-Moonshot-agent scenario, so the agent kept running ~4x past its hourly cap.Changes
Commit 1 (
ec6c7b4) — the pre-dispatch gate:flag_provider_budget_if_exhaustedrunsMeteringEngine::check_provider_budgetbefore dispatch so a breach flags the provider in the shared store. Called from the three dispatch sites (agent_execution.rs,messaging.rsephemeral + streaming).Commit 2 (
4b1d83f) — wires the flag to the two paths that ignored it:llm_drivers.rsresolve_driver): the agent-fallback chain now carries each slot's provider name and builds theFallbackDriverviawith_models_and_providers(...)with the sameMeteringEngine::exhaustion_store()handle the gate flags (mirrorsboot.rs:698-714). Previously it usedwith_models(chain)— no store, empty provider names — sois_slot_exhaustedshort-circuited tofalseand re-dispatched the exhausted provider. When the store is unwired the provider-less builder is kept (no regression).flag_provider_budget_if_exhaustednow returns whether the provider is over cap, and the three dispatch sites hard-returnQuotaExceededwhenresolve_effective_fallbacks(...)is empty. A new pure helperprovider_exhausted_blocks_call(exhausted, manifest, cfg)encapsulates that decision. When a fallback chain exists the call sites do not hard-return — they defer to the store-awareFallbackDriver, preserving the feat(kernel): budget-aware fallback chain — skip exhausted providers, try next in chain #4807 philosophy.Verification
cargo check --workspace --lib— clean.cargo clippy --workspace --all-targets -- -D warnings— 0 warnings.librefang-kerneltestskernel::tests::provider_budget_gate_5980(3 passed):single_provider_over_cap_is_hard_blocked— over-cap single-provider agent is blocked, store reportsBudgetExceeded, and the gate stays blocking on re-check (counter never drains via a leaked dispatch).multi_provider_over_cap_defers_to_fallback_skip— over-cap multi-provider agent is NOT hard-blocked; theFallbackDriverbuilt over the same store handle pre-skips the exhaustedmoonshotslot and serves from healthygroq(actual_provider == Some("groq")).no_budget_configured_gate_is_noop— no provider budget configured ⇒ gate returns false, store never flagged, never blocks.Rebased onto current
origin/mainto clear the stale base.Closes #5980