Skip to content

Commit 279615e

Browse files
Python: remove as_tool budget rollup; fix max_duration in-flight enforcement; clarify thinking-budget framing
- Remove agent.as_tool usage rollup from budget accounting (as_tool API deprecated, microsoft#8950) - Keep BackgroundAgentsProvider rollup: max_tokens is a thinking budget — a uniform bound on total inference work — not a cost proxy; cross-model rollup is intentional - Fix max_duration: now interrupts in-flight model/tool calls via asyncio.wait_for, not only between iterations - Clarify AgentBudget.max_tokens docstring: thinking budget vs cost estimate - Partially addresses eavanvalkenburg feedback on microsoft#8862; see issue for our rationale on keeping sub-agent token accounting Generated with Amplifier Co-Authored-By: Amplifier <[email protected]>
1 parent 98fb54c commit 279615e

5 files changed

Lines changed: 112 additions & 293 deletions

File tree

‎python/packages/core/AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -309,7 +309,7 @@ The vector store API is experimental under the shared `VECTOR_STORES` feature ID
309309
- **`todos_remaining(*, looping_modes=None)`** / **`todos_remaining_message`** - Helper factories for todo-driven loops (the Python counterpart of .NET's `TodoCompletionLoopEvaluator`), designed for `create_harness_agent` but usable with any agent that registers a `TodoProvider` via `context_providers`. They resolve the `TodoProvider`/`AgentModeProvider` from the *running agent* (`agent.context_providers`, via `_resolve_context_provider`) rather than taking the provider as an argument, so they can be wired directly into `loop_should_continue`/`loop_next_message`. `todos_remaining` returns a `should_continue` predicate that loops while any todo is open; pass `looping_modes=[...]` to gate looping to specific operating modes (case-insensitive; honors the `AgentModeProvider`'s `source_id`/`available_modes`), `looping_modes=None` (default) applies in every mode, and an empty sequence raises `ValueError`. `todos_remaining_message` is a `next_message` callable that lists the still-open todo titles and tells the agent to finish them, returning `None` when the session/agent/provider is unavailable or nothing is open (in which case the middleware's default `None` handling applies: reuse the previous iteration's messages verbatim under the default `fresh_context=False`, or `DEFAULT_NEXT_MESSAGE` only when `fresh_context=True`).
310310
- **`background_tasks_running()`** / **`background_tasks_running_message`** - Helper factories for background-agent-driven loops, mirroring the `todos_remaining` pair. They resolve the `BackgroundAgentsProvider` from the *running agent* (`agent.context_providers`, via `_resolve_context_provider`) rather than taking the provider as an argument, so they can be wired directly into `create_harness_agent`'s `loop_should_continue`/`loop_next_message`. `background_tasks_running` returns a `should_continue` predicate that loops while the provider's persisted state shows any task with `status == RUNNING` (pair it with `max_iterations` so the loop is bounded even if a task's persisted status is never refreshed). `background_tasks_running_message` is a `next_message` callable that lists the still-running tasks (`#<id> (<agent_name>): <description>`) and tells the agent to wait for them to finish and retrieve their results, returning `None` when the session/agent/provider is unavailable or no task is running.
311311
- **Approval escape hatch** - `_has_pending_approval_request(result)` checks whether an iteration's response carries a pending tool-approval request (any content with `type == "function_approval_request"`). Both the streaming and non-streaming loops stop and return that response to the caller *before* evaluating `should_continue`/`max_iterations` or injecting `next_message`, so the loop is HITL-safe even when wrapped outermost around a `ToolApprovalMiddleware` (mirrors the C# `LoopAgent`'s `HasPendingApprovalRequests`).
312-
- **Budget** - `budget=AgentBudget` (a `total=False` `TypedDict`; plain dicts work) adds `max_tokens` / `max_duration` limits, validated in `AgentLoopMiddleware.__init__` and checked after each iteration before `should_continue`. `max_tokens` charges each iteration the wrapped agent response's `total_token_count`, falling back to `input_token_count + output_token_count`; the `with_judge` judge call is never charged. Sub-agent usage counts: `as_tool` reports the sub-agent's `usage_details` through a private `FunctionInvocationContext.metadata` key (`_TOOL_USAGE_CONTEXT_KEY`), which the function invocation loop moves onto the function_result and folds into the parent response exactly once (non-streaming `aggregated_usage`; streaming via a `usage` content on the tool update), so nested A->B->C counts each layer once. `BackgroundAgentsProvider` stores each finished run's usage on `BackgroundTaskInfo` (`usage_details`/`usage_reported`) and charges it once, to the parent run that first observes the completion (via the observing tool's context, or a run-scoped chat middleware when `before_run` observes it). `invoke_agent` telemetry spans still report each agent's own usage. Gaps: failed/cancelled sub-agent runs produce no response (tokens lost), background tasks never observed as finished or abandoned at `release_session` are not counted, and custom tools that call `agent.run` directly are not counted. The accounting is not configurable.
312+
- **Budget** - `budget=AgentBudget` (a `total=False` `TypedDict`; plain dicts work) adds `max_tokens` / `max_duration` limits, validated in `AgentLoopMiddleware.__init__` and checked after each iteration before `should_continue`. `max_tokens` is a *thinking budget* (a uniform bound on total inference work, not a cost estimate): each iteration is charged the wrapped agent response's `total_token_count`, falling back to `input_token_count + output_token_count`; the `with_judge` judge call is never charged. `agent.as_tool` usage is NOT rolled up (API deprecated, #8950). `BackgroundAgentsProvider` rollup stays and is intentionally cross-model: it stores each finished run's usage on `BackgroundTaskInfo` (`usage_details`/`usage_reported`) and charges it once, to the parent run that first observes the completion (via the observing tool's context, or a run-scoped chat middleware when `before_run` observes it), using the private `_record_tool_usage` / `_TOOL_USAGE_CONTEXT_KEY` plumbing in `_tools.py`. `invoke_agent` telemetry spans still report each agent's own usage. Gaps: failed/cancelled sub-agent runs produce no response (tokens lost), background tasks never observed as finished or abandoned at `release_session` are not counted, and custom tools that call `agent.run` directly are not counted. The accounting is not configurable. `max_duration` is enforced between iterations and also interrupts the in-flight call via `asyncio.wait_for` (3.10-compatible); it re-raises `TimeoutError` if the first iteration times out with no prior result, otherwise stops cleanly with `loop_exit_reason="time_budget_exceeded"` (streaming: the timeout covers starting the stream, not consuming it).
313313
- **Harness integration** - `create_harness_agent` enables the loop when a `loop_should_continue` callable is passed; it prepends `AgentLoopMiddleware(loop_should_continue, max_iterations=loop_max_iterations, next_message=loop_next_message)` ahead of `ToolApprovalMiddleware` so the loop is the outermost middleware (each iteration is a full agent run including tool approval, and the escape hatch hands pending approvals back to the caller). `loop_next_message` and `loop_max_iterations` only take effect together with `loop_should_continue` (with no `loop_should_continue` there is no loop, so they are ignored); `loop_max_iterations` defaults to the loop's default cap (`None` → unbounded).
314314

315315
### Workflows (`_workflows/`)

‎python/packages/core/agent_framework/_agents.py‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -752,8 +752,6 @@ async def _agent_wrapper(ctx: FunctionInvocationContext, **kwargs: Any) -> str:
752752
ctx: the function invocation context used
753753
**kwargs: only used to dynamically load the argument that is defined for this tool.
754754
"""
755-
from ._tools import _record_tool_usage # pyright: ignore[reportPrivateUsage]
756-
757755
parent_session = ctx.session
758756
session = AgentSession()
759757
child_approval_source_ids = _tool_approval_source_ids(self.middleware)
@@ -840,10 +838,6 @@ async def _agent_wrapper(ctx: FunctionInvocationContext, **kwargs: Any) -> str:
840838
if isawaitable(callback_result):
841839
await callback_result
842840
final_response = await stream.get_final_response()
843-
# Report the sub-agent's token usage (including its own sub-agents) to the calling loop,
844-
# which folds it into the parent response's usage_details exactly once. Recorded before
845-
# the approval/user-input raises below so that usage is not lost on those paths.
846-
_record_tool_usage(ctx, final_response.usage_details)
847841
approval_requests = [
848842
request
849843
for request in final_response.user_input_requests

‎python/packages/core/agent_framework/_harness/_loop.py‎

Lines changed: 69 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828

2929
from __future__ import annotations
3030

31+
import asyncio
3132
import inspect
3233
import math
3334
import time
@@ -138,6 +139,8 @@ class JudgeVerdict(BaseModel):
138139
class AgentBudget(TypedDict, total=False):
139140
"""Budget constraints for an :class:`AgentLoopMiddleware` autonomous loop.
140141
142+
``max_tokens`` is a thinking budget (a bound on total inference work), not a cost estimate.
143+
141144
Both ``max_tokens`` and ``max_duration`` are optional with no default — they are
142145
workload-specific and must be set explicitly for production autonomous sessions.
143146
When a budget dimension is omitted or ``None``, it is unconstrained. Because this is a
@@ -167,35 +170,33 @@ class AgentBudget(TypedDict, total=False):
167170
"""
168171

169172
max_tokens: int | None
170-
"""Maximum token spend (input + output) of the wrapped agent across all loop iterations.
171-
172-
Each iteration is charged the ``total_token_count`` from the agent response's usage details,
173-
falling back to ``input_token_count + output_token_count`` when ``total_token_count`` is
174-
absent. That usage covers every model call the agent makes during the iteration, including
175-
each round of tool calling.
176-
177-
Calls the loop framework makes on its own behalf are never charged: the judge call made by
178-
:meth:`AgentLoopMiddleware.with_judge` does not count toward this budget.
179-
180-
Sub-agent usage counts too: an agent exposed as a tool via :meth:`~agent_framework.Agent.as_tool`
181-
rolls its response usage (including its own sub-agents) into the calling agent's usage details,
182-
and a :class:`~agent_framework.BackgroundAgentsProvider` task's usage is charged once, to the
183-
parent run that first observes the task finished. Remaining gaps: a sub-agent run that fails or
184-
is cancelled produces no response, so its tokens are not counted; background tasks that are never
185-
observed as finished (including tasks abandoned when their session is released) are not counted;
186-
and custom tools that call ``agent.run`` directly do not report the nested run's usage.
187-
188-
This accounting is fixed and not configurable.
173+
"""Maximum token budget for the loop agent across all iterations.
174+
175+
This is a **thinking budget** — a uniform bound on total inference work — not
176+
a cost estimate. Tokens are treated as an interchangeable unit of reasoning work
177+
regardless of model, so usage from background sub-agents via
178+
:class:`~agent_framework.BackgroundAgentsProvider` rolls into the parent's
179+
running total. Applications that need to bound spend should implement their own
180+
model-specific cost accounting outside the framework.
181+
182+
Each iteration is charged the ``total_token_count`` from the response's usage
183+
details, falling back to ``input_token_count + output_token_count`` when
184+
``total_token_count`` is absent. The judge call from
185+
:meth:`AgentLoopMiddleware.with_judge` is never charged.
186+
187+
Known gaps: failed or cancelled sub-agent runs produce no response and are not
188+
counted; background tasks abandoned at session release are not counted; custom
189+
tools that call ``agent.run`` directly do not report nested usage.
189190
"""
190191

191192
max_duration: float | None
192193
"""Maximum wall-clock time in seconds for the entire loop run.
193194
194-
Measured with :func:`time.monotonic` and checked between iterations, after each
195-
iteration completes: once exceeded, no new iteration is started. It does not
196-
interrupt an in-flight model or tool call, so a single slow or hung call can run
197-
past the deadline. Callers that need a hard deadline should wrap the run in their
198-
own cancellation/timeout (for example ``asyncio.timeout``).
195+
Measured with :func:`time.monotonic`. The limit is enforced both between
196+
iterations and during in-flight model or tool calls: when time runs out
197+
mid-iteration, the call is cancelled (for asyncio callers). Callers should
198+
be prepared to handle :exc:`asyncio.CancelledError` or :exc:`TimeoutError`
199+
if the first iteration exceeds the limit and no partial result exists.
199200
"""
200201

201202

@@ -412,15 +413,15 @@ def __init__(
412413
persists server-side across iterations. Used by :meth:`with_judge` to tell the agent
413414
about the criteria its response must satisfy, but available to any loop.
414415
budget: Optional :class:`AgentBudget` (or a plain ``dict`` with the same keys) bounding total
415-
token spend and/or wall-clock time. ``max_tokens`` counts the wrapped agent's
416+
token budget (a thinking budget, not a cost estimate) and/or wall-clock
417+
time. ``max_tokens`` counts the wrapped agent's
416418
``total_token_count`` (falling back to input + output tokens) per iteration; the
417-
judge call made by :meth:`with_judge` is never counted, while sub-agent usage from
418-
``as_tool`` and background agents is included, with the gaps listed on
419-
:attr:`AgentBudget.max_tokens`. Not configurable. Checked after each iteration, before
420-
``should_continue`` is evaluated. When exceeded the loop stops and (non-streaming
421-
only) stamps ``loop_exit_reason``. Budgets do not interrupt an in-flight model/tool
422-
call; use ``asyncio.timeout`` (or similar) around the run for a hard wall-clock
423-
deadline.
419+
judge call made by :meth:`with_judge` is never counted, while
420+
:class:`~agent_framework.BackgroundAgentsProvider` usage is included, with the gaps
421+
listed on :attr:`AgentBudget.max_tokens`. Not configurable. Checked after each
422+
iteration, before ``should_continue`` is evaluated. When exceeded the loop stops and
423+
(non-streaming only) stamps ``loop_exit_reason``. ``max_duration`` also bounds the
424+
in-flight model/tool call of each iteration.
424425
425426
Raises:
426427
ValueError: If ``max_iterations`` is not ``None`` and is less than 1, or if ``budget``
@@ -515,8 +516,8 @@ def with_judge(
515516
baseline between iterations. See :meth:`__init__` for the full semantics. Defaults to
516517
``False``.
517518
budget: Optional :class:`AgentBudget`. See :meth:`__init__`. The judge call's own token
518-
usage is never charged against ``max_tokens``; the agent's sub-agent usage (``as_tool``
519-
and background agents) is, as described on :attr:`AgentBudget.max_tokens`.
519+
usage is never charged against ``max_tokens``; background-agent usage
520+
(:class:`~agent_framework.BackgroundAgentsProvider`) is, as described on :attr:`AgentBudget.max_tokens`.
520521
"""
521522
judge_instructions = (instructions or DEFAULT_JUDGE_INSTRUCTIONS).replace(
522523
CRITERIA_PLACEHOLDER, _render_criteria_block(criteria)
@@ -648,7 +649,25 @@ async def _process_non_streaming(
648649
context.options = stamped_options
649650
try:
650651
while True:
651-
await call_next()
652+
# Bound the call by the remaining wall-clock budget. If time has already run out,
653+
# stop before starting another iteration.
654+
if self._budget_max_duration is not None:
655+
remaining = self._budget_max_duration - (time.monotonic() - budget_start)
656+
if remaining <= 0:
657+
budget_exit = "time_budget_exceeded"
658+
break
659+
try:
660+
# wait_for (not asyncio.timeout) keeps Python 3.10 support.
661+
await asyncio.wait_for(call_next(), timeout=remaining)
662+
except asyncio.TimeoutError:
663+
# Iteration timed out mid-flight. With a prior result, stop cleanly;
664+
# otherwise re-raise so the caller knows nothing was produced.
665+
if final_result is None:
666+
raise
667+
budget_exit = "time_budget_exceeded"
668+
break
669+
else:
670+
await call_next()
652671
iteration += 1
653672

654673
result = context.result
@@ -663,7 +682,7 @@ async def _process_non_streaming(
663682
if result.usage_details is not None:
664683
aggregated_usage = add_usage_details(aggregated_usage, result.usage_details)
665684

666-
# Accumulate token spend for budget enforcement.
685+
# Accumulate token usage for budget enforcement.
667686
if self._budget_max_tokens is not None and result.usage_details is not None:
668687
ud = result.usage_details
669688
tokens_used += ud.get("total_token_count") or (
@@ -771,7 +790,20 @@ async def _generator() -> Any:
771790
try:
772791
while True:
773792
try:
774-
await call_next()
793+
if self._budget_max_duration is not None:
794+
remaining = self._budget_max_duration - (time.monotonic() - budget_start)
795+
if remaining <= 0:
796+
return
797+
try:
798+
# wait_for (not asyncio.timeout) keeps Python 3.10 support.
799+
await asyncio.wait_for(call_next(), timeout=remaining)
800+
except asyncio.TimeoutError:
801+
# Timed out mid-flight: stop cleanly with a prior result, else re-raise.
802+
if holder["final"] is None:
803+
raise
804+
return
805+
else:
806+
await call_next()
775807
inner = context.result
776808
if not isinstance(inner, ResponseStream):
777809
raise TypeError(

0 commit comments

Comments
 (0)