Repository navigation
Python: [Bug]: Background session release waits past its finite timeout during cancellation cleanup #9231
Description
Observed Behavior
BackgroundAgentsProvider.release_session(session, timeout=0.03) waits for a background agent to finish deferred cancellation cleanup instead of returning at the configured timeout. A real Agent.run with an offline BaseChatClient that finishes its cleanup after 0.35 seconds makes release take about 0.35–0.36 seconds. A cooperative cancellation control returns promptly with the same timeout.
The deferred-cleanup client catches repeated CancelledError while finishing a finite cleanup operation. The provider requests cancellation, reaches its timeout, then continues waiting for that task to settle. This extends the host's session teardown beyond its configured bound. The reproduction finishes on its own; it does not run a permanently hanging task.
Expected Behavior
A finite release timeout should bound the provider's wait for cancellation, with unfinished tasks handled using the existing abandonment/exception-observation policy. Cooperative task cancellation, timeout=None, and the cancel_running=False guard should retain their behavior.
Steps to Reproduce
- Run the code below with current
agent-framework-coresource. - It creates a real
Agentand starts itsrunthrough the public background-task tool, first with cooperative cancellation and then with finite deferred cleanup. - Both releases use
timeout=0.03; only the deferred-cleanup case waits until the 0.35-second cleanup finishes.
Minimal Reproduction
import asyncio
import time
from agent_framework import (
Agent, AgentSession, BackgroundAgentsProvider, BaseChatClient,
ChatResponse, Message, SessionContext,
)
class OfflineClient(BaseChatClient):
def __init__(self, defer_cleanup):
super().__init__()
self.defer_cleanup = defer_cleanup
self.started = asyncio.Event()
async def _inner_get_response(self, *, messages, options, **kwargs):
self.started.set()
finish_by = asyncio.get_running_loop().time() + 0.35
cancelled = False
while True:
try:
if cancelled:
await asyncio.sleep(max(0, finish_by - asyncio.get_running_loop().time()))
return ChatResponse(messages=[Message(role='assistant', contents=['cleaned up'])])
await asyncio.sleep(10)
except asyncio.CancelledError:
if not self.defer_cleanup:
raise
cancelled = True
async def _inner_get_streaming_response(self, *, messages, options, **kwargs):
raise RuntimeError('This reproduction uses nonstreaming Agent.run only.')
yield
async def reproduce(defer_cleanup):
client = OfflineClient(defer_cleanup)
agent = Agent(client=client, name='worker')
provider = BackgroundAgentsProvider([agent])
session = AgentSession()
context = SessionContext(session_id=session.session_id, input_messages=[])
await provider.before_run(agent=None, session=session, context=context, state={})
tools = {tool.name: tool for tool in context.tools}
await tools['background_agents_start_task'].invoke(
arguments={'agent_name': 'worker', 'input': 'start', 'description': 'cleanup'},
skip_parsing=True,
)
await client.started.wait()
start = time.monotonic()
await provider.release_session(session, timeout=0.03)
print({'deferred_cleanup': defer_cleanup, 'timeout': 0.03, 'elapsed': time.monotonic() - start})
async def main():
await reproduce(False)
await reproduce(True)
asyncio.run(main())Representative actual output on Windows/Python 3.14.3:
{'deferred_cleanup': False, 'timeout': 0.03, 'elapsed': 0.000076}
{'deferred_cleanup': True, 'timeout': 0.03, 'elapsed': 0.360132}
Error Messages and Stack Traces
No exception is raised by release_session. After waiting for cleanup to finish, it logs:
Session release timed out waiting for 0 task(s). They will be abandoned.
The experimental-provider and offline-client function-invocation warnings are retained; no model or network client is used.
Package Versions
Executed agent-framework-core source from current main 2d9cc3f8ae465baf1aed3022f89b6848db4ffa7b (pyproject.toml version 1.21.0). All 84 core-package Python files match that commit. The existing environment has cached agent-framework-core 1.20.0 dependency metadata; this is a source-based reproduction, without a fresh workspace-lock install.
Python Version
Python 3.14.3
Operating System
Windows
Regression
Unknown
Additional Context
The current _drain_runtime wraps asyncio.gather(*pending, return_exceptions=True) in asyncio.wait_for. Its timeout cancels the gather and then waits for cancellation to finish, so the catch/abandonment branch is not reached at the deadline when the background task defers cancellation cleanup.
This follows the existing release contract introduced by PratikWayase in #7450, addressing antsok's #7385. moonbox3's review asked for a bounded shutdown policy; the accepted implementation added the timeout. The Foundry cleanup guidance in eavanvalkenburg's merged #8899 also relies on the finite provider bound. This report concerns the local release wait; the distributed runtime/lease work in #8760 and karthik-0306's #9040 remains separate.
Additional public controls exercised cooperative cancellation, finite deferred cleanup, timeout=0, timeout=None, and rejecting a release with cancel_running=False while a task is running. Both a protocol-compatible offline agent and a real Agent.run reproduce the finite-timeout discrepancy. No full core suite, external model/service, other operating system, or performance validation was run. Please confirm the scoped direction of enforcing the existing finite release wait before implementation.
AI Assistance
AI-assisted analysis, reproduction, and writing.
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I personally verified this behavior and the reproduction details are authentic.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
github-actions commentedon Oct 9, 2026
🤖 Automated triage reproduction notes (agent-authored — trust but verify)
Agent analysis
Repro:
python/packages/core/agent_framework/_harness/_background_agents.py::BackgroundAgentsProvider._drain_runtimeat lines 405-458 exceeds a finite timeout when a tracked task repeatedly catches cancellation during finite cleanup. The public trigger isrelease_session(session, timeout=0.03)after starting such a task throughbackground_agents_start_task. Minimal repro: run the added test, which defers cleanup for 0.35 seconds and observes release returning after approximately 0.351 seconds.python/packages/core/tests/core/test_harness_background_agents.py::test_release_session_timeout_bounds_deferred_cancellation_cleanup1.21.01.21.0charan-rathore commentedon Oct 10, 2026
I reproduced this on current main (fd52de7) with the issue's script:
release_session(session, timeout=0.03)blocked 0.351s against the deferred-cleanup client, while the cooperative-cancellation control returned promptly. Cross-checked on Linux against the Windows/Python 3.14 report - same behavior.One scoping note from tracing
_drain_runtime: the never-settling task case is already bounded - what escapes the deadline is specifically a task that catchesCancelledErrorand resumes a finite cleanup.wait_for(gather(...))cancels at the deadline and then waits for the gather to settle, and a cancel-swallowing task keeps that gather pending until its cleanup completes, so the timeout stops being a bound. A deadline-based abandon (e.g.asyncio.waitwith the remaining budget, then dropping and observing the stragglers under the existing abandonment policy) would keep the bound without changing the cooperative,timeout=None, orcancel_running=Falsepaths.I see this is assigned to westey (@westey-m) - if there's no internal fix already in flight, I'd be glad to open a PR along those lines. Happy to hold off if you'd rather take it internally.
Instinct assisted with the reproduction and analysis.
eavanvalkenburg commentedon Oct 27, 2026
/plan Verify and scope enforcing the existing finite release_session wait bound when a background task defers or repeatedly catches cancellation during cleanup. Current main documents a bounded host wait but _drain_runtime uses wait_for(gather(...)), which can wait past that bound for cancellation settlement. Evaluate deadline-aware waiting using the existing abandonment and exception-observation policy, without promising forceful task termination. Preserve cooperative cancellation, timeout=None, timeout=0, cancel_running=False, late-task tracking, runtime/session isolation and observation of abandoned failures. Cover finite deferred cleanup and never-settling controls with deterministic offline tests. Coordinate with existing owner westey (@westey-m) and volunteer Charan Rathore (@charan-rathore) before implementation; this requests scope analysis, not a new assignment or implementation-ready status. The related task-identity work in #9441 is distinct.
westey-m commentedon Nov 3, 2026
/plan Verify and scope enforcing the existing finite release_session wait bound when a background task defers or repeatedly catches cancellation during cleanup. Current main documents a bounded host wait but _drain_runtime uses wait_for(gather(...)), which can wait past that bound for cancellation settlement. Evaluate deadline-aware waiting using the existing abandonment and exception-observation policy, without promising forceful task termination. Preserve cooperative cancellation, timeout=None, timeout=0, cancel_running=False, late-task tracking, runtime/session isolation and observation of abandoned failures. Cover finite deferred cleanup and never-settling controls with deterministic offline tests. Coordinate with existing owner westey (@westey-m) and volunteer Charan Rathore (@charan-rathore) before implementation; this requests scope analysis, not a new assignment or implementation-ready status. The related task-identity work in #9441 is distinct.
github-actions commentedon Nov 3, 2026
Summary
BackgroundAgentsProvider.release_session(session, timeout=...)is documented as a bounded wait, so a stuck task can't block host shutdown. The report is that a task which catchesCancelledErrorand then finishes a short cleanup makes release run past the timeout. In the issue's example the cleanup takes about 0.35s and a 0.03s timeout returns after about 0.36s. This is Python-only. I read the code and tests; I did not run anything. The reproduction is confirmed by two earlier comments (the triage bot's and charan-rathore's Linux run), not by me.What the code shows
release_sessionsays: "The default is bounded so a buggy task cannot wedge host eviction or shutdown."_drain_runtimecancels the pending tasks and then awaitsasyncio.wait_for(asyncio.gather(*pending, return_exceptions=True), timeout=remaining).wait_forcancels thegather. Agatherthat is cancelled stays pending until its children actually finish. A task that swallows the cancellation and keeps cleaning up therefore holds thegather, and thewait_foralong with it. Theexcept TimeoutErrorbranch, which logs and abandons, is not reached until cleanup ends. That matches the "timed out waiting for 0 task(s)" log line in the report.timeout=None, andcancel_running=False(which raises before any waiting) don't go through this path, so they are unaffected.Open question about the never-settling control. The existing test
test_release_session_times_out_if_task_ignores_cancellationmay not be a real never-settling control. Its task catches the first cancel and awaits again. The second cancel, whichwait_forsends viagatherat the deadline, then propagates out of theexceptblock and ends the task. So it likely passes today because the task dies on the second cancel. I have not run it. A genuine control must keep absorbing cancellations, for example in awhile Trueloop that catches everyCancelledError.Recommendation
Code change: warranted. This is a small, well-scoped framework defect. The code contradicts its documented contract and the behaviour the Foundry cleanup guidance relies on (the issue cites #8899). There is no workaround inside the framework. Callers can wrap
release_sessionin their ownasyncio.wait_for, but that doesn't clear the provider's state or apply the abandonment policy, so it isn't an adequate fix.Wait deadline-aware instead of cancelling a
gather. This is a proposal, not something I tested. The "no forceful termination" limit stays: a task that never stops cancelling is still only abandoned, as it is today.Proposed plan
_drain_runtime): in the finite-timeout case, replacewait_for(gather(...))withasyncio.wait(pending, timeout=remaining).asyncio.waitdoes not cancel anything on timeout, so the deadline is honoured whether or not tasks absorb cancellation. Keep the existing loop that catches late-added tasks and re-cancels them._log_abandoned_background_taskas a done-callback, the same as the current paths. Check that the callback is added exactly once per task. Repeated loop passes, or a task that was already given the callback on an earlier pass, must not double-register it.timeout=None: keep waiting without a deadline. Usingasyncio.waitwith no timeout there would also be fine, but the key requirement is that behaviour doesn't change. Await completion withgather(return_exceptions=True)(orasyncio.wait). Make sure exceptions from finished tasks are still retrieved.timeout=0: keep the existing early return before any wait. It should cancel, abandon, and attach the observer without yielding.cancel_running=Falseguard, runtime/session isolation, and the "don't pop a replacement runtime" behaviour all stay as they are. They live inrelease_session, outside the part that changes.release_session'sfinally. It already runs after abandonment today, so this should be unchanged.Tests (deterministic, offline, in
test_harness_background_agents.py)Use short timeouts and Events rather than real sleeps where possible, and assert upper bounds with generous margins.
release_session(timeout=small)returns near the deadline, before the Event is set. Then set the Event and check the task finishes and its result/exception is observed.CancelledErrorin a loop. Release returns at the deadline, the log reports it was abandoned, and the test cleans up the task afterwards.timeout=None,timeout=0,cancel_running=False, idempotence, isolation, and replacement runtime.Done when the issue's two-case reproduction returns at about the timeout in both cases, and the added tests pass alongside the existing release tests.
Coordination
@charan-rathoreoffered to open a PR. This comment is scope analysis only; it does not assign anyone or mark the issue ready for implementation. Please confirm who takes it.No changes have been made.
M-Hietala commentedon Nov 10, 2026
This should have been fixed by #9387
Charan Rathore (@charan-rathore) LI (@ktz03)