Skip to content

Commit 028a5fa

Browse files
committed
fix: enforce the grep_file timeout, and stop stripping the pattern
Two defects in GrepFileTool, combined because the test for the second depends on the harness the first one forced. 1. The 30s timeout could not fire. The match ran in a ThreadPoolExecutor, but `re` does not release the GIL while matching. A catastrophic pattern therefore froze every thread in the interpreter, the event loop included, so asyncio.wait_for never got scheduled to time anything out. Reproduced with `(a+)+$` against 32 "a"s plus "!": a 0.5s timeout did not fire and an independent 50ms heartbeat stopped dead until the process was killed. The tool's error message ("pattern took too long to match") was therefore unreachable in exactly the case it described. The match now runs in a single-use worker process, which has its own GIL, so the loop stays responsive and the deadline is honored. Unlike a thread, the runaway can also be killed, and it is -- otherwise the deadline would bound only the caller's wait while the worker kept burning a core. A per-call pool rather than a shared one: a shared pool saves ~100ms of spawn per call but forces a timeout to recycle shared state, killing workers belonging to unrelated greps and surfacing BrokenProcessPool to callers that did nothing wrong. At ~100ms inside a multi-second model turn that trade is not worth making. 2. The pattern was stripped before use. args.get("pattern", "").strip() turned "foo " into "foo", which matches "foobar" -- text the caller's pattern excludes. Whitespace is meaningful in a regex. Trim only to decide emptiness and search with what was asked for, which is the split skills.go already makes, so the two runtimes no longer disagree on identical input. The existing timeout test faked slowness with time.sleep, which *releases* the GIL, so it passed while the real hazard went unguarded. It now runs a genuinely pathological match and asserts the event loop kept ticking. A second, fast test asserts the work leaves this process, because the behavioral test hangs rather than fails if the offload regresses to a thread and there is no pytest-timeout plugin here to cut it short. Tests that patched grep_content were rewritten against real files: a MagicMock cannot be pickled to a worker process, and the child imports the real module regardless. Reported by EItanya in review of #2267. Signed-off-by: brandonkeung <[email protected]>
1 parent 0b0bfe8 commit 028a5fa

2 files changed

Lines changed: 219 additions & 48 deletions

File tree

‎python/packages/kagent-adk/src/kagent/adk/tools/file_tools.py‎

Lines changed: 95 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
import concurrent.futures
1111
import functools
1212
import logging
13+
import multiprocessing
14+
from collections.abc import Callable
1315
from pathlib import Path
1416
from typing import Any, Dict
1517

@@ -32,6 +34,81 @@
3234
logger = logging.getLogger("kagent_adk." + __name__)
3335

3436

37+
def _kill_workers(executor: concurrent.futures.ProcessPoolExecutor) -> None:
38+
"""Terminate the pool's worker processes.
39+
40+
ProcessPoolExecutor has no public way to stop a worker that is mid-call:
41+
shutdown() waits for it rather than ending it, and cancelling the future
42+
does nothing to code that never checks for cancellation. So a runaway
43+
match has to be killed through the private _processes mapping.
44+
45+
If a future Python removes that attribute this must be *loud*. Silently
46+
skipping the kill would restore the original defect -- a pathological
47+
pattern spinning a core until the pod restarts -- with no signal at all.
48+
"""
49+
processes = getattr(executor, "_processes", None)
50+
if processes is None:
51+
logger.warning(
52+
"ProcessPoolExecutor._processes is unavailable; a timed-out grep "
53+
"worker cannot be killed and may spin until the process exits"
54+
)
55+
return
56+
for proc in list(processes.values()):
57+
try:
58+
proc.kill()
59+
except OSError:
60+
# ProcessLookupError when the worker already exited -- expected
61+
# on the success path. Deliberately narrow: an AttributeError
62+
# here would mean kill() changed shape, and that must surface
63+
# rather than be swallowed, same as a missing _processes above.
64+
logger.debug("grep_file: worker already gone", exc_info=True)
65+
66+
67+
async def _run_with_deadline(call: Callable[[], str], timeout_seconds: float) -> str:
68+
"""Run call in a worker process, abandoning and killing it past the deadline.
69+
70+
A process rather than a thread, and that is load-bearing rather than a
71+
tuning choice. `re` does not release the GIL while matching, so a
72+
catastrophic pattern run in a *thread* freezes every thread in the
73+
interpreter -- the event loop included. asyncio.wait_for then cannot fire,
74+
because the coroutine that would fire it never gets scheduled: the timeout
75+
silently becomes unenforceable exactly when it is needed. Reproduced with
76+
`(a+)+$` against 32 "a"s plus "!": a 0.5s timeout did not fire and an
77+
independent 50ms heartbeat stopped dead until the process was killed.
78+
79+
A worker process has its own GIL, so the loop stays responsive, the
80+
deadline is honored, and -- unlike a thread -- the runaway can be killed.
81+
82+
The pool is per-call and single-worker. A shared pool would save ~100ms of
83+
spawn per call, but then a timeout has to recycle shared state: killing
84+
workers belonging to unrelated in-flight greps and surfacing
85+
BrokenProcessPool to callers that did nothing wrong. At ~100ms on a tool
86+
call inside a multi-second model turn, that complexity is not worth
87+
buying; one call owning one process keeps the failure story trivial.
88+
89+
"spawn" rather than the Linux default "fork": this server is threaded, and
90+
forking a threaded process can deadlock a child that inherits a lock held
91+
by a thread that does not exist in it. Callers pass a callable defined in
92+
kagent.skills, so the spawned child imports only that light module, never
93+
this one's google.adk dependency chain.
94+
"""
95+
loop = asyncio.get_running_loop()
96+
executor = concurrent.futures.ProcessPoolExecutor(
97+
max_workers=1,
98+
mp_context=multiprocessing.get_context("spawn"),
99+
)
100+
try:
101+
return await asyncio.wait_for(loop.run_in_executor(executor, call), timeout=timeout_seconds)
102+
finally:
103+
# Unconditional, and not only for the timeout path: a pool worker stays
104+
# alive after finishing a task, so on success there is still an idle
105+
# process here. Killing it is how a single-use pool is reclaimed --
106+
# shutdown() below signals the worker but does not end one that is
107+
# mid-match, which is the case that matters.
108+
_kill_workers(executor)
109+
executor.shutdown(wait=False, cancel_futures=True)
110+
111+
35112
def _resolve_working_path(tool_context: ToolContext, path_str: str) -> tuple[Path, Path]:
36113
"""Resolve path_str relative to the session's working directory.
37114
@@ -189,23 +266,11 @@ class GrepFileTool(BaseTool):
189266

190267
# Bounds regex execution time: the pattern is agent-controlled, and Python's
191268
# backtracking `re` engine can take catastrophically long on adversarial
192-
# patterns (unlike Go's RE2-based regexp, which is linear-time). Note this
193-
# only bounds the *caller's* wait -- CPython can't forcibly stop a running
194-
# thread, so a pathological match keeps running in the background after
195-
# the timeout fires.
269+
# patterns (unlike Go's RE2-based regexp, which is linear-time). The match
270+
# runs in a worker process so this deadline can actually be enforced -- see
271+
# _run_with_deadline for why that is required rather than merely tidy.
196272
_TIMEOUT_SECONDS = 30
197273

198-
# A small dedicated pool, rather than asyncio's shared default executor,
199-
# so a hung or catastrophically slow match can only ever starve other
200-
# grep_file calls -- not unrelated to_thread-based work elsewhere in the
201-
# process (token counting, embeddings, other provider clients). Note
202-
# ThreadPoolExecutor workers are non-daemon threads that CPython's atexit
203-
# hook joins before the interpreter exits, so a permanently-stuck worker
204-
# (e.g. from a pathological pattern) also blocks a clean process shutdown,
205-
# not just steady-state grep_file availability -- bounded in practice by
206-
# Kubernetes' terminationGracePeriodSeconds before SIGKILL.
207-
_EXECUTOR = concurrent.futures.ThreadPoolExecutor(max_workers=4, thread_name_prefix="grep-file")
208-
209274
def __init__(self, skills_directory: str | Path):
210275
super().__init__(
211276
name="grep_file",
@@ -245,33 +310,33 @@ def _get_declaration(self) -> types.FunctionDeclaration:
245310

246311
async def run_async(self, *, args: Dict[str, Any], tool_context: ToolContext) -> str:
247312
"""Search a file or directory for a pattern."""
248-
pattern = args.get("pattern", "").strip()
313+
# Deliberately not stripped: whitespace is meaningful in a regex, and
314+
# trimming turns "foo " into "foo", which then matches text the
315+
# caller's pattern excludes. Trim only to decide emptiness, and search
316+
# with what was actually asked for -- the same split skills.go makes.
317+
pattern = args.get("pattern", "")
249318
path_str = args.get("path", "").strip()
250319
recursive = args.get("recursive", False)
251320
ignore_case = args.get("ignore_case", False)
252321

253-
if not pattern:
322+
if not pattern.strip():
254323
return "Error: No pattern provided"
255324
if not path_str:
256325
return "Error: No file path provided"
257326

258327
try:
259328
path, working_dir = _resolve_working_path(tool_context, path_str)
260329

261-
loop = asyncio.get_running_loop()
262-
return await asyncio.wait_for(
263-
loop.run_in_executor(
264-
self._EXECUTOR,
265-
functools.partial(
266-
grep_content,
267-
path,
268-
pattern,
269-
recursive=recursive,
270-
ignore_case=ignore_case,
271-
allowed_root=[working_dir, Path(self.skills_directory)],
272-
),
330+
return await _run_with_deadline(
331+
functools.partial(
332+
grep_content,
333+
path,
334+
pattern,
335+
recursive=recursive,
336+
ignore_case=ignore_case,
337+
allowed_root=[working_dir, Path(self.skills_directory)],
273338
),
274-
timeout=self._TIMEOUT_SECONDS,
339+
self._TIMEOUT_SECONDS,
275340
)
276341
except (TimeoutError, asyncio.TimeoutError):
277342
# asyncio.TimeoutError is TimeoutError on Python >=3.11, but this
Lines changed: 124 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,14 @@
1-
"""Tests for GrepFileTool's timeout protection against slow/catastrophic regex matching."""
1+
"""Tests for GrepFileTool: timeout protection, and fidelity of the search pattern."""
22

3+
import asyncio
4+
import os
5+
from pathlib import Path
36
from unittest.mock import patch
47

58
import pytest
6-
from kagent.skills import initialize_session_path
9+
from kagent.skills import get_session_path, initialize_session_path
710

8-
from kagent.adk.tools.file_tools import GrepFileTool
11+
from kagent.adk.tools.file_tools import GrepFileTool, _run_with_deadline
912

1013

1114
class MockSession:
@@ -18,45 +21,148 @@ def __init__(self, session_id: str = "test-session-grep-timeout"):
1821
self.session = MockSession(session_id)
1922

2023

24+
# A pattern/subject pair that makes Python's backtracking `re` engine explore
25+
# an astronomical number of ways to split the run of "a"s, none of which can
26+
# match because the line ends in "!". It does not finish in any practical time.
27+
CATASTROPHIC_PATTERN = r"(a+)+$"
28+
CATASTROPHIC_SUBJECT = "a" * 32 + "!"
29+
30+
31+
def _worker_pid() -> int:
32+
"""Module-level so it is picklable to a spawned worker."""
33+
return os.getpid()
34+
35+
36+
@pytest.mark.asyncio
37+
async def test_grep_work_runs_in_a_separate_process():
38+
"""Fast tripwire: the work must leave this process.
39+
40+
`re` holds the GIL while matching, so running a match in a *thread* makes
41+
run_async's timeout unenforceable -- the event loop cannot run to fire it.
42+
The behavioral test below proves that properly but *hangs* if this
43+
regresses, and there is no pytest-timeout plugin here to cut it short.
44+
This returns in about 100ms instead, so a regression names itself.
45+
46+
Asserting on the PID rather than the executor type states the property
47+
that actually matters, and stays true however the offload is implemented.
48+
"""
49+
assert await _run_with_deadline(_worker_pid, 30) != os.getpid()
50+
51+
2152
@pytest.mark.asyncio
22-
async def test_grep_file_tool_times_out_on_slow_match(tmp_path):
53+
async def test_grep_file_tool_times_out_on_catastrophic_backtracking(tmp_path):
54+
"""The timeout must hold against a match that never yields the GIL.
55+
56+
The previous version of this test faked slowness with `time.sleep`, which
57+
*releases* the GIL -- so it passed while the real hazard went unguarded.
58+
This one runs a genuinely pathological match, and additionally asserts the
59+
event loop kept running throughout: that is what a thread pool cannot do,
60+
and it is the difference between a timeout that fires and one that cannot.
61+
"""
2362
skills_dir = tmp_path / "skills"
2463
skills_dir.mkdir()
25-
session_id = "test-session-grep-timeout"
64+
session_id = "test-session-grep-catastrophic"
2665
initialize_session_path(session_id, str(skills_dir))
2766

67+
working_dir = Path(get_session_path(session_id=session_id))
68+
(working_dir / "evil.txt").write_text(CATASTROPHIC_SUBJECT + "\n")
69+
2870
tool = GrepFileTool(skills_directory=str(skills_dir))
29-
tool._TIMEOUT_SECONDS = 0.05
71+
# Comfortably above worker spawn cost, low enough to keep the test quick.
72+
tool._TIMEOUT_SECONDS = 3
3073

31-
def slow_grep(*args, **kwargs):
32-
import time
74+
ticks = 0
3375

34-
time.sleep(1)
35-
return "no matches found"
76+
async def heartbeat():
77+
nonlocal ticks
78+
while True:
79+
await asyncio.sleep(0.05)
80+
ticks += 1
3681

37-
with patch("kagent.adk.tools.file_tools.grep_content", side_effect=slow_grep):
82+
beat = asyncio.create_task(heartbeat())
83+
try:
3884
result = await tool.run_async(
39-
args={"pattern": "foo", "path": "."},
85+
args={"pattern": CATASTROPHIC_PATTERN, "path": "evil.txt"},
4086
tool_context=MockToolContext(session_id),
4187
)
88+
finally:
89+
beat.cancel()
4290

4391
assert "took too long" in result
92+
# If the match had run in a thread, the loop would have been frozen and
93+
# this would be 0 -- assuming the timeout fired at all.
94+
assert ticks > 5, f"event loop was starved during the match (ticks={ticks})"
95+
96+
97+
def _session_with_file(tmp_path, session_id: str, name: str, content: str):
98+
"""Build a session working dir containing one file, and return the tool.
99+
100+
Tests here use real files rather than patching grep_content: the search
101+
runs in a worker *process*, so a MagicMock cannot reach it (it is not
102+
picklable, and the child imports the real module regardless).
103+
"""
104+
skills_dir = tmp_path / "skills"
105+
skills_dir.mkdir()
106+
initialize_session_path(session_id, str(skills_dir))
107+
working_dir = Path(get_session_path(session_id=session_id))
108+
(working_dir / name).write_text(content)
109+
return GrepFileTool(skills_directory=str(skills_dir))
110+
111+
112+
@pytest.mark.asyncio
113+
async def test_grep_file_tool_returns_matches_for_a_normal_pattern(tmp_path):
114+
session_id = "test-session-grep-fast"
115+
tool = _session_with_file(tmp_path, session_id, "data.txt", "hello foo\nunrelated\n")
116+
117+
result = await tool.run_async(
118+
args={"pattern": "foo", "path": "data.txt"},
119+
tool_context=MockToolContext(session_id),
120+
)
121+
122+
assert "data.txt:1:hello foo" in result
123+
assert "unrelated" not in result
124+
125+
126+
@pytest.mark.asyncio
127+
async def test_grep_file_tool_passes_the_pattern_through_verbatim(tmp_path):
128+
"""Whitespace is meaningful in a regex, so the pattern must not be stripped.
129+
130+
Stripping turns `"foo "` into `"foo"`, which matches "foobar" -- a string
131+
the user's pattern explicitly excludes. The Go runtime trims only for its
132+
empty check and searches with the raw pattern (skills.go), so stripping
133+
here would also split the two runtimes on identical input.
134+
"""
135+
session_id = "test-session-grep-pattern"
136+
# "foobar" is the discriminator: the pattern "foo " must NOT match it,
137+
# while a stripped "foo" would. So the assertion fails loudly if the
138+
# trailing space is trimmed anywhere between here and re.search.
139+
tool = _session_with_file(tmp_path, session_id, "data.txt", "foobar\nfoo bar\n")
140+
141+
result = await tool.run_async(
142+
args={"pattern": "foo ", "path": "data.txt"},
143+
tool_context=MockToolContext(session_id),
144+
)
145+
146+
assert "data.txt:2:foo bar" in result
147+
# Line 1 is "foobar": a stripped "foo" would match it, the real pattern must not.
148+
assert ":1:" not in result
44149

45150

46151
@pytest.mark.asyncio
47-
async def test_grep_file_tool_returns_normally_when_fast(tmp_path):
152+
async def test_grep_file_tool_still_rejects_a_whitespace_only_pattern(tmp_path):
153+
"""Keeping the pattern raw must not weaken the empty-pattern guard."""
48154
skills_dir = tmp_path / "skills"
49155
skills_dir.mkdir()
50-
session_id = "test-session-grep-timeout-fast"
156+
session_id = "test-session-grep-blank"
51157
initialize_session_path(session_id, str(skills_dir))
52158

53159
tool = GrepFileTool(skills_directory=str(skills_dir))
54160

55-
with patch("kagent.adk.tools.file_tools.grep_content", return_value="match found") as mocked:
161+
with patch("kagent.adk.tools.file_tools.grep_content") as mocked:
56162
result = await tool.run_async(
57-
args={"pattern": "foo", "path": "."},
163+
args={"pattern": " ", "path": "."},
58164
tool_context=MockToolContext(session_id),
59165
)
60166

61-
assert result == "match found"
62-
assert mocked.called
167+
assert "No pattern provided" in result
168+
assert not mocked.called

0 commit comments

Comments
 (0)