Skip to content

Commit 2e08ad7

Browse files
authored
Python: Samples: terminate only on high/critical matches in ATR validation middleware (#9433)
ATRValidationMiddleware raised MiddlewareTermination on any ATR match, including low and medium severity. Those rules are broad enough to match ordinary tool arguments (for example `curl https://example.com` or `cat ~/.bashrc`), so the sample stopped legitimate tool calls. The middleware now terminates only when the highest-severity match is high or critical (BLOCKING_SEVERITIES). That is the severity at which ATR's reference verdict (determineOutcome in src/verdict.ts) is always deny or ask; "ask" means a person should confirm, and this sample has no confirmation step (approval_mode="never_require"), so both stop the call. Below high, the cut is a false-positive trade-off: ATR also returns ask for some medium matches, but the same rules match ordinary commands. Medium and low matches are logged as a warning with rule id, severity and confidence, and the tool runs. The docstring and README say that ATR is not a dangerous-command filter (`rm -rf ~` matches only a medium rule) and that production code should also require tool approval. detect_attack() now returns the highest-severity pyatr match instead of only its rule id, so the middleware can read severity and confidence. Also update the README row to describe this and drop the mention of a deny-list fallback that the sample does not have. Co-authored-by: eeee2345 <[email protected]>
1 parent 074e112 commit 2e08ad7

2 files changed

Lines changed: 54 additions & 17 deletions

File tree

‎python/samples/02-agents/middleware/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ This folder contains focused middleware samples for `Agent`, chat clients, tools
1111
| [`agent_loop_middleware_todos.py`](./agent_loop_middleware_todos.py) | Demonstrates `AgentLoopMiddleware` with a `should_continue` predicate built from a `TodoProvider` via `todos_remaining`, so the agent keeps working while open todos remain. |
1212
| [`agent_loop_middleware_judge.py`](./agent_loop_middleware_judge.py) | Demonstrates `AgentLoopMiddleware.with_judge`: a ChatClient judge re-runs the agent until it decides the original request was answered, with `criteria` shared between the agent and the judge. |
1313
| [`agent_loop_middleware_report.py`](./agent_loop_middleware_report.py) | Demonstrates composing two `AgentLoopMiddleware` on one agent: an inner `todos_remaining` loop that drafts a report todo-by-todo, wrapped by an outer report-style `with_judge` loop that re-runs it until an editor chat client judges the report publication-ready. |
14-
| [`atr_validation_middleware.py`](./atr_validation_middleware.py) | Demonstrates deterministic validation at the tool-execution boundary: a `FunctionMiddleware` that inspects the validated tool arguments and raises `MiddlewareTermination` before the tool runs when they match an attack rule. Loads the open, MIT-licensed Agent Threat Rules ruleset and runs the real engine locally (`pip install pyatr`), with a built-in deny-list fallback when it is not installed. |
14+
| [`atr_validation_middleware.py`](./atr_validation_middleware.py) | Demonstrates deterministic validation at the tool-execution boundary: a `FunctionMiddleware` that raises `MiddlewareTermination` before the tool runs when the arguments match a high- or critical-severity Agent Threat Rules rule, and logs lower-severity matches as warnings. It is not a dangerous-command filter, so use tool approval in production. Runs the open, MIT-licensed ruleset locally (`pip install pyatr`). |
1515
| [`chat_middleware.py`](./chat_middleware.py) | Shows class-based and function-based chat middleware that can observe, modify, and override model calls. |
1616
| [`class_based_middleware.py`](./class_based_middleware.py) | Shows class-based agent and function middleware. |
1717
| [`decorator_middleware.py`](./decorator_middleware.py) | Demonstrates middleware registration with decorators. |

‎python/samples/02-agents/middleware/atr_validation_middleware.py‎

Lines changed: 53 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,22 @@
3636
point that validates a tool call right before it executes. ATRValidationMiddleware is a
3737
FunctionMiddleware that inspects the validated tool arguments in
3838
``FunctionInvocationContext.arguments`` and raises ``MiddlewareTermination`` BEFORE calling
39-
``call_next()`` when the arguments match a known attack pattern, so the tool never runs.
39+
``call_next()`` when the arguments match a high- or critical-severity attack rule, so the tool
40+
never runs.
41+
42+
Why that threshold: ATR's reference verdict is "deny" or "ask" for every high- or
43+
critical-severity match, where "ask" means a person should confirm the call. This sample has no
44+
confirmation step (``approval_mode="never_require"``), so it treats both as stop. Below high, the
45+
cut is a false-positive trade-off, not ATR's verdict: ATR also returns "ask" for some
46+
medium-severity matches, but those rules also match ordinary commands such as
47+
``curl https://example.com``, so medium- and low-severity matches are logged as warnings and the
48+
tool runs.
49+
50+
Limits: ATR targets agent attacks such as prompt injection, tool-argument tampering, and
51+
exfiltration; it is not a filter for dangerous shell commands. ``rm -rf ~`` matches only a
52+
medium-severity rule, so it runs with a warning, and some destructive commands match no rule at
53+
all. This middleware does not replace tool approval (``approval_mode="always_require"``) for
54+
tools with side effects.
4055
4156
Detection is delegated to Agent Threat Rules (ATR) -- an open, MIT-licensed detection ruleset
4257
for AI-agent threats such as prompt injection, tool-argument tampering, and exfiltration. The
@@ -48,6 +63,10 @@
4863

4964
logger = logging.getLogger(__name__)
5065

66+
# ATR's verdict for a high or critical match is deny or ask; with no confirmation step here, both
67+
# terminate the run. Medium and low matches are logged and the tool runs.
68+
BLOCKING_SEVERITIES = frozenset({"critical", "high"})
69+
5170

5271
def _arguments_to_text(arguments: BaseModel | Mapping[str, Any]) -> str:
5372
"""Flatten tool arguments into a single string for scanning.
@@ -71,27 +90,29 @@ def _load_atr_engine() -> Any:
7190
return engine
7291

7392

74-
def detect_attack(arguments: BaseModel | Mapping[str, Any]) -> str | None:
75-
"""Return the matched ATR rule id, or None when the arguments look benign.
93+
def detect_attack(arguments: BaseModel | Mapping[str, Any]) -> Any | None:
94+
"""Return the highest-severity ATR match, or None when the arguments look benign.
7695
7796
Runs the real ATR engine over the flattened tool arguments. The text is evaluated as a
7897
``tool_call`` event so it is checked against the rules' ``tool_args`` conditions; ``evaluate``
79-
sorts matches critical-first, so the first rule id is the highest-severity hit.
98+
sorts matches critical-first, so the first match is the highest-severity hit. The returned
99+
``pyatr.ATRMatch`` carries ``rule_id``, ``severity`` and ``confidence``; the middleware decides
100+
whether to block.
80101
81102
The ruleset replaces a hand-rolled deny-list. For reference, the shape of the patterns ATR
82-
encodes (and that the earlier version of this sample inlined) is, e.g.::
103+
encodes is, e.g.::
83104
84105
ignore (previous|prior|above) instructions # instruction override / prompt injection
85106
send (secret|token|api_key|password) to http... # credential exfiltration
86107
(cat|read|open) (.env|id_rsa|/etc/passwd) # sensitive-file access
87108
88-
pyatr ships hundreds of such rules and keeps them maintained, so the sample stays a single
89-
straight-line call instead of a local regex list.
109+
pyatr ships and maintains these rules, so the sample stays a single straight-line call
110+
instead of a local regex list.
90111
"""
91112
text = _arguments_to_text(arguments)
92113
event = pyatr.AgentEvent(content=text, event_type="tool_call", fields={"tool_args": text})
93114
matches = _load_atr_engine().evaluate(event)
94-
return matches[0].rule_id if matches else None
115+
return matches[0] if matches else None
95116

96117

97118
# NOTE: approval_mode="never_require" is for sample brevity. Use "always_require" in production;
@@ -107,29 +128,45 @@ def get_weather(
107128

108129

109130
class ATRValidationMiddleware(FunctionMiddleware):
110-
"""Validates tool arguments at the execution boundary and blocks malicious calls.
131+
"""Validates tool arguments at the execution boundary and blocks high- or critical-severity matches.
111132
112-
The check is deterministic and runs before the tool executes: on a match it raises
133+
The check is deterministic and runs before the tool executes: on a match whose severity is in
134+
``BLOCKING_SEVERITIES`` (ATR verdict deny or ask, with no one here to confirm) it raises
113135
``MiddlewareTermination`` so ``call_next()`` is never reached and the tool does not fire.
136+
Medium- and low-severity matches are logged as warnings and the tool runs.
114137
"""
115138

116139
async def process(
117140
self,
118141
context: FunctionInvocationContext,
119142
call_next: Callable[[], Awaitable[None]],
120143
) -> None:
121-
matched = detect_attack(context.arguments)
122-
if matched is not None:
144+
match = detect_attack(context.arguments)
145+
if match is not None and match.severity in BLOCKING_SEVERITIES:
123146
logger.warning(
124-
"[ATRValidationMiddleware] Blocked tool '%s': arguments matched ATR rule %s.",
147+
"[ATRValidationMiddleware] Blocked tool '%s': arguments matched ATR rule %s (%s, %s confidence).",
125148
context.function.name,
126-
matched,
149+
match.rule_id,
150+
match.severity,
151+
match.confidence,
127152
)
128153
# Raise BEFORE call_next() so the tool is never executed. The matched rule id is
129154
# included for auditability.
130-
raise MiddlewareTermination(f"ATR validation blocked tool '{context.function.name}' (rule: {matched})")
155+
raise MiddlewareTermination(
156+
f"ATR validation blocked tool '{context.function.name}' (rule: {match.rule_id})"
157+
)
131158

132-
logger.info("[ATRValidationMiddleware] Tool '%s' passed ATR validation.", context.function.name)
159+
if match is not None:
160+
# Medium or low severity: surface the finding for review, but let the tool run.
161+
logger.warning(
162+
"[ATRValidationMiddleware] Tool '%s' matched ATR rule %s (%s, %s confidence); not blocking.",
163+
context.function.name,
164+
match.rule_id,
165+
match.severity,
166+
match.confidence,
167+
)
168+
else:
169+
logger.info("[ATRValidationMiddleware] Tool '%s' passed ATR validation.", context.function.name)
133170
await call_next()
134171

135172

0 commit comments

Comments
 (0)