Skip to content

Commit 025c65e

Browse files
rng1995claude
andauthored
fix(analyzer): keep Python and Perl syntax from exhausting the shell parser (#778)
* fix(analyzer): keep Python and Perl syntax from exhausting the shell parser Valid host-language source could make has_bounded_parse_exhaustion report static_parse_limit. The file was then recorded as partially inspected and every SKILL.md reference to it became a HIGH AE1 finding. These triggers are new in 2.12.0: - Perl `eval BLOCK` (`eval { require $f; 1 } or die $@;`) reached the shell `eval` command-string reconstruction, and the dynamic `$f` operand made the reparsed string unresolvable. - In Python, a string, docstring or comment containing a Markdown fence, an unpaired backtick or an apostrophe opened a shell quote or command substitution that ran to the end of the view. Any file with more than 4096 characters after it was partial. - Python code that uses a shell wrapper name (`signal.alarm(timeout)`, `if (timeout > 0 ...`) was reparsed as a `timeout` command string, and a `$0` in a comment was treated as a runtime-selected command whose operands ran on into later code. Recognize Perl `eval BLOCK` in .pl files: a clause-initial `eval` followed by `{` is not a string reparse. `{` already starts a new clause, so the statements in the block are still scanned. String eval (`eval $code`, `eval "..."`, `eval qq{...}`) and shell `eval` are unchanged. For a complete .py module that ast.parse accepts (no non-Python shebang, no lone CR, within the AST size bound), compute string, f-string and comment token spans with tokenize and verify them against the source. Valid Python code outside those tokens contains no shell quote or expansion opener, so: - an unresolved shell word is charged only up to the first code-level word break after it, which ends the string or comment token that opened the quote, instead of the end of the view; - a command-string clause (eval, sh -c, wrappers) that begins in Python code is not reparsed as shell; - a runtime-selected command inside a comment is rechecked against that comment alone, because no shell ever receives a comment. Spans are computed lazily, only when one of these checks would otherwise fail closed, and they honor the runtime deadline. Fragments, malformed source, shell-shebang polyglots, a string or comment that itself exceeds the command-word budget, the destructive rm, brace-expansion and printf checks, runtime commands built from strings, and all shell and Markdown behavior keep their existing conservative results. Rule findings are unchanged. Refs #694 Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(analyzer): keep the Python ownership parse from re-emitting warnings The module parse that proves Python token ownership repeated every compiler warning the runner's own parse had already reported, such as an invalid `"\d"` escape. Ordinary skill code printed duplicate warnings, and under `-W error::SyntaxWarning` the parse raised and silently dropped ownership, so the file fell back to static_parse_limit. Run that parse with warnings ignored. `warnings.catch_warnings` swaps the process-global filter list, and analyzer nodes run on concurrent graph worker threads, where two such blocks that exit out of order leave their `ignore` filter installed for the whole process. Serialize the block with a module lock. Pin the existing result for a module that starts with a UTF-8 BOM: the file cache decodes with utf-8, not utf-8-sig, so a leading U+FEFF fails the parse and ownership stays unproven. That gap belongs with a separate decoding fix. Refs #694 Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Signed-off-by: Narendran Raghavan <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 5485cda commit 025c65e

3 files changed

Lines changed: 715 additions & 23 deletions

File tree

‎docs/ANALYSIS_RESOURCE_BOUNDS.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -149,6 +149,27 @@ legacy package separators, incomplete fragments, and real parser limits remain
149149
on the conservative analysis path. A complex helper may therefore still need
150150
its particular ledger reason and expression reviewed.
151151

152+
Perl `eval BLOCK` (for example `eval { require $module; 1 } or die $@;`) traps
153+
exceptions in already-compiled code and is not treated as a shell `eval` of a
154+
string. Its statements are still scanned. String forms such as `eval $code`
155+
and `eval "..."` remain on the conservative path.
156+
157+
### Python strings and comments
158+
159+
For a complete `.py` module that the Python parser accepts, each string literal
160+
(including f-strings) and comment owns its bytes. A shell quote or backtick that
161+
is left open inside one of them, such as a Markdown fence in a string or an
162+
apostrophe in a comment, is charged only up to the end of that token, never to
163+
the code that follows. Python code that happens to use a shell wrapper name, as
164+
in `signal.alarm(timeout)`, is not reparsed as a shell command string. A
165+
runtime-selected command named in a comment takes operands only from that
166+
comment.
167+
168+
Literal payloads remain visible to every security check. A single string or
169+
comment that itself holds more unresolved shell text than the parser bound,
170+
invalid or fragmentary Python, and source with a non-Python shebang keep the
171+
conservative result.
172+
152173
## Structured skill data
153174

154175
AISOP/AISP structured extraction consumes the already-bounded cache and shares the enclosing

0 commit comments

Comments
 (0)