Skip to content

Commit 5d370aa

Browse files
authored
fix(patterns): detect constructed sensitive paths (#545)
* fix(patterns): detect constructed sensitive paths Signed-off-by: Deepak Jain <[email protected]> * fix(patterns): preserve shared Python AST cache Signed-off-by: Deepak Jain <[email protected]> * fix(patterns): resolve constructed joins via AST with per-line PE3 dedup The PE3 constructed-path recognizer missed multiline and imported/aliased os.path.join spellings, and emitted a second PE3 for lines the pattern loop already covered. Resolve literal join calls from the Python AST (handling from/import aliases), and keep the best-confidence PE3 per line. Signed-off-by: Deepak Jain <[email protected]> * fix(patterns): gate constructed-path AST parse behind join pre-check The AST-based os.path.join resolver parsed every Python file, defeating the runner's parse-once invariant (one shared parse per file across the analyzer fan-out). Gate the local parse behind a cheap textual pre-check so files without a plausible join() call never pay for it. The module stays lexical, preserving windowed and normalized-view scans for all file types. Signed-off-by: Deepak Jain <[email protected]> * fix(patterns): alias-aware join gate, shared AST reuse, span dedup Address rng1995's review on constructed sensitive-path detection: - Broaden the join pre-check to cover imported local names: the gate now matches every name the file binds to os.path or os.path.join, so 'from os.path import join as j' followed by j('/etc', 'passwd') reaches alias resolution instead of returning early on the missing literal join(. - Route the supplemental analysis through the scan's shared AST: node() publishes the runner's python_ast_cache_key via a contextvar and _constructed_sensitive_paths consults it with the new peek_python_ast helper (exact-content hit only, never parses or writes). The module stays lexical so windowed and normalized views keep running; windowed view fragments are skipped and standalone callers keep one on-demand parse. - Deduplicate across the call's whole source span: hits now carry (start_line, end_line), so a join call wrapped over several lines no longer produces a second PE3 when the lexical loop fires on a wrapped argument line. Tests: extend test_pe3_aliased_join_imports_are_detected with the renamed import case, add test_pe3_multiline_constructed_join_is_deduplicated, and extend test_graph_scan_parses_python_once_before_parallel_analyzers with a literal os.path.join call to verify the graph still performs one parse. Signed-off-by: Deepak Jain <[email protected]> * fix(patterns): restore constructed-path findings on windowed views Above SECURITY_VIEW_WINDOW_CHARS the runner hands lexical modules window slices instead of the whole file, so peek_python_ast never hits the scan's whole-file cache entry and the constructed-path analysis silently dropped its findings (regression introduced in 56d121c; 40cea88 reported them). On a peek miss, parse the provided content directly behind the same plain textual join gate, restoring the 40cea88 behavior for window slices while keeping the 17:45 improvements for whole files: alias-aware gate, shared AST reuse, multiline span dedup, and scan-count discipline (fragments without a plausible join call never parse). Regression tests: - unit: a windowed fragment under a scan keeps PE3 coverage, and a join-free fragment triggers no parse - graph: a ~376k-char file keeps its constructed-path PE3 through the full scan Signed-off-by: Deepak Jain <[email protected]> * fix(patterns): recognize renamed join imports on windowed cache misses The windowed-fragment fallback gated on the plain join( spelling, so a fragment calling j( bound by from os.path import join as j never parsed and silently dropped its constructed-path PE3 (review P1). The parse gate now also fires on from os.path import spellings: the fragment is parsed to collect its import aliases, and the existing alias-aware join gate filters import-only fragments. Fragments without a plausible join call or join import still never parse. Regression tests: - graph: the ~376k-char large-file regression now spells the call through the renamed import with trailing comment padding - unit: a windowed fragment with from os.path import join as j keeps its PE3 coverage under a scan Signed-off-by: Deepak Jain <[email protected]> * fix(patterns): carry whole-file join aliases into windowed fragments A renamed os.path.join import (from os.path import join as j) and its call site can fall in different runner windows. The call's fragment carries no import aliases of its own and never spells join(, so the textual gate skipped it before alias resolution and the constructed sensitive path was silently dropped. Carry the whole file's import-alias map from the scan cache into the fragment analysis: peek the whole-file parse by filename (no content match), use its aliases for the join pre-check gate, the alias-aware gate, and call resolution. Fragments of the same scanned file share its import bindings, so the renamed call resolves to os.path.join in the later window. Standalone and exact-cache-hit paths are unchanged, and fragments without a plausible join call or join import still never parse. Adds two regression tests: an analyze-level fragment whose import lives in another window, and a full node() graph regression with import and call separated across real runner windows. Signed-off-by: deepujain <[email protected]> * fix(patterns): evaluate windowed join calls on the whole-file AST The windowed fallback parsed each fragment as a standalone module, so a slice starting mid-block (for example inside a function body) failed parsing and its constructed-path PE3 findings were silently dropped. Evaluate the supplemental constructed calls against the whole file's cached tree instead, reusing the existing cache peek: call spans map onto fragment-relative lines and only calls starting inside the fragment are owned by it, preserving full-file syntax, scope and import aliases as well as the runner's source-coordinate and dedupe behavior. Fragments that are not slices of the whole file (normalized views) keep the standalone-parse fallback. Add direct and graph regressions using the enclosing-function fixture: a mid-function window now reports exactly one PE3 at the call site. Signed-off-by: Deepak Jain <[email protected]> --------- Signed-off-by: Deepak Jain <[email protected]> Signed-off-by: deepujain <[email protected]>
1 parent a8a4e9a commit 5d370aa

4 files changed

Lines changed: 614 additions & 3 deletions

File tree

‎src/skillspector/nodes/analyzers/static_patterns_privilege_escalation.py‎

Lines changed: 231 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,21 @@
1717

1818
from __future__ import annotations
1919

20+
import ast
21+
import posixpath
2022
import re
2123
import sys
2224
from bisect import bisect_right
25+
from contextvars import ContextVar
2326

2427
from skillspector.logging_config import get_logger
2528
from skillspector.models import AnalyzerFinding, Location, Severity
29+
from skillspector.python_ast import (
30+
ParsedPythonFile,
31+
parse_python_source,
32+
peek_python_ast,
33+
peek_python_ast_any_content,
34+
)
2635
from skillspector.state import AnalyzerNodeResponse, SkillspectorState
2736

2837
from . import static_runner
@@ -34,13 +43,23 @@
3443
get_context,
3544
get_context_from_lines,
3645
get_line_number,
46+
resolve_call_name,
3747
)
3848
from .pattern_defaults import PatternCategory
3949

4050
logger = get_logger(__name__)
4151

4252
ANALYZER_ID = "static_patterns_privilege_escalation"
4353

54+
# Scan-scoped handle on the runner's shared Python AST cache, published by
55+
# node() for the analyze() calls the runner makes on its behalf. This module
56+
# stays lexical (no USES_PYTHON_AST opt-in) so its windowed and normalized
57+
# views keep running; the constructed-path analysis below consults the shared
58+
# cache through this key instead of reparsing.
59+
_scan_python_ast_cache_key: ContextVar[str | None] = ContextVar(
60+
"privilege_escalation_python_ast_cache_key", default=None
61+
)
62+
4463
PE1_CODE_PATTERNS = [
4564
(r"permissions?\s*:\s*\[?\s*['\"]?\*['\"]?\s*\]?", 0.8),
4665
(
@@ -659,7 +678,179 @@ def _is_qualified_benign_access_requirement(
659678
return heading_index >= 0 and lines[heading_index].strip() == "## Access Requirements"
660679

661680

662-
def analyze(content: str, file_path: str, file_type: str) -> list[AnalyzerFinding]:
681+
# Cheap pre-check before walking for constructed join calls: the AST walk
682+
# below is only worthwhile when the text plausibly contains a join() call.
683+
# The gate covers spelling aliases collected from the imports (``from
684+
# os.path import join as j`` binds ``j`` to ``os.path.join``, so the call
685+
# site reads ``j(`` with no literal ``join`` in sight). Keeping the module
686+
# lexical plus this alias-aware gate preserves the runner's parse-once
687+
# invariant for files without any join call.
688+
_JOIN_CALL_BASE_NAMES = {"join"}
689+
_JOIN_CALL_ALIAS_TARGETS = {"os.path", "os.path.join"}
690+
691+
# Lexical pre-check for a windowed fragment that binds a renamed ``join``
692+
# without ever spelling ``join(`` (``from os.path import join as j`` calls
693+
# ``j(``). On a whole-file cache miss the fragment is parsed to collect its
694+
# import aliases before the alias-aware join gate below runs, so such
695+
# fragments must also clear the parse gate; import-only fragments are
696+
# filtered out again by the alias-aware gate.
697+
_JOIN_IMPORT_HINT = re.compile(r"^\s*from\s+os\.path\s+import\b", re.MULTILINE)
698+
699+
700+
def _join_call_hint(aliases: dict[str, str]) -> re.Pattern[str]:
701+
"""Return a pre-check pattern matching ``join(`` and imported join aliases.
702+
703+
Every local name the file binds to ``os.path`` or ``os.path.join`` is a
704+
possible call spelling (``j(`` for ``from os.path import join as j``),
705+
alongside the plain ``join(`` used by ``os.path.join(``, ``p.join(``,
706+
and direct ``join(`` imports.
707+
"""
708+
names = set(_JOIN_CALL_BASE_NAMES)
709+
for local, qualified in aliases.items():
710+
if qualified in _JOIN_CALL_ALIAS_TARGETS:
711+
names.add(local)
712+
return re.compile(r"\b(?:" + "|".join(sorted(re.escape(name) for name in names)) + r")\s*\(")
713+
714+
715+
def _fragment_whole_file_line_offset(fragment: str, whole_content: str) -> int | None:
716+
"""Return how many whole-file lines precede a windowed view fragment.
717+
718+
Windowed view fragments are contiguous slices of the scanned file, so a
719+
whole-file AST line number maps onto a fragment-relative line by
720+
subtracting this offset. Return ``None`` when the fragment is not a
721+
slice of the whole file (for example a normalized view), so callers keep
722+
the standalone-parse fallback.
723+
"""
724+
start = whole_content.find(fragment)
725+
if start < 0:
726+
return None
727+
return whole_content.count("\n", 0, start)
728+
729+
730+
def _constructed_sensitive_paths(
731+
content: str,
732+
file_path: str,
733+
python_ast: ParsedPythonFile | None = None,
734+
) -> list[tuple[int, int, str, float]]:
735+
"""Return literal sensitive paths assembled with ``os.path.join`` in Python.
736+
737+
Each hit is a ``(start_line, end_line, path, confidence)`` tuple anchored
738+
to the call's source span so callers can deduplicate raw findings across
739+
the whole occurrence, including calls wrapped over several lines.
740+
Resolved from the Python AST so calls split across lines and supported
741+
import spellings (``import os.path as p``, ``from os.path import join``,
742+
``from os.path import join as j``, ``from os import path``) are recognized
743+
without reparsing tricks. The scan's shared parse is reused whenever this
744+
runs inside the runner (the cache key published by node()); standalone
745+
callers get a single on-demand parse. Windowed view fragments under a scan
746+
miss the whole-file cache entry; behind the same textual join-or-import
747+
gate they are evaluated against the whole file's cached tree instead of
748+
being parsed standalone, so a fragment starting mid-block (for example
749+
inside a function body, which is not a valid module on its own) keeps its
750+
findings instead of silently dropping them. The whole file's import-alias
751+
map is carried in from the scan cache for the join gate and call
752+
resolution, so a renamed ``os.path.join`` spelling (``from os.path import
753+
join as j``) is recognized even when the import lives in an earlier window
754+
than the call. Call spans map onto fragment-relative lines and only
755+
calls starting inside the fragment are owned by it, preserving the
756+
runner's source-coordinate and dedupe behavior. When the whole-file tree
757+
is unavailable, or the fragment is not a slice of the whole file
758+
(normalized views), the fragment is parsed directly as before. Only
759+
fully-literal positional argument lists are resolved; anything dynamic is
760+
left to the existing pattern loop. Unparseable content simply yields no
761+
findings here.
762+
"""
763+
whole_file_aliases: dict[str, str] | None = None
764+
whole_file_tree: ast.Module | None = None
765+
whole_file_source: str | None = None
766+
whole_file_line_offset: int | None = None
767+
if python_ast is None:
768+
cache_key = _scan_python_ast_cache_key.get()
769+
if cache_key is not None:
770+
python_ast = peek_python_ast(cache_key, content, file_path)
771+
if python_ast is None:
772+
# A windowed view fragment, not the scan's whole file: the
773+
# shared tree does not cover this slice. Carry the whole
774+
# file's import-alias map and tree from the scan cache first
775+
# (a fragment is a slice of the same scanned file), so the
776+
# textual gate below also fires on renamed call spellings
777+
# like ``j(`` when the ``from os.path import join as j``
778+
# import lives in an earlier window. Fragments without a
779+
# plausible join call or join import never parse.
780+
whole_file = peek_python_ast_any_content(cache_key, file_path)
781+
if whole_file is not None and whole_file.tree is not None:
782+
whole_file_aliases = whole_file.import_aliases
783+
whole_file_tree = whole_file.tree
784+
whole_file_source = whole_file.content
785+
if _join_call_hint(whole_file_aliases or {}).search(
786+
content
787+
) or _JOIN_IMPORT_HINT.search(content):
788+
if whole_file_tree is not None and whole_file_source is not None:
789+
# Evaluate the whole-file tree rather than parsing the
790+
# fragment standalone: a slice starting mid-block is
791+
# not a valid module, so its standalone parse fails
792+
# and the finding would be silently dropped.
793+
whole_file_line_offset = _fragment_whole_file_line_offset(
794+
content, whole_file_source
795+
)
796+
if whole_file_line_offset is None:
797+
python_ast = parse_python_source(content, file_path)
798+
else:
799+
python_ast = parse_python_source(content, file_path)
800+
if whole_file_tree is not None and whole_file_line_offset is not None:
801+
tree = whole_file_tree
802+
aliases = whole_file_aliases or {}
803+
owned_span: tuple[int, int] | None = (
804+
whole_file_line_offset,
805+
whole_file_line_offset + len(content.splitlines()),
806+
)
807+
else:
808+
if python_ast is None:
809+
return []
810+
tree = python_ast.tree
811+
if tree is None:
812+
return []
813+
aliases = (
814+
whole_file_aliases if whole_file_aliases is not None else python_ast.import_aliases
815+
)
816+
owned_span = None
817+
if not _join_call_hint(aliases).search(content):
818+
return []
819+
resolved: list[tuple[int, int, str, float]] = []
820+
for node in ast.walk(tree):
821+
if not isinstance(node, ast.Call):
822+
continue
823+
if resolve_call_name(node, aliases) != "os.path.join":
824+
continue
825+
if node.keywords or len(node.args) < 2:
826+
continue
827+
parts = [arg.value for arg in node.args if isinstance(arg, ast.Constant)]
828+
if len(parts) != len(node.args) or not all(isinstance(part, str) for part in parts):
829+
continue
830+
value = posixpath.join(*parts)
831+
start_line = node.lineno
832+
end_line = node.end_lineno or node.lineno
833+
if owned_span is not None:
834+
# Only calls starting inside this fragment are owned by it; the
835+
# runner's own owned-range filter and cross-window dedupe keep
836+
# exactly one finding per call at whole-file coordinates.
837+
if not owned_span[0] < start_line <= owned_span[1]:
838+
continue
839+
start_line -= owned_span[0]
840+
end_line -= owned_span[0]
841+
for pattern, confidence in PE3_PATTERNS:
842+
if re.search(pattern, value, re.IGNORECASE):
843+
resolved.append((start_line, end_line, value, confidence))
844+
break
845+
return resolved
846+
847+
848+
def analyze(
849+
content: str,
850+
file_path: str,
851+
file_type: str,
852+
python_ast: ParsedPythonFile | None = None,
853+
) -> list[AnalyzerFinding]:
663854
"""Analyze content for privilege escalation patterns (PE1–PE5)."""
664855
findings: list[AnalyzerFinding] = []
665856
line_starts, line_ends = _source_line_metadata(content)
@@ -808,6 +999,40 @@ def context_at(offset: int) -> str:
808999
complete_match=match.group(0),
8091000
)
8101001
)
1002+
if file_type == "python":
1003+
for start_line, end_line, path, confidence in _constructed_sensitive_paths(
1004+
content, file_path, python_ast
1005+
):
1006+
constructed = AnalyzerFinding(
1007+
rule_id="PE3",
1008+
message="Credential Access",
1009+
severity=Severity.HIGH,
1010+
location=loc(start_line),
1011+
confidence=confidence,
1012+
tags=list(tag),
1013+
context=get_context(content, line_starts[start_line - 1]),
1014+
matched_text=path,
1015+
)
1016+
# One PE3 per source occurrence: the pattern loop above may already
1017+
# have fired inside the join call's line span (for example the
1018+
# literal '.ssh/id_rsa' on a wrapped argument line, while this
1019+
# finding anchors to the call's first line). Keep the
1020+
# best-confidence finding per call span, mirroring the PE4/PE5
1021+
# per-line aggregation below; unrelated occurrences on other
1022+
# lines are preserved.
1023+
duplicate = next(
1024+
(
1025+
existing
1026+
for existing in findings
1027+
if existing.rule_id == "PE3"
1028+
and start_line <= existing.location.start_line <= end_line
1029+
),
1030+
None,
1031+
)
1032+
if duplicate is None:
1033+
findings.append(constructed)
1034+
elif confidence > duplicate.confidence:
1035+
findings[findings.index(duplicate)] = constructed
8111036
# Collect best-confidence PE4 finding per line to avoid double-counting lines
8121037
# that match multiple patterns (e.g. DockerClient(base_url=".../docker.sock")).
8131038
pe4_best: dict[int, AnalyzerFinding] = {}
@@ -955,6 +1180,10 @@ def _is_negated_safety_constraint(
9551180

9561181
def node(state: SkillspectorState) -> AnalyzerNodeResponse:
9571182
"""Run privilege_escalation patterns and return findings."""
958-
response = static_runner.run_static_patterns_with_ledger(state, [sys.modules[__name__]])
1183+
token = _scan_python_ast_cache_key.set(state.get("python_ast_cache_key"))
1184+
try:
1185+
response = static_runner.run_static_patterns_with_ledger(state, [sys.modules[__name__]])
1186+
finally:
1187+
_scan_python_ast_cache_key.reset(token)
9591188
logger.info("%s: %d findings", ANALYZER_ID, len(response["findings"]))
9601189
return response

‎src/skillspector/python_ast.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -337,6 +337,46 @@ def get_python_ast(cache_key: str | None, content: str, filename: str) -> Parsed
337337
return parsed
338338

339339

340+
def peek_python_ast(cache_key: str | None, content: str, filename: str) -> ParsedPythonFile | None:
341+
"""Return the scan's cached parse for *filename* without parsing anything.
342+
343+
Only an exact content match hits. Anything else (a windowed view fragment,
344+
an uncached file, an unknown cache key) returns ``None`` so the caller can
345+
apply its own fallback without disturbing the shared cache: unlike
346+
:func:`get_python_ast`, this never parses and never stores.
347+
"""
348+
if cache_key is None:
349+
return None
350+
with _runtime_ast_cache_lock:
351+
cache = _runtime_ast_caches.get(cache_key)
352+
if cache is None:
353+
return None
354+
cached = cache.entries.get(filename)
355+
if cached is not None and cached.content == content:
356+
cache.entries.move_to_end(filename)
357+
return cached
358+
return None
359+
360+
361+
def peek_python_ast_any_content(cache_key: str | None, filename: str) -> ParsedPythonFile | None:
362+
"""Return the scan's cached parse for *filename* regardless of content.
363+
364+
Windowed view fragments never content-match the whole-file entry, but a
365+
fragment is a slice of the same scanned file, so its import-alias map is
366+
still valid for resolving calls that the fragment's own imports cannot
367+
explain (for example a renamed ``os.path.join`` import living in an
368+
earlier window). Like :func:`peek_python_ast`, this never parses and
369+
never stores.
370+
"""
371+
if cache_key is None:
372+
return None
373+
with _runtime_ast_cache_lock:
374+
cache = _runtime_ast_caches.get(cache_key)
375+
if cache is None:
376+
return None
377+
return cache.entries.get(filename)
378+
379+
340380
def clear_python_ast_cache(cache_key: str | None) -> None:
341381
"""Release one scan's process-local parsed trees after its analyzer phase."""
342382
if cache_key is None:

‎tests/nodes/analyzers/test_shared_python_ast.py‎

Lines changed: 39 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
behavioral_taint_tracking,
1717
static_patterns_data_exfiltration,
1818
static_patterns_output_handling,
19+
static_runner,
1920
)
2021
from skillspector.nodes.build_context import build_context
2122
from skillspector.nodes.deduplicate import deduplicate
@@ -128,12 +129,18 @@ def count_parse(*args, **kwargs):
128129

129130

130131
def test_graph_scan_parses_python_once_before_parallel_analyzers(tmp_path, monkeypatch) -> None:
131-
"""The runtime cache shares one parse across the graph's analyzer fan-out."""
132+
"""The runtime cache shares one parse across the graph's analyzer fan-out.
133+
134+
The fixture includes a literal ``os.path.join`` call so the test proves
135+
the supplemental constructed-path analysis reuses the shared parse rather
136+
than parsing again.
137+
"""
132138
(tmp_path / "script.py").write_text(
133139
"import os\n"
134140
"import subprocess\n"
135141
"payload = input()\n"
136142
"environment = os.environ.copy()\n"
143+
"credential = os.path.join('/etc', 'passwd')\n"
137144
"subprocess.run(output)\n"
138145
"exec(payload)\n",
139146
encoding="utf-8",
@@ -151,5 +158,36 @@ def count_parse(*args, **kwargs):
151158
result = graph.invoke({"skill_path": str(tmp_path), "use_llm": False})
152159

153160
assert {"E2", "OH1", "AST1", "TT5"} <= {finding.rule_id for finding in result["findings"]}
161+
assert any(
162+
finding.rule_id == "PE3" and finding.matched_text == "/etc/passwd"
163+
for finding in result["findings"]
164+
)
154165
assert parse_calls == 1
155166
assert JsonPlusSerializer().dumps_typed(result)
167+
168+
169+
def test_graph_scan_reports_constructed_path_above_view_window_chars(tmp_path) -> None:
170+
"""Windowed lexical scans keep constructed-path PE3 above the view window.
171+
172+
Regression test: routing the constructed-path analysis through
173+
``peek_python_ast`` dropped findings once the runner sliced content into
174+
window views (above ``SECURITY_VIEW_WINDOW_CHARS``), because a slice never
175+
matches the scan's whole-file cache entry. The fragment fallback parses
176+
the slice directly so large files keep their findings. The source spells
177+
the call through a renamed import (``from os.path import join as j``) to
178+
pin that the cache-miss fallback recognizes renamed join spellings: the
179+
plain ``join(`` textual gate never fires on the ``j(`` call site.
180+
"""
181+
filler_line = "# " + "x" * 118 + "\n"
182+
body = "from os.path import join as j\ncredential = j('/etc', 'passwd')\n"
183+
target_chars = static_runner.SECURITY_VIEW_WINDOW_CHARS + 120_000
184+
source = body + filler_line * ((target_chars - len(body)) // len(filler_line))
185+
assert len(source) > static_runner.SECURITY_VIEW_WINDOW_CHARS
186+
(tmp_path / "large_script.py").write_text(source, encoding="utf-8")
187+
188+
result = graph.invoke({"skill_path": str(tmp_path), "use_llm": False})
189+
190+
assert any(
191+
finding.rule_id == "PE3" and finding.matched_text == "/etc/passwd"
192+
for finding in result["findings"]
193+
)

0 commit comments

Comments
 (0)