Skip to content

Commit dd95368

Browse files
chrisknvidiarng1995claude
authored
fix(analyzer): stabilize TM1 window identity (#578)
* fix(analyzer): stabilize TM1 window identity Signed-off-by: Christopher Kevin <[email protected]> * fix(analyzer): track eager receiver side effects Signed-off-by: Christopher Kevin <[email protected]> * fix(analyzer): preserve lexical TM1 when dataflow abstains Signed-off-by: Christopher Kevin <[email protected]> * fix(analyzer): require a retained companion owner Signed-off-by: Christopher Kevin <[email protected]> * fix(analyzer): preserve execution scope in binding evidence Signed-off-by: Christopher Kevin <[email protected]> * style(analyzer): simplify binding event deduplication Signed-off-by: Christopher Kevin <[email protected]> * fix: preserve runtime and retained shell evidence Signed-off-by: Christopher Kevin <[email protected]> * fix(analyzer): preserve called-slot identity and cached method evidence Signed-off-by: Christopher Kevin <[email protected]> * fix: preserve cached subprocess slot state across scoped imports Signed-off-by: Christopher Kevin <[email protected]> * fix(analyzer): reconcile explicit cached-slot evidence across scopes Signed-off-by: Christopher Kevin <[email protected]> * fix: abstain from cached slot proof after eager effects Signed-off-by: Christopher Kevin <[email protected]> * test: retain lexical findings after unknown cached-slot effects Signed-off-by: Christopher Kevin <[email protected]> * fix: bound direct called-slot evidence to known execution order Signed-off-by: Christopher Kevin <[email protected]> * fix: limit direct slot proof to single assignment targets Signed-off-by: Christopher Kevin <[email protected]> * fix: invalidate slot evidence on protocols and unsafe releases Signed-off-by: Christopher Kevin <[email protected]> * fix: keep legacy shadow proof isolated from later effects Signed-off-by: Christopher Kevin <[email protected]> * fix: abstain from cached replacement proof after unknown effects Signed-off-by: Christopher Kevin <[email protected]> * fix: retain native callable detection across cached slot stores Signed-off-by: Christopher Kevin <[email protected]> * fix: retain shell findings for native receiver assignments Signed-off-by: Christopher Kevin <[email protected]> * fix: preserve native Popen replacement warnings Signed-off-by: Christopher Kevin <[email protected]> * fix(analyzer): require proven replacement before revoking lexical TM1 Address the three remaining review findings on lexical TM1 reconciliation, where a bound shell=True call still ran natively but lost its HIGH finding: - Untaken expression stores: a walrus in a short-circuited BoolOp operand, an untaken conditional-expression arm, or a comprehension/generator body is no longer recorded as replacement evidence. Only constant operands and tests prove that the store executed. - Identity-preserving stores: unpacked targets are paired with their RHS elements. Self-stores are transparent, truthy constants keep the flag, and native receiver aliases (saved = subprocess, import subprocess as saved, Popen = subprocess.Popen) keep the native receiver. - Deferred bodies: an outer store only proves replacement when every invocation the owner scope can reach sees it. Any reference to the function, method or class (compound statements, main guards, callbacks, other bodies) can invoke it from that point on, and decorators or lambdas can run once defined. This also covers a re-import before a later invocation. The release-trigger test for a called function now matches its module-level twin: the companion abstains and the lexical HIGH remains. Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> * test: account for TM1 Python parsing in marker ownership ledger Main's #784 test expects the tool-misuse ledger event for an invalid Python file to carry obfuscated_instruction_text. This PR makes tool misuse parse Python for TM1 reconciliation, so the runner's higher-precedence syntax_error now names that partial event. Keep the fail-closed contract: assert the marker reading through the lexical-only anti-refusal analyzer and keep tool misuse partial with syntax_error. Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(analyzer): keep one TM1 owner for true-prefixed names without AST proof Port the six true-prefixed TM1 regression tests from #577. This branch already carries test_true_direct_calls_on_one_line_keep_distinct_locations as test_true_direct_calls_around_safe_comprehension_keep_distinct_locations, so only the other five are added. Two of them reported the same call twice on this branch: - A variable window for a name spelled `true` (in any case) duplicated the case-insensitive direct `shell=True` owner at the same call. AST reconciliation removes that window only when the dataflow companion takes the call, so a file that does not parse, or a call after an unknown receiver effect, kept both findings. coalesce_path_findings now drops such a window when a direct lexical owner starts at the same call. A window for a call whose direct candidate was folded into an identical same-line call still reports that call. - A call-anchored window for a longer true-prefixed name (`true_value`) used its raw text as identity, so the raw and normalized security views of one call did not collapse, and reconciliation then moved both to the call. The window now takes its identity from the normalized view text, as the direct owner already does. Also cover both root causes with a test for a nested `true` call after an unknown receiver effect and a normalized `true_value` call in a file that does not parse. Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> * fix(analyzer): report each true-prefixed shell call that #577 reports The `\b`-bounded direct pattern does not match `shell=true_value`, so on this branch such a call was reported only through a variable window. That window pairs an assignment with a single call, skips an assignment inside an earlier window, and was dropped outside Python. Compared with #577 this lost: - calls in JavaScript and Markdown files, fenced or plain; - a later call that the AST companion abstains on, for example after an unknown receiver effect; - all but the last same-line call in a file that does not parse, and the second of two identical same-line `shell=true` calls outside Python, which the direct candidate key folds into one; - a call whose assignment sits inside the window of an earlier one. _variable_shell_matches now pairs every assignment of a true-prefixed name with each call its bounded window reaches, and the nearest assignment owns each call. Candidate generation, analysis and AST reconciliation share that list, and a window's candidate identity uses its full text, so windows to different calls do not collapse on a shared 200-character preview. Outside Python, call-anchored windows for true-prefixed names are kept, and coalescing still drops one when a direct owner reports the same call. With an AST, each added window passes the same visibility, counterevidence, cached replacement and companion checks as before. Windows for other names are unchanged. Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --------- Signed-off-by: Christopher Kevin <[email protected]> Signed-off-by: Christopher Kevin <[email protected]> Signed-off-by: Narendran Raghavan <[email protected]> Co-authored-by: Christopher Kevin <[email protected]> Co-authored-by: Narendran Raghavan <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 8f4ca7d commit dd95368

8 files changed

Lines changed: 4087 additions & 493 deletions

‎src/skillspector/artifacts.py‎

Lines changed: 105 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,8 @@ class SecurityTextView:
9999
text: str
100100
source_offsets: array[int] | None = None
101101
reconstructions: tuple[SecurityTextReconstruction, ...] = ()
102+
right_boundary_is_fixed: bool = False
103+
right_boundary_recovery_start: int | None = None
102104

103105
def source_offset(self, derived_offset: int) -> int:
104106
"""Map a derived character offset to the corresponding source offset."""
@@ -1679,6 +1681,30 @@ def _contextual_default_ignorable_spans(
16791681
yield span_start, end
16801682

16811683

1684+
def _normalization_ignored_spans_in_gap(
1685+
text: str,
1686+
gap_start: int,
1687+
gap_end: int,
1688+
) -> Iterator[tuple[int, int]]:
1689+
"""Yield the normalized-view removals inside one contextual gap."""
1690+
for match in _DEFAULT_IGNORABLE_RUN_PATTERN.finditer(text, gap_start, gap_end):
1691+
start, end = match.span()
1692+
ignored_start = (
1693+
start
1694+
if _is_unconditionally_ignored(text[start])
1695+
or _is_contextual_default_ignorable_offset(text, start)
1696+
else start + 1
1697+
)
1698+
ignored_end = (
1699+
end
1700+
if _is_unconditionally_ignored(text[end - 1])
1701+
or _is_contextual_default_ignorable_offset(text, end - 1)
1702+
else end - 1
1703+
)
1704+
if ignored_start < ignored_end:
1705+
yield ignored_start, ignored_end
1706+
1707+
16821708
def _normalization_ignored_spans(
16831709
text: str,
16841710
check_runtime: Callable[[], None] | None = None,
@@ -1689,22 +1715,7 @@ def _normalization_ignored_spans(
16891715
require_word_boundaries=False,
16901716
check_runtime=check_runtime,
16911717
):
1692-
for match in _DEFAULT_IGNORABLE_RUN_PATTERN.finditer(text, gap_start, gap_end):
1693-
start, end = match.span()
1694-
ignored_start = (
1695-
start
1696-
if _is_unconditionally_ignored(text[start])
1697-
or _is_contextual_default_ignorable_offset(text, start)
1698-
else start + 1
1699-
)
1700-
ignored_end = (
1701-
end
1702-
if _is_unconditionally_ignored(text[end - 1])
1703-
or _is_contextual_default_ignorable_offset(text, end - 1)
1704-
else end - 1
1705-
)
1706-
if ignored_start < ignored_end:
1707-
yield ignored_start, ignored_end
1718+
yield from _normalization_ignored_spans_in_gap(text, gap_start, gap_end)
17081719

17091720

17101721
def _contextual_default_ignorable_offsets(
@@ -1813,6 +1824,55 @@ def normalized_security_view(
18131824
return SecurityTextView("normalized", output.getvalue(), offsets)
18141825

18151826

1827+
def normalized_security_prefix(text: str, max_chars: int) -> str:
1828+
"""Return an exact bounded prefix of the normalized security projection."""
1829+
if max_chars <= 0:
1830+
return ""
1831+
1832+
output = StringIO()
1833+
output_chars = 0
1834+
1835+
def append(character: str) -> bool:
1836+
nonlocal output_chars
1837+
normalized = unicodedata.normalize("NFKC", character).translate(ASCII_CONFUSABLE_SKELETON)
1838+
remaining = max_chars - output_chars
1839+
output.write(normalized[:remaining])
1840+
output_chars += min(len(normalized), remaining)
1841+
return output_chars >= max_chars
1842+
1843+
source_offset = 0
1844+
while source_offset < len(text) and output_chars < max_chars:
1845+
if not _is_token_gap_character(text[source_offset]):
1846+
if append(text[source_offset]):
1847+
break
1848+
source_offset += 1
1849+
continue
1850+
1851+
gap_start = source_offset
1852+
while source_offset < len(text) and _is_token_gap_character(text[source_offset]):
1853+
source_offset += 1
1854+
gap_end = source_offset
1855+
before_is_word = gap_start > 0 and _is_word_character(text[gap_start - 1])
1856+
after_is_word = gap_end < len(text) and _is_word_character(text[gap_end])
1857+
ignored_spans = iter(
1858+
_normalization_ignored_spans_in_gap(text, gap_start, gap_end)
1859+
if before_is_word or after_is_word
1860+
else ()
1861+
)
1862+
next_ignored = next(ignored_spans, None)
1863+
gap_offset = gap_start
1864+
while gap_offset < gap_end and output_chars < max_chars:
1865+
if next_ignored is not None and gap_offset == next_ignored[0]:
1866+
gap_offset = next_ignored[1]
1867+
next_ignored = next(ignored_spans, None)
1868+
continue
1869+
if not _is_unconditionally_ignored(text[gap_offset]) and append(text[gap_offset]):
1870+
break
1871+
gap_offset += 1
1872+
1873+
return output.getvalue()
1874+
1875+
18161876
def obfuscated_instruction_view(
18171877
text: str,
18181878
check_runtime: Callable[[], None] | None = None,
@@ -2171,6 +2231,35 @@ def _requires_normalized_security_view_uncached(
21712231
return not text.translate(_REMOVE_ALLOWED_FORMAT_CHARACTERS).isprintable()
21722232

21732233

2234+
def _has_derived_security_view(text: str) -> bool:
2235+
"""Return whether security projection produces a distinct text view."""
2236+
if text.isascii():
2237+
return (
2238+
_IGNORED_ASCII_CONTROL.search(text) is not None
2239+
or _has_letter_spacing_run(text)
2240+
or next(_obfuscated_instruction_matches(text), None) is not None
2241+
)
2242+
if any(
2243+
unicodedata.normalize("NFKC", character).translate(ASCII_CONFUSABLE_SKELETON) != character
2244+
for character in text
2245+
):
2246+
return True
2247+
if any(_is_unconditionally_ignored(character) for character in text):
2248+
return True
2249+
if next(_normalization_ignored_spans(text), None) is not None:
2250+
return True
2251+
if "\ufffd" in text or next(_compact_gap_offsets(text), None) is not None:
2252+
return True
2253+
return (
2254+
_has_letter_spacing_run(text)
2255+
or next(
2256+
_obfuscated_instruction_matches(text),
2257+
None,
2258+
)
2259+
is not None
2260+
)
2261+
2262+
21742263
def security_text_views(
21752264
text: str,
21762265
check_runtime: Callable[[], None] | None = None,

0 commit comments

Comments
 (0)