Skip to content

Commit 960be30

Browse files
rng1995claude
andcommitted
Merge branch 'main' into yashraj/cache-shell-flag-scope-analysis
Merges main at 2c14261 (#557, #578, #657, #658, #579). Only #578 (stabilize TM1 window identity) conflicted, in static_patterns_tool_misuse.py (7 hunks). Kept main's analyze() logic (_direct_shell_only, per-line context cache, classification_context/_bounded_context, line_number helper) and this PR's USES_RUNTIME_CHECK, check_runtime keyword, runtime_check() calls, and SourceLocationIndex/get_context_from_lines lookups backing line_number/ctx. Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
2 parents f4432cc + 2c14261 commit 960be30

55 files changed

Lines changed: 14980 additions & 1130 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/SUPPRESSION.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,16 @@ skillspector scan ./my-skill/ --baseline .skillspector-baseline.yaml --show-supp
3939
| `skillspector scan <path> --baseline FILE --show-suppressed` | Also list the suppressed findings (they still don't affect the score). |
4040

4141
A missing, malformed, or unsupported baseline file exits with code 2.
42+
Generation also exits with code 2 without creating or replacing the output when
43+
scan execution fails, including failed semantic analysis or unreadable primary
44+
content. Resolve the failure and rerun the scan before accepting its findings.
45+
Use `--no-llm` when you intentionally want to accept only static findings.
46+
47+
A successful but incomplete scan can generate a baseline for its observed
48+
findings, with a warning on stderr. This does not accept uninspected content or
49+
clear coverage gaps: subsequent reports retain their incomplete status and
50+
recommendation even if every observed finding is suppressed.
51+
4252
When a selected baseline or baseline output is stored inside the scan target,
4353
SkillSpector treats that exact file as an explicit scope exclusion. This
4454
prevents sensitive rule text from creating a finding against itself or entering
@@ -103,6 +113,30 @@ Generated by `skillspector baseline`, it is intentionally exact:
103113
editing the source or upgrading SkillSpector keeps the finding active until it
104114
is reviewed and the baseline is regenerated.
105115

116+
LLM and meta-analysis can change explanations, remediation, confidence, or other
117+
evidence between scans of unchanged files. Those results remain active when
118+
their exact fingerprints differ. For repeatable static acceptance, use
119+
`--no-llm` for both baseline generation and subsequent scans.
120+
121+
Generation fingerprints each original finding before report compaction, including
122+
repeated matches on different lines and identical matches in different files.
123+
Only identical exact fingerprints share an entry. A baseline generated by an
124+
affected version may omit these occurrences; regenerate it to include them.
125+
If the complete baseline exceeds the loader's size or record limits, generation
126+
fails before replacing the output file instead of writing a partial baseline.
127+
After validation, generation normally writes a complete temporary file beside the
128+
destination and atomically replaces the output. A failed write or replacement
129+
preserves the existing baseline. Existing destinations must be writable regular
130+
files; symlinks and special files are rejected. Replacement requires a writable
131+
parent directory. On POSIX, new files grant access only to their owner; existing
132+
ordinary permission bits and ownership are preserved, including group-write access.
133+
When a non-owner has write access to an existing file, or the owner cannot
134+
assign the file's group because it is not a member, generation updates its
135+
validated descriptor in place and preserves its ACLs. This shared-file fallback
136+
serializes cooperating writers, but readers can see a partial write and an I/O
137+
failure or interruption can leave a partial baseline. Use an owner-managed output
138+
when atomic replacement is required.
139+
106140
Every v2 entry must be a mapping with a 64-hex-character `sha256:` hash and a
107141
non-empty `reason`. `rule_id` and `file` are informational fields for reviewers.
108142
If source content is unavailable or `scanner_version` does not match, exact

‎src/skillspector/artifacts.py‎

Lines changed: 120 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,8 @@ class SecurityTextView:
9999
text: str
100100
source_offsets: array[int] | None = None
101101
reconstructions: tuple[SecurityTextReconstruction, ...] = ()
102+
right_boundary_is_fixed: bool = False
103+
right_boundary_recovery_start: int | None = None
102104

103105
def source_offset(self, derived_offset: int) -> int:
104106
"""Map a derived character offset to the corresponding source offset."""
@@ -228,6 +230,7 @@ class _ObfuscatedIgnoreState:
228230
".markdown",
229231
".txt",
230232
".py",
233+
".pyw",
231234
".sh",
232235
".json",
233236
".yaml",
@@ -401,6 +404,20 @@ def classify_artifact(path: str, data: bytes, *, referenced: bool = False) -> Ar
401404
}
402405

403406

407+
def promote_artifact_to_decoded_text(artifact: ArtifactRecord) -> None:
408+
"""Apply a successful format-aware text decode without erasing prior limits."""
409+
generic_binary_scope = artifact["content_kind"] is ContentKind.BINARY and (
410+
artifact["disposition"] is ArtifactDisposition.OUT_OF_SCOPE
411+
or artifact["disposition"] is ArtifactDisposition.PARTIAL
412+
and "reason" not in artifact
413+
)
414+
artifact["content_kind"] = ContentKind.TEXT
415+
artifact["decodable"] = True
416+
artifact["misleading_extension"] = _suffix(artifact["path"]) in _BINARY_EXTENSIONS
417+
if generic_binary_scope:
418+
artifact["disposition"] = ArtifactDisposition.ANALYZED
419+
420+
404421
def decode_text(data: bytes) -> str:
405422
"""Return the loss-tolerant local text projection for static analyzers."""
406423
return data.decode("utf-8", errors="replace")
@@ -1679,6 +1696,30 @@ def _contextual_default_ignorable_spans(
16791696
yield span_start, end
16801697

16811698

1699+
def _normalization_ignored_spans_in_gap(
1700+
text: str,
1701+
gap_start: int,
1702+
gap_end: int,
1703+
) -> Iterator[tuple[int, int]]:
1704+
"""Yield the normalized-view removals inside one contextual gap."""
1705+
for match in _DEFAULT_IGNORABLE_RUN_PATTERN.finditer(text, gap_start, gap_end):
1706+
start, end = match.span()
1707+
ignored_start = (
1708+
start
1709+
if _is_unconditionally_ignored(text[start])
1710+
or _is_contextual_default_ignorable_offset(text, start)
1711+
else start + 1
1712+
)
1713+
ignored_end = (
1714+
end
1715+
if _is_unconditionally_ignored(text[end - 1])
1716+
or _is_contextual_default_ignorable_offset(text, end - 1)
1717+
else end - 1
1718+
)
1719+
if ignored_start < ignored_end:
1720+
yield ignored_start, ignored_end
1721+
1722+
16821723
def _normalization_ignored_spans(
16831724
text: str,
16841725
check_runtime: Callable[[], None] | None = None,
@@ -1689,22 +1730,7 @@ def _normalization_ignored_spans(
16891730
require_word_boundaries=False,
16901731
check_runtime=check_runtime,
16911732
):
1692-
for match in _DEFAULT_IGNORABLE_RUN_PATTERN.finditer(text, gap_start, gap_end):
1693-
start, end = match.span()
1694-
ignored_start = (
1695-
start
1696-
if _is_unconditionally_ignored(text[start])
1697-
or _is_contextual_default_ignorable_offset(text, start)
1698-
else start + 1
1699-
)
1700-
ignored_end = (
1701-
end
1702-
if _is_unconditionally_ignored(text[end - 1])
1703-
or _is_contextual_default_ignorable_offset(text, end - 1)
1704-
else end - 1
1705-
)
1706-
if ignored_start < ignored_end:
1707-
yield ignored_start, ignored_end
1733+
yield from _normalization_ignored_spans_in_gap(text, gap_start, gap_end)
17081734

17091735

17101736
def _contextual_default_ignorable_offsets(
@@ -1813,6 +1839,55 @@ def normalized_security_view(
18131839
return SecurityTextView("normalized", output.getvalue(), offsets)
18141840

18151841

1842+
def normalized_security_prefix(text: str, max_chars: int) -> str:
1843+
"""Return an exact bounded prefix of the normalized security projection."""
1844+
if max_chars <= 0:
1845+
return ""
1846+
1847+
output = StringIO()
1848+
output_chars = 0
1849+
1850+
def append(character: str) -> bool:
1851+
nonlocal output_chars
1852+
normalized = unicodedata.normalize("NFKC", character).translate(ASCII_CONFUSABLE_SKELETON)
1853+
remaining = max_chars - output_chars
1854+
output.write(normalized[:remaining])
1855+
output_chars += min(len(normalized), remaining)
1856+
return output_chars >= max_chars
1857+
1858+
source_offset = 0
1859+
while source_offset < len(text) and output_chars < max_chars:
1860+
if not _is_token_gap_character(text[source_offset]):
1861+
if append(text[source_offset]):
1862+
break
1863+
source_offset += 1
1864+
continue
1865+
1866+
gap_start = source_offset
1867+
while source_offset < len(text) and _is_token_gap_character(text[source_offset]):
1868+
source_offset += 1
1869+
gap_end = source_offset
1870+
before_is_word = gap_start > 0 and _is_word_character(text[gap_start - 1])
1871+
after_is_word = gap_end < len(text) and _is_word_character(text[gap_end])
1872+
ignored_spans = iter(
1873+
_normalization_ignored_spans_in_gap(text, gap_start, gap_end)
1874+
if before_is_word or after_is_word
1875+
else ()
1876+
)
1877+
next_ignored = next(ignored_spans, None)
1878+
gap_offset = gap_start
1879+
while gap_offset < gap_end and output_chars < max_chars:
1880+
if next_ignored is not None and gap_offset == next_ignored[0]:
1881+
gap_offset = next_ignored[1]
1882+
next_ignored = next(ignored_spans, None)
1883+
continue
1884+
if not _is_unconditionally_ignored(text[gap_offset]) and append(text[gap_offset]):
1885+
break
1886+
gap_offset += 1
1887+
1888+
return output.getvalue()
1889+
1890+
18161891
def obfuscated_instruction_view(
18171892
text: str,
18181893
check_runtime: Callable[[], None] | None = None,
@@ -2171,6 +2246,35 @@ def _requires_normalized_security_view_uncached(
21712246
return not text.translate(_REMOVE_ALLOWED_FORMAT_CHARACTERS).isprintable()
21722247

21732248

2249+
def _has_derived_security_view(text: str) -> bool:
2250+
"""Return whether security projection produces a distinct text view."""
2251+
if text.isascii():
2252+
return (
2253+
_IGNORED_ASCII_CONTROL.search(text) is not None
2254+
or _has_letter_spacing_run(text)
2255+
or next(_obfuscated_instruction_matches(text), None) is not None
2256+
)
2257+
if any(
2258+
unicodedata.normalize("NFKC", character).translate(ASCII_CONFUSABLE_SKELETON) != character
2259+
for character in text
2260+
):
2261+
return True
2262+
if any(_is_unconditionally_ignored(character) for character in text):
2263+
return True
2264+
if next(_normalization_ignored_spans(text), None) is not None:
2265+
return True
2266+
if "\ufffd" in text or next(_compact_gap_offsets(text), None) is not None:
2267+
return True
2268+
return (
2269+
_has_letter_spacing_run(text)
2270+
or next(
2271+
_obfuscated_instruction_matches(text),
2272+
None,
2273+
)
2274+
is not None
2275+
)
2276+
2277+
21742278
def security_text_views(
21752279
text: str,
21762280
check_runtime: Callable[[], None] | None = None,

‎src/skillspector/cli.py‎

Lines changed: 75 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@
5959
)
6060
from skillspector.logging_config import get_logger, set_level
6161
from skillspector.mcp_registry import scan_registry
62-
from skillspector.models import Finding
62+
from skillspector.models import OCCURRENCE_FINDING_ID_KEY, Finding
6363
from skillspector.multi_skill import (
6464
MultiSkillDetectionResult,
6565
SkillDirectory,
@@ -1001,6 +1001,40 @@ def _ledger_work_identity(entry: dict[str, object]) -> str:
10011001
return f"{record_value}:{entry.get('phase', '')}"
10021002

10031003

1004+
def _ledger_work_identities(value: object) -> dict[str, str]:
1005+
"""Map each child ledger row's own work ID to the identity it was built from.
1006+
1007+
A status ``planned_work`` target carries only the child's work ID, path and
1008+
range, not the identity behind it. Rows whose identity is not the
1009+
analyzer's own -- the static_yara rule-set row is ``rule_set:static``, not
1010+
``static_yara`` -- must be re-scoped with that same identity in the status
1011+
path, or the status target and its ledger row get different scoped IDs
1012+
and the target is dropped as unretained.
1013+
"""
1014+
identities: dict[str, str] = {}
1015+
for event in _coerce_dict_list(value):
1016+
work_id = event.get("work_id")
1017+
if isinstance(work_id, str) and work_id:
1018+
identities[work_id] = _ledger_work_identity(event)
1019+
return identities
1020+
1021+
1022+
def _source_scoped_work_id(identity: str, item: dict[str, object]) -> str:
1023+
"""Build the scoped work ID for an already re-pathed ledger row or status target.
1024+
1025+
Shared by :func:`_source_aware_ledger` and :func:`_source_aware_status_events`
1026+
so the two scoping paths cannot derive different IDs for the same work.
1027+
"""
1028+
start_line = item.get("start_line")
1029+
end_line = item.get("end_line")
1030+
return inspection_work_id(
1031+
identity,
1032+
str(item.get("path", "SKILL.md")),
1033+
start_line if isinstance(start_line, int) else None,
1034+
end_line if isinstance(end_line, int) else None,
1035+
)
1036+
1037+
10041038
def _source_aware_ledger(
10051039
value: object,
10061040
*,
@@ -1026,15 +1060,7 @@ def _source_aware_ledger(
10261060
for item in ids
10271061
if isinstance(item, str)
10281062
]
1029-
scoped_path = str(entry.get("path", "SKILL.md"))
1030-
start_line = entry.get("start_line")
1031-
end_line = entry.get("end_line")
1032-
entry["work_id"] = inspection_work_id(
1033-
_ledger_work_identity(entry),
1034-
scoped_path,
1035-
start_line if isinstance(start_line, int) else None,
1036-
end_line if isinstance(end_line, int) else None,
1037-
)
1063+
entry["work_id"] = _source_scoped_work_id(_ledger_work_identity(entry), entry)
10381064
events.append(entry)
10391065
return events
10401066

@@ -1047,8 +1073,10 @@ def _source_aware_status_events(
10471073
source_digest: str,
10481074
retained_work_ids: set[str],
10491075
max_planned_work: int,
1076+
work_identities: dict[str, str] | None = None,
10501077
) -> list[dict[str, object]]:
10511078
statuses: list[dict[str, object]] = []
1079+
identities = work_identities or {}
10521080
planned_retained = 0
10531081
for status in _coerce_dict_list(value):
10541082
if len(statuses) >= _TRANSITIVE_MAX_STATUS_EVENTS:
@@ -1070,14 +1098,11 @@ def _source_aware_status_events(
10701098
path = scoped_target.get("path")
10711099
if isinstance(path, str) and path:
10721100
scoped_target["path"] = _transitive_component_key(source_identity, path)
1073-
start_line = scoped_target.get("start_line")
1074-
end_line = scoped_target.get("end_line")
1075-
scoped_target["work_id"] = inspection_work_id(
1076-
analyzer_id,
1077-
str(scoped_target.get("path", "SKILL.md")),
1078-
start_line if isinstance(start_line, int) else None,
1079-
end_line if isinstance(end_line, int) else None,
1080-
)
1101+
# Re-scope with the identity the matching ledger row used, so
1102+
# both paths agree on the scoped ID; the analyzer ID is only the
1103+
# fallback for targets with no child ledger row.
1104+
identity = identities.get(str(target.get("work_id", "")), analyzer_id)
1105+
scoped_target["work_id"] = _source_scoped_work_id(identity, scoped_target)
10811106
if scoped_target["work_id"] not in retained_work_ids:
10821107
continue
10831108
scoped_work.append(scoped_target)
@@ -1289,18 +1314,32 @@ def _cache_transitive_result(
12891314
child_filtered = _coerce_findings_list(child_result.get("filtered_findings"))
12901315
child_findings = _coerce_findings_list(child_result.get("findings"))
12911316
all_ids = {finding.finding_id for finding in [*child_filtered, *child_findings]}
1317+
all_ids.update(
1318+
occurrence_id
1319+
for finding in [*child_filtered, *child_findings]
1320+
for occurrence in finding.occurrences
1321+
if isinstance((occurrence_id := occurrence.get(OCCURRENCE_FINDING_ID_KEY)), str)
1322+
)
12921323
all_ids.update(_effective_finding_ids(child_result))
12931324
finding_id_map = {
12941325
finding_id: _scoped_finding_id(source_identity, finding_id) for finding_id in all_ids
12951326
}
12961327

12971328
def _scope_finding(finding: Finding) -> Finding:
1329+
occurrences = []
1330+
for raw in finding.occurrences:
1331+
occurrence = dict(raw)
1332+
occurrence_id = occurrence.get(OCCURRENCE_FINDING_ID_KEY)
1333+
if isinstance(occurrence_id, str):
1334+
occurrence[OCCURRENCE_FINDING_ID_KEY] = finding_id_map[occurrence_id]
1335+
occurrences.append(occurrence)
12981336
return replace(
12991337
finding,
13001338
finding_id=finding_id_map[finding.finding_id],
13011339
source_url=target,
13021340
source_identity=source_identity,
13031341
source_digest=source_digest,
1342+
occurrences=occurrences,
13041343
)
13051344

13061345
scoped_filtered = [_scope_finding(item) for item in child_filtered[:_TRANSITIVE_MAX_FINDINGS]]
@@ -1375,6 +1414,7 @@ def _scope_finding(finding: Finding) -> Finding:
13751414
source_digest=source_digest,
13761415
retained_work_ids=retained_work_ids,
13771416
max_planned_work=len(retained_work_ids),
1417+
work_identities=_ledger_work_identities(child_result.get("inspection_ledger")),
13781418
)
13791419
child_metadata = _decorate_component_metadata(
13801420
_coerce_component_metadata(child_result.get("component_metadata")),
@@ -3417,6 +3457,23 @@ def baseline(
34173457
state = _scan_state(input_path, FormatChoice.json, no_llm)
34183458
state["baseline_path"] = os.path.abspath(output.expanduser())
34193459
result = graph.invoke(state)
3460+
completeness_value = result.get("analysis_completeness")
3461+
completeness = completeness_value if isinstance(completeness_value, dict) else {}
3462+
if (
3463+
result.get("execution_successful") is False
3464+
or completeness.get("execution_successful") is False
3465+
or completeness.get("status") == "failed"
3466+
):
3467+
raise ValueError(
3468+
"Cannot generate baseline because scan execution failed. "
3469+
"Run 'skillspector scan' to inspect analysis completeness, "
3470+
"resolve the failures, and retry."
3471+
)
3472+
if completeness.get("is_complete") is False or completeness.get("status") == "partial":
3473+
err_console.print(
3474+
"[yellow]Warning:[/yellow] Scan analysis is incomplete; the baseline "
3475+
"accepts only observed findings and coverage gaps remain."
3476+
)
34203477
# Fingerprint every occurrence the next scan checks. The reported
34213478
# findings are deduplicated and keep only one occurrence's evidence.
34223479
findings = result["active_findings"]

0 commit comments

Comments
 (0)