Skip to content

Commit 1de92b8

Browse files
committed
Merge remote-tracking branch 'upstream/main' into fix/p2-structural-benign-37
Signed-off-by: Yoseph Zuskin <[email protected]>
2 parents 9f8bbea + 3527006 commit 1de92b8

81 files changed

Lines changed: 7070 additions & 197 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
### 2.12.0 (Unreleased)
22
### Features/Bug Fixes
3+
* fix(baseline): fingerprint every occurrence of a deduplicated finding so the next scan suppresses all of them (#633)
34
* fix(security): retain incomplete coverage for runtime-selected commands and remeasure active Git clones strictly (#514)
45
* fix(scan): preserve required-input failures and multiline prompt uncertainty (#563)
56
* fix(analyzer): preserve Perl print literal ownership and explain referenced-artifact limitations (#615)

‎README.md‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,22 @@ inference gateways.
242242
| `claude_cli` | _(none — uses local CLI auth)_ | local `claude` binary | local Claude runtime fallback, or `SKILLSPECTOR_MODEL` |
243243
| `codex_cli` | _(none — uses local CLI auth)_ | local `codex` binary | local Codex runtime fallback, or `SKILLSPECTOR_MODEL` |
244244
| `gemini_cli` | _(none — uses local CLI auth)_ | local `gemini` binary | local Gemini runtime fallback, or `SKILLSPECTOR_MODEL` |
245-
| `opencode_cli` | _(none — uses local CLI auth)_ | local `opencode` 1.18.31 binary | local OpenCode runtime fallback, or `SKILLSPECTOR_MODEL` |
245+
| `opencode_cli` | _(none — uses local CLI auth)_ | local `opencode` 1.18.33 binary | local OpenCode runtime fallback, or `SKILLSPECTOR_MODEL` |
246+
247+
Structured output is requested through LangChain's `with_structured_output`,
248+
whose default forces a tool call. Some models reject a forced tool call with
249+
HTTP 400 (`tool_choice: type "tool" and "any" are not supported for this
250+
model`). The `anthropic` and `anthropic_proxy` providers route those models
251+
(`claude-fable-5-1`, `claude-mythos-5-1`, or any registry entry with
252+
`structured_output: json_schema`) to the native JSON-schema response format.
253+
Bedrock has no JSON-schema output for them, so the `bedrock` provider leaves
254+
`toolChoice` at `auto`, asks for the tool call in the prompt, and retries a
255+
prose answer; it recognises the model from the model ID, a geo/global
256+
inference-profile ID, or a foundation-model / inference-profile ARN. An
257+
application-inference-profile ARN hides the model, so add that ARN to the
258+
registry (`SKILLSPECTOR_MODEL_REGISTRY`) with `tool_choice: auto`.
259+
`SKILLSPECTOR_STRUCTURED_OUTPUT_METHOD=json_schema|function_calling`
260+
overrides the method for any provider.
246261

247262
```bash
248263
# Stock OpenAI
@@ -635,7 +650,7 @@ Issues (2)
635650

636651
> **CLI providers** (`claude_cli`, `codex_cli`, `gemini_cli`, `opencode_cli`): No API key is needed. Authentication is managed entirely by the agent CLI's own login session. SkillSpector never reads or forwards API keys when these providers are active. The subprocess is run with capabilities restricted, and untrusted skill content is delivered only via stdin.
637652
>
638-
> `opencode_cli` currently fails closed unless the installed OpenCode version is exactly `1.18.31`, the version whose configuration precedence and deny-all semantics are verified by this release.
653+
> `opencode_cli` currently fails closed unless the installed OpenCode version is exactly `1.18.33`, the version whose configuration precedence and deny-all semantics are verified by this release.
639654
640655
### CLI Options
641656

‎docs/DEVELOPMENT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -345,7 +345,7 @@ Base URL env vars are not needed for live provider tests; the tests intentionall
345345
- `claude_cli/` — **local `claude` binary; no API key**. Uses the CLI's own auth session (`claude auth login`). Set `SKILLSPECTOR_PROVIDER=claude_cli`.
346346
- `codex_cli/` — **local `codex` binary; no API key**. Uses the CLI's own auth session (`codex login`). Set `SKILLSPECTOR_PROVIDER=codex_cli`.
347347
- `gemini_cli/` — **local `gemini` binary; no API key**. Uses the CLI's own auth session. Set `SKILLSPECTOR_PROVIDER=gemini_cli`.
348-
- `opencode_cli/` — **local `opencode` 1.18.31 binary; no API key**. Uses the CLI's own auth session (`opencode auth login`) and fails closed on every other runtime version because the deny-all policy is verified against that exact release. Set `SKILLSPECTOR_PROVIDER=opencode_cli`.
348+
- `opencode_cli/` — **local `opencode` 1.18.33 binary; no API key**. Uses the CLI's own auth session (`opencode auth login`) and fails closed on every other runtime version because the deny-all policy is verified against that exact release. Set `SKILLSPECTOR_PROVIDER=opencode_cli`.
349349

350350
CLI providers (`claude_cli`, `codex_cli`, `gemini_cli`, `opencode_cli`) implement the optional `AgentCLICapable` interface (`is_available()` + `complete()`) defined in [providers/base.py](../src/skillspector/providers/base.py). `has_cli_capability(provider)` detects this at runtime. All subprocess calls go through the hardened helper [providers/_agent_cli.py](../src/skillspector/providers/_agent_cli.py) which enforces: no shell (`shell=False`), untrusted content via stdin only, capability stripping (tools disabled / sandboxed), environment scrubbing (no API keys forwarded), per-call timeout, and fail-closed error handling.
351351

‎model_registry.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,27 @@ models:
2020
context_length: 128000
2121
max_output_tokens: 16384
2222

23+
# Pricing (USD per 1M tokens) is informational only; SkillSpector does not
24+
# compute cost. Prompts over 272K input tokens bill at 2x input / 1.5x output.
25+
# Source: https://developers.openai.com/api/docs/models/<model> (as of 2026-09-21)
26+
27+
# $4.00 input / $0.40 cached input / $20.00 output (promotional through
28+
# 2026-11-21). Input is capped at 922K tokens; the 75% input budget
29+
# (~787K) stays under it.
30+
"gpt-5.6-sol":
31+
context_length: 1050000
32+
max_output_tokens: 128000
33+
34+
# $2.00 input / $0.20 cached input / $12.00 output
35+
"gpt-5.6-terra":
36+
context_length: 1050000
37+
max_output_tokens: 128000
38+
39+
# $0.20 input / $0.02 cached input / $1.20 output
40+
"gpt-5.6-luna":
41+
context_length: 1050000
42+
max_output_tokens: 128000
43+
2344
# Provider-prefixed IDs for inference gateways that accept them.
2445
"azure/anthropic/claude-opus-4-5":
2546
context_length: 200000

‎src/skillspector/cleanup.py‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,15 +3,50 @@
33

44
"""Shared cleanup helpers for SkillSpector."""
55

6+
import os
67
import shutil
8+
import stat
9+
from collections.abc import Callable
10+
from pathlib import Path
711

812
from skillspector.python_ast import clear_python_ast_cache
913

1014

15+
def _retry_writable(function: Callable[..., object], path: str, error: BaseException) -> None:
16+
"""Clear a read-only bit and retry a refused removal once; Windows refuses to delete read-only files.
17+
18+
Only a permission error from the removal calls is retried. Any other failure,
19+
such as a directory that is not empty yet, and any other callback ``rmtree``
20+
reports, such as ``os.open``, are left to its best-effort pass, and nothing
21+
raised here may reach the caller.
22+
"""
23+
if function not in (os.unlink, os.rmdir) or not isinstance(error, PermissionError):
24+
return
25+
try:
26+
# chmod follows links, so never touch whatever a link points at.
27+
if os.path.islink(path) or os.path.isjunction(path):
28+
return
29+
# Add the owner-write bit only; replacing the mode would strip read and
30+
# search permission on POSIX and leave the entry harder to remove.
31+
os.chmod(path, stat.S_IMODE(os.lstat(path).st_mode) | stat.S_IWRITE)
32+
function(path)
33+
except Exception:
34+
pass
35+
36+
37+
def remove_temp_tree(path: str | Path) -> None:
38+
"""Best-effort removal of a scan temp directory, including read-only files.
39+
40+
``git clone`` writes its pack files read-only, so ``ignore_errors=True``
41+
alone leaves every cloned repository behind on Windows.
42+
"""
43+
shutil.rmtree(path, onexc=_retry_writable)
44+
45+
1146
def cleanup_result(result: dict[str, object]) -> None:
1247
"""Release scan-local resources and remove a temp dir if set."""
1348
python_ast_cache_key = result.get("python_ast_cache_key")
1449
clear_python_ast_cache(python_ast_cache_key if isinstance(python_ast_cache_key, str) else None)
1550
temp_dir = result.get("temp_dir_for_cleanup")
1651
if temp_dir and isinstance(temp_dir, str):
17-
shutil.rmtree(temp_dir, ignore_errors=True)
52+
remove_temp_tree(temp_dir)

‎src/skillspector/cli.py‎

Lines changed: 69 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -655,22 +655,37 @@ def scan(
655655
"multi-skill scans; scan each sub-skill with its own baseline"
656656
)
657657
raise typer.Exit(code=2)
658-
_scan_multi_skill(
659-
detection,
660-
format=format,
661-
output=output,
662-
no_llm=no_llm,
663-
baseline=baseline,
664-
show_suppressed=show_suppressed,
665-
transitive_enabled=transitive_enabled,
666-
transitive_depth=transitive_depth,
667-
transitive_allow_prefix=transitive_allow_prefix,
668-
transitive_deny_prefix=transitive_deny_prefix,
669-
yara_dir=yara_dir,
670-
verbose=verbose,
671-
fail_on_incomplete=fail_on_incomplete,
672-
fail_on_findings=fail_on_findings,
673-
)
658+
# Same error contract as the single-skill scan below: an error that
659+
# escapes the combined report (e.g. an unwritable --output) exits 2,
660+
# not with a traceback and exit 1, which means "risk found".
661+
try:
662+
_scan_multi_skill(
663+
detection,
664+
format=format,
665+
output=output,
666+
no_llm=no_llm,
667+
baseline=baseline,
668+
show_suppressed=show_suppressed,
669+
transitive_enabled=transitive_enabled,
670+
transitive_depth=transitive_depth,
671+
transitive_allow_prefix=transitive_allow_prefix,
672+
transitive_deny_prefix=transitive_deny_prefix,
673+
yara_dir=yara_dir,
674+
verbose=verbose,
675+
fail_on_incomplete=fail_on_incomplete,
676+
fail_on_findings=fail_on_findings,
677+
)
678+
except typer.Exit:
679+
raise
680+
except (FileNotFoundError, ValueError) as e:
681+
err_console.print(f"[red]Error:[/red] {e}")
682+
raise typer.Exit(code=2) from e
683+
except Exception as e:
684+
if verbose:
685+
err_console.print_exception()
686+
else:
687+
err_console.print(f"[red]Error:[/red] {e}")
688+
raise typer.Exit(code=2) from e
674689
return
675690
if detection.complete and not detection.has_root_skill and len(detection.skills) == 0:
676691
discovery_console.print(
@@ -2710,6 +2725,10 @@ def _scan_multi_skill(
27102725
aggregate_limitations = [
27112726
f"recursive discovery {limitation.resource} limit reached"
27122727
for limitation in detection.limitations[:256]
2728+
# Symlink omissions get their own clearer aggregate message below;
2729+
# listing the generic one too would double-report the same entries.
2730+
if limitation.resource != "multi_skill_symlinked_entry"
2731+
or not detection.omitted_symlink_entries
27132732
]
27142733
retained_public_records = 0
27152734
retained_report_characters = 0
@@ -2843,6 +2862,19 @@ def _scan_multi_skill(
28432862
len(skills) - scanned_skill_count,
28442863
)
28452864
output_omitted_skill_count = max(0, scanned_skill_count - len(processed_skills))
2865+
omitted_symlink_entry_count = detection.omitted_symlink_entries
2866+
if omitted_symlink_entry_count:
2867+
analysis_incomplete = True
2868+
aggregate_limitations.append(
2869+
f"{omitted_symlink_entry_count} symlinked recursive skill(s) omitted "
2870+
"(directory symlinks are not followed)"
2871+
)
2872+
progress_console.print(
2873+
f"[yellow]Warning:[/yellow] {omitted_symlink_entry_count} symlinked skill "
2874+
"directories were skipped during recursive discovery and are not "
2875+
"included in this scan."
2876+
)
2877+
skills_omitted_total = unscanned_skill_count + omitted_symlink_entry_count
28462878
if output_omitted_skill_count:
28472879
analysis_incomplete = True
28482880
aggregate_limitations.append(
@@ -2856,11 +2888,11 @@ def _scan_multi_skill(
28562888
)
28572889
aggregate_limitations = list(dict.fromkeys(aggregate_limitations))[:256]
28582890
aggregate_completeness = _multi_skill_analysis_completeness(
2859-
total_skills=len(skills),
2891+
total_skills=len(skills) + omitted_symlink_entry_count,
28602892
complete_skills=complete_skill_count,
28612893
partial_skills=partial_skill_count,
28622894
failed_skills=failed_skill_count,
2863-
omitted_skills=unscanned_skill_count,
2895+
omitted_skills=skills_omitted_total,
28642896
limitations=aggregate_limitations,
28652897
)
28662898
analysis_incomplete = not bool(aggregate_completeness["is_complete"])
@@ -2898,10 +2930,15 @@ def _scan_multi_skill(
28982930
progress_console.print(
28992931
f" {'<unscanned>':<30} {'—':<8} {'—':<12} {unscanned_skill_count:<10} {'partial':<10}"
29002932
)
2901-
if output_omitted_skill_count or unscanned_skill_count:
2933+
if omitted_symlink_entry_count:
2934+
progress_console.print(
2935+
f" {'<symlink omitted>':<30} {'—':<8} {'—':<12} "
2936+
f"{omitted_symlink_entry_count:<10} {'skipped':<10}"
2937+
)
2938+
if output_omitted_skill_count or unscanned_skill_count or omitted_symlink_entry_count:
29022939
progress_console.print(
29032940
"[yellow]Recursive scan incomplete:[/yellow] one or more skills were omitted "
2904-
"after an aggregate safety limit."
2941+
"after an aggregate safety limit or skipped as symlinks."
29052942
)
29062943

29072944
if format == FormatChoice.json:
@@ -2914,7 +2951,7 @@ def _scan_multi_skill(
29142951
"risk_recommendation": aggregate_risk_assessment["recommendation"],
29152952
"analysis_completeness": aggregate_completeness,
29162953
"skills_scanned": scanned_skill_count,
2917-
"skills_omitted": unscanned_skill_count,
2954+
"skills_omitted": skills_omitted_total,
29182955
"skills_output_omitted": output_omitted_skill_count,
29192956
"public_finding_records": retained_public_records,
29202957
"report_characters": retained_report_characters,
@@ -2965,6 +3002,14 @@ def _scan_multi_skill(
29653002
"reason": "aggregate_scan_limit",
29663003
}
29673004
)
3005+
if omitted_symlink_entry_count:
3006+
combined_skills.append(
3007+
{
3008+
"omitted": True,
3009+
"omitted_count": omitted_symlink_entry_count,
3010+
"reason": "symlink_not_followed",
3011+
}
3012+
)
29683013
rendered = json.dumps(combined, indent=2)
29693014
if len(rendered) > _MULTI_SKILL_MAX_REPORT_CHARACTERS:
29703015
analysis_incomplete = True
@@ -3181,7 +3226,9 @@ def baseline(
31813226
state = _scan_state(input_path, FormatChoice.json, no_llm)
31823227
state["baseline_path"] = os.path.abspath(output.expanduser())
31833228
result = graph.invoke(state)
3184-
findings = effective_findings(result)
3229+
# Fingerprint every occurrence the next scan checks. The reported
3230+
# findings are deduplicated and keep only one occurrence's evidence.
3231+
findings = result["active_findings"]
31853232
data = build_baseline_dict(
31863233
findings,
31873234
reason=reason,

‎src/skillspector/constants.py‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,12 @@
1818
import logging
1919
import os
2020

21-
from skillspector.providers import get_metadata_provider, get_model_config_provider
21+
from skillspector.providers import (
22+
ModelMetadataProvider,
23+
UnknownProviderError,
24+
get_metadata_provider,
25+
get_model_config_provider,
26+
)
2227

2328
logger = logging.getLogger(__name__)
2429

@@ -45,11 +50,17 @@
4550
# ``resolve_model`` runs the waterfall: ``SKILLSPECTOR_MODEL`` env > slot
4651
# default > general default. OSS users pointing at build.nvidia.com or
4752
# stock OpenAI inherit ``NvBuildProvider``'s default model automatically.
48-
_provider = get_metadata_provider()
53+
_provider: ModelMetadataProvider | None
54+
try:
55+
_provider = get_metadata_provider()
56+
except UnknownProviderError:
57+
# Importing static checks and CLI help must not require a valid LLM provider.
58+
# Runtime model configuration still validates the selector in build_model_config.
59+
_provider = None
4960

5061
# Exposed for analyzers that need a final fallback symbol (e.g.,
5162
# ``model = state.model or MODEL_CONFIG[ANALYZER_ID] or _SKILLSPECTOR_DEFAULT_MODEL``).
52-
_SKILLSPECTOR_DEFAULT_MODEL = _provider.DEFAULT_MODEL
63+
_SKILLSPECTOR_DEFAULT_MODEL = _provider.DEFAULT_MODEL if _provider is not None else ""
5364

5465
_MODEL_SLOTS: tuple[str, ...] = (
5566
"default",
@@ -84,7 +95,11 @@ def build_model_config() -> dict[str, str]:
8495
return {slot: _resolve_slot_model(slot, provider) for slot in _MODEL_SLOTS}
8596

8697

87-
MODEL_CONFIG: dict[str, str] = {slot: _resolve_slot_model(slot, _provider) for slot in _MODEL_SLOTS}
98+
MODEL_CONFIG: dict[str, str] = (
99+
{slot: _resolve_slot_model(slot, _provider) for slot in _MODEL_SLOTS}
100+
if _provider is not None
101+
else {}
102+
)
88103

89104

90105
def _validate_model_config() -> None:
@@ -93,6 +108,8 @@ def _validate_model_config() -> None:
93108
When ``SKILLSPECTOR_STRICT_MODEL_VALIDATION=true``, raises
94109
``ValueError`` instead of logging warnings.
95110
"""
111+
if _provider is None:
112+
return
96113
unknown: list[str] = []
97114
for slot, model in MODEL_CONFIG.items():
98115
ctx = _provider.get_context_length(model) # type: ignore[attr-defined]

‎src/skillspector/input_handler.py‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454

5555
import httpx
5656

57+
from skillspector.cleanup import remove_temp_tree
5758
from skillspector.logging_config import get_logger
5859

5960
logger = get_logger(__name__)
@@ -828,7 +829,7 @@ def resolve(self, input_path: str) -> tuple[Path, str]:
828829
def cleanup(self) -> None:
829830
"""Clean up temporary files created during resolution."""
830831
if self._temp_dir and self._temp_dir.exists():
831-
shutil.rmtree(self._temp_dir, ignore_errors=True)
832+
remove_temp_tree(self._temp_dir)
832833
self._temp_dir = None
833834

834835
def temp_dir_for_cleanup(self) -> Path | None:
@@ -1276,6 +1277,12 @@ def _clone_git(self, url: str, *, branch: str | None = None) -> Path:
12761277
self._terminate_git_process(process)
12771278
shutil.rmtree(clone_dir, ignore_errors=True)
12781279
raise ValueError("Failed to clone repository") from exc
1280+
except BaseException:
1281+
# An interrupt or a cancellation while Git is still writing: stop
1282+
# the child before the tree it writes into is removed.
1283+
self._terminate_git_process(process)
1284+
shutil.rmtree(clone_dir, ignore_errors=True)
1285+
raise
12791286
return clone_dir
12801287

12811288
@staticmethod

‎src/skillspector/llm_analyzer_base.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@
6363
_AgentCLIMessage,
6464
_ainvoke_with_usage,
6565
_invoke_with_usage,
66+
bind_structured_output,
6667
chat_model_provider_name,
6768
get_chat_model,
6869
new_inference_usage_collector,
@@ -933,7 +934,9 @@ def __init__(
933934
max_retries=native_retries,
934935
)
935936
self._structured_llm = (
936-
self._llm.with_structured_output(self.response_schema) if self.response_schema else None
937+
bind_structured_output(self._llm, self.response_schema, model)
938+
if self.response_schema
939+
else None
937940
)
938941
self._usage_collector = new_inference_usage_collector(
939942
node=node,
@@ -986,7 +989,9 @@ def _model_for_call(self) -> tuple[object, object | None]:
986989
chat_model_controls(llm),
987990
)
988991
structured = (
989-
llm.with_structured_output(self.response_schema) if self.response_schema else None
992+
bind_structured_output(llm, self.response_schema, self.model)
993+
if self.response_schema
994+
else None
990995
)
991996
return llm, structured
992997

0 commit comments

Comments
 (0)