Repository navigation
fix(sc8): count only the walk's own CPU time against its runtime budget - #828
Conversation
The SC8 shipped-bytecode walk stopped after 5 s of wall-clock time. Analyzers run concurrently in one process, so the walk spent most of that time waiting for the GIL behind CPU-bound analyzers. Skills that ship no bytecode got a static runtime_limit ledger row at a different folder on every run, which SkillEvaluator reports as a Critical "skillspector scan did not execute reliably" (SKILLEVAL-17). Measure the 5 s processing budget with time.thread_time(), as static_yara already does for rule loading. Wall-clock time stays bounded by the remaining workflow deadline, or by MAX_WORKFLOW_SECONDS for callers outside a workflow. Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Narendran Raghavan <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Narendran Raghavan <[email protected]>
chrisknvidia
left a comment
There was a problem hiding this comment.
Reviewed commit 922317e167a47d2ac3819c28edb481519f90d2e2. Looks good to me; no blocking findings.
I checked CPU-time accounting in _scan_shipped_bytecode, propagation of the remaining workflow wall-clock budget, limitation reporting, and the existing traversal/count/finding bounds. All 36 focused SC8 and budget-configuration tests passed.
I also exercised the actual filesystem walk under concurrent CPU-thread load: the baseline hit its wall-clock limit, while this version completed without a limitation. Positive controls still reported shipped bytecode, and an explicit wall-clock allowance still stopped the walk with runtime_limit.
Validation was local on macOS. No live LLM-provider or production validation was performed.
…time-budget Resolve the CHANGELOG.md conflict by keeping both the #828 SC8 entry and the #808 CLI-provider entry from main. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Narendran Raghavan <[email protected]>
fix(sc8): count only the walk's own CPU time against its runtime budget (cherry picked from commit 182a475) Signed-off-by: yashrajbasav <[email protected]>
Since NVIDIA#828, the shipped-bytecode walk counts only its own CPU time against the 5-second budget, and the workflow wall-clock deadline bounds elapsed time. Word the resource-bounds row the same way as the YARA rule-load row. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Narendran Raghavan <[email protected]>
The install docs, the CI recipe and the missing-scanner hint pinned
SkillSpector v2.12.0. That release can make the Tier 1 security check
INCOMPLETE, or fail it, for skills with no real issue:
- valid Python and Perl helpers (for example a `len(x) < 12` comparison
followed by more code, or a Perl `eval { require $file; 1 }` block) are
recorded as `static_parse_limit` or `obfuscated_instruction_text`
ledger exceptions, with a High AE1 at each line that names the helper
(NVIDIA/SkillSpector#777);
- the SC8 bytecode walk charges time spent waiting behind other analyzers
to its budget and can stop with a static `runtime_limit`
(NVIDIA/SkillSpector#828);
- visible text between two HTML comments raises a High P2
hidden-instruction finding (NVIDIA/SkillSpector#452).
SkillSpector v2.13.1 contains all three fixes. Move every install spec to
v2.13.1, keep the rule that docs and code name exactly one release, and
keep the Claude 5.5 caveat: v2.13.1 still lacks the newer structured-output
support, so SKILLSPECTOR_MODEL must still name a supported model when
SKILL_EVAL_LLM_MODEL is a Claude 5.5 model. The SkillSpector model
defaults are unchanged.
Add a troubleshooting entry for false incomplete helper scans on older
releases, and a captured v2.13.1 --no-llm report (from
tests/fixtures/skills/simple) to the captured-report acceptance test so the
strict completeness validator is exercised against the pinned release.
Signed-off-by: Narendran Raghavan <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* chore(deps): pin SkillSpector to v2.13.1
The install docs, the CI recipe and the missing-scanner hint pinned
SkillSpector v2.12.0. That release can make the Tier 1 security check
INCOMPLETE, or fail it, for skills with no real issue:
- valid Python and Perl helpers (for example a `len(x) < 12` comparison
followed by more code, or a Perl `eval { require $file; 1 }` block) are
recorded as `static_parse_limit` or `obfuscated_instruction_text`
ledger exceptions, with a High AE1 at each line that names the helper
(NVIDIA/SkillSpector#777);
- the SC8 bytecode walk charges time spent waiting behind other analyzers
to its budget and can stop with a static `runtime_limit`
(NVIDIA/SkillSpector#828);
- visible text between two HTML comments raises a High P2
hidden-instruction finding (NVIDIA/SkillSpector#452).
SkillSpector v2.13.1 contains all three fixes. Move every install spec to
v2.13.1, keep the rule that docs and code name exactly one release, and
keep the Claude 5.5 caveat: v2.13.1 still lacks the newer structured-output
support, so SKILLSPECTOR_MODEL must still name a supported model when
SKILL_EVAL_LLM_MODEL is a Claude 5.5 model. The SkillSpector model
defaults are unchanged.
Add a troubleshooting entry for false incomplete helper scans on older
releases, and a captured v2.13.1 --no-llm report (from
tests/fixtures/skills/simple) to the captured-report acceptance test so the
strict completeness validator is exercised against the pinned release.
Signed-off-by: Narendran Raghavan <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* chore(ci): install SkillSpector v2.13.1 in the Tier 1 image
The install docs and the missing-scanner hint name SkillSpector v2.13.1,
but the Tier 1 CI image still installed the v2.12.0 source archive
(c7958a3), so CI scans kept the false incomplete and false High findings
that v2.13.1 fixes.
Point ci/requirements/skillspector.in and the hash-locked
skillspector.txt at the archive of the v2.13.1 tag commit
(f3b25697d76256998ca9d42c198d256258d4c035) and update the
THIRD_PARTY_NOTICES row. v2.13.1 declares the same dependencies as
v2.12.0, so only the archive URL and its hash change; the documented
uv pip compile command gives the same lock.
Extend test_skillspector_install_instructions_pin_one_release so the CI
lock's archive commit and the notices row must match the documented
release, and mention the image in the CHANGELOG entry.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
* chore(deps): pin SkillSpector to v2.13.3
Move the install instructions, Tier 1 CI lock (commit 38a1dfe, archive sha256
recomputed), third-party notice, and captured --no-llm fixture from v2.13.1
to v2.13.3.
Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
---------
Signed-off-by: Narendran Raghavan <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Problem
The SC8 shipped-bytecode walk (
_scan_shipped_bytecode) stops afterMAX_SC8_ANALYSIS_SECONDS = 5.0of wall-clock time. All analyzers run in parallel threads in one process. Eachos.scandirstep releases the GIL, and the walk then waits behind CPU-bound analyzers to get it back. A walk that needs a few milliseconds of work can therefore run out of time.In a customer report (SKILLEVAL-17), a skill with no
__pycache__,.pycor.pyofiles got a staticruntime_limitledger row. The folder in it changed on every run, because it was wherever the walk had reached when time ran out. The SC8 walk started about 190 s into the scan. SkillEvaluator reports any staticruntime_limitas a Critical "skillspector scan did not execute reliably", so the row blocked submission.Reproduction: the real v2.12.0
_scan_shipped_bytecodeon a tree with the customer's layout (148 files, 50 folders, no bytecode):runtime_limitatfeatures/safetyruntime_limitFix
time.thread_time(), so only the walk thread's own CPU time counts.static_yaraalready does this for rule loading.timeout_seconds), whichresolve_inputalways sets for graph scans. Callers outside a workflow fall back toMAX_WORKFLOW_SECONDS.Tests
limit_seconds == MAX_SC8_ANALYSIS_SECONDS.MAX_WORKFLOW_SECONDSwhen there is none.tests/nodes/analyzers/test_sc8_shipped_bytecode.py: 19 passed. Supply-chain, static-pattern, workflow and transitive selection oftests/nodes/analyzersandtests/unit: 1018 passed.🤖 Generated with Claude Code