Skip to content

fix(sc8): count only the walk's own CPU time against its runtime budget - #828

Merged
yashrajp22 merged 5 commits into
mainfrom
naren/fix-sc8-thread-time-budget
Oct 16, 2026
Merged

yashrajp22 merged 5 commits into
mainfrom
naren/fix-sc8-thread-time-budget

Conversation

@rng1995

@rng1995 rng1995 commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The SC8 shipped-bytecode walk (_scan_shipped_bytecode) stops after MAX_SC8_ANALYSIS_SECONDS = 5.0 of wall-clock time. All analyzers run in parallel threads in one process. Each os.scandir step releases the GIL, and the walk then waits behind CPU-bound analyzers to get it back. A walk that needs a few milliseconds of work can therefore run out of time.

In a customer report (SKILLEVAL-17), a skill with no __pycache__, .pyc or .pyo files got a static runtime_limit ledger row. The folder in it changed on every run, because it was wherever the walk had reached when time ran out. The SC8 walk started about 190 s into the scan. SkillEvaluator reports any static runtime_limit as a Critical "skillspector scan did not execute reliably", so the row blocked submission.

Reproduction: the real v2.12.0 _scan_shipped_bytecode on a tree with the customer's layout (148 files, 50 folders, no bytecode):

Concurrent load Before After
none 0.06 s, no limitations 0.04 s, no limitations
8 pure-Python CPU threads 5.2 s, runtime_limit at features/safety 7.4 s, no limitations
4 threads compiling large sources 7.5 s, runtime_limit 345 s wall-clock, no limitations (direct call, no workflow deadline)

Fix

  • Measure the 5 s processing budget with time.thread_time(), so only the walk thread's own CPU time counts. static_yara already does this for rule loading.
  • Bound wall-clock time by the remaining workflow deadline (timeout_seconds), which resolve_input always sets for graph scans. Callers outside a workflow fall back to MAX_WORKFLOW_SECONDS.
  • When a limit trips, the ledger row reports the observed and limit seconds of the budget that tripped.

Tests

  • New: waiting time does not count against the budget (fake clock: wall-clock advances, CPU time does not). This test fails on main.
  • New: the CPU budget still stops the walk, with limit_seconds == MAX_SC8_ANALYSIS_SECONDS.
  • New: wall-clock is bounded by the workflow deadline, and by MAX_WORKFLOW_SECONDS when there is none.
  • tests/nodes/analyzers/test_sc8_shipped_bytecode.py: 19 passed. Supply-chain, static-pattern, workflow and transitive selection of tests/nodes/analyzers and tests/unit: 1018 passed.

🤖 Generated with Claude Code

rng1995 and others added 2 commits October 14, 2026 04:51
The SC8 shipped-bytecode walk stopped after 5 s of wall-clock time. Analyzers
run concurrently in one process, so the walk spent most of that time waiting
for the GIL behind CPU-bound analyzers. Skills that ship no bytecode got a
static runtime_limit ledger row at a different folder on every run, which
SkillEvaluator reports as a Critical "skillspector scan did not execute
reliably" (SKILLEVAL-17).

Measure the 5 s processing budget with time.thread_time(), as static_yara
already does for rule loading. Wall-clock time stays bounded by the remaining
workflow deadline, or by MAX_WORKFLOW_SECONDS for callers outside a workflow.

Signed-off-by: Narendran Raghavan <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>

@chrisknvidia chrisknvidia left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed commit 922317e167a47d2ac3819c28edb481519f90d2e2. Looks good to me; no blocking findings.

I checked CPU-time accounting in _scan_shipped_bytecode, propagation of the remaining workflow wall-clock budget, limitation reporting, and the existing traversal/count/finding bounds. All 36 focused SC8 and budget-configuration tests passed.

I also exercised the actual filesystem walk under concurrent CPU-thread load: the baseline hit its wall-clock limit, while this version completed without a limitation. Positive controls still reported shipped bytecode, and an explicit wall-clock allowance still stopped the walk with runtime_limit.

Validation was local on macOS. No live LLM-provider or production validation was performed.

github-actions Bot and others added 3 commits October 15, 2026 06:48
…time-budget

Resolve the CHANGELOG.md conflict by keeping both the #828 SC8 entry and
the #808 CLI-provider entry from main.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
@yashrajp22
yashrajp22 merged commit 182a475 into main Oct 16, 2026
6 checks passed
@yashrajp22
yashrajp22 deleted the naren/fix-sc8-thread-time-budget branch October 16, 2026 10:45
yashrajp22 added a commit that referenced this pull request Oct 16, 2026
fix(sc8): count only the walk's own CPU time against its runtime budget

(cherry picked from commit 182a475)
Signed-off-by: yashrajbasav <[email protected]>
timothybrush pushed a commit to timothybrush/SkillSpector that referenced this pull request Oct 19, 2026
Since NVIDIA#828, the shipped-bytecode walk counts only its own CPU time against
the 5-second budget, and the workflow wall-clock deadline bounds elapsed
time. Word the resource-bounds row the same way as the YARA rule-load row.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>
chrisknvidia pushed a commit to NVIDIA/SkillEvaluator that referenced this pull request Oct 30, 2026
The install docs, the CI recipe and the missing-scanner hint pinned
SkillSpector v2.12.0. That release can make the Tier 1 security check
INCOMPLETE, or fail it, for skills with no real issue:

- valid Python and Perl helpers (for example a `len(x) < 12` comparison
  followed by more code, or a Perl `eval { require $file; 1 }` block) are
  recorded as `static_parse_limit` or `obfuscated_instruction_text`
  ledger exceptions, with a High AE1 at each line that names the helper
  (NVIDIA/SkillSpector#777);
- the SC8 bytecode walk charges time spent waiting behind other analyzers
  to its budget and can stop with a static `runtime_limit`
  (NVIDIA/SkillSpector#828);
- visible text between two HTML comments raises a High P2
  hidden-instruction finding (NVIDIA/SkillSpector#452).

SkillSpector v2.13.1 contains all three fixes. Move every install spec to
v2.13.1, keep the rule that docs and code name exactly one release, and
keep the Claude 5.5 caveat: v2.13.1 still lacks the newer structured-output
support, so SKILLSPECTOR_MODEL must still name a supported model when
SKILL_EVAL_LLM_MODEL is a Claude 5.5 model. The SkillSpector model
defaults are unchanged.

Add a troubleshooting entry for false incomplete helper scans on older
releases, and a captured v2.13.1 --no-llm report (from
tests/fixtures/skills/simple) to the captured-report acceptance test so the
strict completeness validator is exercised against the pinned release.

Signed-off-by: Narendran Raghavan <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>
rng1995 added a commit to NVIDIA/SkillEvaluator that referenced this pull request Nov 2, 2026
* chore(deps): pin SkillSpector to v2.13.1

The install docs, the CI recipe and the missing-scanner hint pinned
SkillSpector v2.12.0. That release can make the Tier 1 security check
INCOMPLETE, or fail it, for skills with no real issue:

- valid Python and Perl helpers (for example a `len(x) < 12` comparison
  followed by more code, or a Perl `eval { require $file; 1 }` block) are
  recorded as `static_parse_limit` or `obfuscated_instruction_text`
  ledger exceptions, with a High AE1 at each line that names the helper
  (NVIDIA/SkillSpector#777);
- the SC8 bytecode walk charges time spent waiting behind other analyzers
  to its budget and can stop with a static `runtime_limit`
  (NVIDIA/SkillSpector#828);
- visible text between two HTML comments raises a High P2
  hidden-instruction finding (NVIDIA/SkillSpector#452).

SkillSpector v2.13.1 contains all three fixes. Move every install spec to
v2.13.1, keep the rule that docs and code name exactly one release, and
keep the Claude 5.5 caveat: v2.13.1 still lacks the newer structured-output
support, so SKILLSPECTOR_MODEL must still name a supported model when
SKILL_EVAL_LLM_MODEL is a Claude 5.5 model. The SkillSpector model
defaults are unchanged.

Add a troubleshooting entry for false incomplete helper scans on older
releases, and a captured v2.13.1 --no-llm report (from
tests/fixtures/skills/simple) to the captured-report acceptance test so the
strict completeness validator is exercised against the pinned release.

Signed-off-by: Narendran Raghavan <[email protected]>
Co-Authored-By: Claude Opus 5.5 <[email protected]>

* chore(ci): install SkillSpector v2.13.1 in the Tier 1 image

The install docs and the missing-scanner hint name SkillSpector v2.13.1,
but the Tier 1 CI image still installed the v2.12.0 source archive
(c7958a3), so CI scans kept the false incomplete and false High findings
that v2.13.1 fixes.

Point ci/requirements/skillspector.in and the hash-locked
skillspector.txt at the archive of the v2.13.1 tag commit
(f3b25697d76256998ca9d42c198d256258d4c035) and update the
THIRD_PARTY_NOTICES row. v2.13.1 declares the same dependencies as
v2.12.0, so only the archive URL and its hash change; the documented
uv pip compile command gives the same lock.

Extend test_skillspector_install_instructions_pin_one_release so the CI
lock's archive commit and the notices row must match the documented
release, and mention the image in the CHANGELOG entry.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>

* chore(deps): pin SkillSpector to v2.13.3

Move the install instructions, Tier 1 CI lock (commit 38a1dfe, archive sha256
recomputed), third-party notice, and captured --no-llm fixture from v2.13.1
to v2.13.3.

Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
Signed-off-by: Narendran Raghavan <[email protected]>

---------

Signed-off-by: Narendran Raghavan <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants