SkillEvaluator 0.9.0

SkillEvaluator 0.9.0 expands how you build evaluation datasets, compare skill versions, and understand why an agent passed or failed. It adds controls for evaluation budgets, reusable baselines, and clearer evidence across terminal, HTML, JSON, Markdown, and benchmark reports.

For PyPI users upgrading from 0.7.2, this release also includes the changes shipped in the GitHub-only 0.8.0 release.

Highlights

• Build and review better evaluation datasets. Declare deterministic checks for script execution, workspace files, and final responses. Generate datasets grounded in skill instructions, author intent, or traces; import production traces as redacted review candidates; and promote expectations only after explicit human review. Preview legacy dataset migrations before writing a new file.
• Track skill improvements over time. Compare retained skill versions with --compare-to and compare-versions, browse run history, and reuse a compatible no-skill baseline with --reuse-baseline. Version comparisons remain separate from Skill Lift against a no-skill baseline.
• Control evaluation spending. Set token, judge-call, attempt, and priced-cost budgets. Reports retain partial results when limits stop a run. Token usage is shown per case and agent, with missing counters and reused-baseline usage identified.
• Understand the evidence behind each result. Reports show met, unmet, unchecked, and inapplicable expectations; dataset coverage and sufficiency; skill activation; failed-arm coverage; and concrete next steps. Skill Lift is the headline comparison, with pass@k shown as supporting reliability. HTML charts work offline.
• Audit graders and judge decisions. Qualify the shipped verifier with tier3 validate-grader, inspect retained judge requests and replies, use opt-in judge retry and fallback policies, and ask questions about a completed run with tier3 feedback. Import externally computed custom metrics with their provenance.
• Extend repository CI. Opt-in GitHub and GitLab setup adds advisory Tier 2 and Tier 3 evaluations alongside required Tier 1 checks. Capability-gated onboarding supports independent chat, judge, and agent models, reviewed scanner baselines, retained benchmark cards, and optional trusted feedback.

Notable fixes

• Align Skill Lift, dimension applicability, verdicts, and publication status across report formats. Failed baselines are labeled incomplete rather than “not run,” and missing evidence stays visible.
• Improve Tier 2 handling of large repeated-content clusters, malformed LLM verdicts, collection coverage, and shell-script chunking. Failed catalog rebuilds preserve the previously committed registry.
• Reduce security and PII false positives while retaining real coverage gaps and scanner failures. Dependency advisories now appear as structured findings in reports and SARIF; stable finding IDs support tracking across runs.
• Improve agent setup retries, cancellation, stop-on-pass concurrency, UTF-8 dataset loading, retained feedback, and Docker setup. Upgrade the Tier 3 backend to Harbor 0.25.0.

Upgrade considerations

• Dataset generation now defaults to auto grounding. Provide substantive author guidance or usable matching traces. Skill-only starter flows need --grounding skill, including with --no-llm. Older datasets that require skill use may need outcome-only expectations to pass strict CI validation; the no-skill baseline receives the same checks.
• Some grading outcomes change. Accuracy evaluates task outcomes without penalizing a no-skill baseline for not identifying the skill. Standard judge scores are computed from verdicts. A scored attempt fails when goal accuracy or security is below 1.0; a scored dimension below 0.40 fails the Tier 3 verdict even when another dimension has no score. Re-evaluate before comparing scores with older releases.
• Baseline timeout tolerance is opt-in. --tolerate-baseline-timeouts applies only to narrowly qualified baseline execution timeouts. It leaves that agent’s baseline and Skill Lift unavailable and does not make the run complete publication evidence.
• Declare build tools explicitly. Generated Debian-based OpenCode images no longer include gcc, g++, or make. Skills or tasks needing them should declare packages in scripts/apt-packages.txt or a custom evaluation Dockerfile.
• All-tier CI keeps Tier 3 advisory. The deprecated --tier3-required, tier3-required, and SKILL_EVAL_TIER3_REQUIRED settings no longer make Tier 3 a required CI gate. Standalone validation and benchmark publication policies are unchanged.
• Python catalog rebuilds replace the selection. Callers reusing a registry must supply the complete intended catalog; successful builds no longer merge into the previous selection.
• Existing CI stays pinned. A Python package upgrade does not silently change an existing repository workflow. Review image and workflow pin updates together, preserving local configuration. The starter walkthrough remains pinned to its tested artifact.
• Hermes support is experimental. The new agent supports Docker with native NVIDIA Build routing; existing agent defaults are unchanged.

Add SkillEvaluator to your repository with CI integration

python -m pip install --upgrade skillevaluator
skillevaluator ci init

The generated workflow uses the release's prebuilt Docker image.

0.9.0 CI artifacts: use these versioned files for manual setup:

• GitHub checks YAML
• GitLab checks YAML
• Optional GitHub report publisher YAML
• Optional LLM checks YAML
• Complete CI bundle ZIP

Docker image: verified 0.9.0 image on GitHub Container Registry.

ghcr.io/nvidia/skillevaluator-tier1@sha256:e96e3aaad89bf78c4f3f8c621e88005d18bd7c485b29c0d7945864700e5944e6

Python package

Published on PyPI: SkillEvaluator 0.9.0. Published through Kitmaker and verified with fresh pip and uv installations.

With Python 3.12, 3.13, or 3.14, create and activate a virtual environment, then install:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade 'skillevaluator==0.9.0'

For optional Tier 2, Tier 3, and Python security dependencies:

python -m pip install --upgrade 'skillevaluator[all]==0.9.0'

The optional dashboard uses the separate dashboard extra. Full security scanning also needs scanner executables; live evaluation needs provider credentials and the selected agent/sandbox runtime. See the installation guide.

See the full changelog and documentation.

SkillEvaluator 0.8.0

Tier 3 now computes every score from judge verdicts, grades accuracy on the outcome alone, and explains failing scores in the HTML report. A new tier3 feedback chat answers questions about a completed run, and opt-in policies control which models a judge may fall back to. Reports give every finding stable IDs. Tier 1 accepts audited SkillSpector baselines and evidence-bound PII waivers. This release also ships a rebuilt Tier 1 scanner image and CI bundle.

pip install --upgrade skillevaluator
skillevaluator ci init

PyPI package · CLI onboarding guide · Ready-to-use CI templates

Before you upgrade

• The shared validation policy is now default (#334). The shared OSS policy for public and private repositories was called external. --profile external, --external, and SKILLEVALUATOR_PROFILE=external still work, with the same rules. Report policy metadata and the policy digest change.
• Tier 3 scores change for the same runs (#303, #286). Accuracy grades the outcome only: FORMAT_FOLLOWED replaces SKILL_IDENTIFIED, so baseline accuracy rises and accuracy lift falls. Judges return verdicts and the verifier computes every score. Eligible OpenAI judges score goal_accuracy with RAGAS.
• A Claude Code refusal now fails the arm (#337, #347). Tier 3 trials set CLAUDE_CODE_DISABLE_REFUSAL_FALLBACK=1. A refused attempt is not scored and fails the arm's attempt coverage, so tier3 evaluate exits 1; the run's result.json records the coverage error. A run that completed before can now fail; rerun it, or run the arm with another model through --agent-model claude-code=MODEL.
• Tier 1 results change (#306, #295). SkillSpector agent-threat matches in files no agent reads or runs no longer fail security. The license check parses SPDX expressions, so malformed expressions and + or -only forms of blocked licenses now fail.

New

• skillevaluator tier3 feedback RUN_DIR opens a terminal chat grounded in one completed run's scores, comparisons, and attempts (#341).
• Opt-in LLM_JUDGE_FALLBACK_POLICY, LLM_JUDGE_RETRY_POLICY, and LLM_JUDGE_FALLBACK_MODELS for the Tier 3 judges. The defaults keep the current behavior (#343).
• Stable finding_id and occurrence_id on every finding. JSON adds finding_counts, finding_groups, and message_counts; Markdown adds a Finding Reconciliation table; and the Markdown and HTML summaries count findings the same way (#310).
• --skillspector-baseline FILE applies an audited SkillSpector suppression baseline (#300). --pii-waivers PATH accepts a reviewed PII value bound to its file and evidence (#307).
• Measured Agent Resources splits agent time into LLM and tool seconds (#344). Rubric scores and Tier 3 insights name the model that wrote them (#308). The verifier saves every judge attempt to judge_attempts.json (#303).
• Private runtime runs keep a sanitized plugin_runtime_report.json that replays offline into every report format (#345).
• check_public_benchmarks.py --expected-source-* checks that each benchmark card names the expected source (#340).

Fixes

• Tier 3 runs. --stop-on-pass honors --n-concurrent (#275). Ctrl+C stops running and queued Harbor jobs (#291). Claude Code and Codex installs retry transient failures (#277). Staging works on older glibc, and sandbox errors explain Bubblewrap failures (#267).
• Tier 3 grading and reports. No false security findings for apply_patch workspace writes (#313) or HTTP method prose (#339). Behavior evidence flags a truncated final answer (#259). HTML reports explain failing scores and show baseline trial rows (#305).
• Tier 2. Large repeated-content clusters are reviewed within the prompt budget (#338, #346). Contained directory aliases such as .agents -> .claude no longer abort a check (#282). JSON verdicts wrapped in prose are recovered (#297).
• Tier 1. Incomplete SkillSpector scans say why, for a runtime limit (#294) or an LLM provider outage (#342). PII scans skip timestamps, digests, matrices, and vectors (#304), GUIDs (#273), and Proton profile paths (#307). argument-hint placeholders are no longer reported as XML tags (#273).

The release pull request, #362, lists all 33 merged pull requests in a table.

Downloads

This single product release contains the Python wheel, source distribution, CI bundle, artifact manifest, and checksums. The wheel and source distribution were built from the tagged commit 8149356e4476a1497fb334727f13f699c6fa6ba5.

The Tier 1 scanner image was rebuilt for this release from main 55d2d046 in run 42158955180, and all five smoke cases passed. The CI bundle pins workflow 55d2d046595e61cce6ba9f5fd9cc84b972cd3915 and image ghcr.io/nvidia/skillevaluator-tier1@sha256:9d5483243918937be4f2178f74cf7c622a531dee41e3176906994d52fa14950a. The report publisher revision is also 55d2d046595e61cce6ba9f5fd9cc84b972cd3915. Scanner versions are unchanged from 0.7.2. The image was built before the version bump, so the package inside it reports version 0.7.2.

Full changelog

What's Changed

• fix(tier3): stage over existing trees on older glibc and explain bwrap failures by @rng1995 in #267
• fix(tier3): stop running and queued Harbor jobs on Ctrl+C by @rng1995 in #291
• fix(license): parse SPDX license expressions by @rng1995 in #295
• fix(tier1): avoid false positives for argument hints and GUIDs by @rng1995 in #273
• feat(tier1): apply audited SkillSpector baselines by @rng1995 in #300
• fix(tier3): retry transient Claude Code and Codex install failures by @rng1995 in #277
• fix(security): explain incomplete SkillSpector scans that hit a runtime limit by @rng1995 in #294
• fix(tier3): flag behavior evidence shortened by the final-response section limit by @xh123637 in #259
• fix(tier3): compute judge scores from verdicts and save judge evidence by @rng1995 in #303
• fix(tier3): stop apply_patch workspace writes scoring as credential access by @rng1995 in #313
• fix(tier2): allow contained directory aliases by @rng1995 in #282
• fix(pii): stop reporting timestamps, matrices, and vectors as PII by @rng1995 in #304
• fix(tier3): honor --n-concurrent with --stop-on-pass by @rng1995 in #275
• fix(tier2): review large repeated-content clusters within the prompt budget by @rng1995 in #338
• fix(tier3): score goal_accuracy with RAGAS for eligible OpenAI judges by @rng1995 in #286
• fix(tier1): scope SkillSpector agent-threat findings to agent control paths by @rng1995 in #306
• feat(reporting): reconcile stable finding IDs across report formats by @rng1995 in #310
• feat(llm): record the model that wrote rubric scores and Tier 3 insights by @rng1995 in #308
• fix(inference): recover one JSON object wrapped in LLM prose by @rng1995 in #297
• fix(ci): use a neutral default validation policy by @...

SkillEvaluator 0.7.2

Add hosted reports to an existing GitHub skill-checks workflow while keeping its shared component, folders, thresholds, and triggers. A centrally maintained component such as @v3 can now update its pinned evaluator without requiring an edit in every consumer repository. Each report retains the signed evaluator identity and resolved workflow revisions.

pip install --upgrade skillevaluator
skillevaluator ci init

PyPI package · CLI onboarding guide · Ready-to-use CI templates

GitHub onboarding and reporting

• Preserve an existing supported scan workflow while adding hosted HTML and one updated PR comment. Direct evaluator calls and shared-component calls are supported.
• Verify the shared component's identity permission before changing repository settings. Existing Actions restrictions, unrelated Pages sites, and deployment approvals remain protected.
• Reuse an administrator-configured private Pages site in eligible GitHub Enterprise Cloud organization repositories. Verify its privacy and expected URL before retaining reports and again before deployment. If private hosting is unavailable, checks and downloadable reports remain available.
• Improve repeat setup, custom scan job names, Windows workflow writes, and CRLF approval hashes.

Real consumer tests covered direct and shared-component setup, changed-skill selection, hosted HTML, updated PR comments, failing and recovered scans, central tag updates, and historical report backfills. Private scanning passed. Private Pages deployment was not tested end to end: an eligible Enterprise sandbox was unavailable.

Other Python package changes since 0.7.1

• Add reproducible local-catalog decoy selection for Tier 3 with tier3 select-decoys, portable manifests, and verified staged content (#263).
• Accept lists in metadata.domain, with explicit limits of 16 entries and 128 characters per entry (#279).
• Fix Tier 2 self-match handling and chunk budgets (#296, #270).
• Fix Tier 3 grading, source-error handling, Codex credential-directory cleanup, and benchmark display behavior (#265, #309, #287, #290, #276, #269).

Downloads

This single product release contains the Python wheel, source distribution, CI bundle, artifact manifest, and checksums.

The CI bundle reuses the existing verified scanner workflow 46a8cf269257d1ec56a0285c508ac9ee48010930 and image ghcr.io/nvidia/skillevaluator-tier1@sha256:a3baec51f48145cdf9de05a5ab054d9576f62f1de0f5276b788410f5c17747d4. No scanner image was rebuilt: the additional Python-package changes above do not change that image. The report publisher revision is 8096c6966a498f35f6f7dc1a596d1cc4da85cf11.

Full changelog

SkillEvaluator 0.7.1 packages guided GitHub onboarding, improves CI report feedback, and includes scanner and evaluation runtime fixes.

Highlights

• Set up repository checks with skillevaluator ci init; use ci status to inspect setup and ci demo to open an optional test PR.
• Check changed skills with Tier 1 quality checks and SkillSpector security scanning. Optional GitHub feedback includes hosted HTML reports and one updated PR comment.
• Improve scanner setup, local agent installation, Tier 3 progress and results, and provider readiness checks. See the changes below.

Python package

SkillEvaluator 0.7.1 on PyPI · NVIDIA Python package index

Requires Python 3.12, 3.13 or 3.14. Install or upgrade:

pip install --upgrade skillevaluator

Both the wheel and source distribution are published on PyPI and NVIDIA's index. Package files and SHA-256 checksums are attached below.

CI integration

From your cloned GitHub repository, run:

skillevaluator ci init

The wizard prepares a setup PR for review. Automatic setup requires an existing GitHub CLI login and repository admin access. Checks-only supports public and private repositories; automatic hosted reports and PR comments support public repositories.

For manual GitHub or GitLab setup, download the ready-to-copy CI templates. The bundle includes immutable workflow/image pins, scanner versions and checksums. Open the template folder to copy individual files. The ZIP and checksums are also stored there.

CLI setup guide · GitHub Actions · GitLab CI

What's Changed

• fix(tier2): report a chat-model refusal as an advisory dedup finding by @rng1995 in #281
• fix(security): make the SkillSpector timeout configurable by @rng1995 in #271
• feat(doctor): add Tier 1 scanner and Tier 2 provider readiness checks by @rng1995 in #293
• fix: decode Git roots as file names and keep PowerShell prefix findings (#312 review) by @rng1995 in #320
• feat(tier3): add --skip-with-skill for baseline-only runs by @rng1995 in #284
• fix(security): accept verified referenced images in partial SkillSpector scans by @rng1995 in #299
• test(security): cover SkillSpector watchdog duration boundaries by @rng1995 in #319
• fix(tier3): write plain progress heartbeats once a minute by @rng1995 in #274
• fix(security): handle read-only passwd mounts with SkillSpector 2.13.3 by @rng1995 in #289
• fix(docs): render README images in PyPI descriptions by @chrisknvidia in #323
• Private runtime: prepared packages, native loading and evidence over the existing engine by @chrisknvidia in #264
• Offer to install a missing agent CLI in local mode by @chrisknvidia in #317
• Show per-case Tier 3 results and a "what to fix" list by @chrisknvidia in #316
• chore(release): reconcile published 0.7.0 metadata by @chrisknvidia in #314
• Set up Tier 1 scanners on first use and add --skip by @chrisknvidia in #318
• fix(ci): require Semgrep in scanner subprocesses by @chrisknvidia in #324
• fix(ci): show concise PR feedback with fresh HTML report links by @chrisknvidia in #326
• feat(ci): ship guided GitHub onboarding in SkillEvaluator 0.7.1 by @chrisknvidia in #327

Full Changelog: 0.7.0...v0.7.1

SkillEvaluator 0.7.0

SkillEvaluator 0.7.0 adds Python 3.14 support alongside Python 3.12 and 3.13, reusable Tier 1 CI integration, configurable quality policies, and clearer evaluation reports.

Highlights

• Python 3.14 support, with full-suite CI and built-package installation checks across Python 3.12, 3.13, and 3.14.
• Reusable GitHub Actions and GitLab CI checks, parallel skill scanning, and configurable quality headings and thresholds.
• Verdict-led HTML reports, case-first Tier 3 summaries, matching-judge comparisons, and actionable recommendations in what_to_fix.json.
• More precise Tier 2 completeness reporting, Tier 3 timeout/recovery and local-mode fixes, and an optional --max-cases limit for exploratory runs.
• SkillSpector 2.13.3 installation and CI pins, preserved NVIDIA Build model-registry handling, and validated LLM provenance in JSON/HTML reports.
• Refreshed locked runtime and dashboard dependencies. The Python package's dependency ranges are unchanged.

Python package

PyPI publication: published and verified. Both the wheel and source distribution are available on PyPI and NVIDIA’s Python package index. Their downloaded SHA-256 hashes match the release files below.

Versioned package page: skillevaluator 0.7.0 on PyPI.

Create a virtual environment using Python 3.12, 3.13, or 3.14 and install the package:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install skillevaluator==0.7.0

For the optional Python evaluation and security dependencies:

python -m pip install 'skillevaluator[all]==0.7.0'

A standalone uv installation is also supported:

uv tool install --python 3.14 'skillevaluator==0.7.0'

The all extra includes the Tier 2, Tier 3, and Python security dependencies. Full static security scanning additionally requires separately installed SkillSpector, Gitleaks, and Semgrep executables. Upgrade existing SkillSpector installations explicitly to v2.13.3; Claude 5.5 requires a supported explicit SKILLSPECTOR_MODEL with this scanner release. Live evaluations require provider credentials and the selected agent/sandbox runtime. The optional dashboard is installed with the separate dashboard extra.

See the installation guide for scanner and runtime setup.

Release files

This release uses the GitHub tag 0.7.0 at commit ad2815fe78c045db91d230681d32863698da08cf. The earlier v0.7.0 tag name was consumed by an immutable release and cannot be reused. The Python package version is 0.7.0.

The Assets section contains the installable Python wheel (skillevaluator-0.7.0-py3-none-any.whl), Python source distribution (skillevaluator-0.7.0.tar.gz), and their SHA-256 checksums. GitHub also supplies repository source archives and a release attestation. The wheel supports all three Python versions listed above.

Installation verification

Fresh PyPI installations were verified with pip using both wheel and source distributions on Python 3.12, 3.13, and 3.14, plus uv tool installation and the all extra on Python 3.14. Checks included dependency consistency, CLI execution, and real offline validation reports in JSON, HTML, Markdown, and SARIF formats. New provider-backed LLM or Tier 3 evaluations were not run for this package publication.

Package checksums

What's Changed

• Fix Windows subprocess decoding, reasoning-model token budget, and harbor bin resolution by @Minervaowl7 in #194
• fix(cli): include nested skills in catalog validation by @YMuskrat in #210
• fix(quality): keep minimum score failures after policy changes by @YMuskrat in #211
• fix(tier3): keep skill lift consistent when rebuilding reports by @YMuskrat in #212
• feat(tier1): score tier 1 quality checks via a configurable YAML file by @westonplatter in #213
• feat(tier3): let installed packages add custom execution backends by @rmalani-nv in #215
• fix(tier3): score unrecovered failures and align recovery verdicts by @mavericksea-ai in #217
• docs: simplify first runs with keyless checks and NVIDIA Build by @chrisknvidia in #218
• fix(judge): treat an "unavailable" schema option as a schema rejection by @MaxFreedomPollard in #219
• Pass Windows program dirs and Python UTF-8 mode to Harbor by @GodfredArhinful in #220
• fix(tier3): retain scores after verified timeout recovery by @chrisknvidia in #222
• fix(reporting): retain completed-agent results in Tier 3 reports by @chrisknvidia in #223
• docs: refresh evaluation guidance and expose dashboard guide by @rng1995 in #224
• fix(quality): score non-mapping metadata instead of crashing by @MaxFreedomPollard in #226
• fix(tier2): bound embedding chunks and preserve source evidence by @chrisknvidia in #227
• fix(tier3): avoid destructive scoring for plain cache cleanup by @chrisknvidia in #228
• fix(tier3): count explicit root searches as wasted tool calls by @chrisknvidia in #229
• fix(tier3): flag deletions that escape temporary paths by @YMuskrat in #230
• fix(tier3): flag destructive rm targets supplied by xargs by @YMuskrat in #231
• fix(tier3): gate nested cleanup findings on symlink evidence by @YMuskrat in #232
• docs(cli): compare direct tier workflows with validate --tiers by @chrisknvidia in #237
• feat(ci): add reusable Tier 1 scans and GitHub onboarding by @chrisknvidia in #238
• fix(tier3): preserve local verifier uploads and shared step files by @chrisknvidia in #239
• Add opt-in Unicode character facts for Tier 3 judges by @mavericksea-ai in #242
• fix(pii): recognize reserved placeholder email domains by @rng1995 in #246
• test: wait for descendant exit after streamed exec cleanup by @rng1995 in #247
• Add reusable GitLab skill evaluation and repository onboarding by @chrisknvidia in #248
• Share deterministic grading and add bounded input readers by @chrisknvidia in #249
• Add private manifest discovery and configuration checks by @chrisknvidia in #250
• fix(reporting): preserve unavailable Tier 3 evaluator scores by @rng1995 in #251
• fix(tier3): preserve local Claude Code instruction variables by @rng1995 in #252
• fix(rubric): preserve source and complete linked guidance by @rng1995 in #254
• feat(tier3): case-first CLI summary and case-based report findings by @chrisknvidia in #255
• perf(ci): faster Tier 1 scans and a clearer HTML report by @rng1995 in #256
• Compose private inventory, audits and catalog checks by @rng1995 in #257
• Share trace evidence, applicable scores and security checks by @rng1995 in #258
• fix: address review findings from #249 and #250 by @rng1995 in #260
• ci: run main test jobs in parallel (-n auto) by @rng1995 in #261
• fix(tier3): write generated test.sh and grader starters with LF endings by @rng1995 in #266
• fix(tier3): disable Claude Code background tasks in eval trials by @rng1995 in #268
• fix(quality): report the 5,000-token skill budget as advisory by @rng1995 in #272
• fix(tier3): launch the Bubblewrap binary that local mode detected by @rng1995 in #278
• fix(tier2): report context-deduplication work limits as incomplete by @rng1995 in #280
• feat(tier3): add opt-in --max-cases limit for exploratory runs by @rng1995 in #283
• fix(reporting): stop labeling recorded Tier 3 baselines as missing by @rng1995 in #285
• fix(security): keep SkillSpector's nv_build model registry for Build scans by @rng1995 in #288
• chore(deps): pin SkillSpector to v2.13.3 by @rng1995 in #292
• fix(security): redact credentials in declared component paths by @rng1995 in https://github.com/NVIDIA/SkillEvaluator/pull...

What's Changed

• docs: bring the published docs in line with v0.5.0 by @rng1995 in #186
• fix(embedding): preserve multiline descriptions in catalogs by @rioyu123 in #191
• test: clear two test-only CodeQL alerts by @chrisknvidia in #184
• feat(report): show the SkillSpector version in the Tier 1 HTML card by @rng1995 in #187
• fix: avoid redundant Node/npm during Docker agent setup by @chrisknvidia in #193
• feat: default to GPT-6.1 Sol and Claude Opus 5.5 across Tiers 1-3 by @rng1995 in #188
• fix(tier3): honor per-step tokens when detecting runtime failures by @YMuskrat in #199
• fix(tier3): include ATIF content parts in judge answer evidence by @YMuskrat in #203
• fix: target dependency coverage and Tier 3 result integrity by @chrisknvidia in #196
• feat(tier3): run hardened Docker trials under Kata Containers by @rng1995 in #182
• fix(tier3): derive accuracy scores from criterion verdicts by @mavericksea-ai in #190
• fix(tier3): exclude failed skill reads from execution credit by @YMuskrat in #201
• fix(reporting): require complete tier evidence for publication by @chrisknvidia in #77
• fix(tier3): use one overall-score formula by @Saibernard in #124
• feat(tier3): expose Harbor's Runta and Mosaic backends by @rng1995 in #183
• feat(tier3): support Claude Code on Agent Platform and add active preflight probe by @kweinmeister in #140
• feat(reporting): add quantitative skill performance dashboard by @chrisknvidia in #179
• fix(tier3): mark never-activated skill evaluations inconclusive by @chrisknvidia in #209
• ci: cover every workflow with the hardening guards, and fix the gaps they surface by @dberzan in #126
• chore(release): prepare v0.6.0 by @rng1995 in #216

New Contributors

• @YMuskrat made their first contribution in #199
• @Saibernard made their first contribution in #124
• @dberzan made their first contribution in #126

Full Changelog: v0.5.0...v0.6.0

Wheel and source distribution for the existing SkillEvaluator v0.6.0 release, built from commit 5320d9329c36ecc72bc94c4f58cc813d6b54224b.

These files provide stable download URLs for managed PyPI publication. Their presence here does not indicate that publication to the package index has completed. Package version and product behavior remain v0.6.0.

What's Changed

• feat(tier3): make evidence bundle budgets and final response limit configurable by @kweinmeister in #160
• feat(tier3): enforce structured JSON schemas and rate-limit retries in LLM judges by @kweinmeister in #145
• feat(tier3): rebuild ATIF from OpenCode and Codex agent logs by @mimran-khan in #119
• fix(tier3): propagate Harbor dual-arm task suffixes and normalize canonical case ID resolution by @kweinmeister in #161
• fix(validation): report non-string YAML keys by @rioyu123 in #178
• feat(tier3): upgrade Harbor to 0.24.0 by @chrisknvidia in #83
• chore(release): prepare v0.5.0 by @rng1995 in #185

Full Changelog: v0.4.0...v0.5.0

SkillEvaluator v0.4.0 simplifies running evaluations and strengthens reporting, security checks, and large-catalog analysis.

Highlights

• Run tiers directly with skillevaluator tier1 PATH, tier2 PATH, or tier3 PATH, with clearer provider setup, gateway model defaults, and evaluation progress.
• Scan larger Tier 2 collections with explicit entry and comparison budgets, faster vector comparisons, and actionable service-failure diagnostics.
• Improve Tier 3 script-invocation evidence, protected-path write detection, token masking, and incomplete-run feedback.
• Harden CLI rendering and secret redaction, run the Docker image as an unprivileged user, and pin the HTML report's Chart.js dependency with integrity verification.

Upgrade notes

• validate PATH now runs all three tiers for skills by default. Use --tiers 1 for keyless static CI; --tiers, --no-tier3, and --no-autopilot provide explicit scope controls.
• When --agents is omitted, Tier 3 selects a provider-native agent. Extra agent runtime preflight is opt-in through --agent-runtime-preflight.
• The NVIDIA Build defaults are now nvidia/nemotron-3-super-120b-a12b for chat and nvidia/nemotron-3-embed-1b for embeddings. Rebuild existing catalogs and caches when switching embedding models.

See the v0.4.0 changelog for the full release notes.

What's Changed

• test: stabilize FIFO subprocess checks by @deepujain in #148
• docs: link SkillEvaluator research paper by @chrisknvidia in #146
• docs: add docs/livestream/blog links to README by @mosheabr in #159
• feat(cli): simplify tier workflows and improve evaluation feedback by @chrisknvidia in #152
• fix(reporting): verify the pinned Chart.js dependency by @deepujain in #162
• fix(container): use an unprivileged runtime account by @deepujain in #163
• Keep fenced code within its Markdown section during chunking by @sylvesterkaczmarek in #165
• fix: keep --no-llm negative cases off-skill by @mimran-khan in #100
• fix: harden all CI workflow guards by @efegokdemir in #158
• fix(tier2): support larger bounded collection scans by @rng1995 in #166
• fix(tier3): require invocation evidence for script execution credit by @mavericksea-ai in #154
• fix(security): stop reporting User-Agent versions as IP addresses by @JayOfTheKeyboard in #168
• fix(reporting): escape untrusted text in Rich CLI output by @rng1995 in #173
• fix(tier3): detect apply_patch writes to protected paths and mask GitHub tokens by @rng1995 in #174
• fix(redaction): make sensitive-key, JWT, and private-key matching linear-time by @rng1995 in #172
• fix(reporting): strip terminal controls from catalog and view output by @rng1995 in #175
• fix(tier3): make log JWT redaction linear-time by @rng1995 in #176
• chore(release): prepare v0.4.0 by @rng1995 in #177

New Contributors

• @sylvesterkaczmarek made their first contribution in #165
• @efegokdemir made their first contribution in #158
• @mavericksea-ai made their first contribution in #154
• @JayOfTheKeyboard made their first contribution in #168

Full Changelog: v0.3.0...v0.4.0

What's Changed

• docs: link the NVIDIA Verified Skills docs from SkillEvaluator by @mosheabr in #46
• Fix Tier 2 scans for contained compatibility alias by @chrisknvidia in #36
• fix: replace OSS boundary line allowlist with stable anchors by @chrisknvidia in #40
• ci: skip heavy jobs for docs-only pull requests by @chrisknvidia in #48
• fix(tier3): isolate per-task inputs and evaluator fixtures by @chrisknvidia in #44
• feat(validate): replay explicit tier gating onto main by @rng1995 in #49
• Ignore short env values in secure Docker redaction by @Hashim1999164 in #51
• fix(quality): avoid substring false positives by @chrisknvidia in #45
• chore(release): prepare v0.2.0 by @rng1995 in #52
• chore: centralize frontier OpenAI/Anthropic chat defaults by @chrisknvidia in #38
• feat(reporting): roll out unified benchmark publication card by @rng1995 in #47
• fix(tier3): propagate judge model overrides to Harbor verifiers by @chrisknvidia in #54
• fix(reporting): use commit SHA for pull request links by @danielgaskins in #60
• fix(tier3): align Anthropic endpoints and fail closed on judge errors by @chrisknvidia in #56
• Fix quality scoring with CommonMark structural parsing by @chrisknvidia in #53
• fix(tier3): handle negative controls in insights by @rookepoole in #58
• fix(tier3): parse SKILL.md frontmatter as YAML by @genkey6 in #68
• chore(release): prepare v0.2.1 by @rng1995 in #76
• fix(tier3): score Harbor single-step null results by @chrisknvidia in #74
• fix(tier3): fail fast on rejected credentials by @chrisknvidia in #75
• feat(tier3): report pass@k uncertainty and paired outcomes by @AdemVessell in #63
• fix(ci): scope Gitleaks to checked-out history by @rioyu123 in #106
• fix(tier3): avoid transcript runtime false positives by @rioyu123 in #84
• fix: scan Markdown headings for PII instead of skipping them by @mimran-khan in #96
• fix(tier3): retry malformed structured judge responses by @chrisknvidia in #78
• fix(tier3): detect a results directory the Docker daemon cannot see by @genkey6 in #80
• fix: skip gitleaks test dirs, not substring names by @mimran-khan in #102
• fix(hygiene): use CommonMark for dead-link checks by @rioyu123 in #109
• fix: keep --llm-verify file reads inside the skill by @mimran-khan in #101
• feat: add SARIF reporter for GitHub Code Scanning by @mimran-khan in #117
• fix: do not let frontmatter MIT hide a GPL LICENSE file by @mimran-khan in #97
• fix: treat tools/ as an executable directory by @mimran-khan in #103
• fix: flag Windows PII usernames that start with s by @mimran-khan in #95
• fix: evaluate full SPDX license expressions by @mimran-khan in #98
• fix(tier3): decode native Codex exec calls by @Tomauskasz in #111
• fix(hygiene): detect unpinned dependencies with environment markers by @rioyu123 in #107
• fix: accept UTF-8 BOM in SKILL.md frontmatter by @mimran-khan in #99
• Honor SkillSpector analysis completeness by @Tomauskasz in #112
• feat(model-catalog): allow plain HTTP for named hosts by @Tomauskasz in #135
• Fix empty process-control environment resets by @RyaliNvidia in #133
• fix: map Harbor trial folders to eval case ids by @mimran-khan in #104
• fix(cli): report malformed policy files cleanly by @rioyu123 in #129
• feat(cli): write catalog-summary.json after catalog validate by @mimran-khan in #121
• feat(cli): parallel catalog validation and fleet rollup by @mimran-khan in #123
• fix(security): accept fully covered documentation scans by @deepujain in #143
• fix(tier3): prevent false positives in network exfiltration checks by @kweinmeister in #141
• fix(embedding): reject non-progressing chunk windows by @deepujain in #142
• feat(reporting): record the evaluated source identity on benchmark cards by @MaxFreedomPollard in #131
• chore(release): prepare v0.3.0 by @rng1995 in #144

New Contributors

• @mosheabr made their first contribution in #46
• @Hashim1999164 made their first contribution in #51
• @danielgaskins made their first contribution in #60
• @rookepoole made their first contribution in #58
• @genkey6 made their first contribution in #68
• @AdemVessell made their first contribution in #63
• @rioyu123 made their first contribution in #106
• @mimran-khan made their first contribution in #96
• @Tomauskasz made their first contribution in #111
• @RyaliNvidia made their first contribution in #133
• @deepujain made their first contribution in #143
• @kweinmeister made their first contribution in #141
• @MaxFreedomPollard made their first contribution in #131

Full Changelog: v0.1.0...v0.3.0