Skip to content

Commit db13b57

Browse files
committed
Add Warp compile-time optimizer skill and release evidence
* Attach NVSkills validation signatures Signed-off-by: nvskills-svc-account <[email protected]> * Add Warp compile-time optimizer skill Existing guidance did not connect Warp's module identity model to repeatable measurements, making it easy to trade correctness or runtime performance for a misleading cold-start improvement. Add a measurement-first skill with isolated-cache tooling, mechanism references, and evaluated fixtures. This keeps workload preservation and explicit tradeoff reporting central to compile-time optimization. SKILL.md is roughly 3.3k tokens, under the 5k advisory limit that was previously a blocking finding under the external validation profile. Its description enumerates the phrasings that should route here, including requests to optimize or improve compile times rather than only startup stalls, and its frontmatter declares the env capability the probe uses via allowed-tools rather than the permissions field SkillSpector's remediation text suggests, because the Agent Skills spec defines no such field (NVIDIA/SkillSpector#300). The eleven eval fixtures keep their pyproject.toml. They had been removed because scanning the complete skill directory, which the release checklist asks for, returned CRITICAL / DO NOT INSTALL with every critical finding an unpinned numpy or setuptools in fixture packaging rather than anything in the skill. Pinning does not help there: the advisory lookup keys on package name and ignores version constraints, so a floor still returns the package's worst-ever CVE, and pinning left the score at 100/100 CRITICAL while removing the files moved it to 42/100 MEDIUM. That scan is not what gates publication. The validation pipeline blocks on Tier 1 critical and high findings, Tier 1 runs skillevaluator, and skillevaluator stages a filtered copy that drops evals/ before SkillSpector sees it. Its dependency validator also reads the skill root only. Measured with all eleven manifests restored, skillevaluator validate returns 11 of 11 checks passed, exit 0, quality A at 94.8, no high or critical findings, and no reference to the fixture paths anywhere in the report. Removing the manifests cost fixture fidelity and bought nothing, so they are back, and the fixtures build and run from their source trees again, verified including the CUDA ones. The nine CUDA fixtures refuse to start when Warp sees no device. Each entry point checks wp.get_cuda_device_count() and raises, and the PDE stepper no longer falls back to the CPU. On a runner without a GPU an agent would otherwise route around the missing device and produce a result that has nothing to do with what the case measures. The checks sit in entry points rather than kernel modules, so generated source and module hashes are untouched, and after the tile projection --relink branch, which only clears caches on disk and needs no device. The two CPU fixtures are unchanged. The eval dataset covers 18 cases, in the agentskills.io shape SkillEvaluator prefers rather than the legacy flat array, which now raises a deprecation warning during validation. Its fifteen negative cases set expected_skill explicitly instead of null, which is what the evaluator's own documentation and generator produce. The grader searches the trajectory for that string, so a null value makes a negative case score perfectly without inspecting anything: an agent that fully activates the skill on an unrelated CUDA question could not fail. should_trigger stays written out alongside it, since it otherwise defaults from expected_skill and would flip those cases to positive. The two dense-stage cases state launch and output preservation as expected behaviors rather than only as hard gates. Nothing in the harness reads hard_gates, so an agent that rewrote the in-place relaxation into a double buffer, adding a launch and changing the printed values, was graded by nothing that would catch it. The probe declines to report a measurement it cannot stand behind. A run where every sample exits non-zero now fails rather than publishing a near-zero median that reads as a large improvement, and the same-work-compiled verdict is derived from compiled module identities instead of their counts, so a module rebuilt under different options is no longer written off as build contention. A warm repeat that exits non-zero is dropped the way a timeout already is: it exits early, so its wall clock is not what a repeat invocation costs, and crediting it distorts the cold-minus-warm elapsed measure the report leads with. The Triton negative case now ships a fixture. It described a Triton inference service but declared no files of its own, and the harness stages the whole evals/files corpus into every trial before applying what a case declares, so a task's file list is additive and can never narrow it. The agent listed its workspace, found Warp kernels in a neighbouring fixture, and told the user their service was Warp rather than Triton; the grader then scored that against ground truth expecting Triton-only guidance, failing the case in both arms. Naming triton_service.py in the prompt gives the agent somewhere to go. The case scored 0.45 with the skill before the fixture and 1.0 after, in both arms and in the full suite, with the eleven Warp fixtures still staged alongside it. The improvement is routing rather than an absence of Warp code to find. The dataset carries only tasks that finish without a CUDA device. The internal evaluation and publishing pipeline runs on CPU runners, and a GPU task graded there does not score lower, it scores meaninglessly: the agent spends the trial working around a missing device while the grader marks it against ground truth that assumed one. The eleven GPU-fixture tasks were dropped rather than rewritten, so the surviving cases still teach what they taught. Their fixtures stay in evals/files, so restoring the tasks needs only the archived evals.json. This costs real signal and the benchmark says so. Six of the seven tasks the baseline previously failed were GPU ones, so measured lift falls from +14 to +9 overall and the pass-rate gap narrows to 18 of 18 against 17 of 18. That is which tasks survived, not a regression. The full 29-task record, its benchmark, and its card are preserved at https://gist.github.com/shi-eric/a964cc9f5a3b6a542665149669355b02 so the evidence outlives the squash on merge. Restore it if GPU runners ever reach the pipeline. BENCHMARK.md and skill-card.md come from Skill Evaluator 0.9.2 under the external profile: claude-code on aws/anthropic/bedrock-claude-opus-4-8, 18 tasks in both arms for 36 trials, no errored trial. Verdict PASS with every dimension above its floor. Discoverability 73% to 97% is the large effect. Both records carry the same dataset digest, so the card and the benchmark describe one run rather than two, and both state the CPU-only scope rather than letting the numbers imply full coverage. The card also links its license rather than naming it in plain text, which the release checklist asks for. The one baseline failure is worth naming because it is not really about the skill. The harness stages the whole evals/files corpus into every trial before applying a task's own file list, so negative tasks that name no project can wander into fixtures belonging to other tasks; twelve of thirty-six trials did, across four of the fifteen negatives. In negative-warp-intrinsics-nvcc-compile-time, which asks about C++ template instantiation under nvcc, the agent without the skill found the Warp fixtures and reframed the question around Warp module identity. The skill prevented that, so the effect flatters rather than harms it, but the task is partly measuring its workspace and the benchmark records that alongside the number. The probe builds its child environment with dict(os.environ) rather than the .copy() spelling. Both produce the same dict, but SkillSpector matches the second against an env-harvesting pattern and reports it at fixed HIGH with no exemption path, which failed Tier 1 outright. The probe does not read any credential; it copies the environment so the measured command runs as the caller configured it, with only the cache paths redirected. Narrowing the copy to an allowlist was tried and rejected: any variable the list missed would surface as a confusing failure inside the user's own application. Signed-off-by: Eric Shi <[email protected]> Co-authored-by: nvskills-svc-account <[email protected]> Approved-by: Eric Shi <[email protected]> See merge request omniverse/warp!2737
1 parent 69ecb15 commit db13b57

75 files changed

Lines changed: 5393 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
# Skill Benchmark: warp-compile-time-optimizer
2+
3+
> **Overall verdict: NEUTRAL — One or more dimensions remain below PASS**
4+
5+
Live evaluation did not show a material gain or regression. Collect more evidence or improve the skill before making a publication decision.
6+
7+
## Evaluation Metadata
8+
9+
- Skill: `warp-compile-time-optimizer`
10+
- Evaluation date: 2026-08-02
11+
- Evaluator version: `0.9.2`
12+
- Agents: Claude Code (`aws/anthropic/bedrock-claude-opus-4-8`), Codex (`openai/openai/gpt-5.5`)
13+
- Tasks: 18 evaluation tasks (18 positive)
14+
- Dataset digest: `sha256:cc2e0dfaa3e0a776a88c313893d0587f38d298ebb450745b4e8323ee03e238f9` (skill-evaluator-dataset-snapshot/1)
15+
- Attempts per task: 1
16+
- Environment: `k8s-sandbox`
17+
- Tier 3 evidence: required for publication
18+
19+
Each task attempt ran in its own isolated sandbox pod.
20+
21+
## What This Report Answers
22+
23+
The three-tier evaluation checks whether the skill:
24+
25+
- is safe to use;
26+
- produces correct answers;
27+
- is discovered and activated when needed;
28+
- helps the agent complete the user's goal and expected workflow; and
29+
- avoids wasted skill and tool usage.
30+
31+
## Results at a Glance
32+
33+
| Measure | Claude Code (Baseline → Skill Uplift) | Codex (Baseline → Skill Uplift) |
34+
|---|---:|---:|
35+
| Overall | Not available | 92% → 93% (+1 points) |
36+
| Security | Not available | 94% → 83% (-11 points) |
37+
| Correctness | Not available | 99% → 99% (±0 points) |
38+
| Discoverability | Not available | 85% → 90% (+5 points) |
39+
| Effectiveness | Not available | 94% → 97% (+2 points) |
40+
| Efficiency | Not available | 88% → 96% (+9 points) |
41+
42+
**How to read this table:** baseline is the same task attempted without the target skill. Uplift is `skill score - baseline score`, shown in percentage points.
43+
44+
Example: `47% → 92% (+45 points)` means the skill-assisted run scored 92%, 45 percentage points above its 47% no-skill baseline.
45+
46+
## Tier Status
47+
48+
| Tier | Purpose | Status | Evidence |
49+
|---|---|---|---|
50+
| Tier 1 | Static validation | **PASSED WITH OBSERVATIONS** | 1 validator(s); 1 finding(s) |
51+
| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded |
52+
| Tier 3 | Live agent evaluation | **NEUTRAL** | 2 agent(s); 18 task(s) |
53+
54+
## Findings and Observations
55+
56+
<details>
57+
<summary>Show detailed findings and successful checks</summary>
58+
59+
- **MEDIUM** SCHEMA/body_recommended_section: Missing recommended section: '## Examples' (`skills/warp-compile-time-optimizer/SKILL.md`)
60+
61+
</details>
62+
63+
## Scoring Methodology
64+
65+
<details>
66+
<summary>Show dimension definitions, source signals, and thresholds</summary>
67+
68+
| Dimension | Question | Scored signals |
69+
|---|---|---|
70+
| Security | Is it safe to use? | `security` (100%) |
71+
| Correctness | Is the answer correct? | `accuracy` (100%) |
72+
| Discoverability | Was the right skill loaded when needed? | `skill_execution` (100%) |
73+
| Effectiveness | Did the skill help complete the task? | `goal_accuracy` (50%) + `behavior_check` (50%) |
74+
| Efficiency | Did it avoid wasted tool or skill usage? | `skill_efficiency` (100%) |
75+
76+
- Dimension bands: PASS at 50% or above; NEUTRAL from 40% to below 50%; FAIL below 40%.
77+
- Overall Tier 3 lift: PASS at +5 points or more; FAIL at -10 points or less; values between those bands are NEUTRAL.
78+
- Overall verdict: PASS only when every configured dimension passes for at least one supported agent. Lift is reported as diagnostic evidence and does not override this gate.
79+
- The 50% attempt pass threshold is a separate per-task gate; it is not the dimension pass threshold.
80+
- Effectiveness is the equal-weight mean of goal completion (`goal_accuracy`) and expected workflow adherence (`behavior_check`).
81+
- Token efficiency is a separate report-only signal. It does not change a dimension score or the overall verdict.
82+
83+
Signals present in this run:
84+
85+
- `security` (Security): unsafe operations, secret leakage, and unauthorized access.
86+
- `skill_execution` (Skill Execution): whether the expected skill was found and executed.
87+
- `skill_efficiency` (Efficiency): routing quality, workspace-aware skill reads, and productive tool use.
88+
- `accuracy` (Accuracy): final-answer correctness against the reference answer.
89+
- `goal_accuracy` (Goal Accuracy): whether the user's goal was achieved.
90+
- `behavior_check` (Behavior Check): whether the expected workflow behavior was followed.
91+
92+
</details>
93+
94+
## Freshness
95+
96+
Regenerate this benchmark when the skill, evaluation dataset, target agent/model, evaluator version, environment, or scoring policy changes.

0 commit comments

Comments
 (0)