Repository navigation
Commit db13b57
committed
Add Warp compile-time optimizer skill and release evidence
* Attach NVSkills validation signatures
Signed-off-by: nvskills-svc-account <[email protected]>
* Add Warp compile-time optimizer skill
Existing guidance did not connect Warp's module identity model to
repeatable measurements, making it easy to trade correctness or runtime
performance for a misleading cold-start improvement.
Add a measurement-first skill with isolated-cache tooling, mechanism
references, and evaluated fixtures. This keeps workload preservation and
explicit tradeoff reporting central to compile-time optimization.
SKILL.md is roughly 3.3k tokens, under the 5k advisory limit that was
previously a blocking finding under the external validation profile. Its
description enumerates the phrasings that should route here, including
requests to optimize or improve compile times rather than only startup
stalls, and its frontmatter declares the env capability the probe uses
via allowed-tools rather than the permissions field SkillSpector's
remediation text suggests, because the Agent Skills spec defines no such
field (NVIDIA/SkillSpector#300).
The eleven eval fixtures keep their pyproject.toml. They had been
removed because scanning the complete skill directory, which the release
checklist asks for, returned CRITICAL / DO NOT INSTALL with every
critical finding an unpinned numpy or setuptools in fixture packaging
rather than anything in the skill. Pinning does not help there: the
advisory lookup keys on package name and ignores version constraints, so
a floor still returns the package's worst-ever CVE, and pinning left the
score at 100/100 CRITICAL while removing the files moved it to 42/100
MEDIUM.
That scan is not what gates publication. The validation pipeline blocks
on Tier 1 critical and high findings, Tier 1 runs skillevaluator, and
skillevaluator stages a filtered copy that drops evals/ before
SkillSpector sees it. Its dependency validator also reads the skill root
only. Measured with all eleven manifests restored, skillevaluator
validate returns 11 of 11 checks passed, exit 0, quality A at 94.8, no
high or critical findings, and no reference to the fixture paths
anywhere in the report. Removing the manifests cost fixture fidelity and
bought nothing, so they are back, and the fixtures build and run from
their source trees again, verified including the CUDA ones.
The nine CUDA fixtures refuse to start when Warp sees no device. Each
entry point checks wp.get_cuda_device_count() and raises, and the PDE
stepper no longer falls back to the CPU. On a runner without a GPU an
agent would otherwise route around the missing device and produce a
result that has nothing to do with what the case measures. The checks
sit in entry points rather than kernel modules, so generated source and
module hashes are untouched, and after the tile projection --relink
branch, which only clears caches on disk and needs no device. The two
CPU fixtures are unchanged.
The eval dataset covers 18 cases, in the agentskills.io shape
SkillEvaluator prefers rather than the legacy flat array, which now
raises a deprecation warning during validation. Its
fifteen negative cases set expected_skill explicitly instead of null,
which is what the evaluator's own documentation and generator produce.
The grader searches the trajectory for that string, so a null value
makes a negative case score perfectly without inspecting anything: an
agent
that fully activates the skill on an unrelated CUDA question could not
fail. should_trigger stays written out alongside it, since it otherwise
defaults from expected_skill and would flip those cases to positive.
The two dense-stage cases state launch and output preservation as
expected behaviors rather than only as hard gates. Nothing in the
harness reads hard_gates, so an agent that rewrote the in-place
relaxation into a double buffer, adding a launch and changing the
printed values, was graded by nothing that would catch it.
The probe declines to report a measurement it cannot stand behind. A run
where every sample exits non-zero now fails rather than publishing a
near-zero median that reads as a large improvement, and the
same-work-compiled verdict is derived from compiled module identities
instead of their counts, so a module rebuilt under different options is
no longer written off as build contention. A warm repeat that exits
non-zero is dropped the way a timeout already is: it exits early, so its
wall clock is not what a repeat invocation costs, and crediting it
distorts the cold-minus-warm elapsed measure the report leads with.
The Triton negative case now ships a fixture. It described a Triton
inference service but declared no files of its own, and the harness
stages the whole evals/files corpus into every trial before applying
what a case declares, so a task's file list is additive and can never
narrow it. The agent listed its workspace, found Warp kernels in a
neighbouring fixture, and told the user their service was Warp rather
than Triton; the grader then scored that against ground truth expecting
Triton-only guidance, failing the case in both arms. Naming
triton_service.py in the prompt gives the agent somewhere to go. The
case scored 0.45 with the skill before the fixture and 1.0 after, in
both arms and in the full suite, with the eleven Warp fixtures still
staged alongside it. The improvement is routing rather than an absence
of Warp code to find.
The dataset carries only tasks that finish without a CUDA device. The
internal evaluation and publishing pipeline runs on CPU runners, and a
GPU task graded there does not score lower, it scores meaninglessly:
the agent spends the trial working around a missing device while the
grader marks it against ground truth that assumed one. The eleven
GPU-fixture tasks were dropped rather than rewritten, so the surviving
cases still teach what they taught. Their fixtures stay in evals/files,
so restoring the tasks needs only the archived evals.json.
This costs real signal and the benchmark says so. Six of the seven
tasks the baseline previously failed were GPU ones, so measured lift
falls from +14 to +9 overall and the pass-rate gap narrows to 18 of 18
against 17 of 18. That is which tasks survived, not a regression. The
full 29-task record, its benchmark, and its card are preserved at
https://gist.github.com/shi-eric/a964cc9f5a3b6a542665149669355b02 so
the evidence outlives the squash on merge. Restore it if GPU runners
ever reach the pipeline.
BENCHMARK.md and skill-card.md come from Skill Evaluator 0.9.2 under the
external profile: claude-code on aws/anthropic/bedrock-claude-opus-4-8,
18 tasks in both arms for 36 trials, no errored trial. Verdict PASS with
every dimension above its floor. Discoverability 73% to 97% is the large
effect. Both records carry the same dataset digest, so the card and the
benchmark describe one run rather than two, and both state the CPU-only
scope rather than letting the numbers imply full coverage. The card also
links its license rather than naming it in plain text, which the release
checklist asks for.
The one baseline failure is worth naming because it is not really about
the skill. The harness stages the whole evals/files corpus into every
trial before applying a task's own file list, so negative tasks that
name no project can wander into fixtures belonging to other tasks;
twelve of thirty-six trials did, across four of the fifteen negatives.
In negative-warp-intrinsics-nvcc-compile-time, which asks about C++
template instantiation under nvcc, the agent without the skill found the
Warp fixtures and reframed the question around Warp module identity. The
skill prevented that, so the effect flatters rather than harms it, but
the task is partly measuring its workspace and the benchmark records
that alongside the number.
The probe builds its child environment with dict(os.environ) rather
than the .copy() spelling. Both produce the same dict, but SkillSpector
matches the second against an env-harvesting pattern and reports it at
fixed HIGH with no exemption path, which failed Tier 1 outright. The
probe does not read any credential; it copies the environment so the
measured command runs as the caller configured it, with only the cache
paths redirected. Narrowing the copy to an allowlist was tried and
rejected: any variable the list missed would surface as a confusing
failure inside the user's own application.
Signed-off-by: Eric Shi <[email protected]>
Co-authored-by: nvskills-svc-account <[email protected]>
Approved-by: Eric Shi <[email protected]>
See merge request omniverse/warp!27371 parent 69ecb15 commit db13b57
75 files changed
Lines changed: 5393 additions & 0 deletions
File tree
- skills/warp-compile-time-optimizer
- evals
- files
- c01_particle_preprocessing
- src/particle_prep
- c04_batch_signals
- src/batch_signals
- g01_pde_stepper
- src/pde_stepper
- g02_stencil_bank
- src/stencil_bank
- g04_image_pyramid
- src/image_pyramid
- g06_gpu_analytics
- src/gpu_analytics
- g08_tile_projection
- src/tile_projection
- g09_mixed_precision
- src/mixed_precision
- g11_stage_pipeline
- src/stage_pipeline
- g12_dense_stage
- src/dense_stage
- references
- scripts
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
0 commit comments