Summary

This adds libs/fast-registry-json, a specialized npm packument indexer, and uses it when loading npm package info. The intent is to avoid eagerly deserializing every version manifest in large packuments when resolution usually only needs a small subset of them.

The new parser is not a general replacement for serde_json. It is a narrow indexer for registry JSON:

• scan JSON quickly to find strings and structural operators
• build an index of top-level packument fields Deno needs for resolution
• keep byte ranges for each versions[version] manifest
• lazily deserialize only selected version manifests with serde_json

SIMD Scanner

fast-registry-json is adapted from simdjson-style stage 1 scanning, but narrowed for npm registry packuments rather than general-purpose JSON parsing.

At a high level, it scans the input in 64-byte blocks, identifies string ranges and structural JSON characters, then feeds those tokens into a small packument-specific state machine. The state machine records only the data Deno needs for npm resolution: name, dist-tags, versions keys and byte ranges, time, _deno.etag, and compact trust evidence.

Target behavior:

• aarch64 uses the NEON path when available via the target feature
• x86_64 uses runtime dispatch once via OnceLock; SSSE3 gets the table-shuffle classifier, otherwise it falls back to comparison-based classification
• x86_64 bitmask extraction uses SSE2 movemask, which is baseline for x86_64
• other targets use the portable wide/comparison path

Unsupported SIMD targets still have a fallback path rather than requiring the fastest classifier.

Deno Integration

NpmPackageInfo::from_packument_slice() now uses the packument index directly. It parses the cheap top-level data eagerly:

• package name
• dist-tags
• publish times for minimumDependencyAge
• version keys and byte ranges
• trust evidence (provenance, trustedPublisher, staged publish approver)

The versions map is now represented by NpmPackageVersionInfos, which can be either materialized or lazy. The lazy variant stores:

• the original packument text in an Arc<str>
• a Version -> Range<usize> map
• a per-version OnceCell<Option<NpmPackageVersionInfo>>
• indexed trust evidence

Resolution can iterate version keys, check publish times, apply trust policy, and then deserialize only the selected manifest. Existing paths that need all manifests can still iterate values or mutate the map; those paths materialize as needed.

Cache Changes

The npm cache path now writes slim packument bytes from the indexed source rather than deserializing the full packument and serializing it again.

The slim cache keeps the resolution-relevant fields:

• dist-tags
• versions
• time
• selected dependency/platform/deprecation/bin fields
• hasInstallScript
• trust evidence compactly projected back into the manifest shape
• _deno.etag

This keeps minimumDependencyAge working for full packuments and avoids defeating laziness when saving the cache.

Benchmarks

Built with cargo build -p deno --profile release-lite.

Compared:

• baseline: system canary /Users/nathanwhit/.deno/bin/deno
• current: target/release-lite/deno

Command shape:

DENO_DIR=<per-binary-cache> NO_COLOR=true DENO_NO_UPDATE_CHECK=1 deno install

For each measured run, node_modules and deno.lock were removed. Each binary used a separate warmed DENO_DIR.

Targeted Fixtures

With lockfiles preserved, both fixtures were effectively neutral because Deno does not need to read packuments on that path.

../vlt-benchmarks Fixtures

Warm-cache, no-lockfile, run in place in ../vlt-benchmarks/fixtures/*:

The large fixture is wall-time neutral because filesystem / node_modules work dominates, though the current branch used much less user CPU. The common app fixtures are consistently about 1.3-1.4x faster.

x86_64 Linux ../vlt-benchmarks Fixtures

Host: [email protected], x86_64 Broadwell, canary upgraded with deno upgrade canary.

Compared:

• baseline: canary deno 2.9.1+4e8b2f4 (canary, release, x86_64-unknown-linux-gnu)
• current: PR release-lite binary deno 2.9.1 (stable, release, x86_64-unknown-linux-gnu)

Command shape:

DENO_DIR=<per-binary-cache> deno install --quiet

Each binary used a separate warmed DENO_DIR populated by that binary. For each measured run, node_modules and deno.lock were removed, preserving normal node_modules behavior.

Full log: /home/bot/work/vlt-deno-install-all-fixtures-20260702-200452.log on the x64 benchmark host.

x86_64 Packument Index Classifier Check

Raw full registry packuments were downloaded directly from https://registry.npmjs.org/<pkg> and benchmarked with cargo bench -p fast-registry-json --bench benchmark -- bench_packument_index_. The optimized x86 path uses SSSE3 runtime dispatch; the fallback measurement temporarily forced the comparison classifier.

Caveats

• Local aarch64 and x86_64 Linux results are both included above. The x86_64 host was shared/noisy, so treat small deltas as directional rather than exact.
• This is a targeted packument indexer, not a validating JSON parser. Selected version manifests are still deserialized through serde_json.

Validation

• cargo test -p fast-registry-json -p deno_npm -p deno_npm_cache
• cargo clippy -p fast-registry-json -p deno_npm -p deno_npm_cache --all-targets -- -D warnings
• cargo test -p specs_tests --test specs -- specs::install::minimum_dependency_age_slim_packument
• cargo test -p specs_tests --test specs -- specs::npm::trust_signals
• cargo fmt --check
• cargo bench -p fast-registry-json --bench benchmark --no-run