Methodology

CompatLab observes exact published packages in a controlled consumer workspace. Each report names its inputs, coverage and limits.

One artifact, one shared snapshot

The service resolves an exact npm version, verifies supplied integrity, and uses a pinned npm installer with lifecycle scripts disabled. Every runtime in a comparison reads the same sealed dependency snapshot. Preparation and package code execute on a dedicated Linux amd64 host behind gVisor, outside the web and database services.

Independent roots, ordered subpaths

ESM import and CommonJS require each start in a fresh sandbox. Explicit executable subpaths are observed in ordered batches; those entries share a module cache and globals. Root success does not imply complete subpath coverage. Wildcard patterns, assets, work limits and interruptions stay visible.

Reading outcomes

Passed

Every applicable planned observation in this group succeeded.

Mixed results

Valid successes and failures coexist.

Failed

An applicable loading operation failed with retained evidence.

Inconclusive

Coverage, resource limits, policy or prerequisites prevented a complete observation.

Unsupported workflow

The requested workflow needs something excluded by this profile, such as native compilation.

Not applicable

No executable public path applies to that group.

Service error

The service could not produce a valid observation. This is separate from a package loading failure.

A declared optional peer can still be required by a particular entry point. When retained error details identify that peer and the installed snapshot confirms its absence, the affected compatibility result is inconclusive and names the missing dependency. The original failed loading observation is retained. Optional peers are not installed automatically; adding one changes the test inputs and does not guarantee success. Empty or unrelated errors remain unexplained failures.

Subpath batches share one process, so a failed load can leave state that affects later entries. When an entry fails after an earlier failure in the same batch process, whatever its error, up to four such entries per batch are loaded again, each in a fresh sandbox. A fresh pass or failure decides that entry. An entry without a completed fresh observation is inconclusive; interrupted confirmations are not retried. Batch observations are retained and labeled separately from fresh ones. The batch failure's cause is not established.

What success does not establish

A pass means import or require resolved the entry and ran its module code. The harness records the type of the loaded value and calls no export, so errors that appear only when an application calls the package are outside this evidence. Loading does not test an application or establish safety; public scans do not run application tests. Package-visible harness observations can be tampered with by malicious code in the same process. Stdout and stderr are logs, never verdicts. Only a named behavioral assertion can earn a separate probe-verified evidence label.

Run the local CLI

The CLI is included in the source repository. It is not currently published as an npm package, and downloading a report does not install a compatlab command. Use Node.js 24.21.0 and pnpm 12.8.1 to build it, then run pnpm cli or the compiled entry point below from the repository directory.

Use a dedicated Linux amd64 host with the qualified Docker/runsc, mount and firewall prerequisites. Ordinary Docker alone does not reproduce this execution profile. The CLI refuses to fall back to host execution. On macOS or Windows, use a separate qualified Linux machine or VM. Do not run another CLI supervisor on the active production worker.

git clone https://github.com/siddiksawani/CompatLab.git
cd CompatLab
pnpm install --frozen-lockfile
pnpm build
sudo "$(command -v node)" apps/cli/dist/bin.js doctor --json
sudo "$(command -v node)" apps/cli/dist/bin.js check [email protected] --json

The example above makes a new check and builds its runtime images locally. Replaying a hosted report additionally requires its exact runtime images. Those images are not yet distributed for public download; an operator must export them from the originating worker and load them on the replay host. Building the same recipe again can produce different image IDs and is not exact replay.

Reproduction can reuse the actual retained snapshot or explicitly rebuild from downloaded inputs and their exact lock. Rebuilding records a new generation and may produce different installed bytes. Missing images, unavailable artifacts or required prerequisites can prevent replay.

Public evidence and retention

Reports, locks and provenance are public and retained as history. Raw package logs expire after 30 days. Sealed worker snapshots have a bounded cache and can become unavailable before report metadata expires. Quarantine and invalidation appear on historical reports.

Maintainer tools: coming soon

We’re planning tools for package authors to monitor releases and add focused, offline behavioral checks. Maintainer accounts and assertion registration are not available through the website yet. See what’s planned for maintainers.

Historical assertions and local CI archives

Existing reports can include a named assertion with its own outcome and immutable source. Successful loading never becomes behavioral verification. Reproduction inputs include the retained assertion bundle when one was selected. The CLI also supports pre-publication archives on a qualified Linux/runsc host. Those reports use a distinct CI artifact identity with caller-supplied workflow provenance. There is no public archive upload endpoint.

Manifest format, examples and CI setup