…tors from scan report

The 3250-line monolith is replaced by a thin entry point + 8 modules in
scripts/build-indicator/:

  integrity.js     (38 lines)  — SHA-256 hash/wrap/unwrap helpers + seeds
  defang.js       (187 lines)  — file defanging: JS/shell/batch/binary logic
  http.js         (359 lines)  — HTTP layer: circuit-breaker, fetchRaw/Json,
                                 fetchChangedNames, getPackageManifest
  lifecycle.js    (313 lines)  — lifecycle script analysis: extract, diff,
                                 match indicators, infer files, suggest signal
  package-cache.js(161 lines)  — loadPackageCache / savePackageCache
  process-lock.js  (64 lines)  — makeLockHelpers / isPidAlive
  deep-cache.js   (421 lines)  — deep fetch/scan pipeline, version hashes,
                                 hashDirTree, deepFetchPackage, deepAnalyzePackage
  main.js        (1685 lines)  — CLI args, run loop, drain, search, output

build-indicator-suggestions.js is now a 3-line entry:
  require('./build-indicator/main')

No behaviour changes — all logic is identical, only the file layout changed.

Co-authored-by: Copilot <[email protected]>

…ckpoint...' remained

'    saving checkpoint...' is 24 chars; the erase write had 23 spaces,
leaving the trailing '.' visible on its own line before the candidates
drain header.

Co-authored-by: Copilot <[email protected]>

…kage-lock.json

CI's npm install removes node_modules/git-raw-commits/node_modules/
conventional-commits-filter (a dev/optional/peer dep that doesn't need
deduplication on the CI runner), leaving the lockfile dirty and failing
the git-dirty check.

Normalized with: npm install --ignore-scripts

Co-authored-by: Copilot <[email protected]>

…ed .meta.json marked stale

The pre-drain cache validity scan read .meta.json with JSON.parse() directly,
then checked meta.state and meta.fetchVersion on the raw envelope object.
Wrapped files have shape { data: {...}, hash: '...' }, so meta.state was
undefined → every wrapped cache entry resolved as 'stale', causing the
entire deep cache to be re-fetched on every --deep run after the integrity
hashing commit.

Fix: apply the same envelope-unwrap logic already present in deepFetchPackage
and deepAnalyzePackage. Also add the missing META_HASH_SEED and unwrapVerified
imports (OUTPUT_HASH_SEED was the only export imported before).

Co-authored-by: Copilot <[email protected]>

…ng deep fetch

Previously deepFetchPackage always wrote state='fetched' regardless of whether
any files were actually downloaded. If unpkg was unreachable or rate-limiting,
all fetchRaw calls returned null and fetchedFiles stayed empty — but the meta
still said 'fetched' with a valid fetchVersion, so subsequent runs treated the
empty cache as valid and never retried.

Fix: package.json is always attempted in step 1. If it's missing from
fetchedFiles after all attempts, every file request failed — write
state='failed' instead of 'fetched' and emit a per-package warning.

The cache validity check already treats any state other than 'fetched' or
'scanned' as 'stale', so failed entries are automatically re-fetched on the
next --deep run without any additional logic.

Co-authored-by: Copilot <[email protected]>

…rovable

approve-scripts/deny-scripts <pkg> was throwing ENOMATCH for packages
listed in the root project's own bundleDependencies (e.g. canvas).
Those packages are fetched from the registry and installed normally;
inBundle=true but inDepBundle=false. The positional-arg node scan was
using the broader inBundle guard, matching the pattern already fixed in
unreviewed-scripts.js and script-allowed.js.

Adds a test asserting root-bundled deps can be approved by name.

Co-authored-by: Copilot <[email protected]>

…d.js comment

The comment on trustedDisplay still said \
pm approve-scripts --allow-scripts-pending\
after the namespace rename in 0c4dd41. Updated to \
pm install-scripts ls\.

Co-authored-by: Copilot <[email protected]>

…!list

npm install-scripts ls --allow-scripts-report-format=json was throwing
EUSAGE because the guard used !pending (only true with --allow-scripts-pending)
instead of !list (true for both --allow-scripts-pending and ls mode).

Also update test: install-scripts ls default output is now markdown report,
and add explicit test cases for the ls+format combo.

Co-authored-by: Copilot <[email protected]>

…detection

- parseSingleCommand: follow
ode install/check and bare paths containing '/'
  (e.g. sharp's
ode install/check, mist's script/install-ninja.js)
- parseSingleCommand: add handler for direct executables without './' prefix when
  the path contains '/' (always a local file, never an npm package name)
- scanFile: track spawnSync/execFile calls using process.execPath as executable;
  resolve and recursively scan the referenced scripts (findExecPathRefs)
- SIGNAL_PATTERNS: extend binary-download to match 'binary-install' npm helper
  and GitHub Releases '/releases/download/' URL pattern
- SIGNAL_PATTERNS: add 'git-hook-setup' signal (husky, lefthook, simple-git-hooks,
  pinst) so lifecycle commands that install hook managers are flagged
- findGitHookScripts: enumerate .husky/ (flat) and .lefthook/ (one level recurse)
  hook script directories and feed them through the recursive scan pipeline
- indicator-definitions.js: add git-hook-runner virtual indicator with commandPatterns
  for husky/lefthook/simple-git-hooks/pinst; add SIGNAL_DESCRIPTIONS entry
- indicator-definitions.js: add 'node install/' and 'node . run' commandPatterns to
  native-addon indicator to cover sharp-style install/check and CLI-driven builds
- deep-cache.js: add fetchUnpkgDirListing to fetch .husky/ and .lefthook/ hook
  directories from unpkg for packages not yet in the local cache; recurse one
  level for lefthook nested hook subdirectories
- defang.js: add case 13 -- prepend exit-1 guard to extensionless files inside
  .husky/, .lefthook/, and hooks/ directories (no-shebang hook scripts)
- DEEP_CACHE_SCHEMA bump to defang-v12 (invalidates stale fetch/scan caches)

…ded hasBuildHint signals

- parseSingleCommand: recognise bun and deno as node-like JS runtimes;
  strip the 'run' subcommand ('bun run ./file.ts', 'deno run ./file.ts')
  before resolving the script path through the existing node handler
- LOCAL_IMPORT_META_RESOLVE_RE: detect import.meta.resolve('./path') — the
  standard ESM way to get local file URLs — and include matched paths in
  the findLocalRefs recursive scan pipeline
- indicator-scanner: replace 7 chained .includes() checks in hasBuildHint
  with a BUILD_HINT_SIGNALS Set + .some(); add five previously-missing
  signals: wasm-load, shell-network-fetch, uses-eval, base64-decode-exec,
  jsfuck-obfuscation so packages whose scanned files emit these signals
  enter the categorisation pipeline instead of silently falling to lifecycleOnly
- tests: add findLocalRefs import.meta.resolve unit test; add bun/deno
  scanner integration tests covering direct invocation and 'run <file>'
  delegation with permission flags

Custom agent that autonomously improves indicator-definitions.js by
reviewing indicator-suggestions.json tasks, tuning commandPatterns and
signal descriptions, and running the build-indicator-suggestions pipeline.

…injection into unpkg URLs

Lifecycle scripts using node -e with inline code containing || or ;
were split naively by SHELL_OPS_RE, isolating the code content as a
standalone sub-command. When that fragment contained / (e.g. a package
name like @scope/pkg inside the inline string), the
!startsWith('@') && includes('/') branch in parseSingleCommand treated
it as a local file path, which was then used to construct a malformed
unpkg URL resulting in HTTP 403.

Replace the naive trimmed.split(SHELL_OPS_RE) in parseCommandFile with
a new quote-aware splitOnShellOps() helper that skips over single- and
double-quoted strings (honouring backslash escapes inside double quotes)
before splitting on shell operators. This keeps node -e "..." as a
single sub-command so only the -e handler runs, which correctly calls
findLocalRefs on the inline code string.

fetchUnpkgDirListing used the default retries=5, so an ECONNRESET
response from unpkg (e.g. for .husky?meta on packages that do not
publish that directory) produced 4 warning lines and a 5-second
sleep-based backoff before silently returning [] via the outer catch.

Pass retries=1 to fetchJson so that a single network failure throws
immediately — no warning is printed, no backoff sleep occurs — and
the outer catch { return [] } handles it in the normal best-effort way.
404 (directory absent on unpkg) was already silent; this brings network
errors in line with the same silent-failure semantics.

Add a SKIP_FETCH_EXTS set (.node, .so, .dll, .dylib, .pyd) so that
when findLocalRefs follows a require('./addon.node') reference the HTTP
fetch is skipped entirely rather than downloading the binary, detecting
the magic bytes in writeDefanged, and printing a noisy "skipped binary"
warning.  Avoids a wasted HTTP round-trip per reference.

…ipt-compiler

indicator-scanner.js: detect the platform-specific-script signal from
scanned lifecycle files and use it to trigger the platform-binary-selector
virtual indicator (triggeredByPlatformSpecificScriptSignal).  Without this
wire-up the signal was detected by the scanner but never caused the
indicator to fire.

indicator-definitions.js: add /\bbun\b/ to the typescript-compiler
commandPatterns.  Bun is a TypeScript-native runtime/bundler (equivalent
to ts-node) and packages using "bun run scripts/build.ts" in prepack or
prepare hooks are TypeScript compilation steps that must not be left
uncategorized.

…dations

Previously, file-tree hash mismatches were detected inside deepFetchPackage
as each package was processed, printing 🗑️ messages interleaved between
"scanning ..." lines and immediately re-fetching in the same worker.

Move the hashDirTree check into the existing cacheStatus pre-check pass,
which already runs concurrently across all manifests via Promise.all before
the fetch loop starts.  Stale directories are now deleted during that pass
so all invalidation messages appear as a single upfront batch — before any
"scanning ..." output.  deepFetchPackage receives either an absent directory
(fresh fetch) or a verified-valid directory, so it never encounters a
tree mismatch mid-scan.  quietTreeWarning is now always true since the
warning path in deepFetchPackage is unreachable from this caller.

…file offload

When unpkg fails to serve package.json (rate-limit, downtime, or 403),
deepFetchPackage now falls back to the npm registry tarball instead of
immediately marking the package failed.

Tarball handling (deep-cache.js):
- fetchNpmTarball(): downloads registry.npmjs.org .tgz, gunzips via
  built-in zlib, returns raw tar Buffer (2 retries, since fallback path).
- buildTarIndex(): single O(n) POSIX ustar parse; returns
  Map<path -> {offset,size}>.  Strips "package/" npm prefix.
- tarFetch(): O(1) index lookup; slices Buffer (small) or fd.read()
  byte-range (large) transparently.
- fetchOne: unpkg retries exhausted first; once tarBuf/tarFd is set,
  skips unpkg entirely for all remaining BFS files.
- TAR_MEMORY_THRESHOLD (10 MB): tarballs larger than this are written to
  os.tmpdir() and served via an open fs.FileHandle so the heap is freed.
- tarballAttempted flag: set when step 1b triggers; used in the failure
  message so "unreachable on unpkg and registry tarball" is printed
  correctly even after tarBuf/tarFd have been cleaned up.
- Cleanup: fd.close() + fs.unlink() after BFS completes; tempFilesSet
  entry removed before unlink so the next checkpoint no longer lists it.

Checkpoint tracking (package-cache.js + main.js):
- savePackageCache: new tempTarFiles=Set param; saves pendingTempFiles[]
  in the checkpoint JSON so orphaned temp files survive a process kill.
- loadPackageCache: returns pendingTempFiles from saved data.
- main.js: activeTempFiles Set shared across all DeepFetch workers;
  pre-flight block on every startup deletes any pendingTempFiles left
  by a previously interrupted run; DeepFetch-mode resume checkpoint
  saves pass activeTempFiles.

fetchedFiles was a plain Array, so if fetchOne was called twice for the
same path (e.g. package.json fetched in step 1, then again when a BFS
file has require("./package.json")), the path appeared twice in the
stored fetchedFiles list.

Two changes:
  1. fetchedFiles: [] -> new Set()  (push->add, includes->has,
     spread to [] at the two JSON/return sites)
  2. fetched BFS Set initialised from fetchedFiles so step-1 files
     (package.json + indicator files) are pre-marked as seen and the
     BFS never calls fetchOne for them again.

Sort fetchedFiles when writing to .meta.json and returning from
deepFetchPackage so the stored list is deterministic and diffs cleanly.
Bump DEEP_CACHE_SCHEMA to defang-v14 to invalidate caches written with
the old unsorted/potentially-duplicate format.

…stop perpetual invalidation

deepAnalyzePackage wrote scan results via `...meta` spread, which
preserved whatever filesHash was in the existing meta.json.  If a
previous scan run had stored a stale or wrong filesHash (e.g. from an
interrupted run, a schema bump transition, or a prior re-scan that
itself used the ...meta spread), the next run's cacheStatus pre-check
would compute the real directory hash, find it differed from the stored
one, delete the cache and re-fetch — only for the cycle to repeat.

Fix: compute a fresh hashDirTree() right before writing the scan results
and store it explicitly (overriding any stale value from ...meta).  This
guarantees the persisted filesHash always matches the actual on-disk
state after a scan, so the next pre-check finds a match and skips the
re-fetch.

…e scripts

The runtime-installer indicator previously only caught npm install when
invoked via node -e "execSync/spawnSync('npm install')" — missing the
equally dangerous pattern of calling npm/yarn/pnpm/bun install directly
as a shell command in a lifecycle hook or in a script reached via npm-run
delegation (extractLifecycleScripts already follows those chains).

A package whose lifecycle runs "cd subdir && npm install && npm run build"
installs arbitrary dependencies at the consumer's install time, completely
outside the parent lockfile and integrity checks — a clear supply-chain red
flag.

Add direct commandPatterns for:
  npm install, npm ci
  yarn install, yarn add
  pnpm install, pnpm add
  bun install, bun add

These close the scanner gap for @fieldwangai/agentflow (and similar
packages) whose build:web-ui delegated script contains npm install.

…gnals/references

Two related fixes in script-risk-scanner.js:

1. referencedFiles dedup: when multiple lifecycle hooks (e.g. postinstall
   and prepare) reference the same script file, scanFile previously pushed
   a shallow copy with a new reason for each hook — resulting in duplicate
   entries with identical path/sha256/signals but different reasons.
   Fix: mutate the existing cached entry to append the new reason ("; "
   separated) instead of pushing a duplicate.

2. Set-based accumulation: replace plain Array push() with Set.add() for
   signals and references throughout scanFile and detectSignals so that
   duplicates are impossible by construction.  Arrays are only produced at
   the output boundary via [...set] so JSON serialization is unaffected.
   Also deduplicates signals in scanInlineCommand via the same pattern.

… scanner dedup fixes

The referencedFiles dedup and Set-based signals/references changes in
script-risk-scanner.js affect scan output but not fetched files.
Bumping DEEP_CACHE_SCHEMA would unnecessarily re-fetch all 8000+
packages; instead, add a SCAN_IMPL_VERSION string included only in
computeDeepScanVersion() so the scan cache is invalidated (forcing a
re-scan with clean deduplicated output) while DEEP_FETCH_VERSION stays
unchanged and fetched file trees are kept on disk.

New module discovery in findLocalRefs / BFS fetch:
- new URL('./file', import.meta.url) — ESM file-relative path pattern
- require.resolve('./path') — local path resolution without loading
Both are followed transitively in deepFetchPackage BFS and in
scanPackageScripts, catching scripts that reference sibling files
via these common patterns.

TypeScript source file support (.ts/.mts/.cts):
- resolveRelPosix (deepFetchPackage BFS) now probes .ts/.mts/.cts
  extensions so ts-node/tsx-executed scripts are fetched
- resolveLocalRef and scanFile extension probing extended to match
- isLocalSpecifier updated to recognise .mts and .cts suffixes

package.json#exports resolution for bare dep entry points:
- deepAnalyzePackage previously only consulted #main when scanning
  a followed bare dep; now resolveExportsEntry() prefers the CJS
  ('require') conditional export, falling back through 'node',
  'default', #main, then index.js — covering modern ESM+CJS packages

TypeScript type-only import filtering:
- stripTypeOnlyImports() strips 'import type ...' / 'export type ...'
  lines before ref extraction in findBareRefs and findLocalRefs
- Prevents @types/* and type-only packages from being queued as bare
  follows, saving fetch budget on deps with no lifecycle scripts
- ESM edge case preserved: 'import type from ./foo' (where 'type' is
  the default binding name) is correctly left intact via lookahead

resolvedFollows sorted before persisting to .meta.json for
deterministic output across concurrent bare-follow resolutions.

Cache invalidation:
- DEEP_CACHE_SCHEMA defang-v14 -> defang-v16 (fetch set may grow)
- SCAN_IMPL_VERSION scan-impl-v1 -> scan-impl-v5 (scan results change)

defang.js: TypeScript files (.ts/.mts/.cts) now receive a top-level
throw new Error() in addition to the null byte.  TypeScript's
transpiler (ts-node, tsx) treats U+0000 as an invalid character and
silently strips it before emitting JS — the null byte alone does not
prevent execution via ts-node.  The throw statement is valid TypeScript,
survives transpilation unchanged, and fires before any code in the
file runs.  JS files (.js/.mjs/.cjs) are unchanged (null byte only).

deep-cache.js: DEEP_CACHE_SCHEMA bumped defang-v16 → defang-v17 to
invalidate existing .ts cache entries and trigger re-fetch with the
updated defanging.

package-cache.js: savePackageCache now uses an atomic write (write to
.new temp file, then rename) instead of fs.writeFile directly on the
target.  On Windows, antivirus/VS Code file watcher/Search indexer can
hold indicator-suggestions.packages.json open briefly, causing
UNKNOWN (ERROR_SHARING_VIOLATION) on the direct open() call.  Writing
to a fresh temp path avoids the contention; rename is near-instantaneous
on NTFS.

…e cache write

main.js: split the DeepFetch pre-check into two serial phases so all
stale cache directories are deleted before any fetch worker starts.

Previously, version-mismatch deletions happened inline inside
deepFetchPackage as each worker processed its package, interleaving
rmReadOnly I/O with network fetches and slowing the overall pipeline.
The pre-check already ran a concurrent meta-read pass but only deleted
tree-hash mismatches there; version mismatches were left for the workers.

New flow:
  Phase 1 — Promise.all: read every .meta.json concurrently, classify
             each package as valid/stale/missing.  Pure reads only, no
             deletions, maximum I/O parallelism.
  Phase 2 — log tree-warning messages, emit a single summary line for
             version-mismatch wipes, then Promise.all: delete all stale
             directories concurrently.  The entire wipe completes before
             the first fetch worker starts.
  Phase 3 — build fetchQueue and start fetch workers against a clean
             directory tree.

On a schema-version bump (e.g. defang-v17 invalidating 7000+ entries)
this eliminates thousands of per-worker delete calls that previously
competed with network I/O.

package-cache.js: savePackageCache now uses an atomic write (write to
.new temp file then rename) instead of fs.writeFile directly on the
target, fixing UNKNOWN (ERROR_SHARING_VIOLATION) on Windows when
antivirus/VS Code file watcher briefly holds the file open.

Temporary workflow demonstrating the deep-scan pipeline check running
'node scripts/build-indicator-suggestions.js --deep --top 1' on PRs.
Includes deep cache files for testing purposes — will be cleaned up.