perf: cut D1 tools-sync runtime via single mise registry --json --security by jdx · Pull Request #130 · jdx/mise-versions

jdx · 2026-04-24T16:06:09Z

Summary

Replace 977 sequential mise tool <name> --json shell-outs (~4 min) with a single mise registry --json --security call (~30 s end-to-end for the entire manifest phase). The --security flag landed upstream in mise v2026.4.22 via jdx/mise#9364 and returns backends + description + security in one JSON array.
Drop the now-unneeded MISE_TOOL_CONCURRENCY env knob, the getSecurity / fetchSecurityMap worker pool, and the execFile + promisify async plumbing.
Add a graceful fallback to mise registry --json (no --security) for any environment running pre-4.22 mise — the rest of the manifest still syncs, just without security info.

Why

The Sync tools to D1 step ran for ~5 min every 30 min in the update workflow. Profiling showed ~4 min was spent in 977 sequential mise process spawns just to read each tool's security array (and a description that was almost always null from mise tool --json).

mise v2026.4.22 exposes everything we need on mise registry --json --security, so the script no longer has any reason to do per-tool work.

Measurement

Local, 978 tools (current registry size):

Before this branch: ~240 s in the manifest phase.
After: ~30 s — the registry call itself is most of that time, file I/O and TOML parsing are the remainder.

In CI the relative speedup will be even larger since the previous design paid 977× the process-spawn + mise-startup cost.

Bonus bug fix

The previous code read description from mise tool --json, which almost always returns null. Pulling descriptions from mise registry --json instead bumps tools-with-descriptions from 17 to 975 locally — purely a side effect of using the right command.

Test plan

bun run test — 42 tests pass
prettier --check + tsc --noEmit clean
Local dry run with v2026.4.22 mise: manifest builds in ~30 s, 845 tools with GitHub, 975 with description, 978 with backends, 184 with package URLs
Local dry run with a wrapper that rejects --security (simulating older mise): warning printed, fallback path runs, manifest still builds
First post-merge scheduled run on main — verify D1 sync step duration drops from ~5 min → well under 1 min, and Upserted: ~977 / Errors: 0 in the result

Notes

The workflow already installs latest mise via curl https://mise.run | sh after the mise-action step, so CI will have v2026.4.22+ on every run.
Followups not in this PR: kill the second mise ls-remote --json call in update.sh's generate_toml_file (different rate-limit issue, separate fix).

🤖 Generated with Claude Code

Note

Medium Risk
Moderate risk because it changes how tool metadata (backends/description/security) is sourced and relies on mise registry --json --security output/alias handling; failures would impact the D1 sync payload, though a fallback path is included for older mise.

Overview
Speeds up the D1 tool-manifest build by replacing per-tool mise tool <name> --json shell-outs with a single mise registry --json --security call that provides backends, description, and security for all tools at once.

Adds a compatibility fallback to mise registry --json when --security is unsupported, and preserves prior alias resolution by mapping registry aliases to the same metadata entry. Metadata derivation is adjusted to pull GitHub/repo URLs from registry backends (or owner/repo tool names) while keeping manual overrides as highest priority.

^{Reviewed by Cursor Bugbot for commit 6e28e7d. Bugbot is set up for automated code reviews on this repo. Configure here.}

gemini-code-assist

Code Review

This pull request optimizes the sync-to-d1.js script by reducing the number of external process calls and introducing parallelism. It replaces individual mise tool calls with a single bulk mise registry --json call to retrieve backends and descriptions. Additionally, it implements a concurrent worker pattern to fetch security metadata for multiple tools in parallel, significantly improving the script's execution time. I have no feedback to provide.

greptile-apps · 2026-04-24T16:09:01Z

Greptile Summary

Replaces 977 sequential mise tool X --json spawns with a single mise registry --json --security call, dropping the worker-pool concurrency machinery and reducing manifest phase from ~4 min to ~30 s. Adds a two-level fallback (no --security → empty Map) and an alias-aware indexing pass so TOML files named after aliases still resolve metadata correctly.

Confidence Score: 5/5

Safe to merge; the refactor is well-tested with both the fast path and the simulated-older-mise fallback path verified locally.

Only P2 findings present. The logic is straightforward — one bulk call replaces many sequential ones — and the alias second-pass and fallback paths are clearly reasoned in comments. No data-loss or correctness issues found.

No files require special attention beyond the note about the broad catch block in getRegistryInfo.

Important Files Changed

Filename	Overview
scripts/sync-to-d1.js	Replaced 977 per-tool `mise tool X --json` spawns with a single `mise registry --json --security` call; adds alias-aware index and graceful fallback; GitHub slug extraction order changed (all backends first, toolName fallback second).

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[main] --> B[getRegistryInfo]
    B --> C{mise registry
--json --security}
    C -->|success| D[JSON.parse output]
    C -->|throws| E[warn: upgrade mise]
    E --> F{mise registry
--json fallback}
    F -->|success| D
    F -->|throws| G[error log
return empty Map]
    D --> H[Pass 1: index by short]
    H --> I[Pass 2: register aliases]
    I --> J[return infoMap]
    J --> K[For each TOML file]
    K --> L[registryMap.get toolName]
    L --> M{has backends?}
    M -->|yes| N[buildPackageUrls
buildAquaLink
extractGithubSlug]
    M -->|no| O{toolName
has slash?}
    N --> O
    O -->|yes| P[tool.github = toolName]
    O -->|no| Q[no github]
    P --> R[apply manualOverrides]
    Q --> R
    N --> R
    R --> S[push to tools array]
    S --> T[POST /api/admin/tools/sync]

_{Reviews (3): Last reviewed commit: "perf: cut D1 tools-sync runtime via sing..." | Re-trigger Greptile}

jdx · 2026-04-25T19:00:20Z

Both greptile review comments above (the negative-MISE_TOOL_CONCURRENCY P1 and the missing-execFileAsync timeout P2) reference code from the pre-rebase commit. Force-push to d7f38616 removed all of it: there's no longer a per-tool getSecurity / fetchSecurityMap worker pool, no execFileAsync call, and no MISE_TOOL_CONCURRENCY env var to validate. The whole concern collapses into the single mise registry --json --security invocation.

Verified at HEAD:

$ grep -n 'MISE_TOOL_CONCURRENCY\|execFileAsync\|getSecurity\|fetchSecurityMap' scripts/sync-to-d1.js
(no matches)

No code change needed.

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d7f3861. Configure here.}

…urity The `Sync tools to D1` step previously shelled out to `mise tool <name> --json` once per tool (977 sequential invocations) just to assemble each tool's GitHub slug, description, and security list. That dominated the step at ~4 of its ~5 min wall time. This PR collapses the whole metadata gather into one mise call: - `mise registry --json --security` (added upstream in mise v2026.4.22 via jdx/mise#9364) returns `{short, backends, description, aliases, security}` for every tool in one JSON array. No per-tool shell-outs, no parallel worker pool, no `MISE_TOOL_CONCURRENCY` env knob. - Falls back to `mise registry --json` (without security) if the installed mise predates the flag, so the rest of the manifest still syncs. Local timing on 978 tools: manifest phase ~30 s end-to-end, vs. ~240 s before any of this work. Code is also dramatically simpler — drops the `getSecurity` / `fetchSecurityMap` worker plumbing and the `execFile` + `promisify` async machinery. Bonus bug fix preserved from the prior version of this PR: the original script read `description` from `mise tool --json`, which almost always returned null. With `mise registry --json` providing descriptions, the count jumps from ~17 tools-with-descriptions to ~975. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

jdx · 2026-04-25T19:12:50Z

Fixed in 6e28e7d7 — getRegistryInfo now does a second pass to index each entry's aliases into the same map (with shorts winning if they collide). Verified locally:

No regression on existing data: counts are identical (845/975/978/184 with github/desc/backends/pkg-urls).
Confirmed the bug is currently latent: zero TOML files in docs/ match an alias-but-not-short, so no rows in D1 would have been clobbered. The fix is defensive against future alias-only TOMLs.

gemini-code-assist Bot reviewed Apr 24, 2026

View reviewed changes

greptile-apps Bot reviewed Apr 24, 2026

View reviewed changes

Comment thread scripts/sync-to-d1.js Outdated

Comment thread scripts/sync-to-d1.js Outdated

jdx force-pushed the sync-to-d1-perf branch from 9567cd0 to d7f3861 Compare April 25, 2026 18:58

jdx changed the title ~~perf: cut D1 tools-sync from ~5 min to ~30s~~ perf: cut D1 tools-sync runtime via single mise registry --json --security Apr 25, 2026

cursor Bot reviewed Apr 25, 2026

View reviewed changes

Comment thread scripts/sync-to-d1.js

jdx force-pushed the sync-to-d1-perf branch from d7f3861 to 6e28e7d Compare April 25, 2026 19:12

jdx merged commit af1c0e5 into main Apr 26, 2026
6 checks passed

jdx deleted the sync-to-d1-perf branch April 26, 2026 01:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: cut D1 tools-sync runtime via single mise registry --json --security#130

perf: cut D1 tools-sync runtime via single mise registry --json --security#130
jdx merged 1 commit intomainfrom
sync-to-d1-perf

jdx commented Apr 24, 2026 •

edited by cursor Bot

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

greptile-apps Bot commented Apr 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

jdx commented Apr 25, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

jdx commented Apr 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jdx commented Apr 24, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Measurement

Bonus bug fix

Test plan

Notes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

greptile-apps Bot commented Apr 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

jdx commented Apr 25, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jdx commented Apr 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

jdx commented Apr 24, 2026 •

edited by cursor Bot

Loading

greptile-apps Bot commented Apr 24, 2026 •

edited

Loading