feat: automatically switch to the right pnpm version by zkochan · Pull Request #8363 · pnpm/pnpm

zkochan · 2024-08-01T00:08:47Z

A new experimental option: manage-package-manager-versions. When set to true, pnpm will automatically download the version requested by the project via the packageManager field in package.json.

close #8360

zkochan · 2024-08-04T13:52:36Z

@jakebailey I've seen that you like corepack, so I guess you'll like to hear that the same feature will work without corepack too.

jakebailey · 2024-08-04T19:01:33Z

cli/cli-utils/src/packageIsInstallable.ts

+  const [name, pmReference] = packageManager.split('@')
+  // pmReference is semantic versioning, not URL
+  if (pmReference.includes(':')) return { name, version: undefined }
+  const [version] = pmReference.split('+')


This omits the integrity; is supporting that possible? Or is the corepack blob different than what pnpm is downloading?

What is that blob about? We could support an sha512 integrity checksum. The same that we use for dependencies in the lockfile. But that would involve a bigger rewrite. If it is required, we can work on it in a separate PR. Also, we will only support installing pnpm from the registry.

I can't recall if corepack downloads the npm package or something else, but it looks like pnpm's npm package is self-contained so likely it was downloading that? (I have not checked)

yes, it must be downloading pnpm from the registry.

Regarding pmReference.includes(':')

I get errors since pmReference is undefined in certain cases. It's kind of hard to nail down...

jakebailey · 2024-08-04T19:04:08Z

I think the main concern I have is the opt-out story. Often, I find myself needing to explicitly run a version of a package manager that is differnet than what's listed in packageManager; corepack has COREPACK_ENABLE_STRICT so I can turn strict checking off and continue on (either by running a specific version or by using corepack pnpm@... or something).

I know this is set by manage-package-manager-versions, but is there an environment variable equivalent? Maybe pnpm settings automatically get them?

If I'm working on pnpm itself and run a dev build, will that also cause a download?

(Okay, well, my main concern is less the opt-out story and more that we're seemingly ending up in this world where I've used a package manager to install a node version manager, which then includes a package manager manager, which then runs a package manager which can then automatically run yet another package manager. It makes me so frustrated that this is the state of things... Trying our best I guess 😞)

zkochan · 2024-08-04T21:22:56Z

I know this is set by manage-package-manager-versions, but is there an environment variable equivalent? Maybe pnpm settings automatically get them?

Yes, all settings can be set via env variables. I will also check the COREPACK_ENABLE_STRICT env variable.

which then includes a package manager manager, which then runs a package manager which can then automatically run yet another package manager.

pnpm will not manage its versions when it is executed by corepack.

zkochan · 2024-08-04T23:28:20Z

@jakebailey there is a problem with the "blob" support. We ship 2 packages for pnpm: pnpm and @pnpm/exe. When we switch to a different pnpm version, we check which type of pnpm package the user is running and download the same package. Hence, every version has 2 packages and two integrities.

jakebailey · 2024-08-04T23:35:27Z

Right, that was the thing I was worried about with the integrity checks, though https://github.com/nodejs/corepack/blob/main/config.json seems to imply that corepack at least always used the registry for pnpm, so wouldn't have been @pnpm/exe?

I'm not sure what the solution here is, though I sort of expect that the integrity to be the one from the npm registry (like corepack does) and not the exe....

zkochan · 2024-08-04T23:36:58Z

@pnpm/exe is in the npm registry as well. These are two npm packages. Both work.

jakebailey · 2024-08-04T23:40:20Z

Right, but I don't think corepack would have ever added @pnpm/exe; I was just thinking about back compat.

zkochan · 2024-08-04T23:43:50Z

Corepack is still experimental, so I think for now it should be OK if we don't support the blob. Maybe print a warning about it. I prefer the @pnpm/exe version as it allows to install pnpm on a system that doesn't have node.js and then use pnpm to install node.js. I know that corepack wasn't going to add support for @pnpm/exe and that is one of the reasons I decided to add this feature to pnpm.

jakebailey · 2024-08-04T23:45:04Z

I'd just ignore it then, yeah.

zkochan · 2024-08-04T23:53:10Z

corepack has COREPACK_ENABLE_STRICT so I can turn strict checking off and continue on (either by running a specific version or by using corepack pnpm@... or something).

I just checked, COREPACK_ENABLE_STRICT=0 doesn't turn off pm version switch. It just allows to run a different pm. This PR doesn't add any new errors, when the pm name doesn't match.

This is an opt-in feature as of now, so the project will need to include manage-package-manager-versions=true in .npmrc. To disable the check, you can run:

pnpm --config.manage-package-manager-versions=false install

or comment out the setting in .npmrc.

zkochan · 2024-08-06T13:55:46Z

I'll do one last change, I need to fix pnpm -v. Unfortunately it will make pnpm -v slower.

KSXGitHub · 2024-08-06T14:00:05Z

I need to fix pnpm -v

Is it strictly necessary? In what situation would the user run pnpm -v? What would the user expect from it?

zkochan · 2024-08-06T14:32:40Z

I predict that users will run pnpm -v to test if the version switch works. If it won't show the version in the package manager field, we'll get a lot of issues filed.

gluxon · 2024-08-06T19:28:13Z

(Okay, well, my main concern is less the opt-out story and more that we're seemingly ending up in this world where I've used a package manager to install a node version manager, which then includes a package manager manager, which then runs a package manager which can then automatically run yet another package manager. It makes me so frustrated that this is the state of things... Trying our best I guess 😞)

I'm also wondering whether we're solving the installation problem comprehensively.

One of the problems in an enterprise setting is that it's tough to teach every new person how to install system-wide tools like corepack and ensure it's on the PATH correctly. This gets hard when an older version of corepack might be installed, or somehow a different pnpm package version takes precedence over the corepack hook. For less experienced developers, we have to help with PATH related problems that become too much for us to do at scale.

Having pnpm version manage itself feels useful, but doesn't entirely solve the problem either since you have to install some version of pnpm system-wide and ensure it's on the PATH.

I've thought about whether it'd be useful to have a ./pnpm shell script that's committed to the root of a git repo and contains similar logic to what's implemented in this PR. Users would then run ./pnpm install, ./pnpm add, etc.

gluxon

Implementation looks good! Looks like a lot of people will be excited for this feature. 🚀

My only question is whether we should do some sort of repo-level entry script (e.g. ./pnpm) in addition or instead this feature. If the repo-level entry script addresses the same pain point, it we might not need this functionality in the pnpm binary itself.

config/config/src/index.ts

Co-authored-by: Brandon Cheng <[email protected]>

zkochan · 2024-08-06T19:37:11Z

One of the problems in an enterprise setting is that it's tough to teach every new person how to install system-wide tools like corepack and ensure it's on the PATH correctly.

Why is it hard at the moment? It is included with Node.js, so they just need to run corepack enable. Can't be easier than that.

gluxon · 2024-08-06T19:44:04Z

One of the problems in an enterprise setting is that it's tough to teach every new person how to install system-wide tools like corepack and ensure it's on the PATH correctly.

Why is it hard at the moment? It is included with Node.js, so they just need to run corepack enable. Can't be easier than that.

For newer, less technical contributors to a repo, you have to teach them how to install node and ensure it's on the PATH. Most of our users are using macOS, which then requires a recommendation to install homebrew, etc. Sometimes people get a bit clever with nvm, but don't know exactly how it works and need help.

We found that it was much easier to write our own script that's committed to the git repo and downloads pnpm for them. That solution completely eliminated the numerous IT-like questions about installing node, pnpm, brew, corepack. etc directed to our team.

I suspect many other teams and companies are running into the exact same problems.

danielbayley · 2024-08-06T22:37:36Z

I've thought about whether it'd be useful to have a ./pnpm shell script that's committed to the root of a git repo and contains similar logic to what's implemented in this PR. Users would then run ./pnpm install, ./pnpm add, etc.

Oh god no, please, not another root .file! 🙈

For newer, less technical contributors to a repo, you have to teach them how to install node and ensure it's on the PATH.

To be fair, installing Node and/or adding something to PATH is basic stuff compared to contributing almost anything on here, and the latter step should be handled by whatever tool is being used to install Node, anyway… Homebrew certainly handles that.

gluxon · 2024-08-06T23:06:54Z

For newer, less technical contributors to a repo, you have to teach them how to install node and ensure it's on the PATH.

To be fair, installing Node and/or adding something to $PATH is basic stuff compared to contributing almost anything on here, and the latter step should be handled by whatever tool is being used to install Node, anyway… Homebrew certainly handles that.

While they may seem like "basic stuff" to seasoned developers like you and I, these sort of tasks can be challenging to drive-by contributors of frontend repos. Imagine a designer who wants to tweak a color in a .css file, but doesn't have unix experience and isn't sure what Node.js is. (I would personally implore them to learn because these tools are very cool, but forcing them to when they're trying to get something else done isn't a great introduction.)

It's usually easier to begin working on a repo when it makes as few assumptions on system state as possible. (Ex: the contents of $PATH.)

As an example, I'm pretty impressed by the Gradle build tool's wrapper concept. To use it, you only need to clone a repo and run ./gradlew build.

danielbayley · 2024-08-07T09:36:55Z

@gluxon I maintain a lot of Homebrew formulae/Casks, and whole taps, and go to great lengths to simplify installation as much as possible, and try to write all my documentation with the same mindset… From a design background myself, I still think in this way—from the user perspective. I'm just saying that over time, so much cruft gets lazily dumped into root .files, it also leads to a bad experience. By all means have that as an optional thing, if it helps less technical people get on with a task, I'm just pleading not to do it in any kind of mandatory way, that requires every other user to have more root .files. Also, hard-coding paths to $HOME/.file is particularly egregious.

gluxon · 2024-08-08T06:15:30Z

@gluxon I maintain a lot of Homebrew formulae/Casks, and whole taps, and go to great lengths to simplify installation as much as possible, and try to write all my documentation with the same mindset… From a design background myself, I still think in this way—from the user perspective. I'm just saying that over time, so much cruft gets lazily dumped into root .files, it also leads to a bad experience. By all means have that as an optional thing, if it helps less technical people get on with a task, I'm just pleading not to do it in any kind of mandatory way, that requires every other user to have more root .files. Also, hard-coding paths to $HOME/.file is particularly egregious.

I think we're on the same page around the importance of the user experience. 🙂

But I think there's a misunderstanding of what I was proposing:

I never stated the file would live at $HOME. In a previous comment, I stated the file would live at the "repo level".
The idea I had in mind would definitely be optional.

I'm imagining a pnpm shell script that's git committed. That shell script would download the correct version + architecture of pnpm and forward all CLI args to it.

Using pnpm in that repo would then become:

git clone [email protected]:<example>/example-project
cd example-project
./pnpm install
./pnpm run <script>

Since pnpm is a node version manager already, this skips the requirement to install node, brew, nvm, etc. In this case, the only system dependency would be git.

danielbayley · 2024-08-08T10:16:32Z

I think we're on the same page around the importance of the user experience. 🙂

I have no doubt. 🙂

But I think there's a misunderstanding of what I was proposing:

I never stated the file would live at $HOME. In a previous comment, I stated the file would live at the "repo level".

Yeah I got that, the ~/.file peeve is just a closely related aside.

The idea I had in mind would definitely be optional.

As long as that’s the case, have at it…

I'm imagining a pnpm shell script that's git committed. That shell script would download the correct version + architecture of pnpm and forward all CLI args to it.

Would this not also be better handled by packageManager/engines.pnpm/devEngines.pnpm, or just a package.json script…?

@Version

@Version

@Version

@Version

@Version

zkochan added 3 commits August 2, 2024 23:37

feat: automatically switch to the right pnpm version

eaef697

close #8360

feat: manage both js and exe pnpm versions

713c79f

refactor: rename global directory

4f4381f

zkochan force-pushed the manage-pnpm-versions branch from 2ef5c6f to 4f4381f Compare August 2, 2024 21:38

zkochan added 7 commits August 3, 2024 00:11

refactor: parsing packageManager

4b5c3b1

refactor: switch pnpm version

70e21cb

refactor: switch pnpm version

749c67a

test: env.path

463a36b

test: switch pnpm version

671e5c4

test: fix

5b171a8

feat: add manage-package-manager-versions setting

0abea66

zkochan marked this pull request as ready for review August 4, 2024 13:49

zkochan requested a review from a team August 4, 2024 13:51

jakebailey reviewed Aug 4, 2024

View reviewed changes

fix: only allow pnpm specified through version

b61efa4

This was referenced Aug 5, 2024

Support devEngines field #8153

Closed

Define spec for development environment field openjs-foundation/package-metadata-interoperability-working-group#15

Closed

refactor: download the correct pnpm artifact

a5d6f65

zkochan requested a review from a team August 5, 2024 12:03

refactor: parse package manager spec

ebcb3f6

refactor: change dir name in .tools

4b9e5ff

fix: version printed by pnpm -v

88142e7

gluxon approved these changes Aug 6, 2024

View reviewed changes

config/config/src/index.ts Show resolved Hide resolved

refactor: update config/config/src/index.ts

dc5160b

Co-authored-by: Brandon Cheng <[email protected]>

zkochan merged commit 26b065c into main Aug 6, 2024

zkochan deleted the manage-pnpm-versions branch August 6, 2024 19:59

jungwoo3490 mentioned this pull request Aug 8, 2024

chore(root): pnpm lockfileVersion 충돌 방지를 위해 버전 조건 명시 및 버전 자동 전환을 도입합니다. Setaday/Setaday-Client#8

Merged

This was referenced Aug 22, 2024

chore: update pnpm version and pin in package.json Arize-ai/phoenix#4333

Merged

chore: update pnpm version and pin in package.json Arize-ai/openinference#946

Merged

roottool mentioned this pull request Sep 18, 2024

Enable pnpm version management roottool/portfolio#1640

Merged

garysassano mentioned this pull request Oct 2, 2024

Add manage-package-manager-versions experimental option when pnpmVersion is specified projen/projen#3871

Open

btea mentioned this pull request May 10, 2025

chore: add packageManager field & move pnpm setting cssnano/cssnano#1700

Merged

lubieowoce mentioned this pull request Jul 11, 2025

[test] Use same pnpm in temporary repo directory vercel/next.js#81540

Merged

jrmajor mentioned this pull request Jul 23, 2025

[feature request] Use version of pnpm specified in package.json's engines section pnpm/action-setup#84

Open

Uh oh!

Conversation

zkochan commented Aug 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Aug 4, 2024

Uh oh!

jakebailey Aug 4, 2024

Choose a reason for hiding this comment

Uh oh!

zkochan Aug 4, 2024

Choose a reason for hiding this comment

Uh oh!

jakebailey Aug 4, 2024

Choose a reason for hiding this comment

Uh oh!

zkochan Aug 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

millette Aug 7, 2024

Choose a reason for hiding this comment

Uh oh!

jakebailey commented Aug 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Aug 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Aug 4, 2024

Uh oh!

jakebailey commented Aug 4, 2024

Uh oh!

zkochan commented Aug 4, 2024

Uh oh!

jakebailey commented Aug 4, 2024

Uh oh!

zkochan commented Aug 4, 2024

Uh oh!

jakebailey commented Aug 4, 2024

Uh oh!

zkochan commented Aug 4, 2024

Uh oh!

zkochan commented Aug 6, 2024

Uh oh!

KSXGitHub commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkochan commented Aug 6, 2024

Uh oh!

gluxon commented Aug 6, 2024

Uh oh!

gluxon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zkochan commented Aug 6, 2024

Uh oh!

gluxon commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielbayley commented Aug 6, 2024

Uh oh!

gluxon commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielbayley commented Aug 7, 2024

Uh oh!

gluxon commented Aug 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danielbayley commented Aug 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

zkochan commented Aug 1, 2024 •

edited

Loading

zkochan Aug 4, 2024 •

edited

Loading

jakebailey commented Aug 4, 2024 •

edited

Loading

zkochan commented Aug 4, 2024 •

edited

Loading

KSXGitHub commented Aug 6, 2024 •

edited

Loading

gluxon commented Aug 6, 2024 •

edited

Loading

gluxon commented Aug 6, 2024 •

edited

Loading

gluxon commented Aug 8, 2024 •

edited

Loading

danielbayley commented Aug 8, 2024 •

edited

Loading