release: [email protected] #16244

zkat · 2017-04-04T21:13:37Z

NOTE: This PR is a WIP. npm@5 beta is currently underway. See #16510 for deets.

Wowowowowow npm@5!

This release marks months of hard work for the young, scrappy, and hungry CLI team, and includes some changes we've been hoping to do for literally years. npm@5 takes npm a pretty big step forward, significantly improving its performance in almost all common situations, fixing a bunch of old errors due to the architecture, and just generally making it more robust and fault-tolerant. It comes with changes to make life easier for people doing monorepos, for users who want consistency/security guarantees, and brings semver support to git dependencies. See below for all the deets!

Breaking Changes

One binary to rule them all: ./cli.js has been removed in favor of ./bin/npm-cli.js. In case you were doing something with ./cli.js itself. (Replace cli.js with bin/npm-cli.js and remove cli.js #12096)
Stub file removed (utils: remove lib/utils/rename.js #16204)
Existing npm caches will no longer be used: you will have to redownload any cached packages. There is no tool or intention to reuse old caches. (Replace metadata + fetch + cache code #15666)
npm install ./packages/subdir will now create a symlink instead of a regular installation. file://path/to/tarball.tgz will not change -- only directories are symlinked. (RFC: file: specifier changes #15900)
npm will now scold you if you capitalize its name. seriously it will fight you.
The "extremely legacy" _token couchToken has been removed. (Remove _token support altogether` #12986)
npm will --save by default now. Additionally, package-lock.json will be automatically created unless an npm-shrinkwrap.json exists. (Replace metadata + fetch + cache code #15666)
git dependencies support semver (Resolve git dependencies with semver ranges #15308, Replace metadata + fetch + cache code #15666)
git dependencies with prepare scripts will have their devDependencies installed, and npm install run in their directory before being packed.
npm cache commands have been rewritten and don't really work anything like they did before. (Replace metadata + fetch + cache code #15666)
--cache-min and --cache-max have been deprecated. (Replace metadata + fetch + cache code #15666)
running npm while offline will no longer insist on retrying network requests (Replace metadata + fetch + cache code #15666)
The contents of ~/.npm have completely changed.
A new loglevel, notice, has been added and set as default.
package locks no longer exclude optionalDependencies that failed to build. This means package-lock.json and npm-shrinkwrap.json should now be cross-platform. (RFC: file: specifier changes #15900)
shrinkwrap and package-lock no longer exclude devDependencies. You can still prevent their installation with --prod/--only=prod. (RFC: file: specifier changes #15900)
If you generated your package lock against registry A, and you switch to registry B, npm will now try to install the packages from registry B, instead of A. If you want to use different registries for different packages, use scope-specific registries (npm config set @myscope:registry=https://myownregist.ry/packages/). Different registries for different unscoped packages are not supported anymore.
shrinkwrap and package-lock no longer warn and exit without saving the lockfile
Local tarballs can now only be installed if they have a file extensions .tar, .tar.gz, or .tgz.

Feature Summary

There's a bunch of exciting stuff going in here! You can track all the changes planned for npm@5 on its project tracker.

Installer changes

A new, standardised lockfile feature meant for cross-package-manager compatibility (package-lock.json), and a new format and semantics for shrinkwrap. (spec: Describe npm-shrinkwrap.json and package-lock.json #16441)
--save is no longer necessary. All installs will be saved by default. You can prevent saving with --no-save. Installing optional and dev deps is unchanged: use -D/--save-dev and -O/--save-optional if you want them saved into those fields instead. Note that since npm@3, npm will automatically update npm-shrinkwrap.json when you save: this will also be true for package-lock.json. (Replace metadata + fetch + cache code #15666)
Installing a package directory now ends up creating a symlink and does the Right Thing™ as far as saving to and installing from npm-shrinkwrap.json goes. If you have a monorepo, this should make things much easier to work with, and probably a lot faster too. 😁 (RFC: file: specifier changes #15900)
git dependencies now support semver-based matching: npm install git://github.com/npm/npm#semver:^5 (Resolve git dependencies with semver ranges #15308, Replace metadata + fetch + cache code #15666)
git dependencies with prepare scripts will now have their devDependencies installed, and their prepare script executed as if under npm pack.
node-gyp now supports node-gyp.cmd on Windows (Change npm_config_node_gyp to use shell script on Windows #14568)
npm no longer blasts your screen with the whole installed tree. Instead, you'll see a summary report of the install that is much kinder on your shell real-estate. Specially for large projects. (Replace post install module list with summary report #15914):

$ npm install
npm added 125, removed 32, updated 148 and moved 5 packages in 5.032s.
$

--parseable and --json now work more consistently across various commands, particularly install and ls.
Indentation is now detected and preserved for package.json, package-lock.json, and npm-shrinkwrap.json. If the package lock is missing, it will default to package.json's current indentation.

Publishing

New publishes will now include both sha512 and sha1 checksums. Versions of npm from 5 onwards will use the strongest algorithm available to verify downloads. (feat(integrity): add integrity field to publish npm-registry-client#157)

Cache Rewrite!

We've been talking about rewriting the cache for a loooong time. So here it is. Lots of exciting stuff ahead. The rewrite will also enable some exciting future features, but we'll talk about those when they're actually in the works. #15666 is the main PR for all these changes. Additional PRs/commits are linked inline.

package metadata, package download, and caching infrastructure replaced
It's a bit faster. Hopefully it will be noticeable. 🤔
With the shrinkwrap and package-lock changes, tarballs will be looked up in the cache by content address (and verified with it)
Corrupted cache entries will automatically be removed and re-fetched on integrity check failure.
npm CLI now supports tarball hashes with any hash function supported by Node.js. That is, it will use sha512 for tarballs from registries that send a sha512 checksum as the tarball hash. Publishing with sha512 is added by feat(integrity): add integrity field to publish npm-registry-client#157 and may be backfilled by the registry for older entries.
remote tarball requests are now properly cached. This means that even if you're missing the integrity field in your npm-shrinkwrap.json, npm will be able to install from the cache.
Downloads for large packages are streamed in and out of disk. npm is now able to install packages of """any""" size without running out of memory. Support for publishing them is pending (due to registry limitations).
Automatic fallback-to-offline mode. npm will seamlessly use your cache if you are offline, or if you lose access to a particular registry (for example, if you can no longer access a private npm repo, or if your git host is unavailable).
A new --prefer-offline option will make npm skip any conditional requests (304 checks) for stale cache data, and only hit the network if something is missing from the cache.
A new --prefer-online option that will force npm to revalidate cached data (with 304 checks), ignoring any staleness checks, and refreshing the cache with revalidated, fresh data.
A new --offline option will force npm to use the cache or exit. It will error with an ENOTCACHED code if anything it tries to install isn't already in the cache.
A new npm cache verify command that will garbage collect your cache, reducing disk usage for things you don't need (-handwave-), and will do full integrity verification on both the index and the content. This will also be hooked into npm doctor as part of its larger suite of checking tools.
The new cache is very fault tolerant and supports concurrent access.
- Multiple npm processes will not corrupt a shared cache.
- Corrupted data will not be installed. Data is checked on both insertion and extraction, and treated as if it were missing if found to be corrupted. I will literally bake you a cookie if you manage to corrupt the cache in such a way that you end up with the wrong data in your installation (installer bugs notwithstanding).
- npm cache clear is no longer useful for anything except clearing up disk space.
Package metadata is cached separately per registry and package type: you can't have package name conflicts between locally-installed packages, private repo packages, and public repo packages. Identical tarball data will still be shared/deduplicated as long as their hashes match.
HTTP cache-related headers and features are "fully" (lol) supported for both metadata and tarball requests -- if you have your own registry, you can define your own cache settings the CLI will obey!
prepublishOnly now runs before the tarball to publish is created, after prepare has run.
...and more!

For release task list

siddharthkp · 2017-04-12T04:44:38Z

@zkat Will any of these flags become default?

Some folks bump into addition flags but it doesn't reach everyone (example: --cache-min)
--prefer-offline is something that would make npm installs faster for everyone and probably should be a default.

I understand that it breaks expectations on how people think npm works right now (which is --prefer-offline right now?) But, do you think major version is probably the time to change this expectation?

zkat · 2017-04-12T05:46:13Z

@siddharthkp There's no significant speed difference between --prefer-offline and --prefer-online when I actually test it (I'm surprised too). And I mean that -- the two installs on the repo I do my usual benchmarks are within 1 second of each other, for a minute-long install. Those 304 check are very fast, and pacote/make-fetch-happen are both very very fast at what they do when it comes to networking.

I see you mentioned that you think users would expect --prefer-offline to be the default, but I don't think that's true in practice: Many/most users install newer versions of packages, for example, by doing npm i -S pkg. In this case, using --prefer-offline will not upgrade anything: latest is already locally cached. pacote.manifest() will be satisfied with that, and the user will receive a very stale version of the package manifest. The intention behind --prefer-offline is to give users an option to reduce network usage in-general, if they're aware of the consequences of doing do.

Now, if you're looking for a speedup, the biggest one you can get with npm@5 is to use npm-shrinkwrap.json -- as of this patch, shrinkwrap will include content addresses for all package data, which means that as long as you have a warm cache, the network will not be used at all, and the entire package manifest phase will be completely skipped (because we don't need manifests if we have a shrinkwrap). The speedup from that is often on the order of 2x a non-shrinkwrap cached --prefer-offline install.

So, no, I probably won't use npm@5 to make --prefer-offline the default, since I'm not sure there's a reason to, in practice. Making npm-shrinkwrap be a default, though... that's in the works. Maybe not for npm@5 but I'm pushing for it for sure.

siddharthkp · 2017-04-12T06:19:29Z

There's no significant speed difference between --prefer-offline and --prefer-online when I actually test it (I'm surprised too).

😮
What were the network conditions like in your benchmarks?

I had a conversation with @iarna a while back (npm 3) where she educated me about the network requests to the registry even when using cache + the effect of latency on it. (Significant for emerging nations - prominently 2G and latency heavy nations - Australia and such)

Are you saying pacote nullifies that effect? (or at least makes it not noticeable) because that would be 💯

I see you mentioned that you think users would expect --prefer-offline to be the default

Ouch! My bad, I meant --prefer-online. Honest mistake.

The speedup from that is often on the order of 2x a non-shrinkwrap cached --prefer-offline install.

This is another interesting part 🤔
Is this because npm still needs to resolve the dependency tree with --prefer-offline which means additional network requests? Not the same with shrinkwrap.

Making npm-shrinkwrap be a default, though... that's in the works.

💯 Is there a github issue that I can upvote for this?

zkat · 2017-04-12T06:47:21Z

What were the network conditions like in your benchmarks?

I'm on a Bay Area network, which is admittedly absurdly good. If the difference is so small now, though, I imagine it wouldn't be so big: even at 1/10th the speed, that would still be barely 10 seconds added (to a 60-second installation).

Are you saying pacote nullifies that effect? (or at least makes it not noticeable) because that would be 💯

I really can't say that for sure right now. tbh, the difference is so surprisingly small that I think I'm gonna have to spend some time crawling through tests and making sure I don't have some bug. That code definitely has tests attached, though...

Ouch! My bad, I meant --prefer-online. Honest mistake.

Ah. As far as this one goes, pacote obeys cache age settings from a given registry. This is generally happy news for all involved (if we hit the registry less without degrading user experience, that's a win!), and those settings tend to be fairly low: the main npm registry sets Cache-Control: max-age=300, which is in seconds (5 minutes). After those 5 minutes, we'll do the 304 checks again. This seems reasonable to me, specially with the --prefer-offline escape hatch. Anyway, no worries. They're kinda easy to get confused 😁

I would care more if it weren't for shrinkwrap being the main target of optimization, which brings me to...

Is this because npm still needs to resolve the dependency tree

Pretty much this. If we don't have the exact hashes for the tarballs we want to end up installing, npm needs to repeat the process of lookup by name --> resolve version --> get shasum + dependency names from version --> repeat for all recursive dependencies. That's a pretty heavy process.

💯 Is there a github issue that I can upvote for this?

Not quite yet. There's a github project for npm@5 and one of my TODOs is to write a proposal for how shrinkwrap-by-default would work. From what I've talked with @iarna, though, we need to fix other stuff (like npm update) before we even consider this. So, it'll probably be specced out after npm@5 is out and we're ready to start thinking about npm@6.

siddharthkp · 2017-04-12T09:24:16Z

If shrinkwrap by default is likely to come in npm6, does it make sense to make --prefer-offline a default in npm5?
Or is that going in the wrong direction?

viktor-ku · 2017-04-12T12:46:53Z

Any reasons why yarn is so fast? I really expect npm@5 to be as fast as yarn. How can I help?

siddharthkp · 2017-04-12T12:54:12Z

@Kuroljov With npm-shrinkwrap, npm will be in serious competition with yarn.

@zkat Can we start playing around with that somewhere? (Can contribute to your benchmarks?)

zkat · 2017-04-12T20:19:26Z

@Kuroljov This release addresses a lot of performance issues. Right now, npm@5 installs are running anywhere between 2x-5x faster than npm@4. There's a few optimizations left that I don't know if we'll have ready by npm@5, but that number is bound to get significantly bigger (as in, 30%+ from one of those changes)

Some sample benchmarks:

react boilerplate install (including lifecycle scripts):
npm@4 i (no cache): 147.23s
npm@4 i (warm cache): 145.70s
npm@5 i (no cache): 74.30s
npm@5 i (warm cache): 70.85s
npm@5 i (no cache, no shrinkwrap): 99.83s

~/code/npm-website $ npm i --ignore-scripts
warm cache:
[email protected]  - 150s
[email protected] - 52s
[email protected] w/ shrinkwrap   - 38s

zkat · 2017-04-12T20:32:14Z

@siddharthkp expect a beta release of npm@5 in the coming weeks. I can't give you the exact date, but I'm currently in the process of reviving npm's massive test suite after such a big change and fixing all my little messups along the way.

What I can tell you is that if you make https://npm.im/make-fetch-happen and https://npm.im/cacache faster and better, it'll be directly reflected on the performance of npm@5. Once https://npm.im/pacote gets its @2 release (probably shortly before npm@beta), the same will apply to that. Those first two projects are relatively stable and usable, though, and I think people can definitely already start contributing significantly to them. There's also https://npm.im/ssri but I don't think that's much of a bottleneck at all rn.

I'll make an announcement somewhere dev-oriented once npm@beta is out, cause I definitely want early adopters to help me find all the things that broke during the rewrite -- because it's a big one.

zkat · 2017-04-20T22:24:22Z

@siddharthkp coming back to this: what are the network conditions where you're at? (since you asked). I'd like to get a better, more realistic perspective on how npm5 handles slow and/or unstable networks and special configurations like proxies, which is all fairly hard to simulate unless you've been deep in that situation, imo. There's now a (semi-closed) beta of npm5 that you can install with npm i -g 'npm/npm#release-beta-5 that will give you an npm5 binary that lives alongside your standard installation. It would be super awesome if folks who are willing to look into weirdness tried it out and gave feedback. I very much want it to be a smooth, seamless experience even in bad network conditions: the worst thing that should happen on unreliable networks is that your download will be slightly slower. Everything else should Just Work™.

…ninstalling

…stall

Previously we were doing this so we could load relationships, but now we can simply request no fake children. This forces a fetch-metadata phase for the lockfile, but does not actually install all the modules.

This can happen when, for example, removing a module from your project that isn't actually in your node_modules at the moment.

It used to be that we did this check in `is-extraneous` but it turns out to be much cleaner to keep that pure. It let's us use `is-extraneous` for things like pruning.

zkat · 2017-05-27T05:59:12Z

This was tagged and released yesterday as npm@5 🎉 thanks everyone!

olegkorol · 2017-06-12T09:06:07Z

Hi,

what is the new format of npm-shrinkwrap.json?
After updating npm to v. 5 it won't npm install anymore...

Thanks in advance!

zkat added breaking release in-progress cache error-handling git installer link network performance platform-integration shasum shrinkwrap supportability windows labels Apr 4, 2017

zkat requested review from ceejbot, iarna and isaacs April 4, 2017 21:24

zkat mentioned this pull request Apr 11, 2017

RFC: file: specifier changes #15900

Closed

iarna force-pushed the release-next-5 branch from b1f6b87 to 8bcef6a Compare April 14, 2017 23:13

zkat and others added 21 commits May 25, 2017 18:55

uninstall: remove stray console.log

f17f4e6

install: Make removing packages work w/o package.json

3116cfe

install,uninstall: Disable fake children when installing one mod or u…

5599538

…ninstalling

test: modernize uninstall-save

e6db682

uninstall: defer arg clearing till after global dir is filtered

1112b9a

test: make no-scan-full-global-dir compat w/ class based install/unin…

f4a6dd5

…stall

test: extraneous deps in legacy shrinkwraps are not installed

237a1cf

install: no need to install non-requested deps

3ada941

Previously we were doing this so we could load relationships, but now we can simply request no fake children. This forces a fetch-metadata phase for the lockfile, but does not actually install all the modules.

test: gently-rm-linked-module: modernize test

80202cd

pkglock: add --no-package-lock option as --no-shrinkwrap alias

4df5934

pacote: send username/password auth through, too

1ab8c41

pacote: look for always-auth too

72ce47c

remove-deps: Update dependencies list even if we're not saving

4c99147

test: gently-rm-linked-module: Cleanup output

1ea953a

install: allow parseable output for nodes w/o paths

da7bbb2

This can happen when, for example, removing a module from your project that isn't actually in your node_modules at the moment.

ls: Nothing is extraneous without a package.json

0d40bf3

It used to be that we did this check in `is-extraneous` but it turns out to be much cleaner to keep that pure. It let's us use `is-extraneous` for things like pruning.

pacote: decode password before passing it on

046f967

pacote: more alwaysAuth logic

a976fa1

preinstall: Runs in the final dest, not the staging folder

0d35975

[email protected]

db76632

lifecycle: added prepack and postpack (#16725)

1e938d6

zkat added the npm5 label May 26, 2017

zkat added 2 commits May 25, 2017 19:25

docs: END OF AN ERA OF CHANGELOGS 😭

5529a6f

doc: update changelog for [email protected]

7903602

zkat force-pushed the release-next-5 branch from 8688036 to 7903602 Compare May 26, 2017 03:03

zkat closed this May 27, 2017

zkat deleted the release-next-5 branch May 27, 2017 05:59

zkat removed the in-progress label May 27, 2017

release: [email protected] #16244

release: [email protected] #16244

Uh oh!

Conversation

zkat commented Apr 4, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

NOTE: This PR is a WIP. npm@5 beta is currently underway. See #16510 for deets.

Breaking Changes

Feature Summary

Installer changes

Publishing

Cache Rewrite!

For release task list

Uh oh!

siddharthkp commented Apr 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkat commented Apr 12, 2017

Uh oh!

siddharthkp commented Apr 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkat commented Apr 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

siddharthkp commented Apr 12, 2017

Uh oh!

viktor-ku commented Apr 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

siddharthkp commented Apr 12, 2017

Uh oh!

zkat commented Apr 12, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zkat commented Apr 12, 2017

Uh oh!

zkat commented Apr 20, 2017

Uh oh!

zkat commented May 27, 2017

Uh oh!

olegkorol commented Jun 12, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

zkat commented Apr 4, 2017 •

edited

Loading

siddharthkp commented Apr 12, 2017 •

edited

Loading

siddharthkp commented Apr 12, 2017 •

edited

Loading

zkat commented Apr 12, 2017 •

edited

Loading

viktor-ku commented Apr 12, 2017 •

edited

Loading

zkat commented Apr 12, 2017 •

edited

Loading