Skip to content

[BUG] NPM 7.x broke the "--json" CLI parameter #2740

Closed

Description

@octogonz

The problem

NPM's --json option allows scripts to invoke an NPM command and receive machine-readable JSON that can be easily processed. In previous NPM versions, the JSON was written to STDOUT and all other messages were written to STDERR, making it easy for a script to extract the JSON.

With NPM 7.x, everything is mixed together on STDERR. This defeats the point of the --json feature, which was to make parsing easy. Instead, a script has to implement heuristics to guess which STDERR lines are JSON or not. Scripts developed for NPM 6 now fail because they are reading the wrong stream.

Steps To Reproduce:

  1. Correct behavior: With NPM 6.x, invoke this command:

    npm search --json
    

    The JSON output is written to STDOUT:

    {
      "error": {
        "code": null,
        "summary": "search must be called with arguments",
        "detail": ""
      }
    }

    The console messages are written to STDERR:

    npm ERR! search must be called with arguments
    npm ERR! A complete log of this run can be found in:
    
  2. Broken behavior: With NPM 7.5.4 (the latest version), invoke the same command. Both outputs are mixed together on STDERR. Nothing is written to STDOUT.

This regression appears to have been introduced with NPM 7.0.0.

Environment:

  • OS: All OS's
  • Node: 12.17.0
  • npm: 7.5.4

Activity

  1. added
    Bugthing that needs fixing
    Needs Triageneeds review for next steps
    Release 7.xwork is associated with a specific npm 7 release
    on Feb 20, 2021
  2. added
    Priority 2secondary priority issue
    and removed
    Needs Triageneeds review for next steps
    on Mar 22, 2021
  3. RammasEchor commented on Jun 14, 2021

    @RammasEchor

    Hi! I've been looking up this issue, and I would like to work on it.

  4. bdkjones commented on Sep 11, 2021

    @bdkjones

    Any progress on this? It's a big regression from 6.x for those of us who integrate npm into tooling. Thanks.

  5. octogonz commented on Sep 12, 2021

    @octogonz
    Author

    This was a significant regression for tools that rely on the NPM installation provided by Node.js: Since we can't assume that the developer has the latest Node.js, even if NPM eventually fixes this bug, it may be years before we can safely remove our workaround.

    Looks like @RammasEchor made a PR but it was rejected.

    snip

    If it helps, here's the workaround that we used:

    https://github.com/microsoft/rushstack/blob/b2aa54fa2a284e166529cdebbc3918bcd7dc7503/apps/rush-lib/src/logic/setup/SetupPackageRegistry.ts#L486

    In brighter news, Node 16 will let you use Yarn without having to install it. yarn info has equivalent functionality, and Yarn's --json parameter is correctly implemented. 👍

  6. rossng commented on Sep 24, 2021

    @rossng

    This regression is causing problems for atlassian/changesets users. For example: changesets/changesets#397

  7. bdkjones commented on Oct 7, 2021

    @bdkjones

    Sounds like switching to yarn is the best approach. "Will get attention when we're freed up" sounds an awful lot like "Will sit in GitHub issues for seven years."

  8. lukekarrys commented on Oct 7, 2021

    @lukekarrys
    Contributor

    I did some digging and found that this was introduced in #2155 (which was to fix #2150).

    It looks like npm ls --json (not sure about other commands yet) would print their own json blob and then the exit handler would print an additional json blob with the error. This caused unparseable json so the second blob was switched to stderr. Obviously, this caused regressions like this one.

    A better solution I think would be to switch the main exit handler back to stdout and special case any commands that are already printing their own json output to not print more. I'm more confident (than I am about #2773) that this fix can come before any output refactor work we're doing. So hopefully it's less than seven years 😄

  9. Andarist commented on Nov 14, 2021

    @Andarist
    Contributor

    The OP suggests that everything was previously sent to stderr - that doesn't sound right though. If I understand the issue correctly the change only affected JSON errors - they are now written to stderr where previously they were written to stdout. So the change is of a smaller scope than OP suggests. Can anyone confirm this? Don't take me wrong - this is still super annoying and affecting me directly (I'm a Changesets maintainer and this has caused problems in our publishing scripts), I just want to make sure that I understand the issue correctly so I can fix it properly on our end.

  10. added 3 commits that reference this issue on Oct 19, 2022
    0db7dcc
    cf39551
    6c870b4
  11. added 2 commits that reference this issue on Oct 19, 2022
    46d038f
    d3543e9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Bugthing that needs fixingPriority 2secondary priority issueRelease 7.xwork is associated with a specific npm 7 release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions