SyntaxError: Unexpected end of JSON input when trying to publish packages

Affected Packages

All of the packages

Problem

When I try to publish my packages using pnpx changesets publish I get this error:

➜ pnpx changeset publish
🦋  info npm info @tinyhttp/app
🦋  info npm info @tinyhttp/cors
🦋  info npm info @tinyhttp/etag
🦋  info npm info @tinyhttp/logger
🦋  info npm info @tinyhttp/static
🦋  info @tinyhttp/app is being published because our local version (0.1.21) has not been published on npm
🦋  info @tinyhttp/cors is being published because our local version (0.1.20) has not been published on npm
🦋  info @tinyhttp/etag is being published because our local version (0.1.20) has not been published on npm
🦋  info @tinyhttp/logger is being published because our local version (0.1.20) has not been published on npm
🦋  info @tinyhttp/static is being published because our local version (0.1.20) has not been published on npm
🦋  info Publishing "@tinyhttp/app" at "0.1.21"
🦋  info Publishing "@tinyhttp/cors" at "0.1.20"
🦋  info Publishing "@tinyhttp/etag" at "0.1.20"
🦋  info Publishing "@tinyhttp/logger" at "0.1.20"
🦋  info Publishing "@tinyhttp/static" at "0.1.20"
🦋  error SyntaxError: Unexpected end of JSON input
🦋  error     at JSON.parse (<anonymous>)
🦋  error     at internalPublish (/home/v1rtl/Coding/tinyhttp/node_modules/.pnpm/@changesets/[email protected]_@[email protected]/node_modules/@changesets/cli/dist/cli.cjs.dev.js:591:19)
🦋  error     at processTicksAndRejections (internal/process/task_queues.js:97:5)

Same happens with ./node_modules/.bin/changeset publish

I don't know what causes it because there's no clear error message.

Helpful info

reproduction repository: https://github.com/talentlessguy/tinyhttp
changesets config: https://github.com/talentlessguy/tinyhttp/blob/master/.changeset/config.json
node version: 14.4.0
pnpm version: 5.0.2
npm version: 6.14.5

I'm having the exact same issue with pnpm

The failing build is here https://github.com/remirror/remirror/pull/251/checks?check_run_id=838235175
The commit causing the issue is here: https://github.com/remirror/remirror/tree/7f4153e0b08ae709dbdd28b09d6ebd3809e9a825
Config: https://github.com/remirror/remirror/blob/7f4153e0b08ae709dbdd28b09d6ebd3809e9a825/support/root/.changeset/config.json
Prerelease config: https://github.com/remirror/remirror/blob/7f4153e0b08ae709dbdd28b09d6ebd3809e9a825/support/root/.changeset/pre.json

Node: 12
pnpm: 5.2.6

The annoying thing is that I also can't run the release command from my own machine. It just freezes at the same point with no error displayed.

Thank you both for those reports. I will make sure to investigate this asap.

build: fix issue in `changesets` with patch

In order to get the release to work I've patched the @changesets/cli package to use npm for publishing always regardless of the project. The publish now works locally.

It's possible something changed in pnpm publish in version 5. Although this is the first time I've actually used pnpm to publish so I'm not sure.

@ifiokjr

@ifiokjr you may just use pnpm to publish the bumped versions. Just run pnpm publish -r.

Ahh that's very useful to know. It looks like pnpm is using changesets as well. How do you manage tags?

@zkochan

@zkochan I just did my first update using pnpm publish -r --tag next and it worked like a charm. Thanks for the tip.

I also like how you're managing releases for the pnpm project. I take it you're using changesets to generate the package specific changelogs and then you're manually adding your own top level changelog and github release. This strategy makes a lot of sense and I think I'll adopt it. With 80+ packages in remirror, creating unfiltered package releases has become too noisy.

@zkochan

@zkochan i think I've already asked this in the past, but... do you see any difference between our publishing script and pnpm publish -r? Would it be safe to just switch to pnpm entirely for this? Is there any required pnpm version for this command? Can the same be used for single package repos?

I would still be interested in investigating why this fails though:https://github.com/atlassian/changesets/blob/81595022e27e4b94a1bc87e56eabafb9a31e8315/packages/cli/src/commands/publish/npm-utils.ts#L126-L138

@Andarist

@Andarist we have discussed it here: #238

To summarize:

pnpm publishes packages in topological order
it works much faster than changesets

I think, when users use pnpm, it is safe to use pnpm publish -r in both single-package cases and multi-package repositories.

We've just released a new version of @changesets/cli with some additional logs for failed JSON.parse calls - if somebody could try running a publish with this new version this would be highly helpful in debugging this problem.

@Andarist

@Andarist hello, I tried it once again, here's the output:

error parsing json: 
🦋  error SyntaxError: Unexpected end of JSON input
🦋  error     at JSON.parse (<anonymous>)
🦋  error     at jsonParse (/home/v1rtl/Coding/tinyhttp/node_modules/.pnpm/@changesets/[email protected]/node_modules/@changesets/cli/dist/cli.cjs.dev.js:480:17)
🦋  error     at internalPublish (/home/v1rtl/Coding/tinyhttp/node_modules/.pnpm/@changesets/[email protected]/node_modules/@changesets/cli/dist/cli.cjs.dev.js:603:14)
🦋  error     at processTicksAndRejections (internal/process/task_queues.js:93:5)

@Andarist

Figured out the bug @Andarist @zkochan: there's a race condition between changesets publishing the manifests, and pnpm rewriting those manifests. That bug (from the perspective of changesets) happens here: https://github.com/pnpm/pnpm/blob/0b8fa3967067da8d4096c30b5e38fb462ed95220/packages/plugin-commands-publishing/src/publish.ts#L212-L223

I think, basically, changesets needs to throw the ball to pnpm on this one, and let it handle all the publishing.

In case that appears to be a non-sequitor, the error actually comes from the manifest being published referencing another dependency in the monorepo, whose manifest is currently missing as it undergoes the pnpm transition.

@devin-fee-ah

@devin-fee-ah how this can result in a JSON.parse fail later? 🤔 I understand that the manifest can not be updated properly but it seems like a potentially different problem.

I think, basically, changesets needs to throw the ball to pnpm on this one, and let it handle all the publishing.

This is probably the way to go for multiple reasons. Would you be willing to help us with this one? I probably won't have time to work on this one any time soon.

@bennypowers

@bennypowers As we already discussed and found out, you actually have to set both NPM_TOKEN and NODE_AUTH_TOKEN

¯_(ツ)_/¯

I'm also seeing this. Unexpected end of JSON input

Seems like input is empty after running the pnpm command with these args

[
  'publish',
  '/x/repos/pnpm-workspaces/src/packages/package-d',
  '--json',
  '--access',
  'restricted',
  '--tag',
  'latest',
  '--no-git-checks'
]

Then the json parser breaks because the command is empty.

🦋  error SyntaxError: Unexpected end of JSON input
🦋  error     at JSON.parse (<anonymous>)
🦋  error     at jsonParse (/x/repos/pnpm-workspaces/node_modules/.pnpm/@[email protected]/node_modules/@changesets/cli/dist/cli.cjs.dev.js:585

The package.json for package-d is correct...

Has anyone found a fix for this issue?

For me, I had to add the following to my package.json's:

  "publishConfig": {
    "access": "public"
  }

Answer from: https://stackoverflow.com/a/53478078

add publish config ref: changesets/changesets#397

I also ran into this issue when trying to publish my pnpm monorepo with changesets. There seem to be at least two issues here:

NPM commands are being run with the --json flag. Unfortunately a regression in NPM 7 (now the default version) means that both structured and unstructured output is all muxed onto stderr. This means the stdout is an empty string which fails to be parsed as JSON.
pnpm does not support the --json flag at all. However there are various places (example) where pnpm is called and the stdout is expected to contain JSON output.

On the first issue, I don't think there's much that can be done until NPM fixes the glaring regression.

I wouldn't mind trying to patch the second issue. I think the best thing would be to special-case pnpm - i.e. listen for the exit code of the pnpm process and, upon failure, pack the entire stdout/stderr output into the error. Would appreciate some guidance though.

There are news? Now I have the same error...

https://github.com/qiskit-community/qiskit-web-components/runs/4026323508?check_suite_focus=true

@rossng

pnpm does not support the --json flag at all.

@rossng where did u get this info from? do u know of any pnpm + other stuff combination that fails to use --json with pnpm? i was just testing it and it seems to work. pnpm simply forwards almost all arguments to the npm (and it uses npm's CLI under the hood for publishing) so in a sense this should "just work". I've tested this manually with pnpm v6 and verified the code for v3, v4, v5, and v6 - they all do roughly the same thing when it comes to this args forwarding to npm publish.

There are news? Now I have the same error...

@abdonrd I'm trying to figure out the proper solution for most of the similar publishing issues to fix this once and for all. Stay tuned

@Andarist

@Andarist I think you are right in the case of publishing. I had noticed that pnpm didn't support --json in the general case - that's what my comment was based on. However, I just had a quick look at the pnpm code and it does appear to forward anything in argv to npm when calling publish.

Glad to hear you are working on this - happy to help/test if needed.

I had the same issue earlier today and it happened for two reasons.

Reason One

There was a package that was set to "private": true in the package.json file. This was unintentional. I attempted to fix this mistake in a recent release by updating the package.json. The problem is that the package already had multiple CHANGELOG.md entries and as a result, changesets attempts to update the package with a PUT rather than publish with a POST request. Since the package doesn't actually exist on the registry this causes an error and npm doesn't respond with json.

Fix: Make sure there is only one entry in your changelog file before publishing a package for the first time.

Reason Two

A package I was trying to publish seemed available, but it actually causes an error response from npm when trying to publish. The naming is too similar to another package that already exists. When an error happens the npm publish command doesn't return json and the json cannot be parsed.

Fix: Rename the package and retry.

Suggestions

I think the main problem is that changesets assumes that npm publish will always return valid json which is not the case. As a result the actual error response is hidden and all we see is an invalid json message.

https://github.com/atlassian/changesets/blob/fe8db7500f81caea9064f8bec02bcb77e0fd8fce/packages/cli/src/commands/publish/npm-utils.ts#L180

My suggestion is that this jsonParse call should be wrapped in a try catch and when failing it should log the original reponse from npm. This way it will be much easier to identify what the problem is.

Updated: Check for stderr before stdout so errors can be diagnosed more easily.

https://github.com/atlassian/changesets/blob/fe8db7500f81caea9064f8bec02bcb77e0fd8fce/packages/cli/src/commands/publish/npm-utils.ts#L169-L175

@Andarist

@Andarist would it help to first check for stderr from the publish command?

https://github.com/atlassian/changesets/blob/fe8db7500f81caea9064f8bec02bcb77e0fd8fce/packages/cli/src/commands/publish/npm-utils.ts#L169-L175

Yep, already pushed out a change like this a couple of hours ago 😉 :

89a70f9

Need to self-review and retest this before publishing a PR.

Thank you also for other comments - i will take a deeper look into everything you have mentioned

I've created a pull request that attempts to fix the issues mentioned here. I will setup snapshot releases for this PR soon-ish but in the meantime I would love to ask if there are any people willing to test this stuff out in their projects.

Does anyone have solutions for this lol

I managed to get the github action to work using this setup.

name: Release

on:
  push:
    branches:
      - main

concurrency: ${{ github.workflow }}-${{ github.ref }}

env:
  GITHUB_TOKEN: ${{ secrets.GH_RELEASE_TOKEN }}
  NPM_TOKEN: ${{ secrets.GH_RELEASE_TOKEN }}
  NODE_AUTH_TOKEN: ${{ secrets.GH_RELEASE_TOKEN }}

jobs:
  release:
    name: Release
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repo
        uses: actions/checkout@v3

      - name: Setup Node.js 18.x
        uses: actions/setup-node@v3
        with:
          node-version: 18.x
          cache: 'npm'
          token: ${{ secrets.GH_RELEASE_TOKEN }}
          registry-url: 'https://npm.pkg.github.com'

      - name: Install Dependencies
        run: npm ci

      - name: Create Release Pull Request or Publish to npm
        id: changesets
        uses: changesets/action@v1
        with:
          publish: npm run release

Navigation Menu

Repository navigation

SyntaxError: Unexpected end of JSON input when trying to publish packages #397

Affected Packages

Problem

Helpful info

26 remaining items

Reason One

Reason Two

Suggestions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SyntaxError: Unexpected end of JSON input when trying to publish packages #397

Description

Affected Packages

Problem

Helpful info

Activity

ifiokjr commented on Jul 5, 2020

Andarist commented on Jul 5, 2020

ifiokjr commented on Jul 5, 2020

zkochan commented on Jul 5, 2020

ifiokjr commented on Jul 5, 2020

ifiokjr commented on Jul 6, 2020

Andarist commented on Jul 7, 2020

zkochan commented on Jul 7, 2020

Andarist commented on Jul 7, 2020

v1rtl commented on Jul 13, 2020

devin-fee-ah commented on Oct 21, 2020

devin-fee-ah commented on Oct 21, 2020

Andarist commented on Oct 21, 2020

26 remaining items

peschee commented on Aug 4, 2021

DavidWells commented on Aug 12, 2021

Decipher commented on Sep 13, 2021

rossng commented on Sep 25, 2021

abdonrd commented on Oct 27, 2021

Andarist commented on Nov 14, 2021

rossng commented on Nov 15, 2021

ifiokjr commented on Nov 15, 2021

Reason One

Reason Two

Suggestions

ifiokjr commented on Nov 15, 2021

Andarist commented on Nov 15, 2021

Andarist commented on Nov 20, 2021

kulasekara02 commented on Jan 19, 2023

stefan-karlsson commented on Mar 19, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions