Skip to content

[Bug]: error uploading artifacts to github releases #2746

Closed

Description

@chetan

What happened?

Artifacts sometimes fail to upload, due to github API failures.

This is similar to the issue reported in #2506. AFAICT, it's an issue on github's end. Here's a larger log of output from my build running in a github action:


• uploading to release      file=dist/checksums.txt name=checksums.txt
• failed to upload artifact, will retry artifact=checksums.txt error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=checksums.txt: 502 Error uploading to https://objects-origin.githubusercontent.com/github-production-release-asset-2e65be: 500 [] try=1
• uploading to release      file=dist/checksums.txt name=checksums.txt
• failed to upload artifact, will retry artifact=checksums.txt error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=checksums.txt: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}] try=2

• uploading to release      file=dist/traefik-kop_0.8.1_Linux_arm64.tar.gz name=traefik-kop_0.8.1_Linux_arm64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_arm64.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_arm64.tar.gz: 502 Error uploading to https://objects-origin.githubusercontent.com/github-production-release-asset-2e65be: 500 [] try=1
• uploading to release      file=dist/traefik-kop_0.8.1_Linux_arm64.tar.gz name=traefik-kop_0.8.1_Linux_arm64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_arm64.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_arm64.tar.gz: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}] try=2

• uploading to release      file=dist/traefik-kop_0.8.1_Linux_x86_64.tar.gz name=traefik-kop_0.8.1_Linux_x86_64.tar.gz

• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv6.tar.gz name=traefik-kop_0.8.1_Linux_armv6.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv6.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv6.tar.gz: 502 Error uploading to https://objects-origin.githubusercontent.com/github-production-release-asset-2e65be: 500 [] try=1
• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv6.tar.gz name=traefik-kop_0.8.1_Linux_armv6.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv6.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv6.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 11; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=2
• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv6.tar.gz name=traefik-kop_0.8.1_Linux_armv6.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv6.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv6.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 13; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=3
• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv6.tar.gz name=traefik-kop_0.8.1_Linux_armv6.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv6.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv6.tar.gz: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}] try=4

• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_arm64.tar.gz name=traefik-kop_0.8.1_Darwin_arm64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_arm64.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_arm64.tar.gz: 502 Error uploading to https://objects-origin.githubusercontent.com/github-production-release-asset-2e65be: 500 [] try=1
• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_arm64.tar.gz name=traefik-kop_0.8.1_Darwin_arm64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_arm64.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_arm64.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 25; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=2
• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_arm64.tar.gz name=traefik-kop_0.8.1_Darwin_arm64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_arm64.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_arm64.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 29; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=3
• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_arm64.tar.gz name=traefik-kop_0.8.1_Darwin_arm64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_arm64.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_arm64.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 33; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=4
• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_arm64.tar.gz name=traefik-kop_0.8.1_Darwin_arm64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_arm64.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_arm64.tar.gz: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}] try=5

• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv7.tar.gz name=traefik-kop_0.8.1_Linux_armv7.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv7.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv7.tar.gz: 502 Error uploading to https://objects-origin.githubusercontent.com/github-production-release-asset-2e65be: 500 [] try=1
• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv7.tar.gz name=traefik-kop_0.8.1_Linux_armv7.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv7.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv7.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 23; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=2
• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv7.tar.gz name=traefik-kop_0.8.1_Linux_armv7.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv7.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv7.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 27; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=3
• uploading to release      file=dist/traefik-kop_0.8.1_Linux_armv7.tar.gz name=traefik-kop_0.8.1_Linux_armv7.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Linux_armv7.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Linux_armv7.tar.gz: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}] try=4

• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_x86_64.tar.gz name=traefik-kop_0.8.1_Darwin_x86_64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_x86_64.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_x86_64.tar.gz: 502 Error uploading to https://objects-origin.githubusercontent.com/github-production-release-asset-2e65be: 500 [] try=1
• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_x86_64.tar.gz name=traefik-kop_0.8.1_Darwin_x86_64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_x86_64.tar.gz error=Post "https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_x86_64.tar.gz": http2: Transport: cannot retry err [stream error: stream ID 39; REFUSED_STREAM; received from peer] after Request.Body was written; define Request.GetBody to avoid this error try=2
• uploading to release      file=dist/traefik-kop_0.8.1_Darwin_x86_64.tar.gz name=traefik-kop_0.8.1_Darwin_x86_64.tar.gz
• failed to upload artifact, will retry artifact=traefik-kop_0.8.1_Darwin_x86_64.tar.gz error=POST https://uploads.github.com/repos/jittering/traefik-kop/releases/54810593/assets?name=traefik-kop_0.8.1_Darwin_x86_64.tar.gz: 422 Validation Failed [{Resource:ReleaseAsset Field:name Code:already_exists Message:}] try=3

I cleaned up and sorted the output so its easier to read. It seems that in each case, a 5xx error is followed by 422 on each successive retry. In this scenario, it might be necessary to check the list of release assets and/or delete it before retrying.

In my case, it took my several rebuilds (5 tries) with manually deleting the release in between until I was able to get a clean build where every artifact uploaded properly.

How can we reproduce this?

Seems to be a github issue, not sure there will be an easy way to reproduce

goreleaser version

goreleaser version 1.1.0
commit: af4a8642bddb171914ee932bfb5768f102c3516f

GoReleaser Check

  • goreleaser check shows no errors

Search

  • I did search for other open and closed issues before opening this.

Code of Conduct

  • I agree to follow this project's Code of Conduct

Additional context

No response

Activity

  1. changed the title [-][Bug]: [/-] [+][Bug]: error uploading artifacts during release[/+] on Dec 7, 2021
  2. changed the title [-][Bug]: error uploading artifacts during release[/-] [+][Bug]: error uploading artifacts to github releases[/+] on Dec 7, 2021
  3. caarlos0 commented on Dec 28, 2021

    @caarlos0
    Member

    [{Resource:ReleaseAsset Field:name Code:already_exists Message:}]

    when this happens, it usually means that goreleaser ran multiple times against the same tag, as the files are already there...

    we should probably drop the retries in this case though

  4. caarlos0 commented on Dec 28, 2021

    @caarlos0
    Member

    hmm, seems to be this google/go-github#2113

  5. avelino commented on Dec 28, 2021

    @avelino
    Contributor

    when this happens, it usually means that goreleaser ran multiple times against the same tag, as the files are already there...

    If the file already exists and you are generating a "new" release, shouldn't you overwrite it?

    Thinking that you are consuming an upload api you should delete the existing file and upload it again

  6. caarlos0 commented on Dec 28, 2021

    @caarlos0
    Member

    If the file already exists and you are generating a "new" release, shouldn't you overwrite it?

    too destructive, rather not... who knows why the file already exists... it may not even have been created by goreleaser

  7. avelino commented on Dec 29, 2021

    @avelino
    Contributor

    @caarlos0 if we put a parameter to force (aka --force) the replacement of the file?

    in my case that solves it, all builds are (and will be) generated "automatically" via pipeline (github actions), if there is one, it was because of some error in the middle of the process, it's ok to overwrite.

  8. caarlos0 commented on Jan 7, 2022

    @caarlos0
    Member

    @caarlos0 if we put a parameter to force (aka --force) the replacement of the file?

    I don't like the idea...

    • if its a flag, it would require the user to either change their CI config and retag (which is not good) or run from their machines instead of CI (also not that good)
    • if its a config, to be safe, they would have to always have that set there... which would still lead to retag if its not there beforehand, or to accidental destruction actions if it is always there...

    to me it seems the right thing is to fix this on go-github

  9. chetan commented on Jan 9, 2022

    @chetan
    Author

    @caarlos0 Is there a way to re-release a specific tag? In my case, when I ran into this issue, I had to manually delete the release from github and then run goreleaser again.

    [{Resource:ReleaseAsset Field:name Code:already_exists Message:}]

    when this happens, it usually means that goreleaser ran multiple times against the same tag, as the files are already there...

    goreleaser already fails-fast when a release already exists, so if it's gotten to the point of uploading artifacts, it means that the release was created during the current run. If that's the case, I don't see any reason why the file can't simply be deleted and retried automatically.

    As you can see from my original problem report, the already_exists error only returns after the initial 522 error. My guess is that despite github throwing 522 on the initial upload, the asset object is actually getting created and stored, but an error was thrown at some other point in the process. Thus, any subsequent retries hit the already_exists error. Obviously the core issue is at github's end, but it can be worked around by simply cleaning up and retrying, possibly with some appropriate exponential backoff between each try.

  10. caarlos0 commented on Jan 10, 2022

    @caarlos0
    Member

    goreleaser already fails-fast when a release already exists

    it does not...

  11. chetan commented on Jan 10, 2022

    @chetan
    Author

    Sorry, you're right, I'm misremembering the sequence from when I was troubleshooting and don't have those logs any more. I just re-ran a release for an existing tag manually and I see this message:

          • scm releases
             • creating or updating release repo=jittering/traefik-kop tag=v0.9.1
    

    All of the artifact uploads indeed fail immediately (only 422 errors, no 5xx) as they already exist. I think in this case, I agree with what @avelino said above -- if it's a new release, overwrite the file; if updating, do not retry a 422 already_exists, just skip or fail the process.

    Unless there is some race condition with multiple releases happening simultaneously, I don't see why this wouldn't be a safe option. For extra caution, a flag which users can decide to always include in their build should be a good compromise.

    I don't think the linked go-github error linked above is relevant here as that appears to be some internal problem while the 502 here is come backing from github's API. This google/go-github#304 issue does mention 502s on upload and the response was (rightly) that it should be handled by the client.

  12. caarlos0 commented on Jan 10, 2022

    @caarlos0
    Member

    Unless there is some race condition with multiple releases happening simultaneously, I don't see why this wouldn't be a safe option. For extra caution, a flag which users can decide to always include in their build should be a good compromise.

    this happens more often than not, people setup goreleaser on commit to main AND tag, and get concurrent releases... if we allow to override, who knows what the end release will look like

    the issue seems to be within go-github google/go-github#2113

  13. caarlos0 commented on Feb 5, 2022

    @caarlos0
    Member

    is this still happening?

  14. avelino commented on Feb 5, 2022

    @avelino
    Contributor

    No reply, I had no more problem (after I reported, ANY more releases were generated using goreleaser)

  15. caarlos0 commented on Feb 5, 2022

    @caarlos0
    Member

    ok, will close this then

    thanks all!

  16. vandot commented on Apr 19, 2022

    @vandot
    Contributor

    @caarlos0 it started happening to me from yesterday here is this fixed in some newer versions of goreleaser?
    I am using troian/golang-cross didn't had time to migrate it to new setup.

    Edit: switching to latest version of goreleaser/goreleaser-cross fixed it, false alarm

  17. caarlos0 commented on Apr 19, 2022

    @caarlos0
    Member

    This is an issue on github api, happens from time time to time to some people in some places in some repos, nothing I can do about it.

    Best thing is open a support ticket for them to make some noise.

    I also opened this discussion a while back: community/community#14341

  18. rfay commented on Jun 11, 2022

    @rfay
    Contributor

    Shoot, I was hoping that goreleaser would have a workaround for this problem at github. I've seen it so very many times- * https://github.com/skx/github-action-publish-binaries/issues/33

    The problem is that when you're doing a release and it fails the recovery is tremendously awkward, can result in deleting the release or at least the stuff already uploaded to the release so you can start over again. But from what I hear there's no workaround in goreleaser, right?

  19. caarlos0 commented on Jun 12, 2022

    @caarlos0
    Member

    But from what I hear there's no workaround in goreleaser, right?

    correct - see #2746 (comment)

  20. github-actions commented on Dec 5, 2022

    @github-actions
    Contributor

    This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

  21. locked as resolved and limited conversation to collaborators on Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions