Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PROTOCOL_ERROR on S3 #4673

Closed
armsby opened this issue Oct 12, 2020 · 5 comments · Fixed by #4674
Closed

PROTOCOL_ERROR on S3 #4673

armsby opened this issue Oct 12, 2020 · 5 comments · Fixed by #4674
Labels
bug Remote: S3
Milestone
v1.54

Comments

@armsby
Copy link

armsby commented Oct 12, 2020

What is the problem you are having with rclone?

Then using rclone mount I have been seeing errors like this: Dir.Stat error: SerializationError: failed to decode REST XML response
status code: 200, request id: 163C63E6DEB02A2C
caused by: stream error: stream ID 19137; PROTOCOL_ERROR

this goes away only with a restart of the rclone mount,
after a small talk with darthShadow, it was suspected that it was http2 that is causing the issue,
after running some test where http2 was disabled in rclone the issue has not appeared.

a flag to disable http2 for s3 would be nice

What is your rclone version (output from rclone version)

rclone v1.53.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

stress an S3 mount and you will see it

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
@darthShadow darthShadow mentioned this issue Oct 12, 2020
Merged
5 tasks
@ncw ncw added this to the v1.54 milestone Oct 12, 2020
@ncw
Copy link
Member

ncw commented Oct 12, 2020

Are you using this with AWS S3?

stress an S3 mount and you will see it

Do you have a recipe for that?

This is almost certainly a bug in one of

  • go http2 library
  • aws s3 sdk
  • s3 servers

It would be nice to get to the bottom of which...

Can you dump the response? You'd do that with -vv --dump responses. I relalise that may be tricky depending on how you are stressing the backend.

Which go version is your rclone compiled with? rclone version will tell you?

@darthShadow did you manage to reproduce this yourself?

@darthShadow
Copy link
Member

This is with minio proxied via Cloudflare. I have heard reports of it happening directly with minio too, however.

I am guessing this is another bug with the golang http/2 library since it looks very similar to the one we are facing with google drive.

I haven't gotten a chance to reproduce it myself. I will be doing some testing for the same the next month so I will try to debug it further then.

@armsby
Copy link
Author

armsby commented Oct 12, 2020

that patch that darthShadow provided worked, I will try to run a rclone without it, both over cloudflare and without, and I hope that I can catch it again

@ncw
Copy link
Member

ncw commented Oct 13, 2020

that patch that darthShadow provided worked

Great. We'll work on getting that merged.

I will try to run a rclone without it, both over cloudflare and without, and I hope that I can catch it again

Super

@ncw ncw closed this as completed in #4674 Oct 13, 2020
ncw pushed a commit that referenced this issue Oct 13, 2020
@darthShadow
s3: Added --s3-disable-http2 to disable http/2
fc5b14b 
Fixes #4673

Signed-off-by: Anagh Kumar Baranwal <[email protected]>
@ncw
Copy link
Member

ncw commented Oct 13, 2020

I've merged @darthShadow 's fix for this this to master now which means it will be in the latest beta in 15-30 mins and released in v1.54

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Remote: S3
Projects
None yet
Milestone
v1.54
Development

Successfully merging a pull request may close this issue.

s3: Added --s3-disable-http2 to disable http/2 darthShadow/rclone
3 participants
@ncw @armsby @darthShadow