This looks very nice :-)

I'm just in the middle of making the 1.53 release - I will review in detail after that is done.

@ivandeex @ncw All inline comments have been addressed except for this one which I'd like to hear your feedback on. #4547 (comment)

Specific questions about the upload cache:

• Who decides to copy source to upload cache or use source directly, backend or generic code in operations.go?
• If a source is "stable", can we avoid copying to cache and keep only json?
• If a source is "dynamic", can we force copying to cache?

UPDATE 2021-03-17
https://forum.rclone.org/t/http-upload-contribution/13271/39

I think for the first version you want to cache the upload in the upload handler. However that doesn't require writing it to disk if all the writing is being done sequentially, it can be written straight to the backend. What will need to be cached for the upload handler is the handle used for writing data to the backend.

As to why I care about resumable accrued (rolling) checksums (if supported by a backend),
please consider this scenario:

• user uploads a large file
• network broken, upload canceled
• source file is changed or another attempt is changing the partial upload on target
• user asks to resume a file
• rclone resumes (here we could have checked validity of partial upload and rewind from start)
• after some hours rclone finds that fingerprint is wrong
• abort, retry

EDIT if backend does not support it, its interface will return a nil accrued hashsum.

@ncw
Can we have this merged to start beta testing and have more time before 1.58 to settle with Tus refactor and finish glitches?

@mcalman
Could you please resolve yet again the permanent merge conflict about global flags? :)
Feel free to drop my "local/resume" patch. If/when we have this on master branch, I will resubmit it in another PR...

My objections to this PR seem to have been collapsed under loads of commit messages.
@ncw please take them into consideration before merging
#4547 (comment)

This "loads of commits and messages" is called working on your PR.

@mcalman

This PR is waiting for @ncw to review

• feat: Add ResumableUpload interface #5299

which is a spin off from

• [WIP] Tus Uploader #3850

which is stuck forever.

Unfortunately #5299 is pretty far from production and altogether invasive on core parts of rclone.
Reviewing it by reading the code is not simple and not enough as it like this PR changes code in the rclone core.
The process will definitely include writing tests and ensuring that current functionality does not get broken.

As far as I can see @ncw actively discussed early design of that PR.
Currently waiting for @ncw to find time for review.
If it's not done by next January, I am going to untangle the knot myself.
Probably I will combine code from both proposals, add appropriate tests and submit as new patch.

In the meantime I have added the patch from

• chunker: implement Resumer interface #5185

on top of this one - at any rate I will consider all parts together.

I do not expect the solution to come before the 1.58 release so I expect the resume feature to appear in 1.59.

@innovate-invent This patch will not be merged on rclone master branch before that, which is at least 3 months from now.

However, the patch of this PR has already solved me hours of work time by resuming downloads ("uploads" onto local backend) failed due to network hiccups.
In the meantime I created a beta branch and a beta release to use this code in practice.
The beta releases will be available at https://beta.rclone.org/branch/resume/

@mcalman By the way, you can open pull requests against the beta branch https://github.com/rclone/rclone/commits/resume

cc @ncw

@mcalman I am sorry about the situation

The PR branch as well as https://beta.rclone.org/branch/resume/ is rebased on the current master.

If it's not done by next January, I am going to untangle the knot myself.

@ivandeex Was just wondering what your status is. Do you still think it is likely the resume feature will be included in 1.59?

I'm hit heavily by anti-Rus sanctions, plans broken. Most of the short-term time I'll be recovering whatever I can w.r.t. migrating work env, accounts and payments etc. Can't promise anything concrete a.t.m.

Hello,
Using resume and bandwidth limiting is not really working as expected (?).
The bandwith limit feature will limit the resume speed... So for now, I'm testing using remote controll to enable bandwith limit at the end of the resume.

The rclone beta that includes the resumer interface posted at https://beta.rclone.org/branch/resume/ is quite out of date now, only at 1.58 while stable is at 1.62.2 with 1.63 just over the horizon. Any chance of getting the resume branch back in line with the master branch?