File transfer order pattern

What is your current rclone version (output from `rclone version`)?

rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7

What problem are you are trying to solve?

Transfer files before their checksum files. Ex files:

dir1/file.tar.gz
dir1/file.tar.gz.md5
dir2/file.tar.gz
dir2/file.tar.gz.md5
dir3/file.tar.gz
dir3/file.tar.gz.md5

I would like the .tar.gz files in each direrctory to finish uploading before transferring the .md5 files, even when using --transfers=n. The reason for this is an automated process on the destination that considers a directory and its files "complete" when the .md5 files appear. For example, the process considers dir1 complete when dir1/file.tar.gz.md5 has been uploaded.

I'm currently using lftp to solve this.
See the man page for lftp: https://lftp.yar.ru/lftp-man.html
Rclone's --order-by feature is similar to lftp's mirror:sort-by feature, but lftp has an additional feature for file transfer order by pattern: mirror:order.

Example command line using lftp:

/bin/lftp user:pass@ftpserver -e "set mirror:order '*.tar.gz *.md5'; mirror -R --parallel=30 --verbose /path/to/source/ /; exit"

My current ideas of solving this with Rclone:

/bin/rclone --transfers=30 --exclude='*.md5' copy /path/to/source/ ftpserver:
/bin/rclone --transfers=30 --include='*.md5' copy /path/to/source/ ftpserver:

But this will not solve the problem for me in a reliable manner, since the source directory might have new files between the rclone runs, thus causing .md5 files to be uploaded for .tar.gz files that haven't been uploaded.

Or I could try using --order-by='size,desc' since the .md5 files will in my case always be smaller than the .tar.gz files.

How do you think rclone should be changed to solve that?

Implement a similar feature as lftp's mirror:order, allowing the user to define a list of file processing order patterns.

Example to transfer all .tar.gz files before transferring .md5 files:

/bin/rclone --transfers=30 --order='*.tar.gz *.md5' copy /path/to/source/ ftpserver:

I could also try using --order-by='size,desc' since the .md5 files will in my case always be smaller than the .tar.gz files.

That was my immediate first thought.

How important is it that the md5 comes last? Does it matter if it is occasionally the other way round?

To implement an order scheme I'd probably want to to do something like --order-by pattern,.tar.gz,.md5

In my case they need to come last in each subdirectory, not necessarily the entire transfer (can be thousands of files and many TB).

@ncw

Hi, I am willing to take a stab at this as it is labeled as good first issue. Can I give it a go? As @ncw pointed out, the --order-by flag functionality should be enhanced. One should be able to run:

rclone copy --order-by pattern,.tar.gz,.md5,.txt /local/path remote:path .

Before jumping into coding, I would like to ask something. Do we want to make pattern matching only based on file suffixes or do we want to support more pattern matching rules like the filtering flags?

I suspect that in order to implement this, we should put an extra case "pattern": in the following switch statement

rclone/fs/sync/pipe.go

Lines 189 to 205 in 04a8859

    
           switch parts[0] { 
        
           case "name": 
        
           	less = func(a, b fs.ObjectPair) bool { 
        
           		return a.Src.Remote() < b.Src.Remote() 
        
           	} 
        
           case "size": 
        
           	less = func(a, b fs.ObjectPair) bool { 
        
           		return a.Src.Size() < b.Src.Size() 
        
           	} 
        
           case "modtime": 
        
           	less = func(a, b fs.ObjectPair) bool { 
        
           		ctx := context.Background() 
        
           		return a.Src.ModTime(ctx).Before(b.Src.ModTime(ctx)) 
        
           	} 
        
           default: 
        
           	return nil, fraction, errors.Errorf("unknown --order-by comparison %q", parts[0]) 
        
           }

and define our new less function. Of course, some docs should be updated too.

sync: add pattern-based transfer ordering

Navigation Menu

Repository navigation

File transfer order pattern #3975

What is your current rclone version (output from `rclone version`)?

What problem are you are trying to solve?

How do you think rclone should be changed to solve that?

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	switch parts[0] {
	case "name":
	less = func(a, b fs.ObjectPair) bool {
	return a.Src.Remote() < b.Src.Remote()
	}
	case "size":
	less = func(a, b fs.ObjectPair) bool {
	return a.Src.Size() < b.Src.Size()
	}
	case "modtime":
	less = func(a, b fs.ObjectPair) bool {
	ctx := context.Background()
	return a.Src.ModTime(ctx).Before(b.Src.ModTime(ctx))
	}
	default:
	return nil, fraction, errors.Errorf("unknown --order-by comparison %q", parts[0])
	}

File transfer order pattern #3975

Description

What is your current rclone version (output from rclone version)?

What problem are you are trying to solve?

How do you think rclone should be changed to solve that?

Activity

ncw commented on Feb 17, 2020

hakong commented on Feb 17, 2020

lkslts64 commented on Feb 22, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

What is your current rclone version (output from `rclone version`)?