Set additional headers using --header-upload and --header-download #59

pquerna · 2015-05-10T17:55:50Z

ncw · 2015-05-10T19:18:16Z

I see. Wouldn't be too hard! I'd probably make an --add-header flag which all the remote storage systems could use not just s3.

Thanks for the suggestion.

schickling · 2015-05-29T15:13:19Z

+1

ncw · 2015-09-08T07:52:33Z

NB this additional header only wants to be set on the operation that actually uploaded the file - not on all the other operations (eg listing directories etc)

nodughere · 2015-09-08T14:34:24Z

Agreed.

The headers should be a dictionary.

Example rclone with Swift
rclone --add-header={'x-meta-object-my-favorite-car':'ford mustang', 'X-Delete-After': 3600} copy /home/doug/data/ SwiftCluster:container

This command would add metadata and allow objects to auto delete in 1 hour as an example

Example rclone with S3
rclone --add-header={'x-amz-meta-my-favorite-car':'ford mustang'} copy /home/doug/data/ S3Cluster:bucket

wouterv · 2016-02-14T19:45:31Z

I have a suggestion for local file system copy actions:

What whould help with backups is to have an additional header file next to the actual file with key=value file, or a json file containing the meta data.

And with that to use those files when uploading to a cloud provider.

Perhaps with an extra option --copy-headers=true --head-file-extension=.header or something for both copying from/to a local file system.

treyd · 2016-06-06T19:35:17Z

+1 for this request, although I would like the additional header to also be set on all requests. The reason behind this is there is some middleware in Swift that uses headers to control functionality, and it would be great if we can have rclone do this as well

ncw · 2016-06-06T20:08:26Z

I'm all in favour of a general purpose option to add headers... Anyone fancy sending in a PR?

fortunto2 · 2017-01-08T00:13:24Z

I'm sorry, but when the scheduled implementation?

ncw · 2017-06-20T10:21:00Z

rclone now has all the machinery to make this relatively straight forward.

I would implement two flags which could be repeated

--upload-header "Header: String"
--download-header "Header: String"

Which would be applied to Uploads (specifically Object.Update or Fs.Put) or Downloads (specifically Object.Open).

These would need to be applied as options (there is a general purpose HTTPOption already). A bit of work would need to be done in each remote to make this work. The options would be applied in the Copy primitive (and possibly elsewhere - this might need factoring).

peixotorms · 2017-09-06T20:13:02Z

+1

browny · 2017-11-16T09:44:13Z

+1

ducktype · 2018-03-13T16:52:49Z

+1

ncw · 2020-04-23T15:08:46Z

I have also implemented the --header flag which affects all transactions on all backends.

bjg2 · 2020-04-30T17:43:57Z

Hey guys, just got to the point to test / use this feature, and I'm having issues.

I'm running version: rclone-v1.51.0-254-g74d9dabd-beta-windows-amd64, and I use the following command:

copyto SOURCE DEST --header-upload "Content-Disposition: attachment; filename='randomstvar.txt'"

On AWS S3 headers work as expected:

On Google Cloud Storage, not as much:

Is there some bug with upload headers with GCS, or am I doing something wrong? This is very important for me at the moment, I should upload a bunch of data to the GCS in a few days... :)

ncw · 2020-05-01T18:15:26Z

s3 - good!

gcs.,, The docs are here: https://cloud.google.com/storage/docs/gsutil/addlhelp/WorkingWithObjectMetadata

I can see the PUT requests are OK with -vv --dump bodies but the headers don't get set...

Just tried a test with gsutil and it puts the headers inline in the object info :-( I'll have a go at fixing that in a bit.

bjg2 · 2020-05-02T08:01:00Z

Thanks for looking into this!

Not to push you, but is it possible to fix this issue until Monday? In Monday I have to upload like a few milion files to GCS, and it would be great if I wouldn't have to track them down and change their content disposition as needed...

Before this code we were settig the headers on the PUT request. However this isn't where GCS needs them. After this fix we set the headers in the object upload request itself. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type It would be possible to support adding metadata also if there is demand.

ncw · 2020-05-02T08:23:22Z

@bjg2 I've tracked down the problem here. Rclone needs to set the headers on the object data in the request not the actual request.

I've had a go at fixing this here - can you have a go? It works in my tests so if it works for you it should be good for your upload!

https://beta.rclone.org/branch/v1.51.0-265-gb40e997f-fix-59-gcs-headers-beta/ (uploaded in 15-30 mins)

bjg2 · 2020-05-02T09:13:57Z

Both S3 and GCS work now! Thanks!

Shareed2k · 2020-05-06T07:22:11Z

what about custom headers, in gcs ?

Before this code we were settig the headers on the PUT request. However this isn't where GCS needs them. After this fix we set the headers in the object upload request itself. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Goog-Meta- Note for the last of those are for setting custom metadata in the form "X-Goog-Meta-Key: value".

ncw · 2020-05-06T16:35:40Z

what about custom headers, in gcs ?

I've added support for custom metadata? Using

x-goog-meta-key: value

Is that what you mean?

I don't think it is possible to set any of the other headers

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52

Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also #59

Before this change we were setting the headers on the PUT request for normal and multipart uploads. For normal uploads this caused the error 403 Forbidden: There were headers present in the request which were not signed After this fix we set the headers in the object upload request itself as the s3 SDK expects. This means that we only support a limited range of headers - Cache-Control - Content-Disposition - Content-Encoding - Content-Language - Content-Type - X-Amz-Tagging - X-Amz-Meta- Note for the last of those are for setting custom metadata in the form "X-Amz-Meta-Key: value". This now works for multipart uploads and single part uploads See also rclone#59

Cinerar · 2020-07-28T19:13:56Z

Am i right that this doesn't implemented for operations.CopyFile but only for operations.Copy and operations.Rcat?

ncw · 2020-07-29T20:11:58Z

Am i right that this doesn't implemented for operations.CopyFile but only for operations.Copy and operations.Rcat?

@Cinerar --header-upload and --header-download should work for copy, copyfile, cat and rcat

Cinerar · 2020-07-29T20:24:54Z

In my experiments i found that --header-upload are not set from Env Variable RCLONE_HEADER_UPLOAD (maybe i used wrong one?), so i was forced to use
fs.Config.UploadHeaders = []*fs.HTTPOption{ &fs.HTTPOption{ Key: "Content-Disposition", Value: "attachement", }}

and i confirm that it works for copyfile too.

ncw · 2020-09-01T17:29:07Z

This is all finished and merged now

  --header stringArray            Set HTTP header for all transactions
  --header-download stringArray   Set HTTP header for download transactions
  --header-upload stringArray     Set HTTP header for upload transactions

Please open a new issue if there is problem with it!

Manojnaik1 · 2023-01-03T05:49:32Z

Hi, can anyone please guide me on how to upload multiple metadata

ncw · 2023-01-03T11:28:10Z

@Manojnaik1 repeat the flag as many times as necessary.

Manojnaik1 · 2023-01-04T11:54:48Z

@ncw thanks for your reply. it's working.

ncw added the enhancement label May 10, 2015

ncw changed the title ~~s3: Set additional headers~~ Set additional headers Sep 8, 2015

ncw mentioned this issue Sep 8, 2015

Add Custom MetaData during a transfer #126

Closed

ncw added this to the Soon milestone Feb 10, 2016

geoffraeder mentioned this issue May 19, 2016

S3 server-side encryption #491

Closed

ncw mentioned this issue Nov 30, 2016

[feature_request] Expiring Object Support #918

Closed

ncw modified the milestones: v1.37, Soon Feb 21, 2017

ncw modified the milestones: v1.37, v1.38 Jul 19, 2017

ncw mentioned this issue Jul 21, 2017

Swift copy : add specific headers ? #1544

Closed

ncw modified the milestones: v1.38, v1.39 Sep 30, 2017

ncw modified the milestones: v1.39, v1.40 Jan 11, 2018

ncw removed this from the v1.40 milestone Mar 19, 2018

ncw added a commit that referenced this issue Apr 23, 2020

putio: add support for --header-upload and --header-download #59

ab6ed25

calebcase mentioned this issue May 8, 2020

Storj Backend #3699

Merged

5 tasks

ncw modified the milestones: v1.52, v1.53 May 29, 2020

ncw closed this as completed Sep 1, 2020

andrewpatto mentioned this issue Oct 24, 2023

Track Egress from Copy Out elsa-data/elsa-data#503

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set additional headers using --header-upload and --header-download #59

Set additional headers using --header-upload and --header-download #59

pquerna commented May 10, 2015 •

edited by ncw

Loading

ncw commented May 10, 2015

schickling commented May 29, 2015

ncw commented Sep 8, 2015

nodughere commented Sep 8, 2015

wouterv commented Feb 14, 2016

treyd commented Jun 6, 2016 •

edited

Loading

ncw commented Jun 6, 2016

fortunto2 commented Jan 8, 2017

ncw commented Jun 20, 2017

peixotorms commented Sep 6, 2017

browny commented Nov 16, 2017

ducktype commented Mar 13, 2018

ncw commented Apr 23, 2020

bjg2 commented Apr 30, 2020 •

edited

Loading

ncw commented May 1, 2020

bjg2 commented May 2, 2020

ncw commented May 2, 2020

bjg2 commented May 2, 2020

Shareed2k commented May 6, 2020

ncw commented May 6, 2020

Cinerar commented Jul 28, 2020

ncw commented Jul 29, 2020

Cinerar commented Jul 29, 2020

ncw commented Sep 1, 2020

Manojnaik1 commented Jan 3, 2023

ncw commented Jan 3, 2023

Manojnaik1 commented Jan 4, 2023

Set additional headers using --header-upload and --header-download #59

Set additional headers using --header-upload and --header-download #59

Comments

pquerna commented May 10, 2015 • edited by ncw Loading

Backends which have and haven't got support for --header-upload and --header-download

ncw commented May 10, 2015

schickling commented May 29, 2015

ncw commented Sep 8, 2015

nodughere commented Sep 8, 2015

wouterv commented Feb 14, 2016

treyd commented Jun 6, 2016 • edited Loading

ncw commented Jun 6, 2016

fortunto2 commented Jan 8, 2017

ncw commented Jun 20, 2017

peixotorms commented Sep 6, 2017

browny commented Nov 16, 2017

ducktype commented Mar 13, 2018

ncw commented Apr 23, 2020

bjg2 commented Apr 30, 2020 • edited Loading

ncw commented May 1, 2020

bjg2 commented May 2, 2020

ncw commented May 2, 2020

bjg2 commented May 2, 2020

Shareed2k commented May 6, 2020

ncw commented May 6, 2020

Cinerar commented Jul 28, 2020

ncw commented Jul 29, 2020

Cinerar commented Jul 29, 2020

ncw commented Sep 1, 2020

Manojnaik1 commented Jan 3, 2023

ncw commented Jan 3, 2023

Manojnaik1 commented Jan 4, 2023

pquerna commented May 10, 2015 •

edited by ncw

Loading

Backends which have and haven't got support for `--header-upload` and `--header-download`

treyd commented Jun 6, 2016 •

edited

Loading

bjg2 commented Apr 30, 2020 •

edited

Loading