Skip to content

S3 touch fails on files greater than 5GB in size #4885

@qodfathr

Description

@qodfathr
Error: buildx failed with: ERROR: failed to solve: failed to touch file: operation error S3: CopyObject,
https response error StatusCode: 400, RequestID: T5TBZT4GG16YVFWR, HostID: /E7Ab6HtmB4mirRGrbve8dmVvEZqnyRrO19i/eWacbsFguCAsdXwYTJ1Q0Hm7413DcoKGZE29X8=,
api error InvalidRequest: The specified copy source is larger than the maximum allowable size for a copy source: 5368709120

I suspect the problem lies here:

func (s3Client *s3Client) touch(ctx context.Context, key string) error {
copySource := fmt.Sprintf("%s/%s", s3Client.bucket, key)
cp := &s3.CopyObjectInput{
Bucket: &s3Client.bucket,
CopySource: &copySource,
Key: &key,
Metadata: map[string]string{"updated-at": time.Now().String()},
MetadataDirective: "REPLACE",
}
_, err := s3Client.CopyObject(ctx, cp)
return err
}

and can be corrected by making use of multi-part uploads, or, alternatively, it's my understanding that renaming a file in S3 to have the exact same name as it currently has will effectively 'touch' the file while avoiding the overhead of copy.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions