I was trying to recursively upload large volume of data to my repo (s3 blockstore) using lakectl.
I noticed that if files in the directory are lager than 5 GB, then the following command gives an error. It start the uploads but after a couple of seconds it fails. The files are around 15 GB each. This issue is happening everytime when the individual files are larger than 5 GB.
Command:
lakectl fs upload --source . lakefs://repo/branch/directory --recursive
Output:
upload file_1.txt ... fail! [123.37MB in 1.068s]
upload file_3.txt ... fail! [154.34MB in 1.068s]
upload file_2.txt ... fail! [173.11MB in 1.069s]
It also returns an URL with Error 400 Bad Request which is something like this
upload file_3.txt failed: upload request failed
https://<bucket>.s3.us-east-1.amazonaws.com/repo/data/<some token>/
<some token>
?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential=<xxxx>
&X-Amz-Date=20250505T083730Z
&X-Amz-Expires=900
&X-Amz-Security-Token=<xxxxx>
&X-Amz-SignedHeaders=host
&x-id=PutObject
&X-Amz-Signature=<xxxxxx>: 400 Bad Request
But, when I upload files individually using the following command, the upload suceeds.
for i in 1 2 3; do
lakectl fs upload --source "file_${i}.txt" lakefs://repo/branch/directory
doneIs there anything I am doing wrong or is this is an issue with lakectl?
I was trying to recursively upload large volume of data to my repo (s3 blockstore) using lakectl.
I noticed that if files in the directory are lager than 5 GB, then the following command gives an error. It start the uploads but after a couple of seconds it fails. The files are around 15 GB each. This issue is happening everytime when the individual files are larger than 5 GB.
Command:
lakectl fs upload --source . lakefs://repo/branch/directory --recursiveOutput:
It also returns an URL with Error 400 Bad Request which is something like this
But, when I upload files individually using the following command, the upload suceeds.
Is there anything I am doing wrong or is this is an issue with lakectl?