esti: test lakectl fs upload of files over 5GiB#10478
Open
highlyavailable wants to merge 1 commit into
Open
Conversation
Covers the multipart path fixed in treeverse#9283 for both pre-signed and regular uploads. Extends the output sanitizers to handle multipart etags (md5-N) and relative physical addresses with part suffixes. Closes treeverse#9284
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9284
Change Description
Background
this adds the Esti coverage requested in #9284 for the scenario fixed by #9283:
lakectl fs uploadof a file over 5GiB, with and without--pre-sign(the test uses a sparse 5GiB+1 temp file and runs on the s3 blockstore only, where the 5GiB single-part limit applies)i think this is also the first golden-file test to hit a multipart upload, so the esti harness needed fixes. Multipart etags look like -, so the checksum matcher now accepts that suffix... multipart physical addresses print relative to the storage namespace, so those now normalize to <OBJECT_KEY> too
Testing
ran locally against lakeFS + MinIO (
blockstore.type: s3). Both subtests pass +TestLakectlPreSignUploadstill passes with the sanitizer changes