use proper buffer size for content get#4664
Merged
mxpv merged 1 commit intocontainerd:masterfrom Oct 27, 2020
Merged
Conversation
Signed-off-by: Avi Deitcher <[email protected]>
|
Hi @deitch. Thanks for your PR. I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Build succeeded.
|
mxpv
approved these changes
Oct 27, 2020
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The command-line
ctr content getusesio.Copy(), with the default buffer size of 32KB. This creates significant slow-downs on large files, as the grpc overhead must be incurred with each 32KB chunk.This PR changes it to 1MB (1<<20), which is what we use for ingesting content, using
io.CopyBuffer()instead.I ran 10 tests on a 1.0GB file,
ctr content geteach with and without the change (yeah, I probably should do a proper go benchmark with hundreds of iterations; if people insist, I can, but it seems unnecessary for such a small change). In all cases I sent the results to/dev/null, to avoid any potential issues with disk writes.I was rather blown away by the results.
Obviously, little difference on blobs not much bigger than 32KB, no difference at all on blobs 32KB or smaller.
Discussed with @dmcgowan and @estesp on Slack.