avoid repeated decompression and further utilize the --GH option to improve conversion speed#2145
Merged
ktock merged 1 commit intocontainerd:mainfrom Oct 18, 2025
Merged
Conversation
escapefreeg
force-pushed
the
main
branch
3 times, most recently
from
210513c to
c8413f3
Compare
ktock
reviewed
Oct 16, 2025
estargz/build.go
Outdated
Comment on lines
+295
to
+296
| uncompressedSizeChan <- uncompressedSize | ||
| close(uncompressedSizeChan) |
Member
There was a problem hiding this comment.
Does this make UncompressedSize return a zero value when it called more than once? And I guess this coroutine leaks with blocking at L295 if UncompressedSize isn't called.
Instead of using channel, I think it can just use an int64 pointer + atomic. UncompressedSize can have a comment that the value is valid only after the full read.
Author
There was a problem hiding this comment.
Thanks for the suggestion. I've updated it based on your advice.
…peed up conversion Signed-off-by: clarehkli <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
During the current image conversion process, the converted blob is decompressed in
estargz/build.goto calculate its hash value. Additionally, the converted blob is decompressed again innativeconverter/estargz/estargz.go(estargz format) andnativeconverter/zstdchunked/zstdchunked.go(zstd format) to calculate its size. However, these two decompression operations are not necessary, and decompression is a time-consuming task. Therefore, the two decompressions can be merged into a single operation to improve image conversion speed.Furthermore, according to #2117 , compared with Go’s built-in gzip decompression library, command-line tools can achieve faster decompression of gzip archives. When the
--estargz-gzip-helper(--GH) option is specified, the gzip helper can be further utilized to speed up decompression of the converted image.I' conducted tests on the related changes. Results from 10 images show that the improvement reduces conversion time by an average of 26.7%. For more details, see:
The calculation method for
Time improvement from applying current optimization on pigz GHis (With pigz GH-With pigz GH and current optimization) /With pigz GHand the calculation method forTime improvement from combining pigz GH and current optimizationis (Without GH-With pigz GH and current optimization) /Without GHNote: Although
ioutils.CountWriterensures correct byte counting in concurrent environments, its usage innativeconverter/estargz/estargz.goandnativeconverter/zstdchunked/zstdchunked.godoes not involve concurrency. Therefore, the size of the converted blob can be directly obtained from the return value ofio.Copy.