Added parallel gzip.#34788
Conversation
See moby#34787 Parallel gzip via pigz external binary greatly accelerates compression/decompression. Signed-off-by: John Boero <[email protected]>
|
Thank you @jboero for the PR and for the feature request! I am assuming that this PR is still in progress as we haven't yet made the Another suggestion I have is perhaps for the title and future easy auditing and inspection of commits:
pkg/archive: add pgzip exec'd as an option for image decompression
Fixes #34787
Parallel gzip via pigz external binary invoked by os/exec
greatly accelerates compression/decompression.
Add this option as toggleable.Thank you. |
|
|
||
| func pigzDecompress(archive io.Reader) (io.ReadCloser, <-chan struct{}, error) { | ||
| args := []string{"unpigz", "-c"} | ||
| return cmdStream(exec.Command(args[0], args[1:]...), archive) |
There was a problem hiding this comment.
I am not thrilled with the idea that the user needs to install additional packages (in this case only unpigz) to get this feature. Maybe https://github.com/klauspost/pgzip is sufficient enough.
There was a problem hiding this comment.
Same here... I even didn't know that unpigz was existing!
There was a problem hiding this comment.
Sorry for the delay - didn't notice this. I think it's nicer to rely on the already hardened and maintained pigz (which uses standard glibz) rather than another piece of Go code that reinvents the wheel. The default behavior for an environment missing pigz/unpigz is just to fallback on the standard zip anyway. I think it's more odd to make users download another implementation of gzip when they likely already have it. Pigz/unpigz is a tiny 114K app that just threads out libz.
Anyway I just tried it out as a PoC so if someone wants to implement it in a supported method with golang, it would make good sense to add both parallel zip and unzip together as was already requested.
There was a problem hiding this comment.
In the Fedora packaging it's as easy as including a dependency package "pigz" to the RPM or better yet just leaving it out as optional and documenting how to enable it.
|
Discussing in the maintainers meeting; we're ok with using an external binary, and use w.r.t. the environment variable; we think we should use this by default (ie., use @jboero can you update the PR with those changes? |
|
ref: similar PR: #35697 |
|
For decompression, this has now been implemented through #35697, so closing this one. Thanks! |
9 participants
See #34787
Parallel gzip via pigz external binary greatly accelerates compression/decompression.
Signed-off-by: John Boero [email protected]
- What I did
- How I did it
- How to verify it
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)