@@ -75,14 +75,47 @@ The above is fine for big encodes. However, whenever possible try to *reuse* the
7575To reuse the encoder, you can use the ` Reset(io.Writer) ` function to change to another output.
7676This will allow the encoder to reuse all resources and avoid wasteful allocations.
7777
78- Currently stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part
79- of a stream. This is independent of the ` WithEncoderConcurrency(n) ` , but that is likely to change
78+ By default, stream encoding has 'light' concurrency, meaning up to 2 goroutines can be working on part
79+ of a stream. This is independent of the ` WithEncoderConcurrency(n) ` , but that is likely to change
8080in the future. So if you want to limit concurrency for future updates, specify the concurrency
8181you would like.
8282
8383If you would like stream encoding to be done without spawning async goroutines, use ` WithEncoderConcurrency(1) `
8484which will compress input as each block is completed, blocking on writes until each has completed.
8585
86+ #### Parallel Stream Compression
87+
88+ For maximum throughput on large streams, use ` WithConcurrentBlocks(true) ` together with
89+ ` WithEncoderConcurrency(n) ` where n is the number of CPU cores you want to use.
90+ This splits the input into large sections (jobs) that are compressed simultaneously by multiple goroutines,
91+ similar to how the C zstd library does multithreaded compression.
92+
93+ ``` Go
94+ enc , err := zstd.NewWriter (out,
95+ zstd.WithEncoderLevel (zstd.SpeedDefault ),
96+ zstd.WithEncoderConcurrency (runtime.GOMAXPROCS (0 )),
97+ zstd.WithConcurrentBlocks (true ),
98+ )
99+ ```
100+
101+ Each non-first job receives an overlap prefix from the previous job for match context,
102+ so compression ratio is only marginally affected. Output is flushed in order,
103+ producing a valid single-frame zstd stream.
104+
105+ Benchmark on 1.8GB GOB stream (AMD Ryzen 9 9950X):
106+
107+ | Level | 1 thread | 4 threads | 16 threads | 1T ratio | 16T ratio |
108+ | ---------| :----------:| :------------------:| :-------------------:| :--------:| :---------:|
109+ | fastest | 783 MB/s | 2950 MB/s (3.8×) | 6939 MB/s (8.9×) | 12.24% | 12.26% |
110+ | default | 728 MB/s | 2533 MB/s (3.5×) | 5340 MB/s (7.3×) | 10.67% | 10.68% |
111+ | better | 434 MB/s | 1105 MB/s (2.5×) | 2206 MB/s (5.1×) | 9.14% | 9.21% |
112+ | best | 129 MB/s | 367 MB/s (2.8×) | 884 MB/s (6.8×) | 8.48% | 8.63% |
113+
114+ Notes:
115+ * Not compatible with dictionary encoding.
116+ * ` Flush() ` dispatches the current partial job, so latency-sensitive callers can force output.
117+ * ` EncodeAll ` is unaffected — it uses its own concurrency via the encoder pool.
118+
86119You can specify your desired compression level using ` WithEncoderLevel() ` option. Currently only pre-defined
87120compression settings can be specified.
88121
0 commit comments