Skip to content

Commit d60ed05

Browse files
101arrowz101arrowz
committed
Improve streaming API
1 parent ef157f0 commit d60ed05

15 files changed

Lines changed: 445 additions & 35 deletions

‎CHANGELOG.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
## 0.8.2
2+
- Fixed broken UMD build
3+
- Fixed edge-case causing skipped data during streaming compression
4+
- Fixed bug in GZIP streaming on member boundary
5+
- Improved streaming performance on inconsistent chunk sizes
6+
- Improved `unzip` performance on undercompressed archives
7+
- Added flushing support into streaming API
8+
- Added backpressure support into async streaming API
9+
- Use new `ondrain` handler and `queuedSize`
110
## 0.8.1
211
- Fixed reallocating on pre-supplied buffer in `inflateSync` and `unzlibSync`
312
- Minor documentation fixes

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -536,9 +536,9 @@ Before you decide that `fflate` is the end-all compression library, you should n
536536
## What about `CompressionStream`?
537537
Like `fflate`, the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API) provides DEFLATE, GZIP, and Zlib compression and decompression support. It's a good option if you'd like to compress or decompress data without installing any third-party libraries, and it wraps native Zlib bindings to achieve better performance than what most JavaScript programs can achieve.
538538

539-
However, browsers do not offer any native non-streaming compression API, and `CompressionStream` has surprisingly poor performance on data already loaded into memory; `fflate` tends to be faster even for files that are dozens of megabytes large. Similarly, `fflate` is much faster for files under a megabyte because it avoids marshalling overheads. Even when streaming hundreds of megabytes of data, the native API usually only performs between 5% slower and 10% faster than `fflate`. And Compression Streams have many other disadvantages - no ability to control compression level, poor support for older browsers, no ZIP support, etc.
539+
However, browsers do not offer any native non-streaming compression API, and `CompressionStream` has surprisingly poor performance on data already loaded into memory; `fflate` tends to be faster even for files that are dozens of megabytes large. Similarly, `fflate` is much faster for files under a megabyte because it avoids marshalling overheads. Even when streaming hundreds of megabytes of data, the native API usually performs between 30% faster and 10% slower than `fflate`. And Compression Streams have many other disadvantages - no ability to control compression level, poor support for older browsers, no ZIP support, etc.
540540

541-
If you'd still prefer to depend upon a native browser API, you can use an `fflate`-based [Compression Streams ponyfill](https://github.com/101arrowz/compression-streams-polyfill) for a seamless transition to `CompressionStream` and `DecompressionStream` if ever they become substantially faster than `fflate`.
541+
If you'd still prefer to depend upon a native browser API but want to support older browsers, you can use an `fflate`-based [Compression Streams ponyfill](https://github.com/101arrowz/compression-streams-polyfill).
542542

543543
## Browser support
544544
`fflate` makes heavy use of typed arrays (`Uint8Array`, `Uint16Array`, etc.). Typed arrays can be polyfilled at the cost of performance, but the most recent browser that doesn't support them [is from 2011](https://caniuse.com/typedarrays), so I wouldn't bother.

‎docs/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@
7171

7272
### Type Aliases
7373

74+
- [AsyncFlateDrainHandler](README.md#asyncflatedrainhandler)
7475
- [AsyncFlateStreamHandler](README.md#asyncflatestreamhandler)
7576
- [AsyncZippableFile](README.md#asynczippablefile)
7677
- [FlateCallback](README.md#flatecallback)
@@ -135,6 +136,28 @@ Renames and re-exports [gzipSync](README.md#gzipsync)
135136

136137
## Type Aliases
137138

139+
### AsyncFlateDrainHandler
140+
141+
Ƭ **AsyncFlateDrainHandler**: (`size`: `number`) => `void`
142+
143+
#### Type declaration
144+
145+
▸ (`size`): `void`
146+
147+
Handler for the asynchronous completion of (de)compression for a data chunk
148+
149+
##### Parameters
150+
151+
| Name | Type | Description |
152+
| :------ | :------ | :------ |
153+
| `size` | `number` | The number of bytes that were processed. This is measured in terms of the input (i.e. compressed bytes for decompression, uncompressed bytes for compression.) |
154+
155+
##### Returns
156+
157+
`void`
158+
159+
___
160+
138161
### AsyncFlateStreamHandler
139162

140163
Ƭ **AsyncFlateStreamHandler**: (`err`: [`FlateError`](interfaces/FlateError.md) \| ``null``, `data`: `Uint8Array`, `final`: `boolean`) => `void`

‎docs/classes/AsyncDecompress.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ Asynchronous streaming GZIP, Zlib, or raw DEFLATE decompression
1111
### Properties
1212

1313
- [ondata](AsyncDecompress.md#ondata)
14+
- [ondrain](AsyncDecompress.md#ondrain)
15+
- [queuedSize](AsyncDecompress.md#queuedsize)
1416

1517
### Methods
1618

@@ -49,6 +51,22 @@ Creates an asynchronous decompression stream
4951

5052
The handler to call whenever data is available
5153

54+
___
55+
56+
### ondrain
57+
58+
• `Optional` **ondrain**: [`AsyncFlateDrainHandler`](../README.md#asyncflatedrainhandler)
59+
60+
The handler to call whenever buffered source data is processed (i.e. `queuedSize` updates)
61+
62+
___
63+
64+
### queuedSize
65+
66+
• **queuedSize**: `number`
67+
68+
The number of compressed bytes buffered in the stream
69+
5270
## Methods
5371

5472
### push

‎docs/classes/AsyncDeflate.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,13 @@ Asynchronous streaming DEFLATE compression
1111
### Properties
1212

1313
- [ondata](AsyncDeflate.md#ondata)
14+
- [ondrain](AsyncDeflate.md#ondrain)
15+
- [queuedSize](AsyncDeflate.md#queuedsize)
1416
- [terminate](AsyncDeflate.md#terminate)
1517

1618
### Methods
1719

20+
- [flush](AsyncDeflate.md#flush)
1821
- [push](AsyncDeflate.md#push)
1922

2023
## Constructors
@@ -52,6 +55,22 @@ The handler to call whenever data is available
5255

5356
___
5457

58+
### ondrain
59+
60+
• `Optional` **ondrain**: [`AsyncFlateDrainHandler`](../README.md#asyncflatedrainhandler)
61+
62+
The handler to call whenever buffered source data is processed (i.e. `queuedSize` updates)
63+
64+
___
65+
66+
### queuedSize
67+
68+
• **queuedSize**: `number`
69+
70+
The number of uncompressed bytes buffered in the stream
71+
72+
___
73+
5574
### terminate
5675

5776
• **terminate**: [`AsyncTerminable`](../interfaces/AsyncTerminable.md)
@@ -61,6 +80,19 @@ push() will silently fail.
6180

6281
## Methods
6382

83+
### flush
84+
85+
▸ **flush**(): `void`
86+
87+
Flushes buffered uncompressed data. Useful to immediately retrieve the
88+
deflated output for small inputs.
89+
90+
#### Returns
91+
92+
`void`
93+
94+
___
95+
6496
### push
6597

6698
▸ **push**(`chunk`, `final?`): `void`

‎docs/classes/AsyncGunzip.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,9 @@ Asynchronous streaming single or multi-member GZIP decompression
1111
### Properties
1212

1313
- [ondata](AsyncGunzip.md#ondata)
14+
- [ondrain](AsyncGunzip.md#ondrain)
1415
- [onmember](AsyncGunzip.md#onmember)
16+
- [queuedSize](AsyncGunzip.md#queuedsize)
1517
- [terminate](AsyncGunzip.md#terminate)
1618

1719
### Methods
@@ -53,6 +55,14 @@ The handler to call whenever data is available
5355

5456
___
5557

58+
### ondrain
59+
60+
• `Optional` **ondrain**: [`AsyncFlateDrainHandler`](../README.md#asyncflatedrainhandler)
61+
62+
The handler to call whenever buffered source data is processed (i.e. `queuedSize` updates)
63+
64+
___
65+
5666
### onmember
5767

5868
• `Optional` **onmember**: [`GunzipMemberHandler`](../README.md#gunzipmemberhandler)
@@ -61,6 +71,14 @@ The handler to call whenever a new GZIP member is found
6171

6272
___
6373

74+
### queuedSize
75+
76+
• **queuedSize**: `number`
77+
78+
The number of compressed bytes buffered in the stream
79+
80+
___
81+
6482
### terminate
6583

6684
• **terminate**: [`AsyncTerminable`](../interfaces/AsyncTerminable.md)

‎docs/classes/AsyncGzip.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,13 @@ Asynchronous streaming GZIP compression
1111
### Properties
1212

1313
- [ondata](AsyncGzip.md#ondata)
14+
- [ondrain](AsyncGzip.md#ondrain)
15+
- [queuedSize](AsyncGzip.md#queuedsize)
1416
- [terminate](AsyncGzip.md#terminate)
1517

1618
### Methods
1719

20+
- [flush](AsyncGzip.md#flush)
1821
- [push](AsyncGzip.md#push)
1922

2023
## Constructors
@@ -52,6 +55,22 @@ The handler to call whenever data is available
5255

5356
___
5457

58+
### ondrain
59+
60+
• `Optional` **ondrain**: [`AsyncFlateDrainHandler`](../README.md#asyncflatedrainhandler)
61+
62+
The handler to call whenever buffered source data is processed (i.e. `queuedSize` updates)
63+
64+
___
65+
66+
### queuedSize
67+
68+
• **queuedSize**: `number`
69+
70+
The number of uncompressed bytes buffered in the stream
71+
72+
___
73+
5574
### terminate
5675

5776
• **terminate**: [`AsyncTerminable`](../interfaces/AsyncTerminable.md)
@@ -61,6 +80,19 @@ push() will silently fail.
6180

6281
## Methods
6382

83+
### flush
84+
85+
▸ **flush**(): `void`
86+
87+
Flushes buffered uncompressed data. Useful to immediately retrieve the
88+
GZIPped output for small inputs.
89+
90+
#### Returns
91+
92+
`void`
93+
94+
___
95+
6496
### push
6597

6698
▸ **push**(`chunk`, `final?`): `void`

‎docs/classes/AsyncInflate.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ Asynchronous streaming DEFLATE decompression
1111
### Properties
1212

1313
- [ondata](AsyncInflate.md#ondata)
14+
- [ondrain](AsyncInflate.md#ondrain)
15+
- [queuedSize](AsyncInflate.md#queuedsize)
1416
- [terminate](AsyncInflate.md#terminate)
1517

1618
### Methods
@@ -52,6 +54,22 @@ The handler to call whenever data is available
5254

5355
___
5456

57+
### ondrain
58+
59+
• `Optional` **ondrain**: [`AsyncFlateDrainHandler`](../README.md#asyncflatedrainhandler)
60+
61+
The handler to call whenever buffered source data is processed (i.e. `queuedSize` updates)
62+
63+
___
64+
65+
### queuedSize
66+
67+
• **queuedSize**: `number`
68+
69+
The number of compressed bytes buffered in the stream
70+
71+
___
72+
5573
### terminate
5674

5775
• **terminate**: [`AsyncTerminable`](../interfaces/AsyncTerminable.md)

‎docs/classes/AsyncUnzlib.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ Asynchronous streaming Zlib decompression
1111
### Properties
1212

1313
- [ondata](AsyncUnzlib.md#ondata)
14+
- [ondrain](AsyncUnzlib.md#ondrain)
15+
- [queuedSize](AsyncUnzlib.md#queuedsize)
1416
- [terminate](AsyncUnzlib.md#terminate)
1517

1618
### Methods
@@ -52,6 +54,22 @@ The handler to call whenever data is available
5254

5355
___
5456

57+
### ondrain
58+
59+
• `Optional` **ondrain**: [`AsyncFlateDrainHandler`](../README.md#asyncflatedrainhandler)
60+
61+
The handler to call whenever buffered source data is processed (i.e. `queuedSize` updates)
62+
63+
___
64+
65+
### queuedSize
66+
67+
• **queuedSize**: `number`
68+
69+
The number of compressed bytes buffered in the stream
70+
71+
___
72+
5573
### terminate
5674

5775
• **terminate**: [`AsyncTerminable`](../interfaces/AsyncTerminable.md)

‎docs/classes/AsyncZlib.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,13 @@ Asynchronous streaming Zlib compression
1111
### Properties
1212

1313
- [ondata](AsyncZlib.md#ondata)
14+
- [ondrain](AsyncZlib.md#ondrain)
15+
- [queuedSize](AsyncZlib.md#queuedsize)
1416
- [terminate](AsyncZlib.md#terminate)
1517

1618
### Methods
1719

20+
- [flush](AsyncZlib.md#flush)
1821
- [push](AsyncZlib.md#push)
1922

2023
## Constructors
@@ -52,6 +55,22 @@ The handler to call whenever data is available
5255

5356
___
5457

58+
### ondrain
59+
60+
• `Optional` **ondrain**: [`AsyncFlateDrainHandler`](../README.md#asyncflatedrainhandler)
61+
62+
The handler to call whenever buffered source data is processed (i.e. `queuedSize` updates)
63+
64+
___
65+
66+
### queuedSize
67+
68+
• **queuedSize**: `number`
69+
70+
The number of uncompressed bytes buffered in the stream
71+
72+
___
73+
5574
### terminate
5675

5776
• **terminate**: [`AsyncTerminable`](../interfaces/AsyncTerminable.md)
@@ -61,6 +80,19 @@ push() will silently fail.
6180

6281
## Methods
6382

83+
### flush
84+
85+
▸ **flush**(): `void`
86+
87+
Flushes buffered uncompressed data. Useful to immediately retrieve the
88+
zlibbed output for small inputs.
89+
90+
#### Returns
91+
92+
`void`
93+
94+
___
95+
6496
### push
6597

6698
▸ **push**(`chunk`, `final?`): `void`

0 commit comments

Comments
 (0)