Skip to content

Commit fe3ccb4

Browse files
maikschulzeclaude
andcommitted
pythongh-155460: Only pre-size zstd1 frames; match libzstd's default window limit
frame_output_size() now requires the zstd1 magic number. libzstd also accepts legacy (v0.5-v0.7) frames in ZSTD_getFrameContentSize(), but their headers have another layout and they are never decoded in a single pass. DEFAULT_MAX_WINDOW_SIZE is now (1 << 27) + 1, matching libzstd's ZSTD_MAXWINDOWSIZE_DEFAULT, so frames that libzstd accepts by default are not refused by the window check. Reword the NEWS entry so it is not specific to Windows. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent 2a2abc6 commit fe3ccb4

2 files changed

Lines changed: 16 additions & 7 deletions

File tree

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
:mod:`compression.zstd` now allocates the output of a frame that records its
2-
decompressed size once, at that size, instead of growing it while
3-
decompressing. This lowers peak memory use and makes decompression faster:
4-
up to 3 times on Windows. The recorded size is only used when the whole frame
5-
is available and could decode to that size.
2+
decompressed size once, at that size, and decodes straight into it, instead
3+
of growing the output while decompressing. This lowers peak memory use and
4+
makes decompressing such frames several times faster. The recorded size is
5+
only used when the whole frame is available and could decode to that size.

‎Modules/_zstd/decompressor.c‎

Lines changed: 12 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -87,9 +87,9 @@ _get_DDict(ZstdDict *self)
8787
}
8888

8989
/* zstd refuses frames that need a larger window than this, unless
90-
window_log_max is set (ZSTD_WINDOWLOG_LIMIT_DEFAULT, in zstd's static
91-
API). */
92-
#define DEFAULT_MAX_WINDOW_SIZE (1ULL << 27)
90+
window_log_max is set (ZSTD_MAXWINDOWSIZE_DEFAULT, in zstd's private
91+
API: (1 << ZSTD_WINDOWLOG_LIMIT_DEFAULT) + 1). */
92+
#define DEFAULT_MAX_WINDOW_SIZE ((1ULL << 27) + 1)
9393

9494
static int
9595
_zstd_set_d_parameters(ZstdDecompressor *self, PyObject *options)
@@ -216,6 +216,15 @@ frame_output_size(ZstdDecompressor *self, ZSTD_inBuffer *in,
216216
{
217217
const unsigned char *src = (const unsigned char *)in->src + in->pos;
218218
size_t avail = in->size - in->pos;
219+
/* Only zstd1 frames: libzstd also accepts legacy (v0.5-v0.7) frames
220+
here, whose headers have another layout and which it never decodes in
221+
a single pass. */
222+
if (avail < 4
223+
|| ((uint32_t)src[0] | (uint32_t)src[1] << 8 | (uint32_t)src[2] << 16
224+
| (uint32_t)src[3] << 24) != ZSTD_MAGICNUMBER)
225+
{
226+
return -1;
227+
}
219228
unsigned long long size = ZSTD_getFrameContentSize(src, avail);
220229
if (size == ZSTD_CONTENTSIZE_UNKNOWN || size == ZSTD_CONTENTSIZE_ERROR
221230
|| size == 0 || size > (unsigned long long)PY_SSIZE_T_MAX

0 commit comments

Comments
 (0)