You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 8da6e6b
Browse filesBrowse the repository at this point in the historyBrowse files
perf: reuse zstd compression contexts across shuffle blocks
Every shuffle block previously created and destroyed its own zstd
context: a fresh CCtx per encoded block and a fresh DCtx per decoded
frame. Context setup is pure overhead that scales with block count, so
high-partition shuffles with small blocks pay the most.
Encode paths now share one context per task, threaded from the
task-level owner so codec memory stays bounded regardless of partition
count. The remote shuffle path still frees the zstd workspace with each
admitted encode, keeping its memory accounting accurate. Decode reuses
a per-thread context behind the existing entry points.
Wire format is unchanged. On a 4M-row hash shuffle with 10,000
partitions at zstd level 3, encode time drops ~10% and wall time ~7%;
larger-block shapes are within noise. lz4 and snappy keep per-block
encoders: no reset API in the pinned crates and much smaller setup
cost.
Part of #5002.
0 commit comments