Skip to content

perf(extract): coalesce archive blocks into large writes - #1681

Merged
tdewey-rpi merged 1 commit into
raspberrypi:mainfrom
unraid:agent/upstream-batched-writes
Aug 7, 2026
Merged

perf(extract): coalesce archive blocks into large writes#1681
tdewey-rpi merged 1 commit into
raspberrypi:mainfrom
unraid:agent/upstream-batched-writes

Conversation

@elibosley

Copy link
Copy Markdown
Contributor

Summary

Multi-file extraction performs one disk write for every small libarchive data block; this coalesces contiguous blocks into 8 MiB writes while preserving exact offsets and error behavior.

Why This Exists

archive_write_data_block() has meaningful per-call overhead, especially on removable media with write caching disabled. In device testing, batching reduced a representative extraction from approximately 26 minutes to 5.5 minutes.

Resolution

Add a small, independently tested BlockBatcher between the archive reader and archive_write_disk. Contiguous blocks are buffered, offset gaps flush immediately to preserve sparse files, and blocks at least as large as the buffer bypass the copy.

Reviewer Considerations

  • The buffer is 8 MiB per active extraction thread.
  • Sparse and backwards offsets force a flush rather than being merged.
  • Pending data is flushed before archive_write_finish_entry().
  • Write errors propagate unchanged and failed data is not retried implicitly.

Behavior Changes

Multi-file images are written using fewer, larger disk operations. Extracted bytes and offsets remain unchanged.

Implementation Summary

  • Add a header-only contiguous block batcher.
  • Integrate it into multi-file extraction.
  • Add 11 focused tests covering batching, sparse offsets, boundaries, bypasses, and failures.
  • Log source block size diagnostics for field validation.

Verification

  • ./build/test/block_batcher_test: 122 assertions in 11 test cases passed.
  • Full macOS build compiled downloadextractthread.cpp successfully; final universal-app link was blocked by the local arm64-only Homebrew Qt installation.
  • git diff --check: passed.

Risk

Low to moderate. The buffering path is isolated and extensively tested; the primary tradeoff is an additional 8 MiB allocation during multi-file extraction.

@elibosley
elibosley marked this pull request as ready for review August 6, 2026 00:27
@tdewey-rpi
tdewey-rpi merged commit e07fd39 into raspberrypi:main Aug 7, 2026
@elibosley
elibosley deleted the agent/upstream-batched-writes branch August 7, 2026 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants