Repository navigation
fix(write): write a small object to S3 in one request, not three - #172
Merged
Merged
Conversation
pyarrow's S3 output stream opens a multipart upload as soon as it opens, so every object a published commit writes — metadata.json, the manifest list, its manifests, version-hint.text — and every small data file was CreateMultipartUpload + UploadPart + CompleteMultipartUpload: three billed write requests. Traced against rustfs, a 5-byte hint was all three, through pyarrow directly and through pyiceberg's FileIO. The FileIO shared_file_io builds is now _SinglePutFileIO, which rebuilds pyiceberg's S3FileSystem from its own options with allow_delayed_open: the stream buffers until it holds a part, so a smaller object is one PutObject and a larger one the multipart upload it was. pyiceberg has no property for it. One publish traced afterwards: ten PutObjects, and the published table's first metadata.json, which pyiceberg's create_table writes through a FileIO of its own, still multipart. Closes #171. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SRe5jfMf2Ro74AL35Us8tC
This was referenced Oct 8, 2026
nhobin219
added a commit
that referenced
this pull request
Oct 8, 2026
**Bug:** #172 rebuilds pyiceberg's S3 filesystem with `allow_delayed_open`, an option that pyarrow 21 introduced. litelink's floor was pyarrow 17 and pyiceberg's is 18, so the resolver still allowed 18–20. On those versions the rebuild raises `TypeError: __init__() got an unexpected keyword argument 'allow_delayed_open'`, and every S3 filesystem fails to build. I reproduced this on pyarrow 20.0.0. #172 is unreleased, so no published version is affected. **Fix:** the floor is now `pyarrow>=21.0.0`. The CHANGELOG notes it as a breaking change. `pyproject.toml` explains the reason in a comment, and `_SinglePutFileIO`'s docstring points to it. **Rejected alternative:** keep an older pyarrow's filesystem as built. The first commit here did that; the second replaces it. That branch would only ever run against a fake in the tests, since CI installs the latest pyarrow. An older pyarrow would also quietly pay three requests per small object. **Downstream:** streamcast doesn't constrain pyarrow itself, and its lock already resolves pyarrow 25.0.1. **Tests:** the packaging tests and the #172 FileIO test pass locally. I didn't run the full suite, since nothing here changes code paths; CI runs it. Upstream: apache/iceberg-python#4093 asks pyiceberg to expose the option. Once that ships, `_SinglePutFileIO` can go. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SRe5jfMf2Ro74AL35Us8tC --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
Merged
nhobin219
added a commit
that referenced
this pull request
Oct 9, 2026
Version bump for 0.12.0: `pyproject.toml`, `uv.lock`, and the CHANGELOG's Unreleased heading becomes `## 0.12.0 — 2026-10-08`. This is a minor release, with one breaking change. Everything is listed in the CHANGELOG: - **Breaking: pyarrow 21 or later is required** (#174), for the single-request upload below. - **A small object goes to S3 in one request, not three** (#172, #171): about a third as many write requests for a log whose published writes are mostly small. - **Published snapshot expiry is batched** (#173): about one expiry commit per quarter of `published_snapshot_retention` instead of one per pass. Merged by Claude on your instruction to cut the release. After `main` CI passes on the merge, pushing tag `v0.12.0` runs `release.yml`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SRe5jfMf2Ro74AL35Us8tC Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #171.
pyarrow's
S3FileSystem.open_output_streamstarts a multipart upload as soon as the stream opens, so every object, even a 5-byte version hint, cost three requests: CreateMultipartUpload, UploadPart and CompleteMultipartUpload (apache/arrow#51029). pyiceberg'sPyArrowFileIObuilds itsS3FileSystemwithout a way to passallow_delayed_open._SinglePutFileIOsubclassesPyArrowFileIOand rebuilds the S3 filesystem withallow_delayed_open=True. An object smaller than a part now goes up as a single PutObject, and a larger one is still a multipart upload.Verified against rustfs, tracing requests with pyarrow's S3 debug log:
The one exception is a published table's very first
metadata.json, which pyiceberg'screate_tablewrites through its own FileIO.On the issue's third item: the version hint is written only after a published commit, not on idle passes. On a log that publishes every pass, most of those commits came from snapshot expiry, which a separate PR batches.
Tests:
test_a_small_object_goes_to_s3_in_one_requestchecks that the FileIO's S3 filesystem hasallow_delayed_openset. The full suite passes locally (713 passed).🤖 Generated with Claude Code
https://claude.ai/code/session_01SRe5jfMf2Ro74AL35Us8tC