Skip to content

Commit 4ac46b7

Browse files
nhobin219claude
andauthored
fix(write): write a small object to S3 in one request, not three (#172)
Closes #171. pyarrow's `S3FileSystem.open_output_stream` starts a multipart upload as soon as the stream opens, so every object, even a 5-byte version hint, cost three requests: CreateMultipartUpload, UploadPart and CompleteMultipartUpload (apache/arrow#51029). pyiceberg's `PyArrowFileIO` builds its `S3FileSystem` without a way to pass `allow_delayed_open`. `_SinglePutFileIO` subclasses `PyArrowFileIO` and rebuilds the S3 filesystem with `allow_delayed_open=True`. An object smaller than a part now goes up as a single PutObject, and a larger one is still a multipart upload. **Verified** against rustfs, tracing requests with pyarrow's S3 debug log: - a small write is 1 PUT instead of 3; - a 12 MiB write is still multipart; - one publish is now 10 PutObjects. The one exception is a published table's very first `metadata.json`, which pyiceberg's `create_table` writes through its own FileIO. On the issue's third item: the version hint is written only after a published commit, not on idle passes. On a log that publishes every pass, most of those commits came from snapshot expiry, which a separate PR batches. **Tests:** `test_a_small_object_goes_to_s3_in_one_request` checks that the FileIO's S3 filesystem has `allow_delayed_open` set. The full suite passes locally (713 passed). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01SRe5jfMf2Ro74AL35Us8tC Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent 8724a4d commit 4ac46b7

3 files changed

Lines changed: 70 additions & 1 deletion

File tree

‎CHANGELOG.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,21 @@ rather than restates it.
77
This project follows [Semantic Versioning](https://semver.org/). Before 1.0 the
88
minor version carries breaking changes.
99

10+
## Unreleased
11+
12+
### Fixed
13+
14+
- **A small object goes to S3 in one request, not three** (#171). pyarrow's
15+
S3 stream opens a multipart upload as soon as it opens, so every
16+
`metadata.json`, manifest list, manifest, `version-hint.text` and small
17+
data file was `CreateMultipartUpload` + `UploadPart` +
18+
`CompleteMultipartUpload`: three billed write requests, even for a 5-byte
19+
hint. The FileIO litelink shares now delays the open, so an object under a
20+
part is one `PutObject`; a larger one is the multipart upload it was. A
21+
log whose published writes are mostly small, such as one publishing every
22+
tick, makes about a third as many write requests. The only exception is a published table's very first
23+
`metadata.json`, which pyiceberg writes through a FileIO of its own.
24+
1025
## 0.11.0 — 2026-10-07
1126

1227
### Levers, and how they relate to RPO

‎src/litelink/_table.py‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@
1919
from dataclasses import dataclass
2020
from typing import TYPE_CHECKING, NamedTuple
2121

22-
from pyarrow.fs import FileSelector, FileType
22+
from pyarrow.fs import FileSelector, FileSystem, FileType
2323
from pyiceberg.catalog import METADATA_LOCATION, Catalog
2424
from pyiceberg.catalog.sql import IcebergTables, SqlCatalog
2525
from pyiceberg.conversions import from_bytes
@@ -140,13 +140,43 @@ def shared_file_io(properties: Mapping[str, str], location: str | None) -> FileI
140140
return io
141141

142142
io = load_file_io(dict(properties), location)
143+
if type(io) is PyArrowFileIO:
144+
io = _SinglePutFileIO(dict(properties))
145+
143146
_FILE_IOS[key] = io
144147
while len(_FILE_IOS) > _FILE_IOS_LIMIT:
145148
_FILE_IOS.popitem(last=False)
146149

147150
return io
148151

149152

153+
class _SinglePutFileIO(PyArrowFileIO):
154+
"""`PyArrowFileIO`, writing an object smaller than a part to S3 in one
155+
request rather than three (#171).
156+
157+
pyarrow's S3 output stream opens a multipart upload as soon as it opens —
158+
`CreateMultipartUpload`, then `UploadPart`, then `CompleteMultipartUpload`,
159+
each a billed write — even for `version-hint.text`, whose whole content is
160+
a version number. Measured against rustfs: a 5-byte write was those three
161+
requests. With `allow_delayed_open` the stream buffers until it holds a
162+
part's worth: a smaller object goes up as one `PutObject`, a larger one as
163+
the multipart upload it was. Every object a commit writes — `metadata.json`,
164+
the manifest list, its manifests, the hint — and every seal a flushed
165+
publish pushes is under a part, so this is most of a log's writes.
166+
167+
pyiceberg offers no property for it, so the filesystem it builds is rebuilt
168+
from its own options with the one changed.
169+
"""
170+
171+
def _initialize_s3_fs(self, netloc: str | None) -> FileSystem:
172+
built = super()._initialize_s3_fs(netloc)
173+
rebuild, (options,) = built.__reduce__()
174+
if options.get("allow_delayed_open"):
175+
return built
176+
177+
return rebuild({**options, "allow_delayed_open": True})
178+
179+
150180
class _Catalog(SqlCatalog):
151181
"""`SqlCatalog`, loading and committing through `shared_file_io`.
152182

‎tests/test_publish.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3393,3 +3393,27 @@ def strip() -> None:
33933393
published.reload()
33943394

33953395
assert published.properties.get(key) == METADATA_PROPERTIES[key]
3396+
3397+
3398+
def test_a_small_object_goes_to_s3_in_one_request(s3: S3Options) -> None:
3399+
"""pyarrow's S3 stream opens a multipart upload as soon as it opens, which
3400+
is three billed writes for a `version-hint.text` (#171). The shared FileIO
3401+
delays the open, so an object under a part is one `PutObject`.
3402+
3403+
Falsify by returning `load_file_io`'s FileIO from `shared_file_io` as is:
3404+
`allow_delayed_open` reads False.
3405+
"""
3406+
from litelink._table import shared_file_io
3407+
3408+
io = shared_file_io(
3409+
{
3410+
"s3.endpoint": s3.endpoint or "",
3411+
"s3.access-key-id": s3.access_key or "",
3412+
"s3.secret-access-key": s3.secret_key or "",
3413+
"s3.region": s3.region or "us-east-1",
3414+
},
3415+
"s3://bucket/prefix",
3416+
)
3417+
filesystem = io.fs_by_scheme("s3", "bucket") # ty: ignore[unresolved-attribute]
3418+
_, (options,) = filesystem.__reduce__()
3419+
assert options["allow_delayed_open"] is True

0 commit comments

Comments
 (0)