Repository navigation
Commit 4ac46b7
fix(write): write a small object to S3 in one request, not three (#172)
Closes #171.
pyarrow's `S3FileSystem.open_output_stream` starts a multipart upload as
soon as the stream opens, so every object, even a 5-byte version hint,
cost three requests: CreateMultipartUpload, UploadPart and
CompleteMultipartUpload (apache/arrow#51029). pyiceberg's
`PyArrowFileIO` builds its `S3FileSystem` without a way to pass
`allow_delayed_open`.
`_SinglePutFileIO` subclasses `PyArrowFileIO` and rebuilds the S3
filesystem with `allow_delayed_open=True`. An object smaller than a part
now goes up as a single PutObject, and a larger one is still a multipart
upload.
**Verified** against rustfs, tracing requests with pyarrow's S3 debug
log:
- a small write is 1 PUT instead of 3;
- a 12 MiB write is still multipart;
- one publish is now 10 PutObjects.
The one exception is a published table's very first `metadata.json`,
which pyiceberg's `create_table` writes through its own FileIO.
On the issue's third item: the version hint is written only after a
published commit, not on idle passes. On a log that publishes every
pass, most of those commits came from snapshot expiry, which a separate
PR batches.
**Tests:** `test_a_small_object_goes_to_s3_in_one_request` checks that
the FileIO's S3 filesystem has `allow_delayed_open` set. The full suite
passes locally (713 passed).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01SRe5jfMf2Ro74AL35Us8tC
Co-authored-by: Claude Opus 5.5 <[email protected]>1 parent 8724a4d commit 4ac46b7
3 files changed
Lines changed: 70 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
10 | 25 | | |
11 | 26 | | |
12 | 27 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| |||
140 | 140 | | |
141 | 141 | | |
142 | 142 | | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
143 | 146 | | |
144 | 147 | | |
145 | 148 | | |
146 | 149 | | |
147 | 150 | | |
148 | 151 | | |
149 | 152 | | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
150 | 180 | | |
151 | 181 | | |
152 | 182 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3393 | 3393 | | |
3394 | 3394 | | |
3395 | 3395 | | |
| 3396 | + | |
| 3397 | + | |
| 3398 | + | |
| 3399 | + | |
| 3400 | + | |
| 3401 | + | |
| 3402 | + | |
| 3403 | + | |
| 3404 | + | |
| 3405 | + | |
| 3406 | + | |
| 3407 | + | |
| 3408 | + | |
| 3409 | + | |
| 3410 | + | |
| 3411 | + | |
| 3412 | + | |
| 3413 | + | |
| 3414 | + | |
| 3415 | + | |
| 3416 | + | |
| 3417 | + | |
| 3418 | + | |
| 3419 | + | |
0 commit comments