You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit bdd4fad
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](spill) Discard the writer of a destroyed spill file and close Iceberg spill writers explicitly
### What problem does this PR solve?
Issue Number: None
Related PR: #68032
Problem Summary: Follow-ups of the review of remote (object storage) spill.
- A SpillFileWriter can outlive its SpillFile: the repartitioner keeps its output writers
while the output files live in the caller, so an error after setup_output() destroys the
files first. Closing the writer later force-charged a footer to no owner (leaking spill
capacity) and, on object storage, could publish a part after the file's prefix was deleted.
SpillFile::gc() now discards its active writer first: the unfinished part is dropped without
footer or close(), in-flight uploads are drained, the budget is reconciled and the multipart
upload is aborted. A writer destroyed after a failed close() unregisters itself so the file
never reaches a dangling writer.
- The Iceberg sort writer relied on the writer destructor to close its spill files, which only
logs a failure. On object storage close() uploads the tail of the file, so a failure could
silently lose rows, and the intermediate merge deleted its inputs before its output was
complete. Both paths now close the writer and return its status.
- The Iceberg spill sink registers the remote spill counters, so its profile shows remote
requests, bytes and times like the other spilling operators.
- The remote spill bytes are aggregated by ProfileManager and shown by the query profile REST
API and show proc '/current_queries'; the HTTP query plan audit event sets them to 0
instead of the -1 default.
- RemoteSpillStatsPoller measures the age of the polled value on the monotonic clock, so a
wall clock moved backward cannot keep a stale value fresh.
- Document that the remote request counters count logical requests: retries inside the object
storage client are not included.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- BE: SpillFileS3Test (new SpillFileDestroyedBeforeWriterDiscardsPart,
FailedCloseThenSpillFileDestroyed), SpillFileTest, SpillRepartitionerTest, partitioned
and spill sort operators, SpillIcebergTableSinkOperatorTest, VIcebergPartitionWriterTest
(213 passed)
- FE: RemoteSpillStatsPollerTest, CurrentQueryStatisticsProcDirTest, AuditLoaderTest
- Behavior changed: Yes. show proc '/current_queries' has two more columns,
SpillWriteBytesToRemoteStorage and SpillReadBytesFromRemoteStorage.
- Does this need documentation: No
Claude-Session: https://claude.ai/code/session_01B5f3dfnSMXhasgsShyagfR
0 commit comments