Skip to content

Commit bdd4fad

Browse files
committed
[fix](spill) Discard the writer of a destroyed spill file and close Iceberg spill writers explicitly
### What problem does this PR solve? Issue Number: None Related PR: #68032 Problem Summary: Follow-ups of the review of remote (object storage) spill. - A SpillFileWriter can outlive its SpillFile: the repartitioner keeps its output writers while the output files live in the caller, so an error after setup_output() destroys the files first. Closing the writer later force-charged a footer to no owner (leaking spill capacity) and, on object storage, could publish a part after the file's prefix was deleted. SpillFile::gc() now discards its active writer first: the unfinished part is dropped without footer or close(), in-flight uploads are drained, the budget is reconciled and the multipart upload is aborted. A writer destroyed after a failed close() unregisters itself so the file never reaches a dangling writer. - The Iceberg sort writer relied on the writer destructor to close its spill files, which only logs a failure. On object storage close() uploads the tail of the file, so a failure could silently lose rows, and the intermediate merge deleted its inputs before its output was complete. Both paths now close the writer and return its status. - The Iceberg spill sink registers the remote spill counters, so its profile shows remote requests, bytes and times like the other spilling operators. - The remote spill bytes are aggregated by ProfileManager and shown by the query profile REST API and show proc '/current_queries'; the HTTP query plan audit event sets them to 0 instead of the -1 default. - RemoteSpillStatsPoller measures the age of the polled value on the monotonic clock, so a wall clock moved backward cannot keep a stale value fresh. - Document that the remote request counters count logical requests: retries inside the object storage client are not included. ### Release note None ### Check List (For Author) - Test: Unit Test - BE: SpillFileS3Test (new SpillFileDestroyedBeforeWriterDiscardsPart, FailedCloseThenSpillFileDestroyed), SpillFileTest, SpillRepartitionerTest, partitioned and spill sort operators, SpillIcebergTableSinkOperatorTest, VIcebergPartitionWriterTest (213 passed) - FE: RemoteSpillStatsPollerTest, CurrentQueryStatisticsProcDirTest, AuditLoaderTest - Behavior changed: Yes. show proc '/current_queries' has two more columns, SpillWriteBytesToRemoteStorage and SpillReadBytesFromRemoteStorage. - Does this need documentation: No Claude-Session: https://claude.ai/code/session_01B5f3dfnSMXhasgsShyagfR
1 parent 422b839 commit bdd4fad

17 files changed

Lines changed: 195 additions & 37 deletions

File tree

‎be/src/exec/operator/spill_iceberg_table_sink_operator.cpp‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#include "exec/operator/iceberg_table_sink_operator.h"
2323
#include "exec/sink/writer/iceberg/viceberg_sort_writer.h"
2424
#include "exec/sink/writer/iceberg/viceberg_table_writer.h"
25+
#include "runtime/runtime_profile_counter_names.h"
2526

2627
namespace doris {
2728

@@ -196,6 +197,11 @@ void SpillIcebergTableSinkLocalState::_init_spill_counters() {
196197
ADD_COUNTER_WITH_LEVEL(profile, "SpillWriteFileBytes", TUnit::BYTES, 1);
197198
ADD_COUNTER_WITH_LEVEL(profile, "SpillWriteRows", TUnit::UNIT, 1);
198199
ADD_COUNTER_WITH_LEVEL(profile, "SpillWriteFileTotalCount", TUnit::UNIT, 1);
200+
ADD_COUNTER_WITH_LEVEL(profile, profile::SPILL_REMOTE_WRITE_REQUESTS, TUnit::UNIT, 1);
201+
ADD_COUNTER_WITH_LEVEL(profile, profile::SPILL_REMOTE_UPLOAD_PART_REQUESTS, TUnit::UNIT, 1);
202+
ADD_COUNTER_WITH_LEVEL(profile, profile::SPILL_REMOTE_UPLOAD_BYTES, TUnit::BYTES, 1);
203+
ADD_TIMER_WITH_LEVEL(profile, profile::SPILL_REMOTE_UPLOAD_TIME, 1);
204+
ADD_TIMER_WITH_LEVEL(profile, profile::SPILL_REMOTE_UPLOAD_WAIT_TIME, 1);
199205

200206
//seems init_spill_read_counters()
201207
ADD_TIMER_WITH_LEVEL(profile, "SpillTotalTime", 1);
@@ -210,6 +216,7 @@ void SpillIcebergTableSinkLocalState::_init_spill_counters() {
210216
ADD_COUNTER_WITH_LEVEL(profile, "SpillReadFileBytes", TUnit::BYTES, 1);
211217
ADD_COUNTER_WITH_LEVEL(profile, "SpillReadRows", TUnit::UNIT, 1);
212218
ADD_COUNTER_WITH_LEVEL(profile, "SpillReadFileCount", TUnit::UNIT, 1);
219+
ADD_COUNTER_WITH_LEVEL(profile, profile::SPILL_REMOTE_READ_REQUESTS, TUnit::UNIT, 1);
213220
ADD_COUNTER_WITH_LEVEL(profile, "SpillWriteFileCurrentBytes", TUnit::BYTES, 1);
214221
}
215222

‎be/src/exec/sink/writer/iceberg/viceberg_sort_writer.cpp‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,9 @@ Status VIcebergSortWriter::_do_spill() {
274274
RETURN_IF_ERROR(writer->write_block(_runtime_state, block));
275275
block.clear_column_data();
276276
}
277+
// On object storage close() uploads the tail of the file; a failure must fail the spill
278+
// instead of being logged by the destructor.
279+
RETURN_IF_ERROR(writer->close());
277280
// Reset the sorter to free memory and accept new data
278281
_sorter->reset();
279282
return Status::OK();
@@ -337,6 +340,8 @@ Status VIcebergSortWriter::_do_intermediate_merge() {
337340
RETURN_IF_ERROR(_merger->get_next(&merge_sorted_block, &eos));
338341
RETURN_IF_ERROR(tmp_spill_writer->write_block(_runtime_state, merge_sorted_block));
339342
}
343+
// The merged output must be complete before its inputs are deleted.
344+
RETURN_IF_ERROR(tmp_spill_writer->close());
340345

341346
// Clean up the files that were consumed during this intermediate merge
342347
for (auto& file : _current_merging_spill_files) {

‎be/src/exec/spill/spill_file.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,12 @@ SpillFile::~SpillFile() {
4343
}
4444

4545
void SpillFile::gc() {
46+
// The writer may outlive this file (e.g. when an error unwinds the file's owner first).
47+
// Drop its unfinished part now: closing it later would charge a footer to no owner and
48+
// could publish an object after the prefix below was deleted.
49+
if (_active_writer != nullptr) {
50+
_active_writer->_discard(this);
51+
}
4652
const int64_t written_bytes = std::exchange(_total_written_bytes, 0);
4753
if (!_dir_created) {
4854
_data_dir->release(written_bytes);

‎be/src/exec/spill/spill_file.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,8 @@ class SpillFile : public std::enable_shared_from_this<SpillFile> {
114114
bool _dir_created = false;
115115
bool _ready_for_reading = false;
116116
// Pointer to the currently-active writer. Mutable to allow checks from const
117-
// methods like create_reader(). Only one writer may be active at a time.
117+
// methods like create_reader(). Only one writer may be active at a time. Cleared when
118+
// the writer finished, was discarded by gc(), or was destroyed.
118119
mutable SpillFileWriter* _active_writer = nullptr;
119120
};
120121
using SpillFileSPtr = std::shared_ptr<SpillFile>;

‎be/src/exec/spill/spill_file_reader.cpp‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,8 @@ void SpillFileReader::_record_read(size_t bytes_read) {
8181
COUNTER_UPDATE(_read_file_size, bytes_read);
8282
ExecEnv::GetInstance()->spill_file_mgr()->update_spill_read_bytes(bytes_read);
8383
if (_is_remote) {
84-
// One read_at() is exactly one GET request on object storage.
84+
// One successful read_at() is one logical GET. Retries inside the object storage client
85+
// and reads that finally failed are not counted (see RemoteWriteStats).
8586
if (_remote_read_requests != nullptr) {
8687
COUNTER_UPDATE(_remote_read_requests, 1);
8788
}

‎be/src/exec/spill/spill_file_writer.cpp‎

Lines changed: 44 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -75,14 +75,36 @@ SpillFileWriter::SpillFileWriter(const std::shared_ptr<SpillFile>& spill_file, R
7575
}
7676

7777
SpillFileWriter::~SpillFileWriter() {
78-
if (_closed) {
79-
return;
78+
if (!_closed) {
79+
Status st = close();
80+
if (!st.ok()) {
81+
LOG(WARNING) << "SpillFileWriter::~SpillFileWriter() failed: " << st.to_string()
82+
<< ", spill_dir=" << _spill_dir;
83+
}
84+
}
85+
// A failed close() leaves this writer registered; the SpillFile must not reach it anymore.
86+
if (auto spill_file = _spill_file_wptr.lock();
87+
spill_file && spill_file->_active_writer == this) {
88+
spill_file->_active_writer = nullptr;
8089
}
81-
Status st = close();
82-
if (!st.ok()) {
83-
LOG(WARNING) << "SpillFileWriter::~SpillFileWriter() failed: " << st.to_string()
84-
<< ", spill_dir=" << _spill_dir;
90+
}
91+
92+
void SpillFileWriter::_discard(SpillFile* spill_file) {
93+
DCHECK_EQ(spill_file->_active_writer, this);
94+
spill_file->_active_writer = nullptr;
95+
_closed = true;
96+
if (!_file_writer) {
97+
return;
8598
}
99+
// The part is dropped without a footer and without close(): destroying an unclosed
100+
// writer waits for its in-flight uploads and publishes nothing, so the ledger below is
101+
// complete afterwards and no object can appear after the SpillFile deleted its prefix.
102+
std::unique_ptr<io::FileWriter> writer = std::move(_file_writer);
103+
MultipartUploadId upload = _multipart_upload_id(writer.get());
104+
writer.reset();
105+
_reconcile_part();
106+
_abort_multipart_upload(upload);
107+
_advance_part();
86108
}
87109

88110
Status SpillFileWriter::_open_next_part(const std::shared_ptr<SpillFile>& spill_file) {
@@ -192,6 +214,20 @@ Status SpillFileWriter::_close_current_part(const std::shared_ptr<SpillFile>& sp
192214
writer.reset();
193215
}
194216

217+
_reconcile_part();
218+
219+
if (!status.ok()) {
220+
LOG(WARNING) << "failed to close spill part " << _current_part_path << ": " << status;
221+
_abort_multipart_upload(upload);
222+
} else if (spill_file) {
223+
spill_file->add_part(_part_written_bytes);
224+
}
225+
226+
_advance_part();
227+
return status;
228+
}
229+
230+
void SpillFileWriter::_reconcile_part() {
195231
// Budget: everything the gate took for this part but the upload callback never gave
196232
// back (buffers that failed before their upload started) is released here. The writer is
197233
// in its final state at this point, so every callback that will ever fire has fired.
@@ -225,24 +261,16 @@ Status SpillFileWriter::_close_current_part(const std::shared_ptr<SpillFile>& sp
225261
ExecEnv::GetInstance()->spill_file_mgr()->update_spill_remote_write(uploaded_bytes,
226262
data_requests);
227263
}
264+
}
228265

229-
if (!status.ok()) {
230-
LOG(WARNING) << "failed to close spill part " << _current_part_path << ": " << status;
231-
_abort_multipart_upload(upload);
232-
} else if (spill_file) {
233-
spill_file->add_part(_part_written_bytes);
234-
}
235-
236-
// Advance to next part
266+
void SpillFileWriter::_advance_part() {
237267
++_current_part_index;
238268
_part_written_blocks = 0;
239269
_part_written_bytes = 0;
240270
_part_max_sub_block_size = 0;
241271
_part_meta.clear();
242272
_part_ledger.reset();
243273
_part_stats.reset();
244-
245-
return status;
246274
}
247275

248276
SpillFileWriter::MultipartUploadId SpillFileWriter::_multipart_upload_id(io::FileWriter* writer) {

‎be/src/exec/spill/spill_file_writer.h‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -44,9 +44,8 @@ class SpillRemoteUploadBudget;
4444
///
4545
/// Part rotation is fully internal. Each part file has its own footer with
4646
/// block offset metadata. Parts are named 0, 1, 2, ... within the SpillFile's
47-
/// directory. Parts are closed non-blocking (FileWriter::close(true)) so that the
48-
/// upload of a finished part overlaps with writing the next one; close() waits for
49-
/// all of them.
47+
/// directory. A part is closed synchronously when it is rotated out or when close() is called.
48+
/// If the SpillFile is destroyed before the writer is closed, the unfinished part is discarded.
5049
///
5150
/// Files are created on the SpillDataDir's file system, which is either a local disk or
5251
/// the object storage of a cloud storage vault. For object storage the writer additionally
@@ -69,6 +68,8 @@ class SpillFileWriter {
6968
Status close();
7069

7170
private:
71+
friend class SpillFile;
72+
7273
/// Remote only: budget bytes taken and given back for one part. Updated from the
7374
/// appending thread (gate) and the upload threads (done callback); shared by value with
7475
/// the FileWriterOptions lambdas so that it outlives the writer.
@@ -88,6 +89,18 @@ class SpillFileWriter {
8889
/// or not.
8990
Status _close_current_part(const std::shared_ptr<SpillFile>& spill_file);
9091

92+
/// Budget and statistics of the current part, once its writer reached a final state.
93+
void _reconcile_part();
94+
95+
/// Reset the per-part state and move on to the next part index.
96+
void _advance_part();
97+
98+
/// Called by SpillFile::gc() when the SpillFile goes away before this writer was closed
99+
/// (e.g. an error unwound the owner of the file first). Drops the current part without
100+
/// a footer and without publishing it, and unregisters from the SpillFile, so that nothing
101+
/// is charged to or written under a SpillFile that no longer exists.
102+
void _discard(SpillFile* spill_file);
103+
91104
/// If current part size >= _max_part_size, close it.
92105
Status _rotate_if_needed(const std::shared_ptr<SpillFile>& spill_file);
93106

‎be/src/io/fs/file_writer.h‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,10 @@ struct FileCacheAllocatorBuilder;
3636
struct EncryptionInfo;
3737

3838
// Request statistics reported by remote file writers when the caller passes an instance
39-
// through FileWriterOptions::remote_write_stats. All fields are cumulative.
39+
// through FileWriterOptions::remote_write_stats. All fields are cumulative. They count logical
40+
// requests, one per call into the object storage client: attempts retried inside the client
41+
// (throttling, transient errors) are not seen by the writer and not counted, so these are
42+
// observability numbers, not the provider's billed request count.
4043
struct RemoteWriteStats {
4144
std::atomic<int64_t> put_object_requests {0};
4245
std::atomic<int64_t> create_multipart_requests {0};

‎be/src/runtime/runtime_profile_counter_names.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,8 @@ inline constexpr char SPILL_WRITE_BLOCK_COUNT[] = "SpillWriteBlockCount";
100100
inline constexpr char SPILL_WRITE_BLOCK_BYTES[] = "SpillWriteBlockBytes";
101101
inline constexpr char SPILL_WRITE_ROWS[] = "SpillWriteRows";
102102
// Remote (object storage) spill write counters. Zero when spill is written to local disks.
103+
// Request counters count logical requests; retries inside the object storage client are not
104+
// included.
103105
inline constexpr char SPILL_REMOTE_WRITE_REQUESTS[] = "SpillRemoteWriteRequests";
104106
inline constexpr char SPILL_REMOTE_UPLOAD_PART_REQUESTS[] = "SpillRemoteUploadPartRequests";
105107
inline constexpr char SPILL_REMOTE_UPLOAD_BYTES[] = "SpillRemoteUploadBytes";

‎be/test/vec/spill/spill_file_s3_test.cpp‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1479,6 +1479,71 @@ TEST_F(SpillFileS3Test, CancelledQueryCloseAbortsOpenMultipart) {
14791479
ASSERT_EQ(_data_dir->get_spill_data_bytes(), 0);
14801480
}
14811481

1482+
// A writer can outlive its SpillFile when an error unwinds the owner of the file first (the
1483+
// repartitioner keeps its writers while the output files live in the caller). Destroying the
1484+
// file discards the unfinished part: the multipart upload is aborted, no footer is charged,
1485+
// and destroying the writer afterwards publishes nothing under the deleted prefix.
1486+
TEST_F(SpillFileS3Test, SpillFileDestroyedBeforeWriterDiscardsPart) {
1487+
config::spill_file_part_size_bytes = 1024 * 1024;
1488+
_create_manager();
1489+
auto* budget = _manager->remote_upload_budget();
1490+
std::mt19937 rng(47);
1491+
SpillFileSPtr spill_file;
1492+
ASSERT_TRUE(_manager->create_spill_file("query_16/join-1-0-1", spill_file).ok());
1493+
SpillFileWriterSPtr writer;
1494+
ASSERT_TRUE(spill_file->create_writer(_runtime_state.get(), _profile.get(), writer).ok());
1495+
for (int i = 0; i < 4; ++i) {
1496+
ASSERT_TRUE(writer->write_block(_runtime_state.get(), _random_string_block(rng, 100, 100))
1497+
.ok());
1498+
}
1499+
ASSERT_EQ(mock_store().create_multipart_requests, 1);
1500+
ASSERT_GT(_data_dir->get_spill_data_bytes(), 0);
1501+
1502+
spill_file.reset();
1503+
ASSERT_EQ(mock_store().abort_multipart_requests, 1);
1504+
ASSERT_EQ(_data_dir->get_spill_data_bytes(), 0);
1505+
ASSERT_EQ(budget->inflight_bytes(), 0);
1506+
ASSERT_EQ(budget->total_acquired_bytes(), budget->total_released_bytes());
1507+
1508+
const int64_t puts_before = mock_store().put_requests;
1509+
ASSERT_TRUE(writer->close().ok());
1510+
writer.reset();
1511+
ASSERT_EQ(mock_store().put_requests, puts_before);
1512+
ASSERT_TRUE(mock_store().keys_with_prefix(kBucket, spill_root() + "/query_16/").empty());
1513+
ASSERT_EQ(_data_dir->get_spill_data_bytes(), 0);
1514+
}
1515+
1516+
// A writer whose close() failed stays registered with its SpillFile; destroying the file
1517+
// afterwards must neither touch the writer's (already reconciled) state nor leak capacity, and
1518+
// the writer destroyed first must not leave a dangling registration behind.
1519+
TEST_F(SpillFileS3Test, FailedCloseThenSpillFileDestroyed) {
1520+
config::spill_file_part_size_bytes = 1024 * 1024;
1521+
_create_manager();
1522+
std::mt19937 rng(53);
1523+
for (bool writer_first : {false, true}) {
1524+
SpillFileSPtr spill_file;
1525+
ASSERT_TRUE(_manager->create_spill_file(fmt::format("query_17/sort-1-0-{}", writer_first),
1526+
spill_file)
1527+
.ok());
1528+
SpillFileWriterSPtr writer;
1529+
ASSERT_TRUE(spill_file->create_writer(_runtime_state.get(), _profile.get(), writer).ok());
1530+
ASSERT_TRUE(writer->write_block(_runtime_state.get(), _random_string_block(rng, 100, 100))
1531+
.ok());
1532+
mock_store().fail_uploads = true;
1533+
ASSERT_FALSE(writer->close().ok());
1534+
mock_store().fail_uploads = false;
1535+
ASSERT_FALSE(spill_file->ready_for_reading());
1536+
if (writer_first) {
1537+
writer.reset();
1538+
spill_file.reset();
1539+
} else {
1540+
spill_file.reset();
1541+
writer.reset();
1542+
}
1543+
ASSERT_EQ(_data_dir->get_spill_data_bytes(), 0);
1544+
}
1545+
}
1546+
14821547
// The scenario behind the fast-path cancellation check: a query cancelled before close() must
14831548
// not start a new upload for its last buffer; the part is refused, nothing is put, the budget
14841549
// stays balanced.

0 commit comments

Comments
 (0)