Skip to content

Commit a78aeb4

Browse files
committed
[fix](be) Ignore Iceberg position deletes when DV exists
### What problem does this PR solve? Issue Number: None Related PR: #65281 Problem Summary: The format v2 Iceberg table reader parsed deletion vector rows and then still merged position delete files for the same data file. Iceberg scan planning applies position delete files only when there is no deletion vector for the data file, so this could filter extra rows in scanner v2. This commit keeps the deletion vector rows and skips position delete files when a deletion vector is present, while continuing to initialize equality delete predicates. The v2 unit test now asserts that position delete rows are ignored in this case. ### Release note None ### Check List (For Author) - Test: Unit Test - Remote BE UT: ./run-be-ut.sh --run --filter="IcebergV2ReaderTest.IcebergTableReaderIgnoresPositionDeleteFilesWhenDeletionVectorPresent:IcebergDeleteFileReaderHelperTest.DeletionVectorCacheKey*" - Behavior changed: No - Does this need documentation: No
1 parent 50cb8ba commit a78aeb4

2 files changed

Lines changed: 9 additions & 8 deletions

File tree

‎be/src/format_v2/table/iceberg_reader.cpp‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -360,16 +360,17 @@ Status IcebergTableReader::_init_delete_predicates(const TTableFormatFileDesc& t
360360
equality_delete_files.push_back(delete_file);
361361
}
362362
}
363-
// `_delete_rows != nullptr` means DeleteVector is parsed
363+
// `_delete_rows != nullptr` means a deletion vector is parsed. Per Iceberg scan planning,
364+
// position delete files apply only when there is no deletion vector for the data file.
364365
if (_delete_rows != nullptr) {
365366
_position_delete_rows_storage = *_delete_rows;
366367
_delete_rows = &_position_delete_rows_storage;
368+
position_delete_files.clear();
367369
}
368-
// Combine position delete rows from both deletion vector and position delete files, and
369-
// initialize equality delete predicates. Position delete files contain row positions of
370-
// deleted rows, which can be directly added to `_delete_rows`. Equality delete files contain
371-
// values of deleted rows, which require reading the files and building predicates for later
372-
// filtering.
370+
// Initialize position and equality delete predicates. Position delete files contain row
371+
// positions of deleted rows, which can be directly added to `_delete_rows`. Equality delete
372+
// files contain values of deleted rows, which require reading the files and building
373+
// predicates for later filtering.
373374
if (!position_delete_files.empty()) {
374375
RETURN_IF_ERROR(_init_position_delete_rows(position_delete_files));
375376
}

‎be/test/format_v2/table/iceberg_reader_test.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1788,7 +1788,7 @@ TEST(IcebergV2ReaderTest, IcebergTableReaderAppliesPositionDeleteFile) {
17881788
std::filesystem::remove_all(test_dir);
17891789
}
17901790

1791-
TEST(IcebergV2ReaderTest, IcebergTableReaderMergesDeletionVectorAndPositionDeleteFiles) {
1791+
TEST(IcebergV2ReaderTest, IcebergTableReaderIgnoresPositionDeleteFilesWhenDeletionVectorPresent) {
17921792
const auto test_dir =
17931793
std::filesystem::temp_directory_path() / "doris_iceberg_delete_files_merge_test";
17941794
std::filesystem::remove_all(test_dir);
@@ -1831,7 +1831,7 @@ TEST(IcebergV2ReaderTest, IcebergTableReaderMergesDeletionVectorAndPositionDelet
18311831
make_iceberg_position_delete_file(position_delete_path)}));
18321832
ASSERT_TRUE(reader.prepare_split(split_options).ok());
18331833

1834-
EXPECT_EQ(read_iceberg_ids(&reader, projected_columns), std::vector<int32_t>({2, 3, 5}));
1834+
EXPECT_EQ(read_iceberg_ids(&reader, projected_columns), std::vector<int32_t>({2, 3, 4, 5}));
18351835

18361836
ASSERT_TRUE(reader.close().ok());
18371837
std::filesystem::remove_all(test_dir);

0 commit comments

Comments
 (0)