Skip to content

Commit 0814e49

Browse files
authored
[feature](iceberg) Support Iceberg position deletes system table (#65135)
Checkpoint conclusions: Goal and scope: the patch adds native Iceberg $position_deletes metadata-table support across FE system-table resolution, split construction, Thrift scan metadata, BE scanner-v1/scanner-v2 readers, regression coverage, and unit tests. I did not find an out-of-scope behavioral change needing a new comment. Existing review context: prior comments already covered the earlier V1 IOContext/profile issue, smooth-upgrade guard, Avro unsupported format, partition JSON/type/evolution/rename concerns, and the ORC row assertion typo. I did not duplicate those threads. FE/BE protocol and compatibility: native $position_deletes ranges use top-level Iceberg content to route into the new readers, while ordinary Iceberg data ranges keep delete-file descriptors nested and do not match that route. The smooth-upgrade backend guard now runs before native position-delete ranges are generated. Data correctness and nullability: partition values are mapped by Iceberg partition field ID into the metadata-table partition struct, missing evolved fields become null, unsupported non-null binary/fixed/UUID partition values fail instead of silently producing wrong data, optional absent row payloads materialize as null, and deletion-vector rows are expanded from the referenced data file plus decoded positions. Scanner and lifecycle paths: scanner V1/V2 routing, Parquet/ORC position-delete files, Puffin deletion vectors, EOF/final-batch handling, IOContext propagation, profile/stat accounting, close/reset state, and row-count behavior were reviewed without a substantiated new defect. Tests and coverage: the changed regression and unit tests cover the main feature surface, including scanner V1/V2, Parquet/ORC deletes, V3 deletion-vector metadata, partition evolution/rename, typed partitions, time travel, empty-delete tables, and privilege checks. A possible Puffin-DV split concern was dismissed because Doris depends on Iceberg 1.10.1, where FileFormat.PUFFIN is non-splittable and BaseContentScanTask.split(...) returns the original task for non-splittable files. Concurrency, persistence, transactions, and config: no new shared mutable concurrency path, edit-log/persistence path, transaction/write path, or user-facing config was introduced by this read-only metadata scan feature. Observability/performance: existing scanner/profile counters remain in use; I did not find a concrete additional metrics requirement or hot-path regression.
1 parent f5c30d7 commit 0814e49

18 files changed

Lines changed: 3218 additions & 46 deletions

File tree

‎be/src/exec/scan/file_scanner.cpp‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@
6969
#include "format/table/hive_reader.h"
7070
#include "format/table/hudi_jni_reader.h"
7171
#include "format/table/hudi_reader.h"
72+
#include "format/table/iceberg_position_delete_sys_table_reader.h"
7273
#include "format/table/iceberg_reader.h"
7374
#include "format/table/iceberg_sys_table_jni_reader.h"
7475
#include "format/table/jdbc_jni_reader.h"
@@ -98,6 +99,20 @@ class ShardedKVCache;
9899
namespace doris {
99100
using namespace ErrorCode;
100101

102+
namespace {
103+
constexpr int kIcebergPositionDeleteContent = 1;
104+
constexpr int kIcebergDeletionVectorContent = 3;
105+
106+
bool is_iceberg_position_deletes_sys_table(const TFileRangeDesc& range) {
107+
return range.__isset.table_format_params &&
108+
range.table_format_params.table_format_type == "iceberg" &&
109+
range.table_format_params.__isset.iceberg_params &&
110+
range.table_format_params.iceberg_params.__isset.content &&
111+
(range.table_format_params.iceberg_params.content == kIcebergPositionDeleteContent ||
112+
range.table_format_params.iceberg_params.content == kIcebergDeletionVectorContent);
113+
}
114+
} // namespace
115+
101116
const std::string FileScanner::FileReadBytesProfile = "FileReadBytes";
102117
const std::string FileScanner::FileReadTimeProfile = "FileReadTime";
103118

@@ -1043,6 +1058,7 @@ Status FileScanner::_get_next_reader() {
10431058
// create reader for specific format
10441059
Status init_status = Status::OK();
10451060
TFileFormatType::type format_type = _get_current_format_type();
1061+
const bool is_position_deletes_sys_table = is_iceberg_position_deletes_sys_table(range);
10461062
// for compatibility, this logic is deprecated in 3.1
10471063
if (format_type == TFileFormatType::FORMAT_JNI && range.__isset.table_format_params) {
10481064
if (range.table_format_params.table_format_type == "paimon" &&
@@ -1168,6 +1184,17 @@ Status FileScanner::_get_next_reader() {
11681184
auto file_meta_cache_ptr = _should_enable_file_meta_cache()
11691185
? ExecEnv::GetInstance()->file_meta_cache()
11701186
: nullptr;
1187+
if (is_position_deletes_sys_table) {
1188+
ReaderInitContext ctx;
1189+
_fill_base_init_context(&ctx);
1190+
auto reader = IcebergPositionDeleteSysTableReader::create_unique(
1191+
_file_slot_descs, _state, _profile, range, _params, _io_ctx,
1192+
file_meta_cache_ptr);
1193+
init_status = static_cast<GenericReader*>(reader.get())->init_reader(&ctx);
1194+
_cur_reader = std::move(reader);
1195+
need_to_get_parsed_schema = false;
1196+
break;
1197+
}
11711198
if (push_down_predicates) {
11721199
RETURN_IF_ERROR(_process_late_arrival_conjuncts());
11731200
}
@@ -1180,6 +1207,17 @@ Status FileScanner::_get_next_reader() {
11801207
auto file_meta_cache_ptr = _should_enable_file_meta_cache()
11811208
? ExecEnv::GetInstance()->file_meta_cache()
11821209
: nullptr;
1210+
if (is_position_deletes_sys_table) {
1211+
ReaderInitContext ctx;
1212+
_fill_base_init_context(&ctx);
1213+
auto reader = IcebergPositionDeleteSysTableReader::create_unique(
1214+
_file_slot_descs, _state, _profile, range, _params, _io_ctx,
1215+
file_meta_cache_ptr);
1216+
init_status = static_cast<GenericReader*>(reader.get())->init_reader(&ctx);
1217+
_cur_reader = std::move(reader);
1218+
need_to_get_parsed_schema = false;
1219+
break;
1220+
}
11831221
if (push_down_predicates) {
11841222
RETURN_IF_ERROR(_process_late_arrival_conjuncts());
11851223
}

‎be/src/exec/scan/file_scanner_v2.cpp‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,7 @@
5555
#include "format_v2/jni/trino_connector_jni_reader.h"
5656
#include "format_v2/table/hive_reader.h"
5757
#include "format_v2/table/hudi_reader.h"
58+
#include "format_v2/table/iceberg_position_delete_sys_table_reader.h"
5859
#include "format_v2/table/iceberg_reader.h"
5960
#include "format_v2/table/paimon_reader.h"
6061
#include "format_v2/table/remote_doris_reader.h"
@@ -71,6 +72,9 @@
7172
namespace doris {
7273
namespace {
7374

75+
constexpr int kIcebergPositionDeleteContent = 1;
76+
constexpr int kIcebergDeletionVectorContent = 3;
77+
7478
std::string table_format_name(const TFileRangeDesc& range) {
7579
return range.__isset.table_format_params ? range.table_format_params.table_format_type
7680
: "NotSet";
@@ -111,6 +115,15 @@ bool is_supported_jni_table_format(const TFileRangeDesc& range) {
111115
table_format == "max_compute" || table_format == "trino_connector";
112116
}
113117

118+
bool is_iceberg_position_deletes_sys_table(const TFileRangeDesc& range) {
119+
return range.__isset.table_format_params &&
120+
range.table_format_params.table_format_type == "iceberg" &&
121+
range.table_format_params.__isset.iceberg_params &&
122+
range.table_format_params.iceberg_params.__isset.content &&
123+
(range.table_format_params.iceberg_params.content == kIcebergPositionDeleteContent ||
124+
range.table_format_params.iceberg_params.content == kIcebergDeletionVectorContent);
125+
}
126+
114127
bool is_csv_format(TFileFormatType::type format_type) {
115128
switch (format_type) {
116129
case TFileFormatType::FORMAT_CSV_PLAIN:
@@ -473,7 +486,9 @@ Status FileScannerV2::_create_table_reader_for_format(
473486
} else if (table_format == "hive") {
474487
*reader = format::hive::HiveReader::create_unique();
475488
} else if (table_format == "iceberg") {
476-
if (get_range_format_type(*_params, range) == TFileFormatType::FORMAT_JNI) {
489+
if (is_iceberg_position_deletes_sys_table(range)) {
490+
*reader = std::make_unique<format::iceberg::IcebergPositionDeleteSysTableV2Reader>();
491+
} else if (get_range_format_type(*_params, range) == TFileFormatType::FORMAT_JNI) {
477492
*reader = std::make_unique<format::iceberg::IcebergSysTableJniReader>();
478493
} else {
479494
*reader = std::make_unique<format::iceberg::IcebergTableReader>();

0 commit comments

Comments
 (0)