Skip to content

Commit 8d2bd1e

Browse files
authored
[fix](iceberg) Publish Parquet split offsets in data file metadata (apache#69615)
### What problem does this PR solve? Doris writes Parquet row groups for Iceberg data files but does not publish their split offsets in commit fragments or manifests. Readers cannot use those known boundaries when planning splits. Collect row-group starts from the closed writer's in-memory Footer, including dictionary pages, and carry them through an optional Thrift field into Iceberg data-file metadata. Keep collection independent of column metrics policy; leave older BE fragments, empty files, ORC and delete files without Parquet offsets. When a file metadata-table scan requests `split_offsets`, also project `file_size_in_bytes`: Iceberg uses the size to validate offsets and otherwise returns null for valid values. Append this dependency after the requested columns to preserve the BE JNI reader's positional contract. ### Release note Publish Parquet row-group split offsets in Iceberg manifests and preserve them in projected file metadata-table scans. ### Validation - Maven reactor build passed: 212 `IcebergScanPlanProviderTest` tests and 31 `IcebergWriterHelperTest` tests, including Manifest round trips and per-file isolation. - The projection regression failed before the fix and passes afterward. It covers files/data_files/all_files/all_data_files, partitioned/unpartitioned tables and five requested-column orders. - 6 targeted BE ASAN tests passed for dictionary/plain pages, multiple/single/empty row groups, metrics-disabled Parquet commits and unchanged ORC behavior. Changed source, generated Thrift and unity objects were linked with cached engine libraries; this was not a clean full BE rebuild. - FE Checkstyle, clang-format 16, BE build hygiene and connector import checks passed. - Checked the optional Thrift field ID against master and release branches 3.1, 4.0, 4.1 and 4.2. - Extended the partitioned-write external regression to assert split offsets, including with column metrics disabled. CI validation of the latest revision and Spark/Trino interoperability remain pending. ### Check List (For Author) - Test: Unit Test and Regression test - Behavior changed: Yes. Newly written Iceberg Parquet data files publish optional split offsets, which projected file metadata scans can return. - Does this need documentation? No.
1 parent 72738b4 commit 8d2bd1e

11 files changed

Lines changed: 295 additions & 7 deletions

File tree

‎be/src/exec/sink/writer/iceberg/viceberg_partition_writer.cpp‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,16 @@ Status VIcebergPartitionWriter::_build_iceberg_commit_data(TIcebergCommitData* c
217217
commit_data->__set_file_size(_file_format_transformer->written_len());
218218
commit_data->__set_file_content(TFileContent::DATA);
219219
commit_data->__set_partition_values(_partition_values);
220+
// Split offsets describe the physical file, even when the table disables all column metrics.
221+
if (_file_format_type == TFileFormatType::FORMAT_PARQUET) {
222+
std::vector<int64_t> offsets;
223+
RETURN_IF_ERROR(static_cast<VIcebergParquetWriter*>(_file_format_transformer.get())
224+
->collect_split_offsets_after_close(&offsets));
225+
// Empty files have no splits; leave the optional field absent, as with an older BE.
226+
if (!offsets.empty()) {
227+
commit_data->__set_split_offsets(offsets);
228+
}
229+
}
220230
// ORC collection reopens the file, so honor the FE policy before any footer work.
221231
if (!_collect_column_stats) {
222232
return Status::OK();

‎be/src/format/transformer/viceberg_parquet_writer.cpp‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,32 @@ std::unique_ptr<ArrowBlockConvertor> VIcebergParquetWriter::_create_arrow_block_
115115
_iceberg_schema, &_iceberg_schema_json, timezone_name, timezone);
116116
}
117117

118+
Status VIcebergParquetWriter::collect_split_offsets_after_close(std::vector<int64_t>* offsets) {
119+
auto metadata = _file_metadata();
120+
// Arrow publishes file metadata only after Close(), so offsets cannot be collected earlier.
121+
if (metadata == nullptr) {
122+
return Status::InternalError("File metadata is not available");
123+
}
124+
offsets->clear();
125+
offsets->reserve(metadata->num_row_groups());
126+
for (int i = 0; i < metadata->num_row_groups(); ++i) {
127+
auto row_group = metadata->RowGroup(i);
128+
// Empty files/groups have no page boundary to advertise as a scan split.
129+
if (row_group->num_rows() == 0 || row_group->num_columns() == 0) {
130+
continue;
131+
}
132+
auto first_column = row_group->ColumnChunk(0);
133+
int64_t offset = first_column->data_page_offset();
134+
// A split must include the dictionary page, not just the first data page that references it.
135+
if (first_column->has_dictionary_page()) {
136+
offset = std::min(offset, first_column->dictionary_page_offset());
137+
}
138+
offsets->push_back(offset);
139+
}
140+
std::sort(offsets->begin(), offsets->end());
141+
return Status::OK();
142+
}
143+
118144
Status VIcebergParquetWriter::collect_file_statistics_after_close(TIcebergColumnStats* stats) {
119145
std::shared_ptr<::parquet::FileMetaData> file_metadata = _file_metadata();
120146
if (file_metadata == nullptr) {

‎be/src/format/transformer/viceberg_parquet_writer.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,8 @@ class VIcebergParquetWriter final : public VParquetWriter {
4444

4545
Status collect_file_statistics_after_close(TIcebergColumnStats* stats);
4646

47+
Status collect_split_offsets_after_close(std::vector<int64_t>* offsets);
48+
4749
protected:
4850
std::unique_ptr<ArrowBlockConvertor> _create_arrow_block_convertor(
4951
DataTypes types, std::vector<std::string> names, const std::string& timezone_name,

‎be/test/exec/sink/writer/iceberg/iceberg_partition_writer_test.cpp‎

Lines changed: 62 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,16 +15,24 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18+
#include <arrow/memory_pool.h>
1819
#include <gtest/gtest.h>
1920

2021
#include <optional>
2122

23+
#include "core/data_type/data_type_number.h"
2224
#include "exec/sink/writer/iceberg/viceberg_partition_writer.h"
2325
#include "exec/sink/writer/iceberg/viceberg_sort_writer.h"
26+
#include "format/table/iceberg/schema_parser.h"
27+
#include "format/transformer/viceberg_parquet_writer.h"
2428
#include "io/fs/local_file_system.h"
29+
#include "runtime/exec_env.h"
2530
#include "testutil/mock/mock_descriptors.h"
2631
#include "testutil/mock/mock_runtime_state.h"
2732
#include "testutil/mock/mock_slot_ref.h"
33+
#include "util/defer_op.h"
34+
#include "util/timezone_utils.h"
35+
#include "util/uid_util.h"
2836

2937
namespace doris {
3038

@@ -73,13 +81,14 @@ class VIcebergPartitionWriterTest : public testing::Test {
7381
static std::unique_ptr<VIcebergPartitionWriter> make_writer(
7482
const TDataSink& sink, const VExprContextSPtrs& output_exprs,
7583
const iceberg::Schema& schema, const std::string* schema_json,
76-
const std::map<std::string, std::string>& hadoop_conf) {
84+
const std::map<std::string, std::string>& hadoop_conf,
85+
TFileFormatType::type format = TFileFormatType::FORMAT_ORC) {
7786
IPartitionWriterBase::WriteInfo write_info;
7887
write_info.file_type = TFileType::FILE_LOCAL;
7988
return std::make_unique<VIcebergPartitionWriter>(
8089
sink, std::vector<std::string> {}, output_exprs, schema, schema_json,
81-
std::vector<std::string> {}, std::move(write_info), "data", 0,
82-
TFileFormatType::FORMAT_ORC, TFileCompressType::ZLIB, hadoop_conf);
90+
std::vector<std::string> {}, std::move(write_info), "data", 0, format,
91+
TFileCompressType::ZLIB, hadoop_conf);
8392
}
8493

8594
static void install_fake_transformer(VIcebergPartitionWriter* writer,
@@ -90,6 +99,11 @@ class VIcebergPartitionWriterTest : public testing::Test {
9099
output_exprs, std::move(close_status), destroyed);
91100
}
92101

102+
static void install_parquet_transformer(VIcebergPartitionWriter* writer,
103+
std::unique_ptr<VIcebergParquetWriter> transformer) {
104+
writer->_file_format_transformer = std::move(transformer);
105+
}
106+
93107
static Status build_commit_data(VIcebergPartitionWriter* writer,
94108
TIcebergCommitData* commit_data) {
95109
return writer->_build_iceberg_commit_data(commit_data);
@@ -114,6 +128,51 @@ TEST_F(VIcebergPartitionWriterTest, OrcSkipsFooterCollectionWhenMetricsAreDisabl
114128
EXPECT_FALSE(commit_data.__isset.column_stats);
115129
}
116130

131+
TEST_F(VIcebergPartitionWriterTest, ParquetKeepsSplitOffsetsWhenColumnMetricsAreDisabled) {
132+
TimezoneUtils::load_timezones_to_cache();
133+
auto* env = ExecEnv::GetInstance();
134+
auto* previous_pool = env->_arrow_memory_pool;
135+
env->_arrow_memory_pool = arrow::default_memory_pool();
136+
const auto path = "./iceberg_split_offsets_" + UniqueId::gen_uid().to_string() + ".parquet";
137+
auto fs = io::global_local_filesystem();
138+
Defer cleanup([&] {
139+
env->_arrow_memory_pool = previous_pool;
140+
static_cast<void>(fs->delete_file(path));
141+
});
142+
auto type = std::make_shared<DataTypeInt32>();
143+
auto output_exprs = MockSlotRef::create_mock_contexts(DataTypes {type});
144+
const std::string json =
145+
R"({"type":"struct","fields":[{"id":1,"name":"value","required":true,"type":"int"}]})";
146+
auto schema = iceberg::SchemaParser::from_json(json);
147+
io::FileWriterPtr file_writer;
148+
ASSERT_TRUE(fs->create_file(path, &file_writer).ok());
149+
RuntimeState state;
150+
state.set_timezone("UTC");
151+
ParquetFileOptions options {.compression_type = TParquetCompressionType::SNAPPY,
152+
.parquet_version = TParquetVersion::PARQUET_1_0,
153+
.parquet_disable_dictionary = false,
154+
.enable_int96_timestamps = false};
155+
auto transformer = std::make_unique<VIcebergParquetWriter>(
156+
&state, file_writer.get(), output_exprs, std::vector<std::string> {"value"}, false,
157+
options, &json, *schema);
158+
ASSERT_TRUE(transformer->open().ok());
159+
auto column = type->create_column();
160+
column->insert(Field::create_field<TYPE_INT>(7));
161+
Block block;
162+
block.insert({std::move(column), type, "value"});
163+
ASSERT_TRUE(transformer->write(block).ok());
164+
ASSERT_TRUE(transformer->close().ok());
165+
auto writer = make_writer(make_table_sink(false), output_exprs, *schema, &json, {},
166+
TFileFormatType::FORMAT_PARQUET);
167+
install_parquet_transformer(writer.get(), std::move(transformer));
168+
TIcebergCommitData commit;
169+
ASSERT_TRUE(build_commit_data(writer.get(), &commit).ok());
170+
EXPECT_FALSE(commit.__isset.column_stats);
171+
ASSERT_TRUE(commit.__isset.split_offsets);
172+
ASSERT_EQ(1, commit.split_offsets.size());
173+
EXPECT_EQ(4, commit.split_offsets[0]);
174+
}
175+
117176
TEST_F(VIcebergPartitionWriterTest, MissingPolicyKeepsCollectionEnabledForRollingUpgrade) {
118177
VExprContextSPtrs output_exprs;
119178
iceberg::Schema schema(std::vector<iceberg::NestedField> {});

‎be/test/format/transformer/vparquet_writer_test.cpp‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
#include <optional>
2929
#include <string_view>
3030

31+
#include "common/config.h"
3132
#include "core/block/block.h"
3233
#include "core/column/column_array.h"
3334
#include "core/column/column_decimal.h"
@@ -50,6 +51,7 @@
5051
#include "runtime/exec_env.h"
5152
#include "runtime/runtime_state.h"
5253
#include "testutil/mock/mock_slot_ref.h"
54+
#include "util/defer_op.h"
5355
#include "util/timezone_utils.h"
5456
#include "util/uid_util.h"
5557

@@ -128,11 +130,75 @@ class VParquetWriterTest : public testing::Test {
128130
}
129131
}
130132

133+
void check_split_offsets(int row_groups, bool dictionary) {
134+
const auto previous_row_group_size = config::min_row_group_size;
135+
config::min_row_group_size = 0;
136+
Defer restore_row_group_size([&] { config::min_row_group_size = previous_row_group_size; });
137+
auto type = std::make_shared<DataTypeInt32>();
138+
auto output_exprs = MockSlotRef::create_mock_contexts(DataTypes {type});
139+
const std::string json =
140+
R"({"type":"struct","fields":[{"id":1,"name":"value","required":true,"type":"int"}]})";
141+
auto schema = iceberg::SchemaParser::from_json(json);
142+
io::FileWriterPtr file_writer;
143+
ASSERT_TRUE(_fs->create_file(_file_path, &file_writer).ok());
144+
RuntimeState state;
145+
state.set_timezone("UTC");
146+
ParquetFileOptions options {.compression_type = TParquetCompressionType::SNAPPY,
147+
.parquet_version = TParquetVersion::PARQUET_1_0,
148+
.parquet_disable_dictionary = !dictionary,
149+
.enable_int96_timestamps = false};
150+
VIcebergParquetWriter writer(&state, file_writer.get(), output_exprs, {"value"}, false,
151+
options, &json, *schema);
152+
ASSERT_TRUE(writer.open().ok());
153+
for (int group = 0; group < row_groups; ++group) {
154+
auto column = type->create_column();
155+
for (int row = 0; row < 100; ++row) {
156+
column->insert(Field::create_field<TYPE_INT>(row % 3));
157+
}
158+
Block block;
159+
block.insert({std::move(column), type, "value"});
160+
ASSERT_TRUE(writer.write(block).ok());
161+
}
162+
ASSERT_TRUE(writer.close().ok());
163+
std::vector<int64_t> offsets {999999};
164+
ASSERT_TRUE(writer.collect_split_offsets_after_close(&offsets).ok());
165+
ASSERT_EQ(row_groups, offsets.size());
166+
auto reader = ::parquet::ParquetFileReader::OpenFile(_file_path, false);
167+
auto metadata = reader->metadata();
168+
for (int i = 0; i < row_groups; ++i) {
169+
auto row_group = metadata->RowGroup(i);
170+
auto column = row_group->ColumnChunk(0);
171+
ASSERT_EQ(dictionary, column->has_dictionary_page());
172+
EXPECT_EQ(dictionary ? column->dictionary_page_offset() : column->data_page_offset(),
173+
offsets[i]);
174+
EXPECT_GT(offsets[i], i == 0 ? 0 : offsets[i - 1]);
175+
}
176+
std::vector<int64_t> repeated;
177+
ASSERT_TRUE(writer.collect_split_offsets_after_close(&repeated).ok());
178+
EXPECT_EQ(offsets, repeated);
179+
}
180+
131181
arrow::MemoryPool* _previous_pool = nullptr;
132182
std::string _file_path;
133183
std::shared_ptr<io::FileSystem> _fs;
134184
};
135185

186+
TEST_F(VParquetWriterTest, IcebergSplitOffsetsIncludeDictionaryPages) {
187+
check_split_offsets(3, true);
188+
}
189+
190+
TEST_F(VParquetWriterTest, IcebergSplitOffsetsWithoutDictionary) {
191+
check_split_offsets(3, false);
192+
}
193+
194+
TEST_F(VParquetWriterTest, IcebergSplitOffsetsSingleRowGroup) {
195+
check_split_offsets(1, true);
196+
}
197+
198+
TEST_F(VParquetWriterTest, IcebergSplitOffsetsEmptyFile) {
199+
check_split_offsets(0, true);
200+
}
201+
136202
TEST_F(VParquetWriterTest, WritesIcebergPrimitiveAndCollectsMetrics) {
137203
auto type = make_nullable(std::make_shared<DataTypeInt32>());
138204
auto output_exprs = MockSlotRef::create_mock_contexts(DataTypes {type});

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -982,7 +982,7 @@ private List<ConnectorScanRange> doPlanSystemTableScan(IcebergTableHandle handle
982982
return doPlanPositionDeletesSystemTableScan(handle, metadataTable, columns, filter, session);
983983
}
984984
TableScan scan = buildScan(metadataTable, handle, filter, session);
985-
// Project the metadata-table scan to ONLY the requested columns, in the SAME order BE lists them in
985+
// Project the metadata-table scan to the requested columns first, in the SAME order BE lists them in
986986
// required_fields (== the scan slot order: buildColumnHandles iterates desc.getSlots(), BE builds
987987
// required_fields from the matching file_slot_descs). This mirrors legacy IcebergScanNode
988988
// .getSystemTableProjectedSchema() + scan.project(...). Two reasons the projection is mandatory AND must
@@ -1011,6 +1011,13 @@ private List<ConnectorScanRange> doPlanSystemTableScan(IcebergTableHandle handle
10111011
projectedFields.add(field);
10121012
}
10131013
}
1014+
// Iceberg returns split_offsets only when the last offset is smaller than file_size_in_bytes.
1015+
// Without that dependency, even valid offsets such as [4, 2048] are hidden as null. Append it
1016+
// after the requested columns so BE's positional reads keep their original order.
1017+
if (seenFieldIds.contains(DataFile.SPLIT_OFFSETS.fieldId())
1018+
&& seenFieldIds.add(DataFile.FILE_SIZE.fieldId())) {
1019+
projectedFields.add(metadataSchema.findField(DataFile.FILE_SIZE.fieldId()));
1020+
}
10141021
scan = scan.project(new Schema(projectedFields));
10151022
}
10161023
List<ConnectorScanRange> ranges = new ArrayList<>();

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergWriterHelper.java‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ private static WriteResult convertToWriterResult(
184184
partitionData = Optional.of(convertToPartitionData(partitionValues, spec, zone));
185185
}
186186
DataFile dataFile = genDataFile(fileFormat, location, spec, partitionData, recordCount, fileSize,
187-
metrics, sortOrder);
187+
metrics, sortOrder, commitData.getSplitOffsets());
188188
dataFiles.add(dataFile);
189189
}
190190
return WriteResult.builder()
@@ -194,14 +194,18 @@ private static WriteResult convertToWriterResult(
194194

195195
private static DataFile genDataFile(FileFormat format, String location, PartitionSpec spec,
196196
Optional<PartitionData> partitionData, long recordCount, long fileSize, Metrics metrics,
197-
SortOrder sortOrder) {
197+
SortOrder sortOrder, List<Long> splitOffsets) {
198198
DataFiles.Builder builder = DataFiles.builder(spec)
199199
.withPath(location)
200200
.withFileSizeInBytes(fileSize)
201201
.withRecordCount(recordCount)
202202
.withMetrics(metrics)
203203
.withSortOrder(sortOrder)
204204
.withFormat(format);
205+
// Older BE fragments omit this optional field; only Parquet data writers report these boundaries.
206+
if (format == FileFormat.PARQUET && splitOffsets != null && !splitOffsets.isEmpty()) {
207+
builder.withSplitOffsets(splitOffsets);
208+
}
205209
partitionData.ifPresent(builder::withPartition);
206210
return builder.build();
207211
}

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderTest.java‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3147,6 +3147,52 @@ public void planSystemTableScanProjectsRequestedColumnsExcludingReadableMetrics(
31473147
+ "BufferUnderflows on complex/boolean bound columns when materialised unrequested");
31483148
}
31493149

3150+
@Test
3151+
public void planSystemTableScanKeepsSplitOffsetsWithTheirFileSizeDependency() throws Exception {
3152+
List<Long> offsets = Arrays.asList(4L, 2048L);
3153+
for (PartitionSpec spec : Arrays.asList(PartitionSpec.unpartitioned(),
3154+
PartitionSpec.builderFor(SCHEMA).identity("id").build())) {
3155+
Table table = createTable("t1", SCHEMA, spec);
3156+
table.newAppend().appendFile(dataFile(table.spec(), "data.parquet", 4096, offsets,
3157+
spec.isPartitioned() ? "id=1" : null)).commit();
3158+
IcebergScanPlanProvider provider = new IcebergScanPlanProvider(
3159+
IcebergCatalogProperties.of(Collections.emptyMap()), opsReturning(table));
3160+
for (String type : Arrays.asList("files", "data_files", "all_files", "all_data_files")) {
3161+
for (List<String> names : Arrays.asList(Collections.singletonList("split_offsets"),
3162+
Arrays.asList("content", "split_offsets"), Arrays.asList("split_offsets", "content"),
3163+
Arrays.asList("split_offsets", "file_size_in_bytes"),
3164+
Arrays.asList("file_size_in_bytes", "split_offsets"))) {
3165+
List<ConnectorColumnHandle> columns = names.stream()
3166+
.map(name -> (ConnectorColumnHandle) new IcebergColumnHandle(name, 1))
3167+
.collect(Collectors.toList());
3168+
List<ConnectorScanRange> ranges = provider.planScan(null, ConnectorScanRequest.builder(
3169+
IcebergTableHandle.forSystemTable("db1", "t1", type, -1L, null, -1L), columns).build());
3170+
int rowCount = 0;
3171+
for (ConnectorScanRange range : ranges) {
3172+
FileScanTask task = IcebergSystemTableSerialization.deserializeFromBase64(
3173+
((IcebergScanRange) range).getSerializedSplit());
3174+
try (CloseableIterable<StructLike> rows = task.asDataTask().rows()) {
3175+
for (StructLike row : rows) {
3176+
Assertions.assertEquals(offsets, row.get(names.indexOf("split_offsets"), List.class),
3177+
type + " " + names);
3178+
if (names.contains("content")) {
3179+
Assertions.assertEquals(0, row.get(names.indexOf("content"), Integer.class));
3180+
}
3181+
if (names.contains("file_size_in_bytes")) {
3182+
Assertions.assertEquals(4096L,
3183+
row.get(names.indexOf("file_size_in_bytes"), Long.class));
3184+
}
3185+
rowCount++;
3186+
}
3187+
}
3188+
Assertions.assertNull(task.schema().findField("readable_metrics"));
3189+
}
3190+
Assertions.assertEquals(1, rowCount);
3191+
}
3192+
}
3193+
}
3194+
}
3195+
31503196
@Test
31513197
public void planSystemTableScanProjectionPreservesRequestedColumnOrderForPositionalBeRead() throws Exception {
31523198
// WHY (regression — External Regression build 1000283): BE's IcebergSysTableJniScanner reads the

0 commit comments

Comments
 (0)