Skip to content

Commit 4aa737f

Browse files
committed
[fix](iceberg) Carry the timestamptz marker into the ancestor read of a missing equality key
### What problem does this PR solve? Issue Number: None Related PR: #69845 Problem Summary: When an Iceberg equality-delete key is a nested field that an older data file does not store, but the file does store the key's parent struct, `_build_missing_equality_delete_key_expr` reads the whole parent so that the key stays NULL under a NULL parent. It added that read through `_append_file_scan_column`, a bare top-level projection without timestamp markers. Earlier in this PR, `annotate_file_schema` started rewriting each marked timestamptz leaf of the Parquet file schema to TIMESTAMPTZ. The file block is built from that schema, but the Parquet reader takes the timestamp semantics only from the projection. Reproduction: an older Parquet file holds a nullable `payload {existing INT, ts INT96 timestamptz}` (Spark's encoding), the table later added `payload.k`, and a non-empty equality delete is keyed on `payload.k`. Scans of that file failed in `check_block_column_type`: Parquet read `payload` as STRUCT<existing, DATETIMEV2(6)> into a STRUCT<existing, TIMESTAMPTZ(6)> file block. Projecting `payload` did not help: adding the root as a predicate column replaced the projection its mapping had marked. Before this PR the file schema kept the physical types, so the same read succeeded. The fix moves the annotated projection the hidden-key path already built (`top_level` plus `attach_timestamp_semantics` from the annotated file column) into `equality_delete_root_projection()`. The missing key's ancestor, the hidden key and the delete file's keys all use it now. The ancestor is still added as a predicate column through `FileScanRequestBuilder`, so a root already in the request is merged or moved as before. A plan without markers builds the same request as before. ### Release note None ### Check List (For Author) - Test: Unit Test (to be run) - New `IcebergV2ReaderTest.IcebergMissingNestedEqualityKeyAncestorKeepsInt96TimestampTzMarker`: in Asia/Shanghai, with the key's parent holding an INT96 timestamptz sibling, checks the surviving rows with the parent hidden and projected, that the projected sibling reads as an instant, and a plan without the marker - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent f2b97b1 commit 4aa737f

2 files changed

Lines changed: 230 additions & 16 deletions

File tree

‎be/src/format_v2/table/iceberg_reader.cpp‎

Lines changed: 30 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1190,6 +1190,23 @@ static Status build_equality_delete_key_expr(
11901190
return build_nested_equality_delete_key_expr(path, std::move(root_expr), key_expr);
11911191
}
11921192

1193+
// Project the root file column of an equality key read without a ColumnMapping, carrying the
1194+
// timestamp markers annotate_file_schema() and _load_equality_delete_file() put on the file schema.
1195+
// The file block and the key expressions take the rewritten types from that schema, but the file
1196+
// reader learns them only from the projection. For example, an older file holds payload
1197+
// {existing INT, ts INT96 timestamptz} and the table later added payload.k: a delete on k reads
1198+
// `payload` for its NULLs, and a bare top-level projection makes Parquet read
1199+
// STRUCT<existing, DATETIMEV2(6)> into a STRUCT<existing, TIMESTAMPTZ(6)> file block, which fails
1200+
// the scan.
1201+
static format::LocalColumnIndex equality_delete_root_projection(
1202+
const format::ColumnDefinition& root) {
1203+
auto projection =
1204+
format::LocalColumnIndex::top_level(format::LocalColumnId(root.file_local_id()));
1205+
format::attach_timestamp_semantics(root.timestamp_is_adjusted_to_utc, root.children,
1206+
&projection);
1207+
return projection;
1208+
}
1209+
11931210
// Whether a missing key's default is NULL. `missing_path` runs from the missing root to the key,
11941211
// and the root's initial default is walked down it with the key expression's child matching. The
11951212
// root having a default is not enough: a missing `payload` defaulting to the struct `{}` takes
@@ -2037,7 +2054,10 @@ Status IcebergTableReader::_build_missing_equality_delete_key_expr(
20372054

20382055
const auto* root = data_path.front();
20392056
const auto field_column_id = format::LocalColumnId(root->file_local_id());
2040-
_append_file_scan_column(request, field_column_id, &request->predicate_columns);
2057+
// The whole ancestor is read for its NULLs, so a marked timestamp leaf inside it, such as an
2058+
// INT96 timestamptz sibling of the missing key, needs its marker as well.
2059+
RETURN_IF_ERROR(format::FileScanRequestBuilder(request).add_predicate_column(
2060+
equality_delete_root_projection(*root)));
20412061
const auto block_position = request->local_positions.at(field_column_id).value();
20422062
VExprSPtr ancestor_expr;
20432063
RETURN_IF_ERROR(build_equality_delete_key_expr(data_path, block_position, &ancestor_expr));
@@ -2079,16 +2099,14 @@ Status IcebergTableReader::_append_equality_delete_predicates(format::FileScanRe
20792099
} else {
20802100
const auto* root = data_path.front();
20812101
const auto field_column_id = format::LocalColumnId(root->file_local_id());
2082-
// A hidden key has no ColumnMapping to carry the timestamp marker that
2083-
// annotate_file_schema() put on the file column. Without it Parquet decodes an
2084-
// INT96 timestamptz key as DATETIMEV2 while the key expression below, built from
2085-
// the annotated file schema, expects TIMESTAMPTZ. A projected key moves into the
2086-
// predicate columns here and must keep the marker its mapping carried.
2087-
auto projection = format::LocalColumnIndex::top_level(field_column_id);
2088-
format::attach_timestamp_semantics(root->timestamp_is_adjusted_to_utc,
2089-
root->children, &projection);
2102+
// A hidden key, like the missing key's ancestor above and the delete file's keys,
2103+
// has no ColumnMapping to carry the timestamp marker annotate_file_schema() put on
2104+
// its file column. Without it Parquet decodes an INT96 timestamptz key as
2105+
// DATETIMEV2 while the key expression below, built from the annotated file schema,
2106+
// expects TIMESTAMPTZ. A projected key, or a projected ancestor of a missing key,
2107+
// moves into the predicate columns and must keep the marker its mapping carried.
20902108
RETURN_IF_ERROR(format::FileScanRequestBuilder(request).add_predicate_column(
2091-
std::move(projection)));
2109+
equality_delete_root_projection(*root)));
20922110
const auto block_position = request->local_positions.at(field_column_id).value();
20932111
RETURN_IF_ERROR(
20942112
build_equality_delete_key_expr(data_path, block_position, &key_expr));
@@ -2427,12 +2445,8 @@ Status IcebergTableReader::_load_equality_delete_file(const TIcebergDeleteFileDe
24272445
format::FileScanRequestBuilder request_builder(request.get());
24282446
for (const auto& path : delete_paths) {
24292447
DORIS_CHECK(!path.empty());
2430-
const auto* root = path.front();
2431-
auto projection =
2432-
format::LocalColumnIndex::top_level(format::LocalColumnId(root->file_local_id()));
2433-
format::attach_timestamp_semantics(root->timestamp_is_adjusted_to_utc, root->children,
2434-
&projection);
2435-
RETURN_IF_ERROR(request_builder.add_non_predicate_column(std::move(projection)));
2448+
RETURN_IF_ERROR(request_builder.add_non_predicate_column(
2449+
equality_delete_root_projection(*path.front())));
24362450
}
24372451
Block file_block_template;
24382452
std::vector<const format::ColumnDefinition*> roots(request->local_positions.size());

‎be/test/format_v2/table/iceberg_reader_test.cpp‎

Lines changed: 200 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1145,6 +1145,56 @@ void write_idless_wrapper_int96_parquet_file(const std::string& file_path,
11451145
::parquet::WriterProperties::Builder().build(), arrow_builder.build()));
11461146
}
11471147

1148+
// Writes `id` (field id 0) and a nullable `payload` STRUCT (1) of `existing` (2) and `ts` (4), with
1149+
// `ts` stored as INT96 the way Spark writes a timestamptz. A true `parent_nulls` entry makes that
1150+
// row's `payload` NULL.
1151+
void write_nested_equality_int96_sibling_parquet_file(const std::string& file_path,
1152+
const std::vector<int32_t>& ids,
1153+
const std::vector<int32_t>& values,
1154+
const std::vector<int64_t>& timestamps,
1155+
const std::vector<bool>& parent_nulls) {
1156+
ASSERT_EQ(ids.size(), values.size());
1157+
ASSERT_EQ(ids.size(), timestamps.size());
1158+
ASSERT_EQ(ids.size(), parent_nulls.size());
1159+
const auto field_id = [](int32_t id) {
1160+
return arrow::key_value_metadata({"PARQUET:field_id"}, {std::to_string(id)});
1161+
};
1162+
const auto timestamp_type = arrow::timestamp(arrow::TimeUnit::MICRO, "UTC");
1163+
const auto payload_type = arrow::struct_(
1164+
{arrow::field("existing", arrow::int32(), false)->WithMetadata(field_id(2)),
1165+
arrow::field("ts", timestamp_type, false)->WithMetadata(field_id(4))});
1166+
arrow::StructBuilder payload_builder(payload_type, arrow::default_memory_pool(),
1167+
{std::make_shared<arrow::Int32Builder>(),
1168+
std::make_shared<arrow::TimestampBuilder>(
1169+
timestamp_type, arrow::default_memory_pool())});
1170+
auto* existing_builder = assert_cast<arrow::Int32Builder*>(payload_builder.field_builder(0));
1171+
auto* ts_builder = assert_cast<arrow::TimestampBuilder*>(payload_builder.field_builder(1));
1172+
for (size_t row = 0; row < ids.size(); ++row) {
1173+
if (parent_nulls[row]) {
1174+
ASSERT_TRUE(payload_builder.AppendNull().ok());
1175+
} else {
1176+
ASSERT_TRUE(payload_builder.Append().ok());
1177+
ASSERT_TRUE(existing_builder->Append(values[row]).ok());
1178+
ASSERT_TRUE(ts_builder->Append(timestamps[row]).ok());
1179+
}
1180+
}
1181+
auto schema = arrow::schema({
1182+
arrow::field("id", arrow::int32(), false)->WithMetadata(field_id(0)),
1183+
arrow::field("payload", payload_type)->WithMetadata(field_id(1)),
1184+
});
1185+
auto table =
1186+
arrow::Table::Make(schema, {build_int32_array(ids), finish_array(&payload_builder)});
1187+
1188+
auto file_result = arrow::io::FileOutputStream::Open(file_path);
1189+
ASSERT_TRUE(file_result.ok()) << file_result.status();
1190+
std::shared_ptr<arrow::io::FileOutputStream> out = *file_result;
1191+
::parquet::ArrowWriterProperties::Builder arrow_builder;
1192+
arrow_builder.enable_force_write_int96_timestamps();
1193+
PARQUET_THROW_NOT_OK(::parquet::arrow::WriteTable(
1194+
*table, arrow::default_memory_pool(), out, static_cast<int64_t>(ids.size()),
1195+
::parquet::WriterProperties::Builder().build(), arrow_builder.build()));
1196+
}
1197+
11481198
void write_iceberg_equality_delete_bigint_parquet_file(const std::string& file_path,
11491199
int32_t field_id, int64_t value) {
11501200
const auto metadata =
@@ -4733,6 +4783,156 @@ TEST(IcebergV2ReaderTest, IcebergMissingNestedEqualityKeyPreservesNullableParent
47334783
}
47344784
}
47354785

4786+
// An older Parquet file holds a nullable payload {existing, ts}, `ts` an INT96 timestamptz as
4787+
// Spark writes it, and the table later added payload.k with the default 7. An equality delete on k
4788+
// reads the whole `payload` for its NULLs: rows under a non-NULL payload take the key 7, the row
4789+
// under the NULL one keeps a NULL key. FE's marker turns `ts` into TIMESTAMPTZ in the annotated
4790+
// file schema, so that ancestor read must carry the marker too, or Parquet still decodes `ts` as
4791+
// DATETIMEV2 and the scan fails the reader's type check. A query projecting `payload` failed the
4792+
// same way: reading the root as a predicate column replaced the projection its mapping had marked.
4793+
// A plan without the marker keeps the file types and reads as before.
4794+
// NOLINTNEXTLINE(readability-function-cognitive-complexity): marker, projection, tombstone cases.
4795+
TEST(IcebergV2ReaderTest, IcebergMissingNestedEqualityKeyAncestorKeepsInt96TimestampTzMarker) {
4796+
TimezoneUtils::load_timezones_to_cache();
4797+
const auto test_dir = std::filesystem::temp_directory_path() /
4798+
"doris_v2_missing_nested_equality_key_int96_sibling";
4799+
std::filesystem::remove_all(test_dir);
4800+
std::filesystem::create_directories(test_dir);
4801+
const auto file_path = (test_dir / "split.parquet").string();
4802+
// 2024-01-01 00:00:00.123456 UTC, then one and two seconds later.
4803+
constexpr int64_t T1_MICROS = 1'704'067'200'123'456L;
4804+
write_nested_equality_int96_sibling_parquet_file(
4805+
file_path, {1, 2, 3}, {10, 20, 30},
4806+
{T1_MICROS, T1_MICROS + 1'000'000L, T1_MICROS + 2'000'000L}, {false, true, false});
4807+
auto physical_reader = ::parquet::ParquetFileReader::OpenFile(file_path, false);
4808+
ASSERT_EQ(::parquet::Type::INT96,
4809+
physical_reader->metadata()->schema()->Column(2)->physical_type());
4810+
physical_reader->Close();
4811+
// The two delete files have the same size and may share an mtime second, so each tombstone
4812+
// gets its own path: one rewritten path could serve the first one from the Parquet caches.
4813+
const auto delete_file_path = [&test_dir](int32_t tombstone) {
4814+
return (test_dir / ("equality-delete-" + std::to_string(tombstone) + ".parquet")).string();
4815+
};
4816+
for (const int32_t tombstone : {7, 8}) {
4817+
write_nested_equality_parquet_file(delete_file_path(tombstone), {}, {tombstone}, {false},
4818+
true, "k", 3);
4819+
}
4820+
4821+
const auto int_type = std::make_shared<DataTypeInt32>();
4822+
const auto tz_type = std::make_shared<DataTypeTimeStampTz>(6);
4823+
const auto nullable_tz_type = make_nullable(tz_type);
4824+
auto payload = make_table_column(1, "payload",
4825+
std::make_shared<DataTypeStruct>(DataTypes {int_type, tz_type},
4826+
Strings {"existing", "ts"}));
4827+
payload.children = {make_table_column(2, "existing", int_type),
4828+
make_table_column(4, "ts", tz_type)};
4829+
4830+
struct Case {
4831+
bool marked;
4832+
bool project_payload;
4833+
int32_t tombstone;
4834+
std::vector<int32_t> ids;
4835+
// `ts` of each surviving row when `payload` is projected; nullopt under a NULL payload.
4836+
std::vector<std::optional<std::string>> timestamps;
4837+
};
4838+
const std::vector<Case> cases = {
4839+
// A plan from an FE that sends no marker.
4840+
{false, false, 7, {2}, {}},
4841+
// The key is the default 7 under a non-NULL payload and NULL under the NULL one.
4842+
{true, false, 7, {2}, {}},
4843+
{true, true, 7, {2}, {std::nullopt}},
4844+
// Instants, not session wall clocks read eight hours early in Asia/Shanghai.
4845+
{true,
4846+
true,
4847+
8,
4848+
{1, 2, 3},
4849+
{"2024-01-01 00:00:00.123456+00:00", std::nullopt,
4850+
"2024-01-01 00:00:02.123456+00:00"}},
4851+
};
4852+
for (const auto& c : cases) {
4853+
SCOPED_TRACE("marked=" + std::to_string(c.marked) +
4854+
" project_payload=" + std::to_string(c.project_payload) +
4855+
" tombstone=" + std::to_string(c.tombstone));
4856+
std::vector<ColumnDefinition> projected_columns {make_table_column(0, "id", int_type)};
4857+
if (c.project_payload) {
4858+
projected_columns.push_back(payload);
4859+
}
4860+
auto ts_field =
4861+
external_schema_field("ts", 4, {}, std::nullopt,
4862+
external_primitive_type(TPrimitiveType::TIMESTAMPTZ, -1, 6));
4863+
if (c.marked) {
4864+
ts_field.field_ptr->__set_timestamp_is_adjusted_to_utc(true);
4865+
}
4866+
auto payload_field = external_struct_schema_field(
4867+
"payload", 1,
4868+
{external_schema_field("existing", 2, {}, std::nullopt,
4869+
external_primitive_type(TPrimitiveType::INT)),
4870+
std::move(ts_field),
4871+
external_schema_field("k", 3, {}, "7",
4872+
external_primitive_type(TPrimitiveType::INT), false, true)},
4873+
true);
4874+
auto scan_params = make_local_parquet_scan_params();
4875+
scan_params.__set_iceberg_scan_semantics_version(ICEBERG_SCAN_SEMANTICS_VERSION_2);
4876+
scan_params.__set_parquet_timestamp_semantics_version(1);
4877+
scan_params.__set_enable_mapping_timestamp_tz(true);
4878+
scan_params.__set_current_schema_id(100);
4879+
scan_params.__set_history_schema_info(
4880+
{external_schema(100, {external_schema_field("id", 0), std::move(payload_field)})});
4881+
4882+
RuntimeProfile profile("test_profile");
4883+
RuntimeState state {TQueryOptions(), TQueryGlobals()};
4884+
state.set_timezone("Asia/Shanghai");
4885+
io::FileReaderStats file_reader_stats;
4886+
io::FileCacheStatistics file_cache_stats;
4887+
auto io_ctx = make_io_context(&file_reader_stats, &file_cache_stats);
4888+
ShardedKVCache cache(1);
4889+
doris::format::iceberg::IcebergTableReader reader;
4890+
init_iceberg_reader(&reader, projected_columns, &scan_params, io_ctx, &state, &profile);
4891+
auto split_options = build_split_options(file_path);
4892+
split_options.cache = &cache;
4893+
split_options.current_range.__set_table_format_params(make_iceberg_table_format_desc(
4894+
file_path, {make_iceberg_equality_delete_file(delete_file_path(c.tombstone), {3})},
4895+
3));
4896+
ASSERT_TRUE(reader.prepare_split(split_options).ok());
4897+
4898+
std::vector<int32_t> ids;
4899+
std::vector<std::optional<std::string>> timestamps;
4900+
bool eos = false;
4901+
while (!eos) {
4902+
Block block = build_table_block(projected_columns);
4903+
const auto status = reader.get_block(&block, &eos);
4904+
ASSERT_TRUE(status.ok()) << status;
4905+
if (block.rows() == 0) {
4906+
continue;
4907+
}
4908+
const auto& id_column =
4909+
assert_cast<const ColumnInt32&>(expect_not_null_table_column(block, 0));
4910+
for (size_t row = 0; row < block.rows(); ++row) {
4911+
ids.push_back(id_column.get_element(row));
4912+
}
4913+
if (!c.project_payload) {
4914+
continue;
4915+
}
4916+
const auto& payload_column =
4917+
assert_cast<const ColumnNullable&>(*block.get_by_position(1).column);
4918+
const auto& payload_struct =
4919+
assert_cast<const ColumnStruct&>(payload_column.get_nested_column());
4920+
for (size_t row = 0; row < block.rows(); ++row) {
4921+
if (payload_column.is_null_at(row)) {
4922+
timestamps.emplace_back(std::nullopt);
4923+
} else {
4924+
timestamps.emplace_back(
4925+
nullable_tz_type->to_string(payload_struct.get_column(1), row));
4926+
}
4927+
}
4928+
}
4929+
EXPECT_EQ(ids, c.ids);
4930+
EXPECT_EQ(timestamps, c.timestamps);
4931+
ASSERT_TRUE(reader.close().ok());
4932+
}
4933+
std::filesystem::remove_all(test_dir);
4934+
}
4935+
47364936
// Keep the shared Parquet/ORC reader setup together so both V2 paths materialize missing struct
47374937
// literals before traversing their children.
47384938
// NOLINTNEXTLINE(readability-function-cognitive-complexity,readability-function-size)

0 commit comments

Comments
 (0)