Skip to content

Commit c2d5878

Browse files
morningmanclaude
andcommitted
[fix](variant) Address review: keep empty arrays out of the empty-object rule
Two follow-ups on the per-part empty-object rule and the root-value scan. The per-part rule keyed off get_base_type_of_array(), which strips every array layer, so it also matched Array(Nothing) - the type an empty array gets, since an empty array carries no element type. That is not an absent value the way an untyped part is: it is `[]`, and the array serde renders it. In an unfinalized subcolumn whose later part promotes the column-level least common type past Array(Nothing), a `[]` sitting in the unpromoted part came back as `{}`. Key the rule off the part's own type instead, which is exactly "scalar Nothing". The root-value scan walked the whole root null map. Only the rows this call converts should have a say: prepare_remove_nullable hands the outer null map over as a separate argument and leaves the masked rows' payloads in place, so a masked row whose root carries a value kept every visible row on the root conversion. Bound the scan by input_rows_count and skip masked rows. ColumnVariantEmptyObjectTest.empty_array_part_is_not_an_empty_object covers the first one: it fails on the previous check with `part type Nullable(Array(Nullable(Nothing))), column type Nullable(Array(Nullable(TINYINT)))` returning `{}` instead of `[]`, and passes with this change. All 6 cases in the suite pass. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01FE4hQXyTkfCbqkMGbokU6e
1 parent e3c52e0 commit c2d5878

3 files changed

Lines changed: 63 additions & 4 deletions

File tree

‎be/src/core/column/column_variant.cpp‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1446,8 +1446,10 @@ size_t ColumnVariant::Subcolumn::serialize_text_json(size_t n, BufferWritable& o
14461446
// the column-level least common type, so the check above stops covering the rows that
14471447
// still sit in an untyped part. Render those rows as an empty JSON object here too,
14481448
// instead of letting the Nothing serde emit an empty string. See #67367.
1449-
if (get_base_type_of_array(data_types[i])->get_primitive_type() ==
1450-
PrimitiveType::INVALID_TYPE) {
1449+
// This has to be the part's own type rather than the base type of its arrays: an empty
1450+
// array is typed Array(Nothing), and it does hold a value - `[]` - that the array serde
1451+
// renders correctly on its own.
1452+
if (data_types[i]->get_primitive_type() == PrimitiveType::INVALID_TYPE) {
14511453
output.write(EMPTY_JSON.data(), EMPTY_JSON.size());
14521454
return EMPTY_JSON.size();
14531455
}

‎be/src/exprs/function/cast/cast_to_variant.h‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -88,11 +88,28 @@ inline Status cast_from_variant_impl(FunctionContext* context, Block& block,
8888
// paths. When no row's root holds a value there is nothing for the root conversion to
8989
// convert, so serialize the tree instead. Rows whose root does hold a value keep the root
9090
// conversion, which is what unwraps a JSON string into its text. See #67367.
91+
// Only the rows this call actually converts get a say. prepare_remove_nullable hands the
92+
// outer null map over as a separate argument and leaves the masked rows' payloads in
93+
// place, so a masked row whose root does carry a value would otherwise keep every visible
94+
// row on the root conversion.
9195
const auto* nullable_root = check_and_get_column<ColumnNullable>(*variant->get_root());
9296
if (nullable_root != nullptr) {
9397
const auto& root_null_map = nullable_root->get_null_map_data();
94-
is_root_valuable = std::any_of(root_null_map.begin(), root_null_map.end(),
95-
[](UInt8 is_null) { return is_null == 0; });
98+
const NullMap::value_type* outer_null_map = null_map;
99+
if (outer_null_map == nullptr && nullable != nullptr) {
100+
outer_null_map = nullable->get_null_map_data().data();
101+
}
102+
const size_t rows_to_scan = std::min(input_rows_count, root_null_map.size());
103+
is_root_valuable = false;
104+
for (size_t row = 0; row < rows_to_scan; ++row) {
105+
if (outer_null_map != nullptr && outer_null_map[row] != 0) {
106+
continue;
107+
}
108+
if (root_null_map[row] == 0) {
109+
is_root_valuable = true;
110+
break;
111+
}
112+
}
96113
}
97114
}
98115

‎be/test/core/column/column_variant_test.cpp‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3706,6 +3706,16 @@ std::string serialize_variant_row(const ColumnVariant& column, size_t row) {
37063706
return serialized;
37073707
}
37083708

3709+
std::string serialize_subcolumn_row(const ColumnVariant::Subcolumn& subcolumn, size_t row) {
3710+
auto tmp_col = ColumnString::create();
3711+
VectorBufferWriter write_buffer(*tmp_col.get());
3712+
DataTypeSerDe::FormatOptions options;
3713+
subcolumn.serialize_text_json(row, write_buffer, options);
3714+
write_buffer.commit();
3715+
auto str_ref = tmp_col->get_data_at(0);
3716+
return std::string(str_ref.data, str_ref.size);
3717+
}
3718+
37093719
std::string describe_variant(const ColumnVariant& column) {
37103720
return fmt::format("rows={} scalar={} null_root={} root_type={} subcolumns={}", column.size(),
37113721
column.is_scalar_variant(), column.is_null_root(),
@@ -3832,4 +3842,34 @@ TEST(ColumnVariantEmptyObjectTest, empty_object_survives_copy_of_mixed_type_part
38323842
expect_all_empty_objects(finalized_copy_variant, "finalized copy of mixed-type parts");
38333843
}
38343844

3845+
// An empty array is typed Array(Nothing), whose base type is also Nothing. It is not an absent
3846+
// value the way an untyped part is - it is the value `[]`, and the array serde renders it. So the
3847+
// per-part empty-object rule above has to key off the part's own type, not the base type of its
3848+
// arrays, or a `[]` sitting in an unpromoted part comes back as `{}`.
3849+
TEST(ColumnVariantEmptyObjectTest, empty_array_part_is_not_an_empty_object) {
3850+
ColumnVariant::Subcolumn subcolumn(0, true /* is_nullable */, false /* is_root */);
3851+
3852+
// An empty array carries no element type, so its part is typed Array(Nothing).
3853+
subcolumn.insert(Field::create_field<TYPE_ARRAY>(Array()));
3854+
ASSERT_EQ(subcolumn.data_types.size(), 1);
3855+
EXPECT_EQ(get_base_type_of_array(subcolumn.data_types[0])->get_primitive_type(),
3856+
PrimitiveType::INVALID_TYPE)
3857+
<< subcolumn.data_types[0]->get_name();
3858+
3859+
// A later typed value promotes the column-level least common type past Array(Nothing) and
3860+
// opens a second part, which is what stops the column-level rule at the top of
3861+
// serialize_text_json from covering row 0 - the same shape as the empty-object case above.
3862+
Field typed_array = Field::create_field<TYPE_ARRAY>(Array());
3863+
typed_array.get<TYPE_ARRAY>().emplace_back(Field::create_field<TYPE_TINYINT>(1));
3864+
subcolumn.insert(typed_array);
3865+
ASSERT_EQ(subcolumn.data_types.size(), 2) << subcolumn.get_least_common_type()->get_name();
3866+
ASSERT_EQ(subcolumn.size(), 2);
3867+
3868+
// Row 0 is still served by the Array(Nothing) part, and it is an empty array, not `{}`.
3869+
EXPECT_EQ(serialize_subcolumn_row(subcolumn, 0), "[]")
3870+
<< "part type " << subcolumn.data_types[0]->get_name() << ", column type "
3871+
<< subcolumn.get_least_common_type()->get_name();
3872+
EXPECT_EQ(serialize_subcolumn_row(subcolumn, 1), "[1]");
3873+
}
3874+
38353875
} // namespace doris

0 commit comments

Comments
 (0)