Skip to content

Commit 54c9572

Browse files
morningmanclaude
andcommitted
[improvement](be) Reserve an object's field spans before checking its fields
The reusable validator dropped the reservation the one-shot validator made for each object. Callers that still validate every value with a fresh validator, such as ColumnVariantV2::deserialize() through insert_data(), then grew the span buffer from zero for each wide object: about ten allocations for an object of 1000 fields instead of one. require_valid_object() now reserves base + count spans before checking the fields. It takes at least twice the current capacity, so sibling objects that each need a little more space do not reallocate one entry at a time: in a root of 26 objects of 30 fields, field k needs k + 30 entries, and the buffer grows 26 -> 52 -> 104 instead of once per sibling. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 714d816 commit 54c9572

2 files changed

Lines changed: 36 additions & 0 deletions

File tree

‎be/src/core/value/variant/variant_field.cpp‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -151,6 +151,15 @@ void VariantPayloadValidator::require_valid_object(VariantRef value, uint32_t de
151151
const size_t values_size = value.value.size - values_offset;
152152
// Entries below base belong to the ancestors of this object; see _object_spans.
153153
const size_t base = _object_spans.size();
154+
// Reserve this object's spans up front. A fresh validator, such as the one each
155+
// validate_variant_payload() call makes, then allocates once for a flat object of 1000
156+
// fields instead of growing ten times (1, 2, 4, ..., 1024). Reserving exactly base + count
157+
// would not be enough for nested objects: in a root of 26 fields that are each objects of
158+
// 30 fields, field k needs k + 30 entries, so every later sibling would reallocate by one
159+
// entry. Taking at least twice the capacity grows 26 -> 52 -> 104 instead.
160+
if (_object_spans.capacity() < base + count) {
161+
_object_spans.reserve(std::max(base + count, 2 * _object_spans.capacity()));
162+
}
154163

155164
StringRef previous_key;
156165
for (uint32_t index = 0; index < count; ++index) {

‎be/test/util/variant/variant_field_test.cpp‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -662,6 +662,33 @@ TEST(VariantFieldTest, ReusableValidatorReusesScratchAcrossRows) {
662662
EXPECT_EQ(validator.object_span_capacity_for_test(), capacity);
663663
}
664664

665+
TEST(VariantFieldTest, FreshValidatorReservesAnObjectOnce) {
666+
// validate_variant_payload() and single-row callers such as ColumnVariantV2::insert_data()
667+
// validate with a fresh validator, so its first object must not grow from zero.
668+
constexpr size_t FIELDS = 1'000;
669+
const VariantField wide = encode_json(wide_object_json(FIELDS));
670+
VariantPayloadValidator flat_validator;
671+
flat_validator.validate(wide.ref());
672+
EXPECT_EQ(flat_validator.object_span_capacity_for_test(), FIELDS);
673+
674+
// 26 objects of 30 fields: the root reserves 26 entries, and its children, which need
675+
// index + 30, double the buffer twice instead of reallocating for every sibling.
676+
std::string nested_json = "{";
677+
for (char key = 'a'; key <= 'z'; ++key) {
678+
if (key != 'a') {
679+
nested_json.push_back(',');
680+
}
681+
nested_json += '"';
682+
nested_json += key;
683+
nested_json += "\":" + wide_object_json(30);
684+
}
685+
nested_json.push_back('}');
686+
const VariantField nested = encode_json(nested_json);
687+
VariantPayloadValidator nested_validator;
688+
nested_validator.validate(nested.ref());
689+
EXPECT_EQ(nested_validator.object_span_capacity_for_test(), 104);
690+
}
691+
665692
TEST(VariantFieldTest, ReusableValidatorScratchIsMemTrackerVisible) {
666693
constexpr size_t FIELDS = 4'096;
667694
// Each field of the object being checked takes one span of two size_t values.

0 commit comments

Comments
 (0)