You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 714d816
Browse filesBrowse the repository at this point in the historyBrowse files
[improvement](be) Reuse memory-tracked scratch when validating Variant payloads
### What problem does this PR solve?
Issue Number: None
Related PR: #68959
Problem Summary: validate_variant_payload() checks that the field values of
every Variant object exactly tile the object's value section. To do that,
require_valid_object() collected the byte spans of the field values into a
fresh std::vector for every object it checked, nested ones included, and freed
it when the check returned. ColumnVariantV2::insert_encoded_rows() (JNI
readers, inserted encoded values, Variant element extraction), the
EncodedRowsAppender that #68959 uses to import unshredded Parquet Variant
values, and the native Arrow Variant reader validate every row this way. A
4096-row batch of object rows therefore paid at least 4096 allocations, and
the scratch (16 bytes per field) came from plain malloc, which Doris memory
tracking does not see. VariantBatchBuilder, which that importer replaced,
checked the same intervals in a reused, tracked DorisVector.
Add VariantPayloadValidator, which keeps the spans in one DorisVector used as
a stack: an object pushes the span of each field after checking the field, so
a nested object pushes above its ancestors and pops back before it returns.
The checks, their order and their errors are unchanged. The batch callers
hold one validator for the batch, and the encoded-row appender holds one for
its whole life; validate_variant_payload() stays as the one-shot form for a
single value. The class is declared in its own header so that core/field.h,
which includes variant_field.h, does not pull in core/custom_allocator.h.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- VariantFieldTest: the reusable validator accepts and rejects the same
payloads as the one-shot call, including a corrupt nested object
followed by a valid one; its scratch capacity stays the same over 4096
rows; the scratch is visible to the memory tracker
- ColumnVariantV2Test.EncodedRowsAppenderReusesValidationScratchAcrossChunks
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
0 commit comments