Repository navigation
Conversation
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: This adds a focused BE unit-test fixture for storage index lifecycle coverage around inverted index probe events, patched tablet schemas, variant sparse/nested metadata, and COUNT_ON_INDEX no-need-read-data behavior. The tests cover index add/drop and build/drop on text and variant path indexes, missing required variant columns, V1/V2 index storage files, rowset map updates after schema patching, and red/green validation that COUNT_ON_INDEX reads only rows filtered by an applied inverted index. The implementation records index probe events for applied, fallback, and not-attempted column predicates, preserves existing SegmentIterator debug points, skips missing physical columns only when nullable/default columns make that valid, and resolves field_pattern variant index build/drop against materialized paths.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `PATH=/tmp/codex-clang-format16:$PATH build-support/clang-format.sh`
- `PATH=/tmp/codex-clang-format16:$PATH build-support/check-format.sh`
- `git diff --check && git diff --cached --check`
- `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*'`
- Attempted `env CLANG_TIDY_BINARY=/mnt/disk1/claude-max/ldb_toolchain20/bin/clang-tidy build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`, but local clang-tidy could not analyze Doris headers due toolchain/PCH diagnostics (`stddef.h` not found from `slice.h` and existing `core/types.h` NOLINT parsing). A clang-tidy-16 probe also failed with incompatible PCH AST.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve? Issue Number: close apache#64205 Related PR: apache#64205 Problem Summary: Add a focused BE unit test that reproduces the variant sparse stats limit reader bug. The test writes a variant column with one materialized path, an exact sparse path v.b, and a hidden child path v.b.c that is omitted from sparse path statistics after the configured limit is reached. Reading v.b through an explicit path-column schema currently returns NULL for the row whose value only exists under b.c, exposing the missing hierarchical read behavior without applying the production fix. ### Release note None ### Check List (For Author) - Test: Unit Test - Ran Target system: Linux; Target arch: x86_64 Python 3.11.13 Check JAVA_HOME version Apache Maven 3.9.12 (848fbb4bf2d427b72bdb2471c22fced7ebd9a7a1) Maven home: /usr/local Java version: 17.0.16, vendor: Oracle Corporation, runtime: /mnt/disk6/common/jdk-17.0.16 Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "5.14.0-694.el9.x86_64", arch: "amd64", family: "unix" cmake version 3.31.8 CMake suite maintained and supported by Kitware (kitware.com/cmake). ccache version 4.5.1 Get params: PARALLEL -- 39 CLEAN -- 0 ENABLE_PCH -- ON EXTRA_BE_MODULES -- Build Backend UT install datasketches-cpp to thirdparty path before build backend ut Update datasketches-cpp submodule ... -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/contrib/datasketches-cpp/build/Release Install the project... -- Install configuration: "Release" -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/version.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/binomial_bounds.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_binomial_proportions.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ceiling_power_of_2.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/common_defs.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/conditional_back_inserter.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/conditional_forward.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_zeros.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/inv_pow2_table.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kolmogorov_smirnov_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kolmogorov_smirnov.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/memory_operations.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/MurmurHash3.h -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/optional.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sorted_view_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sorted_view.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/serde.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/xxhash64.h -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/hll.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/AuxHashMap.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CompositeInterpolationXTable.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/hll.private.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImplFactory.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponHashSet.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponList.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CubicInterpolation.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HarmonicNumbers.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll4Array.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll6Array.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll8Array.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllArray.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImpl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllUtil.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/coupon_iterator.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/RelativeErrorTables.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/AuxHashMap-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CompositeInterpolationXTable-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponHashSet-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponList-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CubicInterpolation-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HarmonicNumbers-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll4Array-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll6Array-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll8Array-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllArray-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketch-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImpl-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllUnion-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/coupon_iterator-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/RelativeErrorTables-internal.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compression_data.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_common.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_compressor.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_compressor_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_confidence.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_util.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/icon_estimator.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kxp_byte_lookup.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/u32_table.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/u32_table_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_helper.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_helper_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/frequent_items_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/frequent_items_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/reverse_purge_hash_map.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/reverse_purge_hash_map_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_a_not_b.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_a_not_b_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_jaccard_similarity.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_comparators.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_constants.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_helpers.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_update_sketch_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_update_sketch_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_set_difference_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_set_difference_base_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_jaccard_similarity_base.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_on_ratios_in_sampled_sets.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_on_ratios_in_theta_sketched_sets.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compact_theta_sketch_parser.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compact_theta_sketch_parser_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bit_packing.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sample.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sample_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_intersection.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_intersection_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_a_not_b.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_a_not_b_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_jaccard_similarity.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_of_doubles_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_union.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_union_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_intersection.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_intersection_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_a_not_b.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_a_not_b_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_common.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_compactor.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_compactor_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_min.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_min_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/density_sketch.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/density_sketch_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tdigest.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tdigest_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter_builder_impl.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bit_array_ops.hpp -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketches.cmake -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketchesConfigVersion.cmake -- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketchesConfig.cmake Update apache-orc submodule ... Update clucene submodule ... -- Make program: /usr/bin/make -- Use ccache: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache and -DCMAKE_C_COMPILER_LAUNCHER=ccache -- Extra cxx flags: -- GLIBC_COMPATIBILITY is ON -- USE_LIBCPP is OFF -- USE_JEMALLOC is OFF -- USE_UNWIND is ON -- ENABLE_PCH is ON -- USE_AVX2 is ON -- Build type is ASAN_UT -- Build target arch is x86_64 -- DORIS_HOME is /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework -- THIRDPARTY_DIR is /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed -- make test: ON -- make benchmark: OFF -- build fs benchmark tool: OFF -- build task executor simulator: OFF -- build file cache lru tool: OFF -- build gensrc if necessary make: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc' make -C script make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script' /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script/gen_build_version.sh make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script' make -C proto make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/proto' make[1]: Nothing to be done for 'all'. make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/proto' make -C thrift make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift' /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/bin/thrift -I /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift -I /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/../build//thrift/ --gen cpp:moveable_types,no_skeleton -out /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/../build//gen_cpp --allow-64bit-consts -strict /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/Normalization.thrift make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift' make: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc' -- Configuring incomplete, errors occurred!; expected failure before the apache#64205 fix: v.b values are scalar-0, scalar-1, scalar-2, scalar-3, NULL and do not include child-0. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Migrate variant array inverted index writer debug point coverage from the variant_p0 regression suite into the BE index storage test framework. The new test drives typed variant array rowset writes, verifies array writer field counts through a scoped debug point, covers null elements, empty arrays, null arrays, top-level null Variant rows, mixed typed JSON elements, and checks index file materialization across cumulative compaction. The shared index storage test utility now supports scoped debug points, array typed variant path templates, and direct Variant-column batches with shape validation for future typed Variant storage tests.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*'
- ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*' (26/27 passed; IndexStorageLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated failed and reproduced in single-test rerun with unchanged failure)
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Split Variant index storage lifecycle coverage out of the non-Variant index storage lifecycle test file so text/non-Variant and Variant storage cases can evolve independently. Add a storage-level migration for the variant_p0 multi-index compaction debug-point scenario by verifying that equality predicates on a predefined Variant string path choose the plain field-pattern index, not the parent, phrase, sibling, or array path indexes, before and after cumulative compaction.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
- `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.VariantPathEqualityChoosesStringIndexAfterCompaction'`
- `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- `./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*'`
- `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` (expected failure, reproduces hidden child loss before fix)
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary: Split pure index storage schema patch unit tests out of the lifecycle test files. The lifecycle files now focus on rowset write/read/index/compaction behavior, while text and variant schema patch construction checks live in dedicated files. This keeps future migrations from variant debug point and storage tests easier to place without mixing schema-only checks with lifecycle coverage.
### Release note
None
### Check List (For Author)
- Test: Unit Test
- `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
- `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageSchemaPatchTest.*:IndexStorageVariantSchemaPatchTest.*'`
- `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- Red test, expected failure retained: `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64205 Problem Summary: Add storage-level coverage for typed Variant field-pattern inverted indexes across cumulative and full compaction. The test writes typed glob predefined paths, verifies that an INT predicate uses only the matching field-pattern index before compaction, then reloads the compacted rowset and verifies the same index behavior after compaction. Shared Variant index assertion helpers are moved into the index storage UT utility layer so later Variant index storage cases can reuse them instead of duplicating probe-event scans. ### Release note None ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantFieldPatternIndexTest.*'` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` - Expected red test: `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` fails with `has_hidden_child` false. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64205 Problem Summary: Extend the index storage unit-test fixture to pass common expression pushdown contexts into rowset readers, then add a separate Variant expression-pushdown test for IS NULL and IS NOT NULL over a field-pattern indexed typed path. This keeps expression-level index coverage separate from field-pattern column-predicate and compaction coverage, and lets future Variant expression tests reuse the same write/read/assert framework. ### Release note None ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh` - `git diff --cached --check` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*'` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64205 Problem Summary: Add storage-level unit coverage for Variant array_contains common expression pushdown over a field-pattern array inverted index. The test migrates the array_func.array_contains debugpoint coverage from variant_p0 into the index storage fixture by asserting the index bitmap cardinality, rows filtered by the inverted index, and returned rows without relying on SQL profile polling. ### Release note None ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh` - `git diff --check` - `git diff --cached --check` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*'` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64205 Problem Summary: Remove the production behavior fixes from the index-storage test framework branch so the new BE unit tests keep covering the identified storage/index bugs as intentional reproducers. The remaining non-test code is limited to index probe observability needed by the tests, while Variant sparse stats, field_pattern index build/drop, patched schema compaction, and COUNT_ON_INDEX behavior are left unfixed and represented by red tests. ### Release note None ### Check List (For Author) - Test: Unit Test - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh` - `git diff --check` - `git diff --cached --check` - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.CountOnIndexSkipsReadingKeyDataWhenIndexApplied:IndexStorageVariantLifecycleTest.BuildAndDropVariantPathIndexAfterExistingRows:IndexStorageVariantLifecycleTest.PatchedSchemaAddDropVariantColumnCompactsNewRows:IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` (expected red: 4 failed tests) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64348 Problem Summary: Refine the reusable index-storage unit-test fixture so future Index and Variant storage tests can assert structured index probe events instead of profile timer side channels. The change records selected inverted-index readers through the index query context, emits expr-pushdown probe events with source/path/index id metadata, removes the fragile last-read-index-id path, and makes the test row batch/schema injection helpers more explicit. ### Release note None ### Check List (For Author) - Test: Unit Test - PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh - git diff --check - git diff --cached --check - env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*' (passed: 2 tests) - env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageSchemaPatchTest.*:IndexStorageVariantSchemaPatchTest.*' (current draft red: 29 passed, 6 failed repro tests) - env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.CountOnIndexSkipsReadingKeyDataWhenIndexApplied:IndexStorageVariantLifecycleTest.BuildAndDropVariantPathIndexAfterExistingRows:IndexStorageVariantLifecycleTest.PatchedSchemaAddDropVariantColumnCompactsNewRows:IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated' (expected red: 4 failed tests) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64348 Problem Summary: Preserve index probe observability while extracting probe recording into a reusable recorder. The previous recorder shape could collapse null-bitmap probes and duplicate index reads into one event, weakening Variant and Index unit tests that need to assert every probe. This change records every index read probe, keeps one primary probe responsible for filtered-row attribution to avoid double counting, and adds a focused unit test covering null-bitmap and duplicate probe preservation. ### Release note None ### Check List (For Author) - Test: Unit Test - PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh - git diff --check - git diff --cached --check - env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexProbeRecorderTest.*:IndexStorageVariantExprPushdownTest.*' (passed: 3 tests) - env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.DropVariantPathIndexWithoutMaterializedPathRemovesMetadata:IndexStorageVariantLifecycleTest.DropOneVariantPathIndexKeepsSiblingPathIndexUsable:IndexStorageVariantLifecycleTest.VariantPathEqualityChoosesStringIndexAfterCompaction' (passed: 1 test; current draft still has 2 drop-index lifecycle failures) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64348 Problem Summary: Tighten the index probe test framework after extracting reusable probe events. The previous helper flow could conflate applied indexes that actually reduced filter statistics with supporting index reads, which weakened assertions around duplicate or null-bitmap probes. This change records each probe event with explicit filter-stat attribution, keeps strict no-applied-index assertions, adds a separate non-filtering assertion helper, and binds index query context with column ids through a single API on production SegmentIterator paths. Focused tests cover iterator probe column-id binding, null-bitmap and duplicate probes, expr pushdown attribution, and variant path index selection after compaction. ### Release note None ### Check List (For Author) - Test: Unit Test - PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:/mnt/disk1/claude-max/bin:/mnt/disk1/claude-max/.local/bin:/mnt/disk1/claude-max/.npm-global/bin:/mnt/disk1/claude-max/.opencode/bin:/mnt/disk1/claude-max/.nvm/versions/node/v24.14.1/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin build-support/clang-format.sh - git diff --check - git diff --cached --check - env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexProbeRecorderTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantLifecycleTest.VariantPathEqualityChoosesStringIndexAfterCompaction' - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64348 Problem Summary: Reduce the production-code footprint of the index probe test framework. The previous probe integration recorded synthetic NOT_ATTEMPTED events and distinguished search expression sources inside SegmentIterator, which made the reader path more invasive than needed for the current unit-test coverage. This change keeps probe events for real index reads on predicate and expr pushdown paths, preserves iterator column-id binding, and moves missing-index, disabled-index, and drop-index assertions back to existing OlapReaderStatistics, schema, and index-file checks. ### Release note None ### Check List (For Author) - Test: Unit Test - PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:/mnt/disk1/claude-max/.local/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path:/mnt/disk1/claude-max/bin:/mnt/disk1/claude-max/.local/bin:/mnt/disk1/claude-max/.npm-global/bin:/mnt/disk1/claude-max/.opencode/bin:/mnt/disk1/claude-max/.nvm/versions/node/v24.14.1/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin build-support/clang-format.sh - git diff --check - git diff --cached --check - env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageSchemaPatchTest.*:IndexStorageVariantSchemaPatchTest.*' (29 passed, 6 expected-red failures) - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#64348 Problem Summary: Extend the Variant index storage BE unit-test framework with structured probe-event assertions for array_contains expression pushdown, typed field-pattern index selection, segment-level zone-map pruning, count-on-index probe recording, and compaction/reload behavior. The change also adds reusable probe expectation helpers and keeps currently failing product behaviors as disabled expected-red reproducers instead of fixing them in this test-only branch. ### Release note None ### Check List (For Author) - Test: Unit Test - /mnt/disk1/claude-max/tmp/doris/.github/actions/clang-format-lint-action/clang-format/clang-format16 -i be/test/storage/index_probe_recorder_test.cpp be/test/storage/index_storage_lifecycle_test.cpp be/test/storage/index_storage_variant_expr_pushdown_test.cpp be/test/storage/index_storage_variant_field_pattern_index_test.cpp be/test/storage/index_storage_variant_lifecycle_test.cpp be/test/testutil/index_storage_test_util.cpp be/test/testutil/index_storage_test_util.h - git diff --check - git diff --cached --check - DORIS_THIRDPARTY=/mnt/disk6/common/doris-thirdparties/doris-thirdparty-master ./run-be-ut.sh --run --filter='IndexProbeRecorderTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.VariantPathCountOnIndexRecordsAppliedEvent:IndexStorageVariantLifecycleTest.VariantPathEqualityChoosesStringIndexAfterCompaction:IndexStorageVariantLifecycleTest.VariantPathIndexHitAfterCumulativeCompaction' (passed: 12 tests, 3 disabled) - DORIS_THIRDPARTY=/mnt/disk6/common/doris-thirdparties/doris-thirdparty-master ./run-be-ut.sh --run --coverage --filter='IndexProbeRecorderTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.VariantPathCountOnIndexRecordsAppliedEvent:IndexStorageVariantLifecycleTest.VariantPathEqualityChoosesStringIndexAfterCompaction:IndexStorageVariantLifecycleTest.VariantPathIndexHitAfterCumulativeCompaction' (passed: 12 tests, 3 disabled; coverage report generated manually after script merge command fallback) - Expected-red disabled repros: 10 tests / 4 suites failed as expected. - Behavior changed: No - Does this need documentation: No
eldenmoon
force-pushed
the
branch-doris-26466-regression
branch
from
June 16, 2026 10:59
0c7662f to
4a8cd0a
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes Variant compaction metadata aggregation for schema-patched rowsets where a newly added nullable VARIANT column is missing from some segments (no segment metadata). Instead of failing compaction with NOT_FOUND when fetching a column reader directly, the compaction stats/extended-info aggregation now skips those missing readers when the column can be filled as default/NULL by the normal read path.
Changes:
- Skip
NOT_FOUNDsegment column readers during Variant compaction stats and extended-info aggregation when the missing column is nullable or has a default value. - Enable the previously-disabled BE unit test covering patched schema add/drop Variant compaction.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
be/test/storage/variant/index_storage_variant_lifecycle_test.cpp |
Enables the regression test for patched schema add/drop Variant compaction. |
be/src/exec/common/variant_util.cpp |
Avoids failing compaction metadata aggregation when nullable/default VARIANT columns are missing from a segment. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
eldenmoon
force-pushed
the
branch-variant-gtest-framework-master
branch
3 times, most recently
from
June 24, 2026 04:24
235a069 to
a22fbee
Compare
eldenmoon
pushed a commit
that referenced
this pull request
Sep 1, 2026
Problem Summary: Streaming `ROWS` window aggregates retain state across
frame evaluations. The eviction path previously considered only whether
buffered blocks had been emitted, so it could erase either the outgoing
row needed by a bounded sliding frame or the next unread row needed by
an `UNBOUNDED PRECEDING ... N PRECEDING` frame. After rebasing, negative
partition and outgoing positions could allow a nullable aggregate to
access its null map out of bounds. Evicting either kind of required row
could also produce incorrect aggregate results.
Root cause: `_remove_unused_rows()` did not account for the earliest row
required by the next ROWS frame evaluation, and
`BoundaryPose::remove_unused_rows()` allowed retained-column coordinates
to become negative.
This change defers block-aligned eviction when the candidate prefix
contains either `frame_start - 1`, the outgoing row required by a
bounded sliding update, or the next unread row required by an `UNBOUNDED
PRECEDING ... N PRECEDING` frame. It also rebases partition and order
boundaries to nonnegative physical-column coordinates. The BE unit
coverage exercises both ROWS executors across eviction boundaries and
verifies boundary rebasing.
Observed ASAN failure before this change (`output/be/log/be.out`):
```text
ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 1
#0 doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
be/src/exprs/aggregate/aggregate_function_null_v2.h:595
#1 doris::AggFnEvaluator::execute_function_with_incremental(...)
be/src/exprs/vectorized_agg_fn.cpp:334
#2 doris::AnalyticSinkLocalState::_execute_for_function<true>(...)
be/src/exec/operator/analytic_sink_operator.cpp:385
#3 doris::AnalyticSinkLocalState::_get_next_for_sliding_rows(...)
be/src/exec/operator/analytic_sink_operator.cpp:203
#4 doris::AnalyticSinkLocalState::_execute_impl(...)
be/src/exec/operator/analytic_sink_operator.cpp:358
#5 doris::AnalyticSinkOperatorX::sink_impl(...)
be/src/exec/operator/analytic_sink_operator.cpp:757
SUMMARY: AddressSanitizer: heap-buffer-overflow in
doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Why
DORIS-26466 exposes a schema-patched old rowset that does not contain segment metadata for a newly added nullable Variant column. The normal read path can fill that column with null values, but the Variant compaction stats path fetched the column reader directly and returned
NOT_FOUND, causing compaction to fail.Tests
GTEST_ALSO_RUN_DISABLED_TESTS=1 ./run-be-ut.sh --run --filter=IndexStorageVariantLifecycleTest.DISABLED_PatchedSchemaAddDropVariantColumnCompactsNewRows./run-be-ut.sh --run --filter=IndexStorageVariantLifecycleTest.PatchedSchemaAddDropVariantColumnCompactsNewRows:IndexStorageVariantLifecycleTest.MissingRequiredVariantColumnFailsExtendedInfoAggregation./run-be-ut.sh --run --filter=IndexStorageVariantLifecycleTest.*