Skip to content

[test](be) Migrate variant index storage debugpoint tests - #2

Open
eldenmoon wants to merge 8 commits into
masterfrom
branch-variant-debugpoint-ut-migration
Open

eldenmoon wants to merge 8 commits into
masterfrom
branch-variant-debugpoint-ut-migration

Conversation

@eldenmoon

@eldenmoon eldenmoon commented Jun 10, 2026 •

Copy link
Copy Markdown
Owner

What problem does this PR solve?

Issue Number: None

Related PR: apache#64205

Problem Summary: Build a reusable BE index-storage unit-test fixture and migrate the Variant variant_p0 debugpoint/index/compaction coverage that belongs at storage-reader/writer level into focused unit tests. The migrated coverage includes array inverted-index writer debugpoint checks, Variant path index lifecycle and compaction coverage, field-pattern typed index coverage for cumulative/full compaction, common expression pushdown for IS NULL / IS NOT NULL, and array_contains common-expression index filtering. Variant and non-Variant coverage is split into independent test files, with shared rowset write/read/reload/compact/index-probe helpers kept in index_storage_test_util.

The bloom_filter_must_filter_data profile/debugpoint portion in variant_p0/predefine/test_types_with_indexes_profile.groovy remains regression/bloom-filter scope rather than inverted-index storage fixture scope. Existing BE tests cover bloom-filter reader/writer primitives separately; this PR migrates the inverted-index storage paths. The Cloud schema-change transaction-conflict debugpoint remains regression-scope because it validates cloud schema-change job conflict handling rather than BE rowset/index storage behavior. The sparse-stats apache#64205 reproducer is kept as an intentional red BE UT and is excluded from the green storage/index verification command until the product fix lands.

Release note

None

Check List (For Author)

  • Test: Unit Test
    • PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh
    • git diff --check
    • git diff --cached --check
    • env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*' (2 tests passed)
    • env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated' (36 tests passed)
  • Behavior changed: No
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: This adds a focused BE unit-test fixture for storage index lifecycle coverage around inverted index probe events, patched tablet schemas, variant sparse/nested metadata, and COUNT_ON_INDEX no-need-read-data behavior. The tests cover index add/drop and build/drop on text and variant path indexes, missing required variant columns, V1/V2 index storage files, rowset map updates after schema patching, and red/green validation that COUNT_ON_INDEX reads only rows filtered by an applied inverted index. The implementation records index probe events for applied, fallback, and not-attempted column predicates, preserves existing SegmentIterator debug points, skips missing physical columns only when nullable/default columns make that valid, and resolves field_pattern variant index build/drop against materialized paths.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `PATH=/tmp/codex-clang-format16:$PATH build-support/clang-format.sh`
    - `PATH=/tmp/codex-clang-format16:$PATH build-support/check-format.sh`
    - `git diff --check && git diff --cached --check`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*'`
    - Attempted `env CLANG_TIDY_BINARY=/mnt/disk1/claude-max/ldb_toolchain20/bin/clang-tidy build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`, but local clang-tidy could not analyze Doris headers due toolchain/PCH diagnostics (`stddef.h` not found from `slice.h` and existing `core/types.h` NOLINT parsing). A clang-tidy-16 probe also failed with incompatible PCH AST.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: close apache#64205

Related PR: apache#64205

Problem Summary: Add a focused BE unit test that reproduces the variant sparse stats limit reader bug. The test writes a variant column with one materialized path, an exact sparse path v.b, and a hidden child path v.b.c that is omitted from sparse path statistics after the configured limit is reached. Reading v.b through an explicit path-column schema currently returns NULL for the row whose value only exists under b.c, exposing the missing hierarchical read behavior without applying the production fix.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - Ran Target system: Linux; Target arch: x86_64
Python 3.11.13
Check JAVA_HOME version
Apache Maven 3.9.12 (848fbb4bf2d427b72bdb2471c22fced7ebd9a7a1)
Maven home: /usr/local
Java version: 17.0.16, vendor: Oracle Corporation, runtime: /mnt/disk6/common/jdk-17.0.16
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.14.0-694.el9.x86_64", arch: "amd64", family: "unix"
cmake version 3.31.8

CMake suite maintained and supported by Kitware (kitware.com/cmake).
ccache version 4.5.1
Get params:
    PARALLEL            -- 39
    CLEAN               -- 0
    ENABLE_PCH          -- ON
    EXTRA_BE_MODULES    --

Build Backend UT
install datasketches-cpp to thirdparty path before build backend ut
Update datasketches-cpp submodule ...
-- Configuring done (0.0s)
-- Generating done (0.0s)
-- Build files have been written to: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/contrib/datasketches-cpp/build/Release
Install the project...
-- Install configuration: "Release"
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/version.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/binomial_bounds.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_binomial_proportions.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ceiling_power_of_2.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/common_defs.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/conditional_back_inserter.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/conditional_forward.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_zeros.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/inv_pow2_table.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kolmogorov_smirnov_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kolmogorov_smirnov.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/memory_operations.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/MurmurHash3.h
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/optional.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sorted_view_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sorted_view.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/serde.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/xxhash64.h
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/hll.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/AuxHashMap.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CompositeInterpolationXTable.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/hll.private.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImplFactory.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponHashSet.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponList.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CubicInterpolation.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HarmonicNumbers.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll4Array.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll6Array.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll8Array.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllArray.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImpl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllUtil.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/coupon_iterator.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/RelativeErrorTables.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/AuxHashMap-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CompositeInterpolationXTable-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponHashSet-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CouponList-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/CubicInterpolation-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HarmonicNumbers-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll4Array-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll6Array-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/Hll8Array-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllArray-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketch-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllSketchImpl-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/HllUnion-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/coupon_iterator-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/RelativeErrorTables-internal.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compression_data.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_common.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_compressor.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_compressor_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_confidence.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_union.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_union_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/cpc_util.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/icon_estimator.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kxp_byte_lookup.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/u32_table.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/u32_table_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_helper.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/kll_helper_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/frequent_items_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/frequent_items_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/reverse_purge_hash_map.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/reverse_purge_hash_map_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_a_not_b.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_a_not_b_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_jaccard_similarity.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_comparators.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_constants.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_helpers.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_update_sketch_base.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_update_sketch_base_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_base.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_union_base_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_base.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_intersection_base_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_set_difference_base.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_set_difference_base_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/theta_jaccard_similarity_base.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_on_ratios_in_sampled_sets.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bounds_on_ratios_in_theta_sketched_sets.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compact_theta_sketch_parser.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/compact_theta_sketch_parser_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bit_packing.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_union.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/var_opt_union_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sample.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sample_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/ebpps_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_union.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_union_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_intersection.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_intersection_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_a_not_b.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_a_not_b_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tuple_jaccard_similarity.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_of_doubles_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_union.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_union_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_intersection.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_intersection_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_a_not_b.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/array_tuple_a_not_b_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_common.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_compactor.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/req_compactor_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/quantiles_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_min.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/count_min_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/density_sketch.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/density_sketch_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tdigest.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/tdigest_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bloom_filter_builder_impl.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/include/DataSketches/bit_array_ops.hpp
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketches.cmake
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketchesConfigVersion.cmake
-- Installing: /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/lib/DataSketches/cmake/DataSketchesConfig.cmake
Update apache-orc submodule ...
Update clucene submodule ...
-- Make program: /usr/bin/make
-- Use ccache: -DCMAKE_CXX_COMPILER_LAUNCHER=ccache and -DCMAKE_C_COMPILER_LAUNCHER=ccache
-- Extra cxx flags:
-- GLIBC_COMPATIBILITY is ON
-- USE_LIBCPP is OFF
-- USE_JEMALLOC is OFF
-- USE_UNWIND is ON
-- ENABLE_PCH is ON
-- USE_AVX2 is ON
-- Build type is ASAN_UT
-- Build target arch is x86_64
-- DORIS_HOME is /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework
-- THIRDPARTY_DIR is /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed
-- make test: ON
-- make benchmark: OFF
-- build fs benchmark tool: OFF
-- build task executor simulator: OFF
-- build file cache lru tool: OFF
-- build gensrc if necessary
make: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc'
make -C script
make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script'
/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script/gen_build_version.sh
make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/script'
make -C proto
make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/proto'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/proto'
make -C thrift
make[1]: Entering directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift'
/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/thirdparty/installed/bin/thrift -I /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift -I /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/../build//thrift/ --gen cpp:moveable_types,no_skeleton -out /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/../build//gen_cpp --allow-64bit-consts -strict /mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift/Normalization.thrift
make[1]: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc/thrift'
make: Leaving directory '/mnt/disk1/claude-max/tmp/doris-variant-gtest-framework/gensrc'
-- Configuring incomplete, errors occurred!; expected failure before the apache#64205 fix: v.b values are scalar-0, scalar-1, scalar-2, scalar-3, NULL and do not include child-0.
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Migrate variant array inverted index writer debug point coverage from the variant_p0 regression suite into the BE index storage test framework. The new test drives typed variant array rowset writes, verifies array writer field counts through a scoped debug point, covers null elements, empty arrays, null arrays, top-level null Variant rows, mixed typed JSON elements, and checks index file materialization across cumulative compaction. The shared index storage test utility now supports scoped debug points, array typed variant path templates, and direct Variant-column batches with shape validation for future typed Variant storage tests.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*'
    - ./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*' (26/27 passed; IndexStorageLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated failed and reproduced in single-test rerun with unchanged failure)
- Behavior changed: No
- Does this need documentation: No
Copilot AI review requested due to automatic review settings June 10, 2026 09:16

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR migrates/extends Variant-array inverted-index debug-point coverage into the BE index storage test framework and adjusts several regression/FE areas to improve correctness around Variant/nullable wrapping, date/time diff behavior, and Iceberg partition parsing.

Changes:

  • Updates regression suites/expected outputs for Variant compaction and Nereids date-diff edge cases (timezone-sensitive + nullable inputs).
  • Adds/adjusts FE helpers for unwrapping Nullable/NonNullable around literals during signature matching, plus date/datetime literal diff helpers and Iceberg float/double special-value parsing tests.
  • Extends an Iceberg regression suite with a new “UNSUPPORTED” expectation for a Variant repro table.

Reviewed changes

Copilot reviewed 45 out of 47 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
regression-test/suites/variant_p0/compaction/test_compaction.groovy Adjusts Variant compaction query projections/casts and includes mixed-type array case in results.
regression-test/data/variant_p0/compaction/test_compaction.out Updates golden output for the added mixed-type array row(s).
regression-test/suites/nereids_function_p0/scalar_function/D.groovy Adds timezone-scoped date/datetime diff regression coverage (nullable + timestamptz forms).
regression-test/data/nereids_function_p0/scalar_function/D.out Golden output for the new timezone/date-diff queries.
regression-test/suites/external_table_p0/iceberg/test_iceberg_varbinary.groovy Adds a Variant repro query and an expected “UNSUPPORTED” failure case.
regression-test/pipeline/external/conf/regression-conf.groovy Updates external regression exclusions list.
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergUtilsTest.java Adds unit test coverage for NaN/Infinity float/double partition parsing.
fe/fe-core/src/main/java/org/apache/doris/nereids/util/ExpressionUtils.java Adds helper to unwrap Nullable/NonNullable wrappers when inspecting literals.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/SearchSignature.java Uses the new unwrap helper during string-literal coercion for signature matching.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateV2Literal.java Adds day-diff helper based on total-days arithmetic.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeV2Literal.java Adds date/datetime diff helpers with “round toward zero” semantics.
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/literal/DateTimeLiteral.java Exposes microsecond time-part extraction used by diff helpers.
Comments suppressed due to low confidence (1)

regression-test/pipeline/external/conf/regression-conf.groovy:80

  • excludeSuites includes test_paimon_gcs twice, which is redundant and makes the exclusion list harder to maintain.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +165 to +168
sql """ use test_db; """
qt_select23 """
select id from test_variant_repro;
"""
Comment on lines +261 to +263
Optional<Literal> literalAfterUnwrapNullable = ExpressionUtils.getLiteralAfterUnwrapNullable(argument);
if (!argument.isNullLiteral() && literalAfterUnwrapNullable.isPresent() && realType.isStringLikeType()) {
String literalValue = literalAfterUnwrapNullable.get().getStringValue();
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Split Variant index storage lifecycle coverage out of the non-Variant index storage lifecycle test file so text/non-Variant and Variant storage cases can evolve independently. Add a storage-level migration for the variant_p0 multi-index compaction debug-point scenario by verifying that equality predicates on a predefined Variant string path choose the plain field-pattern index, not the parent, phrase, sibling, or array path indexes, before and after cumulative compaction.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
    - `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.VariantPathEqualityChoosesStringIndexAfterCompaction'`
    - `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
    - `./run-be-ut.sh --run --filter='IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*'`
    - `./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` (expected failure, reproduces hidden child loss before fix)
- Behavior changed: No
- Does this need documentation: No

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0295758fd1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

}
const auto full_path = column.path_info_ptr()->get_path();
const auto relative_path = column.path_info_ptr()->copy_pop_front().get_path();
if (field_pattern == full_path || field_pattern == relative_path) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Match glob field patterns when resolving variant subcolumns

When an inverted index is defined on a variant field declared with MATCH_NAME_GLOB (for example a child pattern like metrics.* and an index property field_pattern='metrics.*'), this equality check never matches the materialized subcolumn path such as metrics.score, so the resolver falls back to the parent variant column. The subsequent index-build path skips unsupported variant roots, leaving the ALTER/link operation without building the requested subcolumn index for glob-pattern schemas; this should use the same glob matching logic as generate_sub_column_info() rather than exact string comparison only.

Useful? React with 👍 / 👎.

Comment on lines +399 to +400
- name: Sync Codex auth back to OSS
if: ${{ always() }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make auth sync non-blocking after review

When the Codex review step succeeds but this post-review OSS persistence step hits a transient upload failure or the auth file validation changes, the job still fails because the step is not continue-on-error like the memories/Litefuse syncs. In that scenario a review has already been submitted, but the reusable workflow reports failure and the caller leaves the required code-review status pending, so successful reviews can still block PRs for an unrelated auth-cache write failure.

Useful? React with 👍 / 👎.

Comment on lines +39 to +40
if (!_retry_slow_down && error.GetExceptionName() == "SlowDown" &&
error.GetResponseCode() == Aws::Http::HttpResponseCode::SERVICE_UNAVAILABLE) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Honor SlowDown retry opt-out for HTTP 429 throttling

When an S3-compatible service reports throttling as HTTP 429, setting retry_slow_down to false still retries because the early return only catches SlowDown with 503, and the following IsRetryableHttpResponseCode branch treats 429 as retryable. This means the new cloud s3_client_retry_slow_down=false switch does not actually suppress all 429/503 SlowDown-style throttling retries as advertised, so deployments trying to fail fast on throttling can continue generating retry traffic.

Useful? React with 👍 / 👎.

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Split pure index storage schema patch unit tests out of the lifecycle test files. The lifecycle files now focus on rowset write/read/index/compaction behavior, while text and variant schema patch construction checks live in dedicated files. This keeps future migrations from variant debug point and storage tests easier to place without mixing schema-only checks with lifecycle coverage.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageSchemaPatchTest.*:IndexStorageVariantSchemaPatchTest.*'`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
    - Red test, expected failure retained: `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: apache#64205

Problem Summary: Add storage-level coverage for typed Variant field-pattern inverted indexes across cumulative and full compaction. The test writes typed glob predefined paths, verifies that an INT predicate uses only the matching field-pattern index before compaction, then reloads the compacted rowset and verifies the same index behavior after compaction. Shared Variant index assertion helpers are moved into the index storage UT utility layer so later Variant index storage cases can reuse them instead of duplicating probe-event scans.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantFieldPatternIndexTest.*'`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
    - Expected red test: `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'` fails with `has_hidden_child` false.
- Behavior changed: No
- Does this need documentation: No
@eldenmoon
eldenmoon force-pushed the branch-variant-debugpoint-ut-migration branch from de9957e to 5c7c0e5 Compare June 10, 2026 10:10

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c7c0e5825

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread be/src/util/s3_util.cpp
Comment on lines 527 to +528
aws_config.retryStrategy = std::make_shared<S3CustomRetryStrategy>(
config::max_s3_client_retry /*scaleFactor = 25*/);
config::max_s3_client_retry /*scaleFactor = 25*/, /*retry_slow_down=*/false);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve BE SlowDown retries

When a BE S3 client receives a 503 SlowDown response, this explicit retry_slow_down=false makes S3CustomRetryStrategy::ShouldRetry() return false before the normal retryable-HTTP branch, so BE reads/writes fail immediately regardless of max_s3_client_retry. Cloud recycler has a new config knob for this behavior, but the BE path has no corresponding opt-out and previously retried 503 SlowDown through IsRetryableHttpResponseCode, so S3 throttling can now surface as user-visible query/load failures instead of being retried.

Useful? React with 👍 / 👎.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#64205

Problem Summary: Extend the index storage unit-test fixture to pass common expression pushdown contexts into rowset readers, then add a separate Variant expression-pushdown test for IS NULL and IS NOT NULL over a field-pattern indexed typed path. This keeps expression-level index coverage separate from field-pattern column-predicate and compaction coverage, and lets future Variant expression tests reuse the same write/read/assert framework.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
    - `git diff --cached --check`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*'`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- Behavior changed: No
- Does this need documentation: No
@eldenmoon
eldenmoon force-pushed the branch-variant-debugpoint-ut-migration branch from a69280c to a9a5ffa Compare June 11, 2026 01:45
### What problem does this PR solve?

Issue Number: None

Related PR: apache#64205

Problem Summary: Add storage-level unit coverage for Variant array_contains common expression pushdown over a field-pattern array inverted index. The test migrates the array_func.array_contains debugpoint coverage from variant_p0 into the index storage fixture by asserting the index bitmap cardinality, rows filtered by the inverted index, and returned rows without relying on SQL profile polling.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - `PATH=/mnt/disk1/claude-max/ldb_toolchain16/bin:$PATH build-support/clang-format.sh`
    - `git diff --check`
    - `git diff --cached --check`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantExprPushdownTest.*'`
    - `env DORIS_CLANG_HOME=/mnt/disk1/claude-max/ldb_toolchain20 DORIS_THIRDPARTY=/mnt/disk1/claude-max/tmp/doris/thirdparty ./run-be-ut.sh --run --filter='IndexStorageVariantDebugPointTest.*:IndexStorageVariantExprPushdownTest.*:IndexStorageVariantFieldPatternIndexTest.*:IndexStorageVariantLifecycleTest.*:IndexStorageVariantSchemaPatchTest.*:IndexStorageLifecycleTest.*:IndexStorageSchemaPatchTest.*-IndexStorageVariantLifecycleTest.ExactSparsePathReadsHiddenChildAfterSparseStatsLimitTruncated'`
- Behavior changed: No
- Does this need documentation: No
@eldenmoon eldenmoon changed the title [test](be) Migrate variant array index debug point test [test](be) Migrate variant index storage debugpoint tests Jun 11, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 399d715bfc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +99 to +100
if (!index_metas.empty()) {
return index_metas;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Check field-pattern indexes before returning root indexes

When a rowset has a parent/root variant inverted index and the ALTER targets a field_pattern path that is not materialized in that rowset, tablet_schema->inverted_indexs(column) returns the parent index here, so this early return never consults inverted_index_by_field_pattern(). The drop path then filters by the requested index_id, finds no match, and leaves the requested field-pattern index in the output rowset schema, so a dropped path index can continue to be advertised whenever a parent variant index coexists.

Useful? React with 👍 / 👎.

Comment thread be/src/util/s3_util.cpp

Azure::Storage::Blobs::BlobClientOptions options;
options.Retry.StatusCodes.insert(Azure::Core::Http::HttpStatusCode::TooManyRequests);
options.Retry.MaxRetries = config::max_s3_client_retry;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve Azure 429 retries in BE clients

When BE is built with USE_AZURE and Azure Blob returns HTTP 429 TooManyRequests, this client path now only sets MaxRetries after removing the explicit retry-status insertion, and BE has no replacement config like the cloud recycler path. As a result, Azure throttling can surface immediately as read/write failures despite max_s3_client_retry being configured, which regresses the previous retry behavior for transient 429 responses.

Useful? React with 👍 / 👎.

eldenmoon pushed a commit that referenced this pull request Sep 1, 2026
Problem Summary: Streaming `ROWS` window aggregates retain state across
frame evaluations. The eviction path previously considered only whether
buffered blocks had been emitted, so it could erase either the outgoing
row needed by a bounded sliding frame or the next unread row needed by
an `UNBOUNDED PRECEDING ... N PRECEDING` frame. After rebasing, negative
partition and outgoing positions could allow a nullable aggregate to
access its null map out of bounds. Evicting either kind of required row
could also produce incorrect aggregate results.

Root cause: `_remove_unused_rows()` did not account for the earliest row
required by the next ROWS frame evaluation, and
`BoundaryPose::remove_unused_rows()` allowed retained-column coordinates
to become negative.

This change defers block-aligned eviction when the candidate prefix
contains either `frame_start - 1`, the outgoing row required by a
bounded sliding update, or the next unread row required by an `UNBOUNDED
PRECEDING ... N PRECEDING` frame. It also rebases partition and order
boundaries to nonnegative physical-column coordinates. The BE unit
coverage exercises both ROWS executors across eviction boundaries and
verifies boundary rebasing.

Observed ASAN failure before this change (`output/be/log/be.out`):

```text
ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 1
    #0 doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
       be/src/exprs/aggregate/aggregate_function_null_v2.h:595
    #1 doris::AggFnEvaluator::execute_function_with_incremental(...)
       be/src/exprs/vectorized_agg_fn.cpp:334
    #2 doris::AnalyticSinkLocalState::_execute_for_function<true>(...)
       be/src/exec/operator/analytic_sink_operator.cpp:385
    #3 doris::AnalyticSinkLocalState::_get_next_for_sliding_rows(...)
       be/src/exec/operator/analytic_sink_operator.cpp:203
    #4 doris::AnalyticSinkLocalState::_execute_impl(...)
       be/src/exec/operator/analytic_sink_operator.cpp:358
    #5 doris::AnalyticSinkOperatorX::sink_impl(...)
       be/src/exec/operator/analytic_sink_operator.cpp:757
SUMMARY: AddressSanitizer: heap-buffer-overflow in
doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
```


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
eldenmoon pushed a commit that referenced this pull request Oct 8, 2026
### What problem does this PR solve?

Issue Number: N/A

Problem Summary:

**Context.** `run-be-ut.sh` is the only entry point for the BE unit
tests, and on macOS it is also the only way to run them: the `BE UT
(macOS)` workflow runs `build.sh --be`, which configures with
`-DMAKE_TEST=OFF`, so nothing under `be/test` is compiled there (the
workflow says so in its own comment at
`.github/workflows/be-ut-mac.yml:123`). On a clean worktree the script
therefore fails twice before a single test executes, and then stops
seven more times while compiling `be/test`.

**1. The problem, and what it cost**

*The script never initializes two of the submodules it needs.*
`run-be-ut.sh` initializes three submodules (`contrib/datasketches-cpp`,
`contrib/apache-orc`, `contrib/clucene`); `build.sh` initializes five,
and the two it adds — `contrib/faiss` and `contrib/openblas` — are
required before a single file is compiled. `be/CMakeLists.txt:1038` adds
`be/src/storage/index/ann` unconditionally, and that directory's
`cmake-protect/CMakeLists.txt:48,66` calls `add_subdirectory()` on both.
On a fresh clone or a new git worktree the configure step therefore ends
with

CMake Error at src/storage/index/ann/cmake-protect/CMakeLists.txt:48
(add_subdirectory):
The source directory .../contrib/openblas does not contain a
CMakeLists.txt file.
CMake Error at src/storage/index/ann/cmake-protect/CMakeLists.txt:66
(add_subdirectory):
The source directory .../contrib/faiss does not contain a CMakeLists.txt
file.

Running `build.sh` once first hides it, which is why it goes unnoticed.

*Seven `be/test` translation units do not compile on macOS.* The Linux
BE UT build is green for all seven. `run-be-ut.sh` invokes ninja without
`-k`, so the developer fixes one file, waits out another full compile,
and meets the next one — seven round trips, each of them tens of
minutes, for a task whose whole point is to run a test.

| # | TU | Error | Root cause |
|---|---|---|---|
| 1 | `exprs/function/function_map_inner_product_test.cpp` |
`'faiss/impl/platform_macros.h' file not found` | The `APPLE` branch of
`be/test/CMakeLists.txt:170` links `vector_search_test` as
`-Wl,-force_load,$<TARGET_FILE:vector_search_test>`: a file path, which
carries no usage requirements. Every other platform links
`-Wl,--whole-archive vector_search_test`, which is a target in the link
list and does propagate them. |
| 2 | `exprs/function/function_format_round_test.cpp` | `ambiguous
conversion for functional-style cast from 'long' to 'Decimal64'` (11
sites) | `Decimal64` is `Decimal<int64_t>`, so on Darwin the constructor
parameter is `long long` while these literals are `long`; every integral
and floating constructor is then an equally ranked conversion away. |
| 3 | `storage/index/snii/bench/bkd_native_vs_clucene_bench_test.cpp` |
six `-Werror,-Wformat` | `uint64_t` is `unsigned long long` on Darwin
and `unsigned long` on Linux, so `%lu` / `%ld` is right on one and wrong
on the other. |
| 4 | `storage/index/snii/bench/snii_vs_v3_benchmark_test.cpp` | `no
member named 'posix_fadvise'`, `mincore` argument type | `posix_fadvise`
/ `POSIX_FADV_DONTNEED` do not exist on Darwin; `mincore(2)` takes
`char*` there and `unsigned char*` on Linux. |
| 5 | `storage/index/snii/bkd/bkd_corruption_test.cpp` | `deduced
conflicting types ('size_type' vs 'uint64_t')` | An initializer list
mixing `payload.size()` (size_t) with `uint64_t` literals. |
| 6 | `storage/index/snii/writer/spimi_locality_bench_test.cpp` | libc++
`no matching function for call to object of type 'key_equal'` | The
transparent equality functor defines one argument order only. libc++
calls `key_equal` as `(key, lookup key)`, libstdc++ as `(lookup key,
key)`. |
| 7 | `storage/segment/hierarchical_data_iterator_test.cpp` | three `no
matching function for call to 'min'` | `kRows` is `size_t` and
`current_ordinal` is `ordinal_t` (`uint64_t`), which are distinct types
on Darwin, so `std::min` cannot deduce a common type. |

None of the seven is related to a local change; a clean master
reproduces all of them. #2 arrived on 2026-09-24 with apache#68397 (`[fix](be)
Preserve decimal precision in format_round`, by TengJianPing), which is
the commit that added `function_format_round_test.cpp` — after the last
full macOS sweep. So this class of breakage is still being produced:
"green on Linux, red on macOS" is invisible to every CI job we have, and
the two obstacles above make the one manual path that would find it
expensive enough that nobody walks it.

**2. What this PR does, and why it helps**

- `run-be-ut.sh`: the `update_submodule` list gains `contrib/openblas`
and `contrib/faiss`, with the same archive URLs `build.sh:941` already
uses, plus a comment that the list has to stay in sync with `build.sh`.
- `be/test/CMakeLists.txt`: the `APPLE` branch links `ann_index` in
addition to the `force_load` flag. `ann_index` is the target that
exports the faiss include directories `PUBLIC`, so the APPLE link line
now feeds the same compile and link inputs as every other platform,
instead of being a macOS special case.
- Six test files: the smallest change that removes the platform
difference, each spelled the way the file's neighbours already spell it
— `PRIu64` / `PRId64` (`core/decimal12.h`, `util/histogram.cpp`),
`int64_t(...)` around the literals (`storage/key_coder_test.cpp:392`,
`runtime/jsonb_document_test.cpp:433`), `#if
defined(POSIX_FADV_DONTNEED)` around the call.

Linux behaviour is unchanged: every edit is either inside an
`APPLE`-only path or a strictly more portable spelling of the same
value. On macOS `run-be-ut.sh` becomes a usable test line for the first
time.

One cost worth naming: because the APPLE link line now propagates
include directories, the compile command of every `be/test` TU changes
once, so the first macOS build after this lands rebuilds `be/test`
(about 1230 TUs). Later builds are incremental again.

**Branch pick.** apache#68397 was never picked to `branch-4.1`, so
`function_format_round_test.cpp` does not exist there. Of the seven
files, only `storage/segment/hierarchical_data_iterator_test.cpp` exists
on `branch-4.1`, with the same three `std::min(*rows, ROWS -
current_ordinal)` sites and the same types (`ROWS` is `size_t`,
`ordinal_t` is `uint64_t`), so it fails to compile on macOS there too;
that branch also has no `APPLE` `force_load` branch in
`be/test/CMakeLists.txt`, so the faiss part is master-only. If 4.1 is
meant to run BE UT on macOS, the pick is that one file's hunk.

**3. The classes, and how they call each other**

- `run-be-ut.sh` (changed): `update_submodule()` calls `git submodule
update --init --recursive <path>` and falls back to the branch tarball;
the two added lines are identical in shape to the three that were
already there.
- `be/test/CMakeLists.txt` (changed): the `APPLE` branch of the
`doris_be_test` link step.
- `be/src/storage/index/ann/cmake-protect/CMakeLists.txt` (unchanged,
the reason the submodules are mandatory): `add_subdirectory()` on
`contrib/openblas` and `contrib/faiss`, reached unconditionally from
`be/CMakeLists.txt:1038`.
- `ann_index` (unchanged target in `be/src/storage/index/ann`): links
`faiss` `PUBLIC`, so its `INTERFACE_INCLUDE_DIRECTORIES` carry the faiss
include path.
- `vector_search_test` (unchanged target in
`be/test/storage/index/ann`): links `ann_index` `PUBLIC`; on APPLE it
reaches `doris_be_test` as forced-in objects rather than as a target.
- The seven test TUs: unchanged in behaviour, three of them compiled in
a portable spelling.

```
run-be-ut.sh
  |- update_submodule contrib/{datasketches-cpp, apache-orc, clucene}   [already there]
  '- update_submodule contrib/{openblas, faiss}                        [added here]
        '- git submodule update --init --recursive

be/CMakeLists.txt:1038  add_subdirectory(be/src/storage/index/ann)
  '- storage/index/ann/CMakeLists.txt:18  add_subdirectory(cmake-protect)
        '- cmake-protect/CMakeLists.txt:48,66  add_subdirectory(contrib/{openblas, faiss})
              '- target faiss --(PUBLIC)--> target ann_index --(PUBLIC)--> vector_search_test
                                                                              |
be/test/CMakeLists.txt:170  if (APPLE)                                        |
  |- before: -Wl,-force_load,<path to libvector_search_test.a>  -- propagates nothing
  '- after:  the same flag + ann_index                          -- propagates the faiss include dirs
                                                                              |
                                        function_map_inner_product_test.cpp <--'
                                        (and every other be/test TU, which now
                                         compiles with the same inputs as on Linux)

be/test/CMakeLists.txt  add_executable(doris_be_test ${UT_FILES})   <- the other six TUs, no link involvement
```
eldenmoon pushed a commit that referenced this pull request Oct 8, 2026
…pache#68613)

Unfiltered ANN searches scoped to complete index segments can spend time
materializing redundant row-ID allowlists. Update lance-c to an
immutable upstream revision containing the merged Lance
development-branch fix lance-format/lance#9599 and ANN diagnostics from
lance-format/lance#9602, through lance-format/lance-c#89.

Keep all master changes in `thirdparty/`. Remove the superseded local
lance-c patch chain and retain the Foyer cache integration while its
interface remains outside upstream main. The upstream revision also
carries the merged lance-format/lance#9537 PQ scoring change and ANN
stage timing diagnostics; no Doris FE or BE code changes are included
here.

The prefilter fast path requires complete visible coverage of every
selected segment and no predicate. Partial coverage and predicates
retain row-ID prefiltering, while deletions and unindexed fallback
remain effective.

Pin lance-c `9bd730add2ac70316c1d642b8459011e2dd92022`, using the merged
Lance apache#9602 revision `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. This
includes parallel-path partition-preparation timing. The upstream
callback regression checks timer presence and nanosecond units while
accepting valid zero durations.

The Foyer wrapper caches immutable ObjectMeta and Attributes
independently of HTTP response extensions, avoiding repeated HEAD
requests on warm range reads. It does not replay transport extensions
from cached metadata. The HTTP regression checks HEAD/GET counts,
returned bytes/ranges/metadata, fresh dataset scopes, and explicit HEAD
bypass.

Generate the retained Foyer patch from source commit
`24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f` relative to the pinned
lance-c baseline. The header records both revisions and the regeneration
command. The source is submitted as
zhangstar333/lance-c#3 against the branch behind
lance-format/lance-c#73, on top of merged source-alignment PR #2. Doris
consumes the exact reviewed source revision while that supplementary PR
awaits merge.

Scope metadata, size, and block keys to a random namespace for each live
underlying object-store instance. The wrapping API omits a complete
stable backend identity; identical bucket/path names alone cannot
distinguish S3-compatible endpoints. Weak identity records preserve
sharing for the same live store without retaining it, while new stores
and process restarts start cold. A fresh Dataset open that creates a new
store consequently needs to warm its own cache. This deliberately
reduces reuse across instances to prevent returning another origin's
data; it does not claim unchanged cache-hit rates or production latency.

Avoid inserting an unchanged size entry into the WriteOnInsertion hybrid
cache: look up both tiers first and populate only absent or invalid size
records. Regression tests cover real HTTP endpoints sharing
bucket/path/ETag, batched data and NotFound isolation, replaced origins
after disk recovery, weak-reference lifetime, and actual disk-write
bytes. Five warm range reads wrote 40 KB before the fix and zero bytes
after it in the regression.

Fingerprint the patch in the downloader so existing source trees refresh
after updates, including legacy empty markers. Identical patches reuse
cached sources. Check platform definitions under nounset with simulated
Darwin x86_64/arm64, and make the optional ADBC source guard safe when
unset on master. Downloader lifecycle and platform handling remain
downstream in Doris.

Validation: 461 regular Rust tests passed on the final source; Rust
formatting and diff checks passed. GNU patch and git apply checks
passed, and all 87 tracked source files match the recorded source
commit. Downloader tests passed on master, branch-4.1, and the
downstream hotfix branch for fresh extraction, idempotence,
re-extraction, generic markers, legacy/mismatched Foyer markers, and
patch failure. Shell syntax and simulated macOS initialization passed.
All three opt-in native consumer tests passed: C calls, C++ calls, and
static OSS HTTP transport. Full Doris builds and BE integration
execution remain in PR CI.
eldenmoon pushed a commit that referenced this pull request Oct 19, 2026
…pache#69100)

### What problem does this PR solve?

Problem Summary:

`RleEncoder::FlushRepeatedRun()` wrote `repeat_count_ << 1` into an
`int32_t`. `repeat_count_` was an `int`, so a repeated run of 2^30 or
more values overflowed the indicator into a negative number.
`BitWriter::PutVlqInt()` then took that as an `int32_t` and shifted it
right arithmetically:

```cpp
inline void BitWriter::PutVlqInt(int32_t v) {
    while ((v & 0xFFFFFF80) != 0L) {
        PutAligned<uint8_t>((v & 0x7F) | 0x80, 1);
        v >>= 7;
    }
    PutAligned<uint8_t>(v & 0x7F, 1);
}
```

An arithmetic right shift of a negative value converges to `-1` and
stays there, and `-1 & 0xFFFFFF80` is non-zero, so **the loop never
exits**. It appends `0xFF` to the output buffer forever, pinning a core
at 100% and growing the buffer without bound.

#### How it was hit

A base compaction in a production cluster wedged on a table with many
nullable `text` columns. Two `BaseCompactionT` threads sat in this stack
for over a month:

```
#0  inline_memcpy (size=348057208)
#2  doris::faststring::GrowArray (newcapacity=502825728)
#4  doris::BitWriter::GetNextBytePtr (num_bytes=1)
#5  doris::BitWriter::PutAligned<unsigned char> (val=255 '\377')
#6  doris::BitWriter::PutVlqInt (v=-1)              bit_stream_utils.inline.h:91
#7  doris::RleEncoder<bool>::FlushRepeatedRun       rle_encoding.h:499
#9  doris::segment_v2::NullBitmapBuilder::finish
#11 doris::segment_v2::ScalarColumnWriter::finish
#15 doris::Merger::vertical_compact_one_group (is_key=false, max_rows_per_segment=1073741824)
apache#20 doris::CloudBaseCompaction::execute_compact
```

`v=-1` together with `val=255` pins it down exactly: the loop is
emitting `0xFF` with `v` stuck at `-1`.

Two existing behaviours combined to produce a 2^30-value run:

- Vertical compaction derives the segment row limit from
`vertical_compaction_max_segment_size / avg_bytes_per_row`
([`Compaction::get_avg_segment_rows()`](https://github.com/apache/doris/blob/master/be/src/olap/compaction.cpp)).
For this table the average row size integer-divided to `0`, so the
divisor degenerated to `1` and the limit became the config default
`1073741824` = 2^30 rows — visible as `max_rows_per_segment=1073741824`
in the stack above.
- `ScalarColumnWriter::append_nulls()` never checks whether the page is
full, unlike `append_data()`, and NULLs never reach the page builder. So
an all-NULL column accumulated a single null-bitmap run spanning the
whole 2^30-row segment in one page.

Because the stuck tasks kept the base compaction slots occupied
(`max_base_compaction_task_num_per_disk` defaults to 2),
`_generate_cloud_compaction_tasks()` computed `base_n = 0` and stopped
picking any tablet, so **no base compaction ran on that backend again**
and compaction scores climbed to the cap across the instance.

### What this PR changes

1. **`PutVlqInt()` takes `uint32_t`.** A logical shift always converges
to 0, so the loop terminates for every possible input. This is the
backstop that makes a hang impossible regardless of what callers pass.

2. **`repeat_count_` widened to `int64_t`, and long runs are split.**
Runs longer than `MAX_VALUES_PER_REPEATED_RUN` (2^30 - 1) are emitted as
several back-to-back repeated runs of the same value. Readers already
merge adjacent runs carrying the same value (`RleDecoder::GetNextRun`),
so splitting is invisible to them. The cap keeps the indicator inside
`int32_t`'s positive range — exactly the range previous versions could
encode — so **output stays byte-identical for every run length that used
to work**, and no reader change is needed.

3. **`faststring::reserve()` grows geometrically** via the existing
`GrowToAtLeast()` instead of reallocating to exactly the requested
capacity. `BitWriter` reserves a handful of bytes at a time, so exact
growth reallocated and memcpy'd the entire buffer every 8 bytes, making
an n-byte append cost O(n^2). This is what turned the infinite loop into
a thread stuck in `memcpy` with a ~500MB buffer, and it slows down every
other large RLE buffer as well. Measured: reaching 1 MiB now takes 26
reallocations instead of ~131,000.

Deliberately **not** in this PR, each worth its own change:
- `ScalarColumnWriter::append_nulls()` still lets a page grow without
bound.
- `Compaction::get_avg_segment_rows()` still degenerates to the
configured byte size when the average row size rounds down to zero.

### Release note

Fix a BE hang in RLE encoding when a column produces a run of 2^30 or
more identical values (for example an all-NULL column in a very large
segment), which could permanently stall base compaction on that backend.

### Check List (For Author)

- Test
    - [x] Unit Test

New tests:

| Test | Covers |
|---|---|
| `TestRle.TestRepeatedRunExceedingIndicatorRange` | Run lengths around
and far beyond 2^30 encode compactly and decode back to the original
length |
| `TestBitStreamUtil.TestPutVlqIntRoundTrip` | `PutVlqInt` round-trips
every `uint32_t`, including values `>= 2^31` that used to hang |
| `FaststringTest.TestReserve_ExponentiallyExpand` | `reserve()` does
not reallocate on every call |

Verified that the new tests actually catch the bug: built the exact
repro (`RleEncoder<bool>::Put(true, 1 << 30)` followed by `Flush()`)
against unpatched master and it never returns (killed by timeout); with
this patch it completes and emits 8 bytes. All pre-existing tests in
`rle_encoding_test`, `bit_stream_utils_test` and `faststring_test` pass
(29 tests, with `LOOP_LESS_OR_MORE` forced to the slow/thorough branch).

Also confirmed no format regression: a short run encoded via `Put(v)` in
a loop and via `Put(v, n)` in one call still produce byte-identical
output.

- Behavior changed:
- [x] No. Encoded output is unchanged for every run length that
previously encoded correctly; only run lengths that used to hang now
produce (correct, readable) output.

- Does this need documentation?
    - [x] No.

Co-authored-by: Xin Liao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants