You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3c43f4c
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](storage) align pruned complex column types with sub-column iterators on merge/agg read paths
When a query prunes nested sub-columns of a complex column (EXPLAIN shows
"pruned type"), the scan's struct sub-column iterators are filtered down to
the pruned subset (set_access_paths + remove_pruned_sub_iterators), while the
FE-pruned data type is recorded only in TabletSchema::_pruned_columns_data_type
and honored only by TabletSchema::create_block*. Read paths that materialize
column shapes from other sources still used the FULL storage type, so the
positional pairing in StructFileColumnIterator::next_batch worked on two
different ordinal spaces:
1. VMergeIteratorContext::block_reset built per-segment merge blocks from the
storage Schema (full type). A topn ordered read over a rowset with multiple
OVERLAPPING segments then decoded pruned children into the wrong dst
columns:
- "[E-3110] Method insert_many_dict_data is not supported for TINYINT"
(dict-encoded text child decoded into a tinyint child)
- "... is not supported for INT" (single-child pruning variant)
- BE SIGSEGV when the shifted child types happen to match and the loop
runs past the pruned iterator vector (same family as DORIS-26595)
This is the root cause of DORIS-26889: the CI's s3 insert naturally
produces overlapping multi-segment rowsets, while single-segment local
loads never take the merge path. On master the default plan usually
shields this path via TopN lazy materialization (apache#63736 + apache#64242), but
e.g. raising topn_opt_limit_threshold, disabling
topn_lazy_materialization_threshold, or light_schema_change=false tables
expose it; branch-4.1 hits it by default.
2. BlockReader::_init_agg_state built the reader_replace argument type from
the raw TabletColumn (full type) while the stored block uses the pruned
type. On AGG tables any pruned read that goes through the aggregate merge
path (>= 2 data rowsets, no special session vars needed) failed with
"Aggregate function reader_replace argument 0 type check failed:
ColumnStruct has 2 elements, but DataTypeStruct has 3 elements".
Fix: make the shape-materializing read paths use the same FE-pruned type as
the sub-column iterators, so dst blocks and iterators stay in one ordinal
space by construction.
- VMergeIteratorContext::init reads the FE-pruned complex types from
StorageReadOptions::tablet_schema (TabletSchema::pruned_columns_data_type(),
already set on the query read path), and block_reset then builds each merge
block with the pruned type for pruned columns and the full storage type for
the rest, so the block's struct children align with the pruned sub-column
iterators. No new plumbing through the merge iterator or the storage Schema.
- TabletColumn::get_aggregate_function now takes the column's actual data type
explicitly. BlockReader::_init_agg_state passes the stored block's column
type (already the pruned type for pruned complex columns), while the
load/compaction/schema-change paths pass the full column type.
The new regression case constructs an OVERLAPPING multi-segment rowset
deterministically (MemTable.need_flush debug point + NULL keys in every
batch, segment count kept below the segcompaction threshold), self-checks the
layout via tablet meta, and covers all variants: the dict-error and
SIGSEGV topn shapes on the inline path, the AGG default-session shape across
two rowsets, and the VMaterializeNode default path as a guard. All variants
fail before this fix and pass after it.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
0 commit comments