Repository navigation
[fix](variant) extract element from scalar JSONB-rooted variant columns - #4
raghav-reglobe wants to merge 2 commits into
Conversation
Iceberg/parquet variant readers produce scalar JSONB roots with no subcolumns or sparse/doc maps; element access fell through to the empty shredded-column machinery and silently returned NULL for every key while CAST(col AS STRING) worked. Add a JSONB-root branch mirroring the STRING-root branch: serialize each row via JsonbToJson and reuse the simdjson extraction path.
…ess object result nullability Two defects in variant element access surfaced by Iceberg/parquet reads of shredded variant files: 1. The scalar-root branches of get_element_column call get_root()/ get_root_type() on the input column. The parquet lazy-read predicate path (RowGroupReader::_do_lazy_read -> execute_conjuncts) hands the function a just-read, NON-finalized ColumnVariant; get_root() on a non-finalized root throws "Subcolumn is not finalized", failing the whole scan for e.g. WHERE col['k'] IS NOT NULL. Finalize up-front (the hierarchical branch already cloned finalized). 2. wrap_variant_nullable returned an ALL-NULL map whenever the result has a null/absent root. An OBJECT-shaped element result (col['k'] where k holds a nested object, e.g. a shredded typed_value sub-object) has no scalar root by construction but carries real data in subcolumns, so every such result read as NULL under CAST/IS NULL while chained access (col['k']['sub']) read through fine. Derive the null map per row from subcolumn + sparse/doc data; all-null only when there is genuinely no data source.
|
Pushed a follow-up commit with two more fixes found while testing element access against shredded variant parquet files (DuckDB-shredded, deep layout like 1. 2. Object-shaped element results read as NULL. Both verified against real shredded files: |
…projects (apache#66531) ### What problem does this PR solve? Related PR: apache#63690 Problem Summary: Eager aggregation pushdown may fail when different aggregate functions become the same expression after passing through a Project. report error: ```text 2026-08-06 04:26:54,285 INFO (mysql-nio-pool-14|325) [PushDownAggregation.visitLogicalAggregate():280] PushDownAggregation failed: Cannot invoke "org.apache.doris.nereids.trees.expressions.NamedExpression.toSlot()" because "namedExpression" is null at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalProject(EagerAggRewriter.java:718) at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalProject(EagerAggRewriter.java:90) at org.apache.doris.nereids.trees.plans.logical.LogicalProject.accept(LogicalProject.java:160) at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalUnion(EagerAggRewriter.java:582) at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalUnion(EagerAggRewriter.java:90) at org.apache.doris.nereids.trees.plans.logical.LogicalUnion.accept(LogicalUnion.java:155) ``` For example: ```text Aggregate: SUM(x)#4, SUM(y)#5 Union All Project: 0 AS x, 0 AS y Join ``` After pushing the aggregates through the Project, both functions become`SUM(0)`: ```text functions: [SUM(0), SUM(0)] aliasMap: SUM(0) -> #5 ``` Because `aliasMap` uses expression equality, only one entry is retained. The Project still tries to read both `#4` and `#5` from `BilateralState`, causing a null lookup. This PR deduplicates the child aggregate and records the ExprId mapping: ```text child aggregate: SUM(0) -> #8 ExprId mapping: #4 -> #8, #5 -> #8 ``` The Project then restores both required outputs: ```text slot#8 AS slot#4 slot#8 AS slot#5 ``` When no aggregate functions are merged, the original ExprIds are reused to avoid unnecessary aliases. The same fix also covers cases such as: ```text Project: a#1 AS x, a#1 AS y ``` where `SUM(x)` and `SUM(y)` both become `SUM(a#1)` after pushdown. ### Release note None ### Check List (For Author) - Test - [x] Regression test - `query_p0/eager_agg/bilateral_eager_agg` - Covers two aggregate functions that become the same function after Project pushdown. - [ ] Unit Test - [ ] Manual test - [ ] No need to test or manual test. - Behavior changed: - [x] Yes. - Prevents eager aggregation pushdown from failing when multiple aggregate functions become identical after Project rewriting. - `eager_aggregation_mode=1` can force eligible pushdown after a UNION. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
Problem Summary: Streaming `ROWS` window aggregates retain state across
frame evaluations. The eviction path previously considered only whether
buffered blocks had been emitted, so it could erase either the outgoing
row needed by a bounded sliding frame or the next unread row needed by
an `UNBOUNDED PRECEDING ... N PRECEDING` frame. After rebasing, negative
partition and outgoing positions could allow a nullable aggregate to
access its null map out of bounds. Evicting either kind of required row
could also produce incorrect aggregate results.
Root cause: `_remove_unused_rows()` did not account for the earliest row
required by the next ROWS frame evaluation, and
`BoundaryPose::remove_unused_rows()` allowed retained-column coordinates
to become negative.
This change defers block-aligned eviction when the candidate prefix
contains either `frame_start - 1`, the outgoing row required by a
bounded sliding update, or the next unread row required by an `UNBOUNDED
PRECEDING ... N PRECEDING` frame. It also rebases partition and order
boundaries to nonnegative physical-column coordinates. The BE unit
coverage exercises both ROWS executors across eviction boundaries and
verifies boundary rebasing.
Observed ASAN failure before this change (`output/be/log/be.out`):
```text
ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 1
#0 doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
be/src/exprs/aggregate/aggregate_function_null_v2.h:595
#1 doris::AggFnEvaluator::execute_function_with_incremental(...)
be/src/exprs/vectorized_agg_fn.cpp:334
#2 doris::AnalyticSinkLocalState::_execute_for_function<true>(...)
be/src/exec/operator/analytic_sink_operator.cpp:385
#3 doris::AnalyticSinkLocalState::_get_next_for_sliding_rows(...)
be/src/exec/operator/analytic_sink_operator.cpp:203
#4 doris::AnalyticSinkLocalState::_execute_impl(...)
be/src/exec/operator/analytic_sink_operator.cpp:358
#5 doris::AnalyticSinkOperatorX::sink_impl(...)
be/src/exec/operator/analytic_sink_operator.cpp:757
SUMMARY: AddressSanitizer: heap-buffer-overflow in
doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
```
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
…#68677) Related PR: apache/doris-skills#15; shuke987/codex#4 Problem Summary: A pipeline-equivalent local review performed with `gpt-6.1-sol` is currently rejected by the PASS receipt validator. Add its exact model ID with the existing Codex effort gate (`xhigh`, `max`, or a host-exposed and actually used `ultra`). Preserve all existing qualified models and receipt checks. Tests exercise all three accepted efforts, lower/unknown effort rejection, and rejection of similar unlisted model IDs. The sender policy is updated in a companion apache/doris-skills PR. Merge this receiver change before deploying the sender update. This PR does not change the CI Runner model or install the Codex runtime; rollout of the fork waits for Codex #4 and live-model acceptance.
…pache#69100) ### What problem does this PR solve? Problem Summary: `RleEncoder::FlushRepeatedRun()` wrote `repeat_count_ << 1` into an `int32_t`. `repeat_count_` was an `int`, so a repeated run of 2^30 or more values overflowed the indicator into a negative number. `BitWriter::PutVlqInt()` then took that as an `int32_t` and shifted it right arithmetically: ```cpp inline void BitWriter::PutVlqInt(int32_t v) { while ((v & 0xFFFFFF80) != 0L) { PutAligned<uint8_t>((v & 0x7F) | 0x80, 1); v >>= 7; } PutAligned<uint8_t>(v & 0x7F, 1); } ``` An arithmetic right shift of a negative value converges to `-1` and stays there, and `-1 & 0xFFFFFF80` is non-zero, so **the loop never exits**. It appends `0xFF` to the output buffer forever, pinning a core at 100% and growing the buffer without bound. #### How it was hit A base compaction in a production cluster wedged on a table with many nullable `text` columns. Two `BaseCompactionT` threads sat in this stack for over a month: ``` #0 inline_memcpy (size=348057208) #2 doris::faststring::GrowArray (newcapacity=502825728) #4 doris::BitWriter::GetNextBytePtr (num_bytes=1) #5 doris::BitWriter::PutAligned<unsigned char> (val=255 '\377') #6 doris::BitWriter::PutVlqInt (v=-1) bit_stream_utils.inline.h:91 #7 doris::RleEncoder<bool>::FlushRepeatedRun rle_encoding.h:499 #9 doris::segment_v2::NullBitmapBuilder::finish #11 doris::segment_v2::ScalarColumnWriter::finish #15 doris::Merger::vertical_compact_one_group (is_key=false, max_rows_per_segment=1073741824) apache#20 doris::CloudBaseCompaction::execute_compact ``` `v=-1` together with `val=255` pins it down exactly: the loop is emitting `0xFF` with `v` stuck at `-1`. Two existing behaviours combined to produce a 2^30-value run: - Vertical compaction derives the segment row limit from `vertical_compaction_max_segment_size / avg_bytes_per_row` ([`Compaction::get_avg_segment_rows()`](https://github.com/apache/doris/blob/master/be/src/olap/compaction.cpp)). For this table the average row size integer-divided to `0`, so the divisor degenerated to `1` and the limit became the config default `1073741824` = 2^30 rows — visible as `max_rows_per_segment=1073741824` in the stack above. - `ScalarColumnWriter::append_nulls()` never checks whether the page is full, unlike `append_data()`, and NULLs never reach the page builder. So an all-NULL column accumulated a single null-bitmap run spanning the whole 2^30-row segment in one page. Because the stuck tasks kept the base compaction slots occupied (`max_base_compaction_task_num_per_disk` defaults to 2), `_generate_cloud_compaction_tasks()` computed `base_n = 0` and stopped picking any tablet, so **no base compaction ran on that backend again** and compaction scores climbed to the cap across the instance. ### What this PR changes 1. **`PutVlqInt()` takes `uint32_t`.** A logical shift always converges to 0, so the loop terminates for every possible input. This is the backstop that makes a hang impossible regardless of what callers pass. 2. **`repeat_count_` widened to `int64_t`, and long runs are split.** Runs longer than `MAX_VALUES_PER_REPEATED_RUN` (2^30 - 1) are emitted as several back-to-back repeated runs of the same value. Readers already merge adjacent runs carrying the same value (`RleDecoder::GetNextRun`), so splitting is invisible to them. The cap keeps the indicator inside `int32_t`'s positive range — exactly the range previous versions could encode — so **output stays byte-identical for every run length that used to work**, and no reader change is needed. 3. **`faststring::reserve()` grows geometrically** via the existing `GrowToAtLeast()` instead of reallocating to exactly the requested capacity. `BitWriter` reserves a handful of bytes at a time, so exact growth reallocated and memcpy'd the entire buffer every 8 bytes, making an n-byte append cost O(n^2). This is what turned the infinite loop into a thread stuck in `memcpy` with a ~500MB buffer, and it slows down every other large RLE buffer as well. Measured: reaching 1 MiB now takes 26 reallocations instead of ~131,000. Deliberately **not** in this PR, each worth its own change: - `ScalarColumnWriter::append_nulls()` still lets a page grow without bound. - `Compaction::get_avg_segment_rows()` still degenerates to the configured byte size when the average row size rounds down to zero. ### Release note Fix a BE hang in RLE encoding when a column produces a run of 2^30 or more identical values (for example an all-NULL column in a very large segment), which could permanently stall base compaction on that backend. ### Check List (For Author) - Test - [x] Unit Test New tests: | Test | Covers | |---|---| | `TestRle.TestRepeatedRunExceedingIndicatorRange` | Run lengths around and far beyond 2^30 encode compactly and decode back to the original length | | `TestBitStreamUtil.TestPutVlqIntRoundTrip` | `PutVlqInt` round-trips every `uint32_t`, including values `>= 2^31` that used to hang | | `FaststringTest.TestReserve_ExponentiallyExpand` | `reserve()` does not reallocate on every call | Verified that the new tests actually catch the bug: built the exact repro (`RleEncoder<bool>::Put(true, 1 << 30)` followed by `Flush()`) against unpatched master and it never returns (killed by timeout); with this patch it completes and emits 8 bytes. All pre-existing tests in `rle_encoding_test`, `bit_stream_utils_test` and `faststring_test` pass (29 tests, with `LOOP_LESS_OR_MORE` forced to the slow/thorough branch). Also confirmed no format regression: a short run encoded via `Put(v)` in a loop and via `Put(v, n)` in one call still produce byte-identical output. - Behavior changed: - [x] No. Encoded output is unchanged for every run length that previously encoded correctly; only run lengths that used to hang now produce (correct, readable) output. - Does this need documentation? - [x] No. Co-authored-by: Xin Liao <[email protected]>
Follow-up to the root-cause comment on apache#63192: the parquet variant reader emits scalar JSONB roots (no subcolumns/sparse maps), but
FunctionVariantElement::get_element_column's scalar fallback only handles STRING-typed roots — sov['key']on Iceberg-read VARIANT silently returned NULL for every key whileCAST(v AS STRING)worked.This adds a JSONB-root branch mirroring the existing STRING-root branch: serialize each row via
JsonbToJson::to_json_stringand reuse the same simdjsonextract_from_documentpath — same STRING-rooted output shape, so chained access (v['a']['b']) works through the existing branch on the second hop. Empty/null rows insert defaults.Verified against Iceberg V3 tables with both shredded and unshredded parquet variants written by an external engine. Feel free to fold this into your PR directly if that's easier than merging here.