Skip to content

[fix](variant) extract element from scalar JSONB-rooted variant columns - #4

Open
raghav-reglobe wants to merge 2 commits into
eldenmoon:codex/iceberg-v3-variantfrom
raghav-reglobe:variant-jsonb-element-fix
Open

raghav-reglobe wants to merge 2 commits into
eldenmoon:codex/iceberg-v3-variantfrom
raghav-reglobe:variant-jsonb-element-fix

Conversation

@raghav-reglobe

Copy link
Copy Markdown

Follow-up to the root-cause comment on apache#63192: the parquet variant reader emits scalar JSONB roots (no subcolumns/sparse maps), but FunctionVariantElement::get_element_column's scalar fallback only handles STRING-typed roots — so v['key'] on Iceberg-read VARIANT silently returned NULL for every key while CAST(v AS STRING) worked.

This adds a JSONB-root branch mirroring the existing STRING-root branch: serialize each row via JsonbToJson::to_json_string and reuse the same simdjson extract_from_document path — same STRING-rooted output shape, so chained access (v['a']['b']) works through the existing branch on the second hop. Empty/null rows insert defaults.

Verified against Iceberg V3 tables with both shredded and unshredded parquet variants written by an external engine. Feel free to fold this into your PR directly if that's easier than merging here.

Iceberg/parquet variant readers produce scalar JSONB roots with no
subcolumns or sparse/doc maps; element access fell through to the empty
shredded-column machinery and silently returned NULL for every key while
CAST(col AS STRING) worked. Add a JSONB-root branch mirroring the
STRING-root branch: serialize each row via JsonbToJson and reuse the
simdjson extraction path.
…ess object result nullability

Two defects in variant element access surfaced by Iceberg/parquet reads
of shredded variant files:

1. The scalar-root branches of get_element_column call get_root()/
   get_root_type() on the input column. The parquet lazy-read predicate
   path (RowGroupReader::_do_lazy_read -> execute_conjuncts) hands the
   function a just-read, NON-finalized ColumnVariant; get_root() on a
   non-finalized root throws "Subcolumn is not finalized", failing the
   whole scan for e.g. WHERE col['k'] IS NOT NULL. Finalize up-front
   (the hierarchical branch already cloned finalized).

2. wrap_variant_nullable returned an ALL-NULL map whenever the result
   has a null/absent root. An OBJECT-shaped element result (col['k']
   where k holds a nested object, e.g. a shredded typed_value
   sub-object) has no scalar root by construction but carries real data
   in subcolumns, so every such result read as NULL under CAST/IS NULL
   while chained access (col['k']['sub']) read through fine. Derive the
   null map per row from subcolumn + sparse/doc data; all-null only when
   there is genuinely no data source.
@raghav-reglobe

Copy link
Copy Markdown
Author

Pushed a follow-up commit with two more fixes found while testing element access against shredded variant parquet files (DuckDB-shredded, deep layout like typed_value.k.typed_value.d):

1. Subcolumn is not finalized crash on predicate pushdown. WHERE col['k'] IS NOT NULL fails the whole scan: the lazy-read path (RowGroupReader::_do_lazy_read → execute_conjuncts) evaluates element_at on the just-read, non-finalized ColumnVariant, and the scalar-root branches of get_element_column call get_root() / get_root_type(), which throw on a non-finalized root. My earlier JSONB-root branch widened the exposure, but the pre-existing STRING-root branch has the same defect. Fix: finalize up-front — the hierarchical branch already worked on clone_finalized().

2. Object-shaped element results read as NULL. col['k'] where k holds a nested object produces a rootless result (data lives in subcolumns), and wrap_variant_nullable returns an all-null map for any is_null_root() column — so CAST(col['k'] AS STRING) and IS NOT NULL saw NULL on every row while chained col['k']['sub'] read through fine. Fix: derive the null map per row from subcolumn + sparse/doc data; all-null only when there is genuinely no data source.

Both verified against real shredded files: col['k'] returns the sub-object, CAST renders it (`{"d":"..."}"), and the predicate form scans without error.

eldenmoon pushed a commit that referenced this pull request Aug 13, 2026
…projects (apache#66531)

### What problem does this PR solve?

Related PR: apache#63690

Problem Summary:

Eager aggregation pushdown may fail when different aggregate functions
become
the same expression after passing through a Project. report error:

```text
2026-08-06 04:26:54,285 INFO (mysql-nio-pool-14|325) [PushDownAggregation.visitLogicalAggregate():280] PushDownAggregation failed: Cannot invoke "org.apache.doris.nereids.trees.expressions.NamedExpression.toSlot()" because "namedExpression" is null
        at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalProject(EagerAggRewriter.java:718)
        at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalProject(EagerAggRewriter.java:90)
        at org.apache.doris.nereids.trees.plans.logical.LogicalProject.accept(LogicalProject.java:160)
        at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalUnion(EagerAggRewriter.java:582)
        at org.apache.doris.nereids.rules.rewrite.eageraggregation.EagerAggRewriter.visitLogicalUnion(EagerAggRewriter.java:90)
        at org.apache.doris.nereids.trees.plans.logical.LogicalUnion.accept(LogicalUnion.java:155)
```

For example:

```text
Aggregate: SUM(x)#4, SUM(y)#5
  Union All
    Project: 0 AS x, 0 AS y
      Join
```

After pushing the aggregates through the Project, both functions
become`SUM(0)`:

```text
functions: [SUM(0), SUM(0)]
aliasMap:  SUM(0) -> #5
```

Because `aliasMap` uses expression equality, only one entry is retained.
The Project still tries to read both `#4` and `#5` from
`BilateralState`,
causing a null lookup.

This PR deduplicates the child aggregate and records the ExprId mapping:

```text
child aggregate: SUM(0) -> #8
ExprId mapping:  #4 -> #8, #5 -> #8
```

The Project then restores both required outputs:

```text
slot#8 AS slot#4
slot#8 AS slot#5
```

When no aggregate functions are merged, the original ExprIds are reused
to
avoid unnecessary aliases.

The same fix also covers cases such as:

```text
Project: a#1 AS x, a#1 AS y
```

where `SUM(x)` and `SUM(y)` both become `SUM(a#1)` after pushdown.

### Release note

None


### Check List (For Author)

- Test
    - [x] Regression test
        - `query_p0/eager_agg/bilateral_eager_agg`
- Covers two aggregate functions that become the same function after
          Project pushdown.
    - [ ] Unit Test
    - [ ] Manual test
    - [ ] No need to test or manual test.

- Behavior changed:
    - [x] Yes.
        - Prevents eager aggregation pushdown from failing when multiple
          aggregate functions become identical after Project rewriting.
- `eager_aggregation_mode=1` can force eligible pushdown after a UNION.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
eldenmoon pushed a commit that referenced this pull request Sep 1, 2026
Problem Summary: Streaming `ROWS` window aggregates retain state across
frame evaluations. The eviction path previously considered only whether
buffered blocks had been emitted, so it could erase either the outgoing
row needed by a bounded sliding frame or the next unread row needed by
an `UNBOUNDED PRECEDING ... N PRECEDING` frame. After rebasing, negative
partition and outgoing positions could allow a nullable aggregate to
access its null map out of bounds. Evicting either kind of required row
could also produce incorrect aggregate results.

Root cause: `_remove_unused_rows()` did not account for the earliest row
required by the next ROWS frame evaluation, and
`BoundaryPose::remove_unused_rows()` allowed retained-column coordinates
to become negative.

This change defers block-aligned eviction when the candidate prefix
contains either `frame_start - 1`, the outgoing row required by a
bounded sliding update, or the next unread row required by an `UNBOUNDED
PRECEDING ... N PRECEDING` frame. It also rebases partition and order
boundaries to nonnegative physical-column coordinates. The BE unit
coverage exercises both ROWS executors across eviction boundaries and
verifies boundary rebasing.

Observed ASAN failure before this change (`output/be/log/be.out`):

```text
ERROR: AddressSanitizer: heap-buffer-overflow
READ of size 1
    #0 doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
       be/src/exprs/aggregate/aggregate_function_null_v2.h:595
    #1 doris::AggFnEvaluator::execute_function_with_incremental(...)
       be/src/exprs/vectorized_agg_fn.cpp:334
    #2 doris::AnalyticSinkLocalState::_execute_for_function<true>(...)
       be/src/exec/operator/analytic_sink_operator.cpp:385
    #3 doris::AnalyticSinkLocalState::_get_next_for_sliding_rows(...)
       be/src/exec/operator/analytic_sink_operator.cpp:203
    #4 doris::AnalyticSinkLocalState::_execute_impl(...)
       be/src/exec/operator/analytic_sink_operator.cpp:358
    #5 doris::AnalyticSinkOperatorX::sink_impl(...)
       be/src/exec/operator/analytic_sink_operator.cpp:757
SUMMARY: AddressSanitizer: heap-buffer-overflow in
doris::AggregateFunctionNullUnaryInlineV2<...>::execute_function_with_incremental(...)
```


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
eldenmoon pushed a commit that referenced this pull request Oct 8, 2026
…#68677)

Related PR: apache/doris-skills#15;
shuke987/codex#4

Problem Summary: A pipeline-equivalent local review performed with
`gpt-6.1-sol` is currently rejected by the PASS receipt validator. Add
its exact model ID with the existing Codex effort gate (`xhigh`, `max`,
or a host-exposed and actually used `ultra`). Preserve all existing
qualified models and receipt checks.

Tests exercise all three accepted efforts, lower/unknown effort
rejection, and rejection of similar unlisted model IDs. The sender
policy is updated in a companion apache/doris-skills PR. Merge this
receiver change before deploying the sender update. This PR does not
change the CI Runner model or install the Codex runtime; rollout of the
fork waits for Codex #4 and live-model acceptance.
eldenmoon pushed a commit that referenced this pull request Oct 19, 2026
…pache#69100)

### What problem does this PR solve?

Problem Summary:

`RleEncoder::FlushRepeatedRun()` wrote `repeat_count_ << 1` into an
`int32_t`. `repeat_count_` was an `int`, so a repeated run of 2^30 or
more values overflowed the indicator into a negative number.
`BitWriter::PutVlqInt()` then took that as an `int32_t` and shifted it
right arithmetically:

```cpp
inline void BitWriter::PutVlqInt(int32_t v) {
    while ((v & 0xFFFFFF80) != 0L) {
        PutAligned<uint8_t>((v & 0x7F) | 0x80, 1);
        v >>= 7;
    }
    PutAligned<uint8_t>(v & 0x7F, 1);
}
```

An arithmetic right shift of a negative value converges to `-1` and
stays there, and `-1 & 0xFFFFFF80` is non-zero, so **the loop never
exits**. It appends `0xFF` to the output buffer forever, pinning a core
at 100% and growing the buffer without bound.

#### How it was hit

A base compaction in a production cluster wedged on a table with many
nullable `text` columns. Two `BaseCompactionT` threads sat in this stack
for over a month:

```
#0  inline_memcpy (size=348057208)
#2  doris::faststring::GrowArray (newcapacity=502825728)
#4  doris::BitWriter::GetNextBytePtr (num_bytes=1)
#5  doris::BitWriter::PutAligned<unsigned char> (val=255 '\377')
#6  doris::BitWriter::PutVlqInt (v=-1)              bit_stream_utils.inline.h:91
#7  doris::RleEncoder<bool>::FlushRepeatedRun       rle_encoding.h:499
#9  doris::segment_v2::NullBitmapBuilder::finish
#11 doris::segment_v2::ScalarColumnWriter::finish
#15 doris::Merger::vertical_compact_one_group (is_key=false, max_rows_per_segment=1073741824)
apache#20 doris::CloudBaseCompaction::execute_compact
```

`v=-1` together with `val=255` pins it down exactly: the loop is
emitting `0xFF` with `v` stuck at `-1`.

Two existing behaviours combined to produce a 2^30-value run:

- Vertical compaction derives the segment row limit from
`vertical_compaction_max_segment_size / avg_bytes_per_row`
([`Compaction::get_avg_segment_rows()`](https://github.com/apache/doris/blob/master/be/src/olap/compaction.cpp)).
For this table the average row size integer-divided to `0`, so the
divisor degenerated to `1` and the limit became the config default
`1073741824` = 2^30 rows — visible as `max_rows_per_segment=1073741824`
in the stack above.
- `ScalarColumnWriter::append_nulls()` never checks whether the page is
full, unlike `append_data()`, and NULLs never reach the page builder. So
an all-NULL column accumulated a single null-bitmap run spanning the
whole 2^30-row segment in one page.

Because the stuck tasks kept the base compaction slots occupied
(`max_base_compaction_task_num_per_disk` defaults to 2),
`_generate_cloud_compaction_tasks()` computed `base_n = 0` and stopped
picking any tablet, so **no base compaction ran on that backend again**
and compaction scores climbed to the cap across the instance.

### What this PR changes

1. **`PutVlqInt()` takes `uint32_t`.** A logical shift always converges
to 0, so the loop terminates for every possible input. This is the
backstop that makes a hang impossible regardless of what callers pass.

2. **`repeat_count_` widened to `int64_t`, and long runs are split.**
Runs longer than `MAX_VALUES_PER_REPEATED_RUN` (2^30 - 1) are emitted as
several back-to-back repeated runs of the same value. Readers already
merge adjacent runs carrying the same value (`RleDecoder::GetNextRun`),
so splitting is invisible to them. The cap keeps the indicator inside
`int32_t`'s positive range — exactly the range previous versions could
encode — so **output stays byte-identical for every run length that used
to work**, and no reader change is needed.

3. **`faststring::reserve()` grows geometrically** via the existing
`GrowToAtLeast()` instead of reallocating to exactly the requested
capacity. `BitWriter` reserves a handful of bytes at a time, so exact
growth reallocated and memcpy'd the entire buffer every 8 bytes, making
an n-byte append cost O(n^2). This is what turned the infinite loop into
a thread stuck in `memcpy` with a ~500MB buffer, and it slows down every
other large RLE buffer as well. Measured: reaching 1 MiB now takes 26
reallocations instead of ~131,000.

Deliberately **not** in this PR, each worth its own change:
- `ScalarColumnWriter::append_nulls()` still lets a page grow without
bound.
- `Compaction::get_avg_segment_rows()` still degenerates to the
configured byte size when the average row size rounds down to zero.

### Release note

Fix a BE hang in RLE encoding when a column produces a run of 2^30 or
more identical values (for example an all-NULL column in a very large
segment), which could permanently stall base compaction on that backend.

### Check List (For Author)

- Test
    - [x] Unit Test

New tests:

| Test | Covers |
|---|---|
| `TestRle.TestRepeatedRunExceedingIndicatorRange` | Run lengths around
and far beyond 2^30 encode compactly and decode back to the original
length |
| `TestBitStreamUtil.TestPutVlqIntRoundTrip` | `PutVlqInt` round-trips
every `uint32_t`, including values `>= 2^31` that used to hang |
| `FaststringTest.TestReserve_ExponentiallyExpand` | `reserve()` does
not reallocate on every call |

Verified that the new tests actually catch the bug: built the exact
repro (`RleEncoder<bool>::Put(true, 1 << 30)` followed by `Flush()`)
against unpatched master and it never returns (killed by timeout); with
this patch it completes and emits 8 bytes. All pre-existing tests in
`rle_encoding_test`, `bit_stream_utils_test` and `faststring_test` pass
(29 tests, with `LOOP_LESS_OR_MORE` forced to the slow/thorough branch).

Also confirmed no format regression: a short run encoded via `Put(v)` in
a loop and via `Put(v, n)` in one call still produce byte-identical
output.

- Behavior changed:
- [x] No. Encoded output is unchanged for every run length that
previously encoded correctly; only run lengths that used to hang now
produce (correct, readable) output.

- Does this need documentation?
    - [x] No.

Co-authored-by: Xin Liao <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant