Skip to content

[Bug](High) BE crashes in VMergeIteratorContext::compare while initializing an OLAP scan #66390

New issue
New issue
Closed
#66432
Closed
[Bug](High) BE crashes in VMergeIteratorContext::compare while initializing an OLAP scan#66390
#66432

Description

@MoanasDaddyXu
MoanasDaddyXu
opened on Aug 3, 2026

Search before asking

  • I searched the existing issues. [Bug] VMergeIteratorContext::compare core dump #16495 reports a crash in the same comparator, but it is closed and its stack reaches VMergeIteratorContext::compare() from _next_batch(). The crash reported here occurs while VMergeIterator::init() is building its heap on a current master commit.

Version

Reproduced on Apache Doris commit 1590e7e3228f8f0cafcaa89e841fece09c2475be in a non-cloud deployment with 1 FE and 3 BEs.

What's Wrong?

During a ROW binlog regression workload, a valid query caused a BE to receive SIGSEGV. The first Doris business frame is VMergeIteratorContext::compare() at src/storage/iterator/vgeneric_iterators.cpp:128.

The comparator is called by std::push_heap() while VMergeIterator::init() initializes rowset readers. The crash terminates the BE instead of returning a query error.

What You Expected?

A valid OLAP scan should initialize its rowset merge iterators and complete normally. If a rowset or iterator state is invalid, Doris should return a diagnostic error instead of dereferencing invalid memory and terminating the BE process.

How to Reproduce?

The crash was observed during a ROW binlog regression workload on a 1 FE / 3 BE cluster. The original SQL could not be recovered from the retained audit/profile records, so a standalone deterministic reproduction is not available yet.

The current reproduction gap is to recover or minimize the query and identify the exact rowset/segment combination passed to VMergeIterator::init().

Regression Test Result

Sanitized continuous stack:

doris::signal::FailureSignalHandler(...)
libc.so.6
doris::VMergeIteratorContext::compare(...) const
  at src/storage/iterator/vgeneric_iterators.cpp:128
std::__push_heap(...)
  at bits/stl_heap.h:140
std::push_heap(...)
doris::VMergeIterator::init(...)
  at src/storage/iterator/vgeneric_iterators.cpp:373
doris::BetaRowsetReader::_init_iterator()
  at src/storage/rowset/beta_rowset_reader.cpp:407
doris::BetaRowsetReader::_init_iterator_once()
  at src/storage/rowset/beta_rowset_reader.cpp:368
doris::BetaRowsetReader::_next_batch<doris::BlockWithSameBit>(...)
  at src/storage/rowset/beta_rowset_reader.h:102
doris::BetaRowsetReader::next_batch(...)
  at src/storage/rowset/beta_rowset_reader.h:59
doris::VCollectIterator::Level0Iterator::refresh_current_row()
  at src/storage/iterator/vcollect_iterator.cpp:537
doris::VCollectIterator::Level0Iterator::init(bool)
  at src/storage/iterator/vcollect_iterator.cpp:489
doris::VCollectIterator::build_heap(...)
  at src/storage/iterator/vcollect_iterator.cpp:142
doris::BlockReader::_init_collect_iter(...)
  at src/storage/iterator/block_reader.cpp:481
doris::BlockReader::init(...)
  at src/storage/iterator/block_reader.cpp:601
doris::OlapScanner::_open_impl(...)
  at src/exec/scan/olap_scanner.cpp:316
doris::Scanner::open(...)
doris::ScannerScheduler::_scanner_scan(...)
doris::ScannerSplitRunner::process_for(...)
  at src/exec/scan/scanner_scheduler.cpp:384
doris::PrioritizedSplitRunner::process()
doris::TimeSharingTaskExecutor::_dispatch_thread()
  at src/exec/scan/task_executor/time_sharing/time_sharing_task_executor.cpp:574
doris::Thread::supervise_thread(void*)
  at src/util/thread.cpp:461
start_thread
clone3

Initial Analysis

Confirmed:

  • The direct crash point is VMergeIteratorContext::compare().
  • The crash happens during heap construction in VMergeIterator::init(), before the scanner starts returning normal batches.
  • The running BE commit matches the public Apache Doris commit listed above.
  • This differs from [Bug] VMergeIteratorContext::compare core dump #16495, where the comparator was reached from _next_batch().

Suspected:

  • One merge context may contain an invalid current-row reference, comparison column, or underlying column object when the heap comparator runs.
  • The lifetime and initialization state of the contexts inserted into the heap should be checked, especially after each rowset reader performs its first batch load.

Anything Else?

Tracking issue: #65265

Related historical issue: #16495

Additional evidence still needed:

  • A minimized SQL/DDL/data reproduction.
  • The exact rowset and segment metadata involved in the failing comparison.
  • Core-variable inspection for both comparator operands at the crash frame.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Activity

  1. Doris-Breakwater commented on Aug 3, 2026

    @Doris-Breakwater
    Doris-Breakwater
    on Aug 3, 2026
    Collaborator

    Initial maintainer analysis

    Triage: This is a credible high-severity BE crash. The issue currently has no labels; bug, BE/storage, and the project's high-priority label would be appropriate. I found a concrete code-level crash condition at the reported commit that closely matches the stack, but the missing SQL/core operands prevent attributing this particular incident with certainty.

    Verified code facts

    • VMergeIterator::init() initializes each context, skips invalid/EOF contexts, and then pushes a shared_ptr into the heap. This makes a simple context-lifetime or "uninitialized context was pushed" explanation unlikely: vgeneric_iterators.cpp#L294-L303, vgeneric_iterators.cpp#L357-L373.
    • Every ROW-binlog scan forces a key-ordered read, sets the order-by prefix count to zero, and can force a per-rowset merge even for non-overlapping segments: olap_scanner.cpp#L568-L578, beta_rowset_reader.h#L62-L65. With a zero prefix, _compare_columns remains null, so the comparator takes the exact line-128 branch and compares _num_key_columns.
    • VMergeIteratorContext::_num_key_columns comes from _output_schema->num_key_columns(): vgeneric_iterators.h#L86-L98. However, Schema(columns, col_ids) counts key columns across the entire tablet schema, not across the projected col_ids: schema.h#L51-L81.
    • For MIN_DELTA/DETAIL, the scanner explicitly widens the storage projection with all key columns. For other binlog modes, notably APPEND_ONLY, the direct_mode branch can leave return_columns as only the SQL projection: olap_scanner.cpp#L471-L525. This is reachable for the normal no-storage-merge DUP/UNIQUE-MOW path.
    • Block::compare_at() only has DCHECK bounds checks and then uses unchecked positional access. In a release BE, _num_key_columns > block.columns() can therefore dereference past the block's column array instead of returning a Status: block.h#L375-L385.

    Leading root-cause hypothesis (strong, not incident-confirmed)

    An APPEND_ONLY ROW-binlog query projected fewer columns than the tablet's key count, or projected columns without all leading keys. The output block was built from that projection, while the merge comparator retained the full tablet key count. Once a rowset with at least two segments was initialized, the second heap insertion called compare() and read a missing/wrong positional column.

    This explains all distinctive evidence in the report:

    1. ROW-binlog workload;
    2. VMergeIterator::init() / std::push_heap() rather than _next_batch();
    3. line 128, the default key-column comparison branch;
    4. a release-build SIGSEGV rather than a diagnostic error.

    Even when block.columns() >= _num_key_columns, a projection that omits/reorders keys can make the comparator sort by value columns, so this is also a correctness invariant issue, not only an out-of-bounds issue. This mechanism is different from #16495, whose fix corrected a sequence-column schema-ID/block-position mismatch.

    Evidence still needed to confirm this crash

    • The original SQL or at least its incrementType and projected columns, plus EXPLAIN VERBOSE.
    • SHOW CREATE TABLE/full schema, including key type, key columns, ROW-binlog properties, and hidden binlog columns.
    • The full signal header (query ID included), relevant BE log interval, and the scan tablet/rowset ID.
    • Rowset metadata: segment count, segment overlap flag, and selected segment offsets.
    • At the crashing frame, for both operands: _valid, _index_in_block, _block, _block->rows(), _block->columns(), _num_key_columns, _compare_columns, _sequence_id_idx, _output_schema->column_ids(), and _output_schema->num_column_ids(). The decisive check is whether either block has fewer columns than _num_key_columns, or whether its first _num_key_columns positions are not the full ordered key prefix.

    Recommended next steps

    1. First validate the above values in the retained core. This can confirm or reject the projected-schema hypothesis without recovering the whole SQL.
    2. Minimize from the existing ROW-binlog regression tests with an APPEND_ONLY query that omits one or all key columns and a rowset forced to contain at least two segments. Cover both DUP and UNIQUE-MOW where practical.
    3. Fix the invariant at its source: every path that requests/forces key-ordered merging must supply the full key prefix to the storage projection, while the existing output projection removes those internal columns before returning rows. Do not only catch the comparator failure, because comparing projected value columns would still silently misorder rows.
    4. Add an init-time validation that all key/explicit comparison/sequence positions exist in the blocks and return an error containing tablet, rowset, segment, projection, and comparison-index details if the contract is violated.
    5. Add a focused BE unit test using a projected output schema plus two merge inputs, and a regression test for ROW-binlog APPEND_ONLY with a key-omitting projection and a multi-segment rowset.

    Breakwater-GitHub-Analysis-Slot: slot_2156593f0d29

  2. morningman
    closed this as completedin #66432on Aug 6, 2026
  3. morningman
    added a commit that references this issue on Aug 6, 2026

    [fix](binlog) Fix BE crash when APPEND_ONLY row-binlog scan omits key…

    86c78b9
  4. wyxxxcat
    added a commit that references this issue on Aug 17, 2026

    [fix](binlog) Fix BE crash when APPEND_ONLY row-binlog scan omits key…

    9b564bb
  5. csun5285
    added 3 commits that reference this issue on Nov 11, 2026

    [refactor](storage) Shape the AGG_KEYS heap merge and fold like StarR…

    d2e0469

    [refactor](storage) Merge the rows of an AGG_KEYS key in write order …

    3b3fb60

    [refactor](storage) Name the heap merge's helpers for what they do an…

    f967864
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions