Skip to content

[Bug](Highest) APPEND_ONLY table stream query can crash BE in VCollectIterator with multi-segment rowsets #65901

New issue
New issue
Closed
#66338
Closed
[Bug](Highest) APPEND_ONLY table stream query can crash BE in VCollectIterator with multi-segment rowsets#65901
#66338

Description

@MoanasDaddyXu
MoanasDaddyXu
opened on Jul 22, 2026

Tracking

Search before asking

  • I searched the existing issues and found no issue with the same assertion or iterator-mode conflict.

Version

Reproduced on Doris commit 87d57a84729.

What's Wrong?

Querying an APPEND_ONLY table stream backed by ROW binlog can trigger a fatal assertion in the BE scan path when a data rowset contains multiple segments:

Check failed: !_get_data_by_ref

The BE then receives SIGABRT and exits.

The outer BlockReader sees a NONOVERLAPPING rowset and selects Block-based batch reading. ROW binlog requires stable ordered output, so the inner VCollectIterator selects merge/RowRef reading when the rowset has multiple segments. These independently selected modes conflict: the child iterator remains in by-reference mode while the parent calls the Block interface, causing the assertion.

The assertion prevents the engine from continuing in an inconsistent state that could otherwise produce missing, duplicated, or incorrect rows.

What You Expected?

The table stream query should complete without crashing the BE. BlockReader and all child VCollectIterator instances should use a consistent Block or RowRef mode while preserving ROW binlog ordering semantics.

How to Reproduce?

  1. Create a table with ROW binlog enabled.
  2. Create an APPEND_ONLY table stream on the table.
  3. Write enough data for a data rowset to contain multiple segments, with the rowset classified as NONOVERLAPPING.
  4. Query the table stream.
  5. Observe the BE assertion in VCollectIterator::Level0Iterator::next(Block*) followed by SIGABRT.

The issue does not reproduce when the data rowset has only one segment because the inner iterator does not enter merge/RowRef mode.

be.out

The following excerpt is limited to the relevant, sanitized be.out content:

Check failed: !_get_data_by_ref
*** Current BE git commitID: 87d57a84729 ***
*** SIGABRT received; stack trace: ***
 0# doris::signal::FailureSignalHandler(int, siginfo_t*, void*)
 1# pthread_kill
 2# raise
 3# abort
 4# google::LogMessage::SendToLog()
 5# google::LogMessage::Flush()
 6# google::LogMessageFatal::~LogMessageFatal()
 7# doris::VCollectIterator::Level0Iterator::next(doris::Block*)
 8# doris::VCollectIterator::Level1Iterator::_normal_next(doris::Block*)
 9# doris::VCollectIterator::Level1Iterator::next(doris::Block*)
10# doris::VCollectIterator::next(doris::Block*)
11# doris::BlockReader::_direct_next_block(doris::Block*, bool*)
12# doris::BlockReader::next_block_with_aggregation(doris::Block*, bool*)
13# doris::OlapScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*)
14# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*)
15# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*)
16# doris::ScannerScheduler::_scanner_scan(...)
17# doris::ScannerSplitRunner::process_for(...)
18# doris::PrioritizedSplitRunner::process()
19# doris::TimeSharingTaskExecutor::_dispatch_thread()
20# doris::Thread::supervise_thread(void*)

Root Cause

ROW binlog is the direct trigger, but the code-level defect is inconsistent mode selection between BlockReader and VCollectIterator. The parent must explicitly propagate whether Block or RowRef output is required, and Block mode must prevent every child iterator from entering by-reference mode.

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Activity

  1. zclllyybb commented on Jul 22, 2026

    @zclllyybb
    zclllyybb
    on Jul 22, 2026
    Contributor

    Breakwater-GitHub-Analysis-Slot: slot_c3914a919850
    This content is generated by AI for reference only.

    Initial assessment: this looks like a real BE storage scan bug, not a resource issue or an invalid SQL/table-stream usage. The reported stack is consistent with a caller/callee mode contract violation: the outer scan path calls the Block interface while a child rowset iterator has already been initialized for row-reference output.

    Code-path check on commit 87d57a84729:

    • APPEND_ONLY table stream is propagated as TBinlogScanType::APPEND_ONLY.
    • OlapScanner treats any row-binlog scan as ordered: it sets read_orderby_key = true and force_key_ordered_read = true.
    • For a single-version, non-overlapping layout, TabletReader::ReaderParams::has_single_version() can make the scan use direct_mode = true; BlockReader::init() only diverts MIN_DELTA and DETAIL to row-by-row binlog-specific readers, so APPEND_ONLY can still use _direct_next_block() and call VCollectIterator::next(Block*).
    • Independently, BetaRowsetReader::is_merge_iterator() returns true when ordered output is required, the rowset has more than one segment, and force_key_ordered_read is set. Therefore a multi-segment ROW-binlog rowset can become an inner merge iterator even when its rowset metadata is NONOVERLAPPING.
    • In the non-merge/union setup, a later Level0Iterator can be initialized with get_data_by_ref = true. If an earlier child is empty and skipped, that by-reference child can become the current child while the parent still uses the Block interface. VCollectIterator::Level0Iterator::next(Block*) then hits CHECK(!_get_data_by_ref), which matches the reported abort.

    So the high-confidence root cause is an inconsistent read-mode contract between BlockReader/VCollectIterator and the rowset reader under APPEND_ONLY ROW-binlog scans. The fatal check should not be removed; it is protecting against continuing after the storage layer has mixed two incompatible output protocols.

    Recommended fix direction:

    • Make the selected output protocol explicit and consistent from BlockReader down to every VCollectIterator::Level0Iterator.
    • For an APPEND_ONLY scan that will use the Block interface, prevent every child from entering _get_data_by_ref mode, while still preserving the ordered multi-segment row-binlog merge inside BetaRowsetReader.
    • Alternatively, route APPEND_ONLY through a row-reference/materialization path similar in contract to the MIN_DELTA/DETAIL readers, but without changing APPEND_ONLY semantics.
    • Add a regression test that creates a ROW-binlog table stream with a multi-segment NONOVERLAPPING data rowset, including the empty-initial-rowset plus data-rowset shape if that is the actual failing layout, then queries the APPEND_ONLY stream and verifies both no BE abort and stable ordered output.

    Useful missing information for a self-contained upstream repro:

    • Exact base table DDL, binlog/table-stream DDL, and stream query SQL.
    • The load/insert script or segment-size settings needed to force the failing multi-segment rowset.
    • Failing tablet rowset metadata: version range, row count, segment count, segments_overlap, and whether an empty rowset precedes the data rowset.
    • FE plan or scan range snippet showing read_row_binlog=true and binlog_scan_type=APPEND_ONLY.

  2. zclllyybb commented on Jul 22, 2026

    @zclllyybb
    zclllyybb
    on Jul 22, 2026
    Contributor

    Breakwater-GitHub-Analysis-Slot: slot_c3914a919850
    This content is generated by AI for reference only.

    Initial assessment: this looks like a real BE storage scan bug, not a resource issue or an invalid SQL/table-stream usage. The reported stack is consistent with a caller/callee mode contract violation: the outer scan path calls the Block interface while a child rowset iterator has already been initialized for row-reference output.

    Code-path check on commit 87d57a84729:

    • APPEND_ONLY table stream is propagated as TBinlogScanType::APPEND_ONLY.
    • OlapScanner treats any row-binlog scan as ordered: it sets read_orderby_key = true and force_key_ordered_read = true.
    • For a single-version, non-overlapping layout, TabletReader::ReaderParams::has_single_version() can make the scan use direct_mode = true; BlockReader::init() only diverts MIN_DELTA and DETAIL to row-by-row binlog-specific readers, so APPEND_ONLY can still use _direct_next_block() and call VCollectIterator::next(Block*).
    • Independently, BetaRowsetReader::is_merge_iterator() returns true when ordered output is required, the rowset has more than one segment, and force_key_ordered_read is set. Therefore a multi-segment ROW-binlog rowset can become an inner merge iterator even when its rowset metadata is NONOVERLAPPING.
    • In the non-merge/union setup, a later Level0Iterator can be initialized with get_data_by_ref = true. If an earlier child is empty and skipped, that by-reference child can become the current child while the parent still uses the Block interface. VCollectIterator::Level0Iterator::next(Block*) then hits CHECK(!_get_data_by_ref), which matches the reported abort.

    So the high-confidence root cause is an inconsistent read-mode contract between BlockReader/VCollectIterator and the rowset reader under APPEND_ONLY ROW-binlog scans. The fatal check should not be removed; it is protecting against continuing after the storage layer has mixed two incompatible output protocols.

    Recommended fix direction:

    • Make the selected output protocol explicit and consistent from BlockReader down to every VCollectIterator::Level0Iterator.
    • For an APPEND_ONLY scan that will use the Block interface, prevent every child from entering _get_data_by_ref mode, while still preserving the ordered multi-segment row-binlog merge inside BetaRowsetReader.
    • Alternatively, route APPEND_ONLY through a row-reference/materialization path similar in contract to the MIN_DELTA/DETAIL readers, but without changing APPEND_ONLY semantics.
    • Add a regression test that creates a ROW-binlog table stream with a multi-segment NONOVERLAPPING data rowset, including the empty-initial-rowset plus data-rowset shape if that is the actual failing layout, then queries the APPEND_ONLY stream and verifies both no BE abort and stable ordered output.

    Useful missing information for a self-contained upstream repro:

    • Exact base table DDL, binlog/table-stream DDL, and stream query SQL.
    • The load/insert script or segment-size settings needed to force the failing multi-segment rowset.
    • Failing tablet rowset metadata: version range, row count, segment count, segments_overlap, and whether an empty rowset precedes the data rowset.
    • FE plan or scan range snippet showing read_row_binlog=true and binlog_scan_type=APPEND_ONLY.
  3. MoanasDaddyXu
    closed this as completedon Aug 11, 2026
  4. gavinchou
    added a commit that references this issue on Aug 25, 2026

    [fix](table stream) Fix multi-segment row-binlog reads for MoW tables (…

    369f887
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions