Repository navigation
[Bug](Highest) BE can crash in OrdinalIndexWriter::finish during binlog workload flush #65951
Description
Tracking
- Tracking issue: [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265
- Internal reference: DORIS-27426
Search before asking
- I searched the existing issues and found no issue with the same
OrdinalIndexWriter::finishcrash stack.
Version
Reproduced on Doris commit a8b1fd95e2a.
What's Wrong?
A binlog workload can crash the BE while a memtable is being flushed into a segment. The fatal path reaches:
doris::segment_v2::OrdinalIndexWriter::finish(...)
The implementation at the affected commit contains this fatal precondition:
CHECK(_page_builder->count() > 0) << "no entry has been added";The crash therefore means ordinal-index finalization was invoked before any data-page entry had been appended. The call stack passes through ScalarColumnWriter::write_ordinal_index() and ArrayColumnWriter::write_ordinal_index() while SegmentWriter is finalizing column indexes.
What You Expected?
Flushing imported data should not terminate the BE. The column writer should either guarantee that at least one ordinal-index entry exists before finalization or safely handle a column/index state with no data pages.
How to Reproduce?
A reduced SQL reproducer is not yet available. The observed failure sequence is:
- Run an import workload against a binlog-enabled table.
- Allow a memtable flush to create and finalize a segment.
- Reach ordinal-index finalization through the array-column writer path.
- Observe the zero-entry assertion in
OrdinalIndexWriter::finish, followed bySIGABRT.
be.out
The following excerpt contains only the relevant, sanitized be.out stack:
*** Current BE git commitID: a8b1fd95e2a ***
*** SIGABRT received; stack trace: ***
0# doris::signal::FailureSignalHandler(int, siginfo_t*, void*)
1# pthread_kill
2# raise
3# abort
4# google::LogMessage::SendToLog()
5# google::LogMessage::Flush()
6# google::LogMessageFatal::~LogMessageFatal()
7# doris::segment_v2::OrdinalIndexWriter::finish(...)
8# doris::segment_v2::ScalarColumnWriter::write_ordinal_index()
9# doris::segment_v2::ArrayColumnWriter::write_ordinal_index()
10# doris::segment_v2::SegmentWriter::finalize_columns_index(...)
11# doris::segment_v2::SegmentWriter::finalize(...)
12# doris::SegmentFlusher::_flush_segment_writer(...)
13# doris::SegmentFlusher::flush_single_block(...)
14# doris::SegmentCreator::flush_single_block(...)
15# doris::BaseBetaRowsetWriter::flush_memtable(...)
16# doris::FlushToken::_flush_memtable_impl(...)
17# doris::FlushToken::_flush_group_memtable(...)
18# doris::PartOfGroupMemtableFlushTask::run()
19# doris::ThreadPool::dispatch_thread()
20# doris::Thread::supervise_thread(void*)
Impact
The issue converts an index-writer invariant violation into a process-level crash during ingestion, interrupting writes and reducing cluster availability.
Are you willing to submit PR?
- Yes, I am willing to submit a PR!
Activity
zclllyybb commentedon Jul 23, 2026
Breakwater-GitHub-Analysis-Slot: slot_681b70d25561
This content is generated by AI for reference only.
Initial analysis after checking the reported commit:
This should be treated as a real BE segment-writer bug, not as an expected empty-import case. At
a8b1fd95e2a,OrdinalIndexWriter::finish()still has a fatal precondition that_page_builder->count() > 0. Entries are appended only when a scalar writer actually writes a data page via_write_data_page(). The flush layer already skipsblock->rows() == 0, and_flush_segment_writer()also returns beforefinalize()when the segment writer has zero rows, so reachingSegmentWriter::finalize_columns_index()means the segment writer believed this segment had rows.The array part of the stack is important.
ArrayColumnWriter::write_ordinal_index()unconditionally finalizes the array offset writer and nullable null writer, and only skips the item writer whenhas_empty_items()is true. Therefore the observedArrayColumnWriter -> ScalarColumnWriter -> OrdinalIndexWriterstack means some scalar subwriter under that array column had zero data-page entries at index finalization time. For a normally appended array column, even all-empty arrays or all-null arrays should still append offset/null rows, so this points to a column append/finalize mismatch rather than a valid empty-array case.The most suspicious code area for this workload is the row-binlog segment writer path.
RowBinlogSegmentWriter::append_block()builds row-binlog prefix columns, normal AFTER columns, and optional BEFORE columns itself, then increments_num_rows_written. If an ARRAY column in the row-binlog schema is skipped by that selective fill path,SegmentWriter::_write_ordinal_index()later still walks every column writer and can finalize the untouched array offset/null scalar writer, matching the fatal ordinal-index precondition. I would not reuse the older segcompaction ordinal-index RCA here: this stack is a memtable flush path throughSegmentFlusher/FlushToken, not aSegcompactionWorker/Mergerpath.Suggested next steps:
_num_rows_written += num_rows: every top-level row-binlog column writer that belongs to the row-binlog schema should have advanced to the old row count plusnum_rows. This should identify the exact missing column id before the fatal ordinal-index finish.RowBinlogSegmentWriter::append_block()for partial-update and historical-BEFORE cases with ARRAY/complex columns, especially whether every normal/BEFORE row-binlog column is appended exactly once.Information still needed to pin the exact subwriter:
filepath=...and any nearby tablet/rowset/segment ids;[fix](be) Scope nullable AFTER conversion to Row Binlog
[fix](binlog) wrap Row Binlog AFTER columns as nullable before conver…
[fix](binlog) wrap Row Binlog AFTER columns as nullable before conver…