Repository navigation
[Bug](High) Large ROW binlog Broker Load fails with COW::assert_mutable: use_count() > 1 #66368
Description
Search before asking
- I had searched in the issues and found no similar issues.
Version
Reproduced on doris-0.0.0-1590e7e3228 in a non-cloud deployment with 1 FE and 3 BEs.
Regression case:
regression-test/suites/stress/job_and_mtmv/ivm_ab/ivm_performance_scale_matrix_test.groovy
What's Wrong?
A 107-column UNIQUE KEY Merge-on-Write table was configured with ROW binlog and historical values. A large Broker Load completed successfully in the first round, but a second load of the same source data with a different business-key prefix was cancelled after the BE hit an internal copy-on-write assertion:
[INTERNAL_ERROR][E6] COW::assert_mutable: use_count() > 1
The first failure occurred while preparing ROW binlog source data. The same assertion was later observed in the normal vertical segment flush path.
This is not a regression-test assertion failure: the load itself entered CANCELLED after the BE returned the internal error.
What You Expected?
A valid UNIQUE KEY MOW + ROW binlog Broker Load should complete successfully. If the workload exceeds a supported resource limit, Doris should return a clear resource-limit error instead of violating the copy-on-write ownership invariant in IColumn::mutate().
How to Reproduce?
Create a wide UNIQUE KEY MOW table with the following relevant properties:
enable_unique_key_merge_on_write = true
binlog.enable = true
binlog.format = ROW
binlog.need_historical_value = true
inverted_index_storage_format = V3
AUTO PARTITION BY MONTH
DISTRIBUTED BY HASH(trace_id) BUCKETS 128
Run two large Broker Loads against the table. The reproduced workload used 1,000 source files and the following load settings:
load_parallelism = 3
exec_mem_limit = 8589934592
timeout = 43200
max_filter_ratio = 0.1
Use a different business-key prefix in the second round so that it does not overwrite rows from the first round.
Current reproduction scale:
round 1: ScannedRows = 2,500,000,000, succeeded
round 2: ScannedRows = 1,052,674,279, cancelled
round 2: LoadBytes = 906,586,501,986
round 2: FilteredRows = 0
The issue has reproduced at approximately 1 TiB input scale. A smaller standalone reproduction has not been identified yet.
Regression Test Result
Primary stack:
doris::IColumn::mutate() const && at ../src/core/cow.h:325
doris::OlapBlockDataConvertor::OlapColumnDataConvertorSimple<...>::convert_to_olap()
doris::OlapBlockDataConvertor::convert_column_data(unsigned long)
doris::segment_v2::RowBinlogSourceDataWriter::prepare_by_source_block(...)
doris::segment_v2::RowBinlogSegmentWriter::append_block(...)
doris::SegmentFlusher::flush_single_block(...)
doris::SegmentCreator::flush_single_block(...)
doris::BaseBetaRowsetWriter::flush_memtable(...)
doris::FlushToken::_flush_memtable_impl(...)
doris::FlushToken::_flush_group_memtable(...)
doris::PartOfGroupMemtableFlushTask::run()
doris::ThreadPool::dispatch_thread()
The assertion was also observed through the regular segment writer path:
doris::IColumn::mutate()
doris::OlapBlockDataConvertorSimple<...>::convert_to_olap()
doris::segment_v2::VerticalSegmentWriter::write_batch()
doris::SegmentFlusher::_add_rows()
doris::SegmentFlusher::flush_single_block()
Initial Analysis
The first load used the same schema, data source, field mapping, and load configuration and completed successfully. The failed load had FilteredRows = 0. All BEs remained alive, disk usage was about 50%, and the cluster still had available memory; there was no OOM, no-space error, node loss, or connection timeout.
Under large concurrent memtable flushes, the ROW binlog source block and the regular segment writer may share a column object that is later mutated in place. This would cause IColumn::mutate() const && to observe use_count() > 1.
Please check the ownership and clone/mutate boundaries among:
RowBinlogSourceDataWriter::prepare_by_source_block
shared/group memtable flush
OlapBlockDataConvertorSimple
VerticalSegmentWriter
Anything Else?
Tracking issues:
- [Tracking][Binlog] Binlog feature umbrella — all sub-issues tracked here #65265
- [Feature] Track incremental computation with row binlog, Table Stream, and MTMV #65418
The remaining reproduction work is to reduce the data size and isolate the effect of bucket count, ROW binlog, and inverted indexes.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Activity
Doris-Breakwater commentedon Aug 3, 2026
Initial assessment: confirmed BE copy-on-write race on the reported commit, with high confidence in the root cause. The failure is not explained by a supported resource limit. The same unsafe ownership pattern is still present on current
master(dc4fc151b997a6d1909e01ebdc6bcc60eed5b8a2). This should be treated as a high-priority binlog/storage stability defect. The issue currently has no labels, assignee, milestone, or linked fix.Verified code path
When ROW binlog uses a
GroupRowsetWriter,FlushToken::submit()creates two independently scheduled tasks, one for the normal data writer and one for the row-binlog writer._memtable2block()constructs the block once and returns the sameSharedMemtable::blockto both tasks; each task then passes it to its child writer (call site).SegmentFlushermakes only a shallowBlockcopy. With binlog disabled in the data child context it selectsVerticalSegmentWriter; with binlog enabled in the binlog child context it selectsRowBinlogSegmentWriter(selection). This accounts for both reported stacks reaching the same converter from the two concurrent sub-tasks.OlapBlockDataConvertor::set_source_column()retains anotherColumnPtrto the selected top-level column (code). For a nullable simple column, however,OlapColumnDataConvertorSimple::convert_to_olap()obtains the nested column by reference and calls:(code). This does not transfer or replace the
ColumnNullableowner slot. Copies of the top-levelColumnPtrtherefore do not raise the nested column's refcount; both flush threads can observe that nested node as uniquely owned and try to acquire a mutable pointer to it.The exact exception follows from the two separate refcount observations in COW:
shallow_mutate()first checksuse_count(), then the unique path callsassert_mutable(), which checks it again (code). One thread can pass the first check while the count is 1; another thread then returns a mutable pointer and increments the count; the first thread reachesassert_mutable()with a count greater than 1 and emits the reported error.A different interleaving can let both threads pass the assertion and create simultaneous mutable aliases to the nested column. That is a genuine ownership violation, not merely an over-strict diagnostic. It is especially relevant for nullable
FLOAT/DOUBLE, because this converter normalizes NaNs in place in the same function; other simple types still take an unnecessary mutable alias.This also explains why round 1 may succeed and round 2 fail: the trigger is scheduling-dependent, and a wide schema plus many memtable flushes creates many opportunities for the race. The present evidence does not show that the business-key prefix, historical-value lookup, 128 buckets, inverted indexes, memory pressure, or input bytes are independently causal. There is no supplied evidence of persisted data corruption, but the possible shared mutable alias means the problem should not be classified as only a load-cancellation diagnostic.
Information still useful
The code and stacks are sufficient to confirm the ownership defect. The following artifacts would identify the exact runtime column and make the regression portable:
OlapColumnDataConvertorSimple<T>specialization, plus the failing column name/id, type, and nullability.SHOW CREATE TABLE, especially all nullable simple columns and anyFLOAT/DOUBLEcolumns.ivm_performance_scale_matrix_test.groovy, its data generator, and any required source-file manifest. That path is not present in the public repository at the reported commit or on current master, so maintainers cannot currently run the stated case.A Doris query profile is not required to establish this COW race.
Recommended next steps
Fix the converter ownership boundary. Do not call rvalue
mutate()on a borrowed nested-column reference. Either detach through an ownedColumnPtr/the parent owner slot before accessing a writable nested node, or keep the normal conversion path const and create a private writable temporary only for types that actually require normalization. A whole-block deep copy or serializing the two flush tasks would mask the race at substantially broader cost and should not substitute for fixing the converter contract.Add a deterministic BE test using the real grouped memtable flush path, a two-thread pool, and nullable fixed-width columns. Synchronize the data and row-binlog writers at conversion so both touch the same nested node. Include nullable
FLOAT/DOUBLEwith NaN, verify the source block remains unchanged, and verify that both the data and binlog segments finish correctly. The existing group-flush test uses non-nullable columns and mock writers, so it does not exercise this converter race.Exercise both normal segment-writer variants after the fix: vertical enabled and disabled, ROW binlog with historical values enabled and disabled, and repeated concurrent flushes under TSAN if available.
Audit other calls that apply rvalue
mutate()to a reference returned from a composite column. The safe API must either consume an owning pointer or replace the composite's owner slot.Add the repository's existing
binlog,area/storage,area/load,kind/stability, andkind/need-regression-testlabels as appropriate, then assign a BE storage/binlog owner. Backport evaluation should be based on supported branches containing both the grouped ROW-binlog flush path and the enforced COW contract; no internal branch/tag naming should be inferred from this development build.Breakwater-GitHub-Analysis-Slot: slot_ce2778b8bd08
[chore](be) Document OLAP conversion ownership
[fix](binlog) Avoid mutating source columns during OLAP conversion (#…