Repository navigation
[Bug](High) Flexible Partial Update fails with ROW binlog due to illegal partial update block columns #65866
Description
Tracking
Search before asking
- I searched the existing issues and found no issue with the same error or reproduction.
Version
Reproduced while testing #62606 on Doris commit a78b707ed040c3206df99cf43ea3101ab2ba5b90.
Test environment: non-Cloud deployment with 1 FE and 3 BEs.
What's Wrong?
Flexible Partial Update fails when it is used on a UNIQUE KEY merge-on-write table with ROW binlog enabled. The Stream Load reaches the BE flush stage and returns:
[INTERNAL_ERROR] illegal partial update block columns: 10, num key columns: 1, total schema columns: 12
The entire batch fails with NumberLoadedRows=0.
The same schema accepts Flexible Partial Update when ROW binlog is disabled, and fixed-column partial update works with ROW binlog enabled. This narrows the failure to the combination of ROW binlog and UPDATE_FLEXIBLE_COLUMNS.
IVM requires ROW binlog, so the issue blocks IVM base tables from using Flexible Partial Update.
What You Expected?
Flexible Partial Update should write successfully while ROW binlog records sufficient before/after data for downstream incremental consumption. A later incremental refresh should produce results consistent with the base table.
How to Reproduce?
Create the table and initial row:
DROP DATABASE IF EXISTS ivm_flexible_row_binlog_repro;
CREATE DATABASE ivm_flexible_row_binlog_repro;
USE ivm_flexible_row_binlog_repro;
CREATE TABLE flexible_base (
id BIGINT NOT NULL,
seq BIGINT NOT NULL,
amount INT NULL,
status VARCHAR(32) NULL,
comment VARCHAR(64) NULL
)
UNIQUE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES (
'replication_num'='1',
'enable_unique_key_merge_on_write'='true',
'enable_unique_key_skip_bitmap_column'='true',
'function_column.sequence_col'='seq',
'binlog.enable'='true',
'binlog.format'='ROW',
'binlog.need_historical_value'='true'
);
INSERT INTO flexible_base VALUES (1, 10, 100, 'new', 'one');Prepare flexible.json:
{"id":1,"seq":11,"amount":111}
{"id":2,"seq":1,"status":"inserted"}Run Flexible Partial Update:
curl -u root: -H 'format:json' -H 'read_json_by_line:true' \
-H 'unique_key_update_mode:UPDATE_FLEXIBLE_COLUMNS' \
-H 'Expect:100-continue' \
-T flexible.json \
http://FE_HOST:8030/api/ivm_flexible_row_binlog_repro/flexible_base/_stream_loadThe Stream Load fails during BE flush with the error above and writes zero rows.
Regression Test Result
- Suite:
ivm_dml_flexible_partial_row_binlog_blocking - Jenkins:
regression-debug #793 - Result: 4 DML suites were run; the other 3 passed and only this suite failed with the INTERNAL_ERROR.
Anything Else?
The failure occurs below the IVM refresh layer in the storage flush path, so changing the IVM refresh strategy cannot avoid it.
Are you willing to submit PR?
- Yes, I am willing to submit a PR!
Activity
zclllyybb commentedon Jul 22, 2026
Breakwater-GitHub-Analysis-Slot: slot_8ed64919122e
This content is generated by AI for reference only.
Initial code-path analysis:
I checked the reported commit
a78b707ed040c3206df99cf43ea3101ab2ba5b90and the current head of PR #62606 for the relevant BE storage writer code. I did not run a local cluster reproduction, but the reported error matches a concrete writer-path mismatch.This looks like a real BE ROW-binlog handling bug for
UPDATE_FLEXIBLE_COLUMNS, not an IVM refresh-layer problem.What happens:
SegmentFlusher::flush_single_block()disablesVerticalSegmentWriterfor that row-binlog write path. It then createsRowBinlogSegmentWriter.RowBinlogSegmentWriter::append_block()treats every partial update as a sparse fixed-column partial-update block and rejects:block->columns() <= source_schema->num_key_columns() || block->columns() >= source_schema->num_columns()That is valid for fixed-column partial update, but not for flexible partial update. Flexible partial update intentionally carries a full source-schema block plus the skip-bitmap column, so
block->columns() == source_schema->num_columns()is the expected shape.The non-binlog flexible path is different:
VerticalSegmentWriter::batch_block()explicitly acceptsblock->columns() == _tablet_schema->num_columns(), and_append_block_with_flexible_partial_content()uses the skip bitmap plus historical-row lookup to build the full AFTER row before writing the segment.For flexible mode,
PartialUpdateInfo::init()does not populateupdate_cidsas a fixed sparse column list. It records all non-key columns inmissing_cids, while the per-row updated/missing cells are represented by the skip bitmap. So the row-binlog writer cannot be fixed by only relaxing the column-count check; it also needs flexible-specific skip-bitmap handling to avoid writing placeholder values or double-writing columns.This explains the matrix in the issue:
Suggested fix direction:
is_flexible_partial_update()branch inRowBinlogSegmentWriter::append_block().block->columns() == source_schema->num_columns()only for flexible partial update.VerticalSegmentWriter::_append_block_with_flexible_partial_content()/FlexibleReadPlan, then append each row-binlog normal column exactly once from that completed AFTER image.binlog.need_historical_value/ BEFORE-image behavior by using historical rows for BEFORE columns, and verify op generation for existing-row update, new-row insert, and delete rows.Useful extra validation material: