[Bug](High) Enabling binlog reduces import throughput by about 65% #65947

Activity

1. zclllyybb commented on Jul 23, 2026

   

   zclllyybb

   on Jul 23, 2026

   Contributor

   More actions

   Breakwater-GitHub-Analysis-Slot: slot_2543afc8eebe
   This content is generated by AI for reference only.

   Initial triage for 87d57a84729b61fcdd2e00e4e10697b46b2a82e9:

   This looks like a real row-binlog write-path amplification issue, but the current report does not include enough workload detail or profiles to pin the single dominant bottleneck yet.

   What the code path shows:

   • If the table uses binlog.enable=true with binlog.format=ROW, FE creates a separate row-binlog index/schema for streaming binlog. That schema contains the source key/visible columns, optional BEFORE columns when binlog.need_historical_value=true, and hidden TSO/LSN/op columns.
   • On the BE load path, VTabletWriter detects the row-binlog index and allocates one row-binlog LSN per input row through the auto-increment buffer. Those LSNs are copied through the add-block request, TabletsChannel, the memtable sidecar, and finally into the segment writer options at memtable flush time.
   • The rowset writer is wrapped as a GroupRowsetWriter, so each flush/build is synchronous for both the normal data rowset and the row-binlog rowset. In other words, row-binlog generation is on the import critical path, not an asynchronous post-commit task.
   • RowBinlogSegmentWriter::append_block() reconstructs and writes a second segment stream: it converts source columns again, writes the TSO/LSN/op columns, builds a key index, and, when historical values are requested or partial update needs full AFTER rows, performs historical-row retrieval/fill work before writing BEFORE/AFTER columns.
   • FE transaction commit also fetches a commit TSO for row-binlog tables. That is worth measuring, especially for workloads made of many small transactions, but for large bulk import the code suggests the main suspects are BE row-binlog segment construction, sidecar LSN handling, and optional historical-row lookup.

   So the 65% throughput drop is plausible from the current implementation, but I would not classify the exact regression boundary from the provided numbers alone. The strongest code-level suspect is that the current ROW binlog implementation synchronously writes a full secondary rowset per flush and may additionally do per-row LSN copying plus historical lookup. That can directly throttle import throughput when the workload is CPU-bound in conversion/sorting/index writing or I/O-bound on the extra segment files.

   Missing information needed to make this actionable:

   • Exact CREATE TABLE DDL, especially key type, number/types of columns, partitions/buckets/tablets, replica count, and the binlog properties: binlog.enable, binlog.format, and binlog.need_historical_value.
   • Exact load method and workload shape: Stream Load/Broker Load/Routine Load/Insert, total rows/bytes, file format, batch size, concurrency, transaction count, and whether both tests created the same number of rowsets/segments.
   • FE/BE configs that affect this path, including enable_feature_binlog, memtable flush settings, compaction settings, load parallelism, auto-increment prefetch settings, and disk/network layout.
   • Side-by-side BE load profiles or flamegraphs for enabled vs disabled runs, with attention to VTabletWriter, TabletsChannel::add_block, MemTableFlushExecutor, GroupRowsetWriter::flush_memtable/build_rowsets, RowBinlogSegmentWriter::append_block, OlapBlockDataConvertor, build_key_index, and historical-row retriever code if need_historical_value=true.
   • FE transaction timing for commit/visible/publish phases, especially TSO-fetch latency if the workload uses many small transactions.

   Recommended next checks:

   1. First confirm whether this is ROW binlog. If it is STATEMENT_AND_SNAPSHOT, this investigation should pivot to the CCR-style binlog path instead of the row-binlog writer path above.
   2. Run the same benchmark with binlog.format=ROW and binlog.need_historical_value=false versus true. A large gap there would point to historical-row retrieval/BEFORE-column filling rather than basic secondary-rowset writing.
   3. Compare flush count, segment count, rowset count, CPU time, and disk write bytes between the two runs. If the slowdown scales with segment/flush count, the synchronous secondary rowset build is the likely limiter.
   4. Add or collect timing around the row-binlog writer stages before optimizing: LSN allocation/copy, source-column conversion, binlog meta-column fill, key-index build, BEFORE/AFTER historical lookup, and normal-vs-row-binlog flush/build time.
   5. If profiles confirm the BE row-binlog writer dominates, likely optimization directions are batching/reducing LSN sidecar copies, avoiding repeated column conversion where possible, bulk-filling LSN/op columns, reducing key-index build cost for append-only cases, or decoupling row-binlog materialization from the synchronous import critical path while preserving ordering and correctness.
2. MoanasDaddyXu

   closed this as completedon Sep 2, 2026