Skip to content

Commit 26d02dc

Browse files
morningmanclaude
andcommitted
[refactor](row-binlog) Derive binlog column indexes from column names
### What problem does this PR solve? Issue Number: None Related PR: #65810 Problem Summary: A row binlog tablet has three hidden columns, `__DORIS_BINLOG_TSO__`, `__DORIS_BINLOG_LSN__` and `__DORIS_BINLOG_OP__`. `TabletSchema` keeps their positions in `_binlog_{tso,lsn,op}_col_idx`. The positions came from thrift `TTabletSchema` 28-30 when a tablet was created, and were persisted in `TabletSchemaPB` 41-43 (`TabletSchemaCloudPB` 42-44). Since #65810 the FE no longer sets `TTabletSchema` 28-30. Every row binlog tablet created since then has -1 in its tablet meta schema (and in the meta service schema entry), and so do the schemas loaded from it: rowsets synced from the meta service, compaction outputs. Nothing reads these -1 today. Every reader gets a schema whose indexes are derived from the column names: `build_current_tablet_schema` for loads, `append_column` for scans and the publish-phase writer. But: 1. Any new reader of the tablet schema would get -1. 2. If a load ever came with an index schema without column unique ids, `RowsetBuilder` would fall back to copying the tablet schema, and row binlog derivation would hit `CHECK(idx >= 0)`. 3. BE unit tests set 28-30 and the PB fields by hand, so they exercise a path that production never takes. This PR makes `TabletSchema::init_from_pb` derive the three indexes from the column names, as `append_column` and `build_current_tablet_schema` already do. The stored values are then not needed: - thrift `TTabletSchema` 28-30 are retired (commented out); - proto `TabletSchemaPB` 41-43 and `TabletSchemaCloudPB` 42-44 are reserved; `TabletSchema::to_schema_pb`, `TabletMeta` and the cloud `pb_convert` stop writing and copying them; - BE unit tests stop setting them. A new `TabletSchemaTest` case checks the indexes after `init_from_pb` and after a round trip through the PB. Compatibility: none of these ids exists on branch-4.2, branch-4.1, branch-4.0 or branch-3.1. Meta written by an older master still loads: protobuf keeps the reserved ids as unknown fields, and the indexes are derived anyway. An older master BE that reads meta written by this one gets -1, which is what it already has for every row binlog tablet created since #65810. ### Release note None ### Check List (For Author) - Test: - Build: `build.sh --fe --be` (BE Release; FE with checkstyle). - Regression test: `row_binlog_p0` (23 suites) and `table_stream_p0` (16 suites) pass. - Manual test: create a row binlog table and load it; the row binlog tablet meta no longer carries the three fields. Restart the cluster, load again, run a binlog compaction ([0-1]...[5-5] into [0-5]) and read through `binlog()`: every event comes back. - Unit Test: BE, 191 tests in 19 suites, all pass: the new `TabletSchemaTest.test_binlog_col_idx_from_pb_column_names`, and every suite that builds a row binlog schema or touches the changed code (`TabletSchemaTest`, `TabletMetaTest`, `PbConvert`, `RowBinlogDeriveTest`, `RowBinlogVmergeCompactionTest`, `SegmentFlusher(Transform)FormatTest`, `(Cloud)GroupRowset*Test`, `TestEngineStorageMigrationTask`, the merge-on-write transform tests, ...). - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 15eb3f4 commit 26d02dc

10 files changed

Lines changed: 56 additions & 99 deletions

File tree

‎be/src/cloud/pb_convert.cpp‎

Lines changed: 0 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -511,15 +511,6 @@ void doris_tablet_schema_to_cloud(TabletSchemaCloudPB* out, const TabletSchemaPB
511511
if (in.has_row_lsn_col_idx()) {
512512
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
513513
}
514-
if (in.has_binlog_tso_col_idx()) {
515-
out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
516-
}
517-
if (in.has_binlog_lsn_col_idx()) {
518-
out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
519-
}
520-
if (in.has_binlog_op_col_idx()) {
521-
out->set_binlog_op_col_idx(in.binlog_op_col_idx());
522-
}
523514

524515
if (in.has___split_schema()) {
525516
out->mutable___split_schema()->CopyFrom(in.__split_schema());
@@ -575,15 +566,6 @@ void doris_tablet_schema_to_cloud(TabletSchemaCloudPB* out, TabletSchemaPB&& in)
575566
if (in.has_row_lsn_col_idx()) {
576567
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
577568
}
578-
if (in.has_binlog_tso_col_idx()) {
579-
out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
580-
}
581-
if (in.has_binlog_lsn_col_idx()) {
582-
out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
583-
}
584-
if (in.has_binlog_op_col_idx()) {
585-
out->set_binlog_op_col_idx(in.binlog_op_col_idx());
586-
}
587569

588570
if (in.has___split_schema()) {
589571
out->mutable___split_schema()->CopyFrom(in.__split_schema());
@@ -652,15 +634,6 @@ void cloud_tablet_schema_to_doris(TabletSchemaPB* out, const TabletSchemaCloudPB
652634
if (in.has_row_lsn_col_idx()) {
653635
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
654636
}
655-
if (in.has_binlog_tso_col_idx()) {
656-
out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
657-
}
658-
if (in.has_binlog_lsn_col_idx()) {
659-
out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
660-
}
661-
if (in.has_binlog_op_col_idx()) {
662-
out->set_binlog_op_col_idx(in.binlog_op_col_idx());
663-
}
664637

665638
if (in.has___split_schema()) {
666639
out->mutable___split_schema()->CopyFrom(in.__split_schema());
@@ -717,15 +690,6 @@ void cloud_tablet_schema_to_doris(TabletSchemaPB* out, TabletSchemaCloudPB&& in)
717690
if (in.has_row_lsn_col_idx()) {
718691
out->set_row_lsn_col_idx(in.row_lsn_col_idx());
719692
}
720-
if (in.has_binlog_tso_col_idx()) {
721-
out->set_binlog_tso_col_idx(in.binlog_tso_col_idx());
722-
}
723-
if (in.has_binlog_lsn_col_idx()) {
724-
out->set_binlog_lsn_col_idx(in.binlog_lsn_col_idx());
725-
}
726-
if (in.has_binlog_op_col_idx()) {
727-
out->set_binlog_op_col_idx(in.binlog_op_col_idx());
728-
}
729693

730694
if (in.has___split_schema()) {
731695
out->mutable___split_schema()->CopyFrom(in.__split_schema());

‎be/src/storage/tablet/tablet_meta.cpp‎

Lines changed: 0 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -403,15 +403,6 @@ void TabletMeta::init_schema_from_thrift(const TTabletSchema& tablet_schema,
403403
if (tablet_schema.__isset.row_lsn_col_idx) {
404404
tablet_schema_pb->set_row_lsn_col_idx(tablet_schema.row_lsn_col_idx);
405405
}
406-
if (tablet_schema.__isset.binlog_tso_idx) {
407-
tablet_schema_pb->set_binlog_tso_col_idx(tablet_schema.binlog_tso_idx);
408-
}
409-
if (tablet_schema.__isset.binlog_lsn_idx) {
410-
tablet_schema_pb->set_binlog_lsn_col_idx(tablet_schema.binlog_lsn_idx);
411-
}
412-
if (tablet_schema.__isset.binlog_op_idx) {
413-
tablet_schema_pb->set_binlog_op_col_idx(tablet_schema.binlog_op_idx);
414-
}
415406
auto p_seq_map = tablet_schema_pb->mutable_seq_map(); // ColumnGroupsPB
416407
for (auto& it : tablet_schema.seq_map) { // std::vector< ::doris::TColumnGroup>
417408
uint32_t key = it.sequence_column;

‎be/src/storage/tablet/tablet_schema.cpp‎

Lines changed: 4 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1124,9 +1124,10 @@ void TabletSchema::init_from_pb(const TabletSchemaPB& schema, bool ignore_extrac
11241124
_skip_bitmap_col_idx = schema.skip_bitmap_col_idx();
11251125
_commit_tso_col_idx = schema.commit_tso_col_idx();
11261126
_row_lsn_col_idx = schema.row_lsn_col_idx();
1127-
_binlog_tso_col_idx = schema.binlog_tso_col_idx();
1128-
_binlog_lsn_col_idx = schema.binlog_lsn_col_idx();
1129-
_binlog_op_col_idx = schema.binlog_op_col_idx();
1127+
// The binlog column cids are not persisted; derive them from the column names.
1128+
_binlog_tso_col_idx = field_index(BINLOG_TSO_COL);
1129+
_binlog_lsn_col_idx = field_index(BINLOG_LSN_COL);
1130+
_binlog_op_col_idx = field_index(BINLOG_OP_COL);
11301131
_sort_type = schema.sort_type();
11311132
_sort_col_num = schema.sort_col_num();
11321133
_compression_type = schema.compression_type();
@@ -1463,9 +1464,6 @@ void TabletSchema::to_schema_pb(TabletSchemaPB* tablet_schema_pb) const {
14631464
tablet_schema_pb->set_skip_bitmap_col_idx(_skip_bitmap_col_idx);
14641465
tablet_schema_pb->set_commit_tso_col_idx(_commit_tso_col_idx);
14651466
tablet_schema_pb->set_row_lsn_col_idx(_row_lsn_col_idx);
1466-
tablet_schema_pb->set_binlog_tso_col_idx(_binlog_tso_col_idx);
1467-
tablet_schema_pb->set_binlog_lsn_col_idx(_binlog_lsn_col_idx);
1468-
tablet_schema_pb->set_binlog_op_col_idx(_binlog_op_col_idx);
14691467
tablet_schema_pb->set_inverted_index_storage_format(_inverted_index_storage_format);
14701468
tablet_schema_pb->mutable_row_store_column_unique_ids()->Assign(
14711469
_row_store_column_unique_ids.begin(), _row_store_column_unique_ids.end());

‎be/test/storage/row_binlog_vmerge_compaction_test.cpp‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -136,7 +136,6 @@ class RowBinlogVmergeCompactionTest : public testing::Test {
136136
tso->set_length(8);
137137
tso->set_is_nullable(false);
138138
tso->set_visible(false);
139-
pb.set_binlog_tso_col_idx(2);
140139

141140
auto schema = std::make_shared<TabletSchema>();
142141
schema->init_from_pb(pb);

‎be/test/storage/rowset/segment_flusher_format_test.cpp‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -937,13 +937,10 @@ ComplexRowBinlogSchemas create_complex_row_binlog_schemas() {
937937
}
938938
auto* tso = add_column(&row_binlog_pb, 4, BINLOG_TSO_COL, kKeyTypes[4], false, true);
939939
tso->set_visible(false);
940-
row_binlog_pb.set_binlog_tso_col_idx(4);
941940
auto* lsn = add_column(&row_binlog_pb, 5, BINLOG_LSN_COL, kKeyTypes[4], false, false);
942941
lsn->set_visible(false);
943-
row_binlog_pb.set_binlog_lsn_col_idx(5);
944942
auto* op = add_column(&row_binlog_pb, 6, BINLOG_OP_COL, kKeyTypes[4], false, false);
945943
op->set_visible(false);
946-
row_binlog_pb.set_binlog_op_col_idx(6);
947944
row_binlog_pb.set_next_column_unique_id(7);
948945

949946
auto source = std::make_shared<TabletSchema>();
@@ -3166,11 +3163,7 @@ class SegmentFlusherTransformFormatTest : public SegmentFlusherFormatTest {
31663163
binlog_request.tablet_schema.columns.insert(
31673164
binlog_request.tablet_schema.columns.begin() + binlog_tso_idx,
31683165
std::move(before_v1));
3169-
binlog_tso_idx += 2;
31703166
}
3171-
binlog_request.tablet_schema.__set_binlog_tso_idx(binlog_tso_idx);
3172-
binlog_request.tablet_schema.__set_binlog_lsn_idx(binlog_tso_idx + 1);
3173-
binlog_request.tablet_schema.__set_binlog_op_idx(binlog_tso_idx + 2);
31743167

31753168
RuntimeProfile profile("SegmentFlusherTransformFormatTest");
31763169
EXPECT_TRUE(_engine->create_tablet(request, &profile).ok());

‎be/test/storage/tablet/tablet_schema_test.cpp‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
#include "storage/schema.h"
3333
#include "storage/tablet/tablet_schema_helper.h"
3434
#include "storage/tablet_info.h"
35+
#include "storage/utils.h"
3536
#include "util/json/path_in_data.h"
3637

3738
namespace doris {
@@ -70,6 +71,46 @@ TEST_F(TabletSchemaTest, test_commit_tso_col_idx_from_current_index_schema) {
7071
EXPECT_EQ(1, current.commit_tso_col_idx());
7172
}
7273

74+
TEST_F(TabletSchemaTest, test_binlog_col_idx_from_pb_column_names) {
75+
TabletSchemaPB schema_pb;
76+
schema_pb.set_keys_type(DUP_KEYS);
77+
for (const std::string& name :
78+
{std::string("k1"), BINLOG_TSO_COL, BINLOG_LSN_COL, BINLOG_OP_COL}) {
79+
ColumnPB* column = schema_pb.add_column();
80+
column->set_unique_id(schema_pb.column_size() - 1);
81+
column->set_name(name);
82+
column->set_type("BIGINT");
83+
column->set_is_key(name == "k1");
84+
column->set_length(8);
85+
column->set_aggregation("NONE");
86+
}
87+
88+
// The binlog column cids are not stored in the PB, so they must come from the column names
89+
// both for a new schema and after a round trip through the PB.
90+
TabletSchema schema;
91+
schema.init_from_pb(schema_pb);
92+
EXPECT_EQ(1, schema.binlog_tso_col_idx());
93+
EXPECT_EQ(2, schema.binlog_lsn_col_idx());
94+
EXPECT_EQ(3, schema.binlog_op_col_idx());
95+
96+
TabletSchemaPB round_trip_pb;
97+
schema.to_schema_pb(&round_trip_pb);
98+
TabletSchema round_trip;
99+
round_trip.init_from_pb(round_trip_pb);
100+
EXPECT_EQ(1, round_trip.binlog_tso_col_idx());
101+
EXPECT_EQ(2, round_trip.binlog_lsn_col_idx());
102+
EXPECT_EQ(3, round_trip.binlog_op_col_idx());
103+
EXPECT_TRUE(schema == round_trip);
104+
105+
TabletSchemaPB data_schema_pb;
106+
data_schema_pb.add_column()->CopyFrom(schema_pb.column(0));
107+
TabletSchema data_schema;
108+
data_schema.init_from_pb(data_schema_pb);
109+
EXPECT_EQ(-1, data_schema.binlog_tso_col_idx());
110+
EXPECT_EQ(-1, data_schema.binlog_lsn_col_idx());
111+
EXPECT_EQ(-1, data_schema.binlog_op_col_idx());
112+
}
113+
73114
TEST_F(TabletSchemaTest, test_tablet_column_init_from_pb) {
74115
ColumnPB column_pb;
75116
column_pb.set_unique_id(1001);

‎be/test/storage/transform/row_binlog_derive_test.cpp‎

Lines changed: 0 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -120,11 +120,6 @@ class RowBinlogDeriveTest : public MowTransformTestBase {
120120
add_col(5, BINLOG_TSO_COL, "BIGINT", false, true);
121121
add_col(6, BINLOG_LSN_COL, "BIGINT", false, false);
122122
add_col(7, BINLOG_OP_COL, "BIGINT", false, false);
123-
// init_from_pb reads this straight from the PB (it is not inferred from
124-
// the column name). [k1,v1,v2,__BEFORE__v1__,__BEFORE__v2__,TSO,LSN,OP]
125-
pb.set_binlog_tso_col_idx(5);
126-
pb.set_binlog_lsn_col_idx(6);
127-
pb.set_binlog_op_col_idx(7);
128123

129124
auto schema = std::make_shared<TabletSchema>();
130125
schema->init_from_pb(pb);
@@ -167,9 +162,6 @@ class RowBinlogDeriveTest : public MowTransformTestBase {
167162
add_col(1, BINLOG_TSO_COL, "BIGINT", false, true);
168163
add_col(2, BINLOG_LSN_COL, "BIGINT", false, false);
169164
add_col(3, BINLOG_OP_COL, "BIGINT", false, false);
170-
pb.set_binlog_tso_col_idx(1);
171-
pb.set_binlog_lsn_col_idx(2);
172-
pb.set_binlog_op_col_idx(3);
173165

174166
auto schema = std::make_shared<TabletSchema>();
175167
schema->init_from_pb(pb);
@@ -283,9 +275,6 @@ class RowBinlogDeriveTest : public MowTransformTestBase {
283275
add_col(2, BINLOG_TSO_COL, "BIGINT", false, true);
284276
add_col(3, BINLOG_LSN_COL, "BIGINT", false, false);
285277
add_col(4, BINLOG_OP_COL, "BIGINT", false, false);
286-
pb.set_binlog_tso_col_idx(2);
287-
pb.set_binlog_lsn_col_idx(3);
288-
pb.set_binlog_op_col_idx(4);
289278

290279
auto schema = std::make_shared<TabletSchema>();
291280
schema->init_from_pb(pb);
@@ -485,9 +474,6 @@ TEST_F(RowBinlogDeriveTest, PlainMapsHiddenKeysAndSkipsHiddenNonKeys) {
485474
add_system_column(10, BINLOG_TSO_COL, true);
486475
add_system_column(11, BINLOG_LSN_COL, false);
487476
add_system_column(12, BINLOG_OP_COL, false);
488-
binlog_pb.set_binlog_tso_col_idx(4);
489-
binlog_pb.set_binlog_lsn_col_idx(5);
490-
binlog_pb.set_binlog_op_col_idx(6);
491477
auto binlog_schema = std::make_shared<TabletSchema>();
492478
binlog_schema->init_from_pb(binlog_pb);
493479

@@ -684,10 +670,6 @@ TEST_F(RowBinlogDeriveTest, MowHiddenKeyColumnIsPartOfTheProbeKey) {
684670
add_col(&binlog_pb, 4, BINLOG_TSO_COL, "BIGINT", false, false, true);
685671
add_col(&binlog_pb, 5, BINLOG_LSN_COL, "BIGINT", false, false, false);
686672
add_col(&binlog_pb, 6, BINLOG_OP_COL, "BIGINT", false, false, false);
687-
// init_from_pb reads these straight from the PB, not from the column names
688-
binlog_pb.set_binlog_tso_col_idx(4);
689-
binlog_pb.set_binlog_lsn_col_idx(5);
690-
binlog_pb.set_binlog_op_col_idx(6);
691673
auto binlog_schema = std::make_shared<TabletSchema>();
692674
binlog_schema->init_from_pb(binlog_pb);
693675
ASSERT_EQ(binlog_schema->binlog_lsn_col_idx(), 5);

‎be/test/testutil/creators.h‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -223,13 +223,10 @@ inline TTabletSchema create_row_binlog_tablet_schema(const TTabletSchema& base_s
223223

224224
row_binlog_schema.columns.push_back(create_tablet_column(
225225
{BINLOG_TSO_COL, TPrimitiveType::BIGINT, false, true, TAggregationType::NONE, true}));
226-
row_binlog_schema.__set_binlog_tso_idx(row_binlog_schema.columns.size() - 1);
227226
row_binlog_schema.columns.push_back(create_tablet_column(
228227
{BINLOG_LSN_COL, TPrimitiveType::BIGINT, false, true, TAggregationType::NONE}));
229-
row_binlog_schema.__set_binlog_lsn_idx(row_binlog_schema.columns.size() - 1);
230228
row_binlog_schema.columns.push_back(create_tablet_column(
231229
{BINLOG_OP_COL, TPrimitiveType::BIGINT, false, true, TAggregationType::NONE}));
232-
row_binlog_schema.__set_binlog_op_idx(row_binlog_schema.columns.size() - 1);
233230
return row_binlog_schema;
234231
}
235232

‎gensrc/proto/olap_file.proto‎

Lines changed: 8 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -578,14 +578,10 @@ message TabletSchemaPB {
578578
// cid of the base-table row lsn column (__DORIS_ROW_LSN_COL__).
579579
optional int32 row_lsn_col_idx = 40 [default = -1];
580580

581-
// cid of the binlog tso column (__DORIS_BINLOG_TSO__).
582-
optional int32 binlog_tso_col_idx = 41 [default = -1];
583-
584-
// cid of the binlog lsn column (__DORIS_BINLOG_LSN__).
585-
optional int32 binlog_lsn_col_idx = 42 [default = -1];
586-
587-
// cid of the binlog op column (__DORIS_BINLOG_OP__).
588-
optional int32 binlog_op_col_idx = 43 [default = -1];
581+
// The cids of the binlog columns are derived from the column names.
582+
reserved 41; // deprecated binlog_tso_col_idx
583+
reserved 42; // deprecated binlog_lsn_col_idx
584+
reserved 43; // deprecated binlog_op_col_idx
589585

590586
optional SplitSchemaPB __split_schema = 1000; // A special field, DO NOT change it.
591587
}
@@ -640,14 +636,10 @@ message TabletSchemaCloudPB {
640636
// cid of the base-table row lsn column (__DORIS_ROW_LSN_COL__).
641637
optional int32 row_lsn_col_idx = 41 [default = -1];
642638

643-
// cid of the binlog tso column (__DORIS_BINLOG_TSO__).
644-
optional int32 binlog_tso_col_idx = 42 [default = -1];
645-
646-
// cid of the binlog lsn column (__DORIS_BINLOG_LSN__).
647-
optional int32 binlog_lsn_col_idx = 43 [default = -1];
648-
649-
// cid of the binlog op column (__DORIS_BINLOG_OP__).
650-
optional int32 binlog_op_col_idx = 44 [default = -1];
639+
// The cids of the binlog columns are derived from the column names.
640+
reserved 42; // deprecated binlog_tso_col_idx
641+
reserved 43; // deprecated binlog_lsn_col_idx
642+
reserved 44; // deprecated binlog_op_col_idx
651643

652644
optional bool is_dynamic_schema = 100 [default=false];
653645

‎gensrc/thrift/AgentService.thrift‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -54,9 +54,9 @@ struct TTabletSchema {
5454
25: optional list<Types.TColumnGroup> seq_map
5555
26: optional i32 commit_tso_col_idx = -1
5656
27: optional i32 row_lsn_col_idx = -1
57-
28: optional i32 binlog_tso_idx = -1
58-
29: optional i32 binlog_lsn_idx = -1
59-
30: optional i32 binlog_op_idx = -1
57+
// 28: deprecated binlog_tso_idx
58+
// 29: deprecated binlog_lsn_idx
59+
// 30: deprecated binlog_op_idx
6060
}
6161

6262
// this enum stands for different storage format in src_backends

0 commit comments

Comments
 (0)