Repository navigation
Commit 26d02dc
[refactor](row-binlog) Derive binlog column indexes from column names
### What problem does this PR solve?
Issue Number: None
Related PR: #65810
Problem Summary:
A row binlog tablet has three hidden columns, `__DORIS_BINLOG_TSO__`,
`__DORIS_BINLOG_LSN__` and `__DORIS_BINLOG_OP__`. `TabletSchema` keeps
their positions in `_binlog_{tso,lsn,op}_col_idx`. The positions came
from thrift `TTabletSchema` 28-30 when a tablet was created, and were
persisted in `TabletSchemaPB` 41-43 (`TabletSchemaCloudPB` 42-44).
Since #65810 the FE no longer sets `TTabletSchema` 28-30. Every row
binlog tablet created since then has -1 in its tablet meta schema (and
in the meta service schema entry), and so do the schemas loaded from it:
rowsets synced from the meta service, compaction outputs.
Nothing reads these -1 today. Every reader gets a schema whose indexes
are derived from the column names: `build_current_tablet_schema` for
loads, `append_column` for scans and the publish-phase writer. But:
1. Any new reader of the tablet schema would get -1.
2. If a load ever came with an index schema without column unique ids,
`RowsetBuilder` would fall back to copying the tablet schema, and
row binlog derivation would hit `CHECK(idx >= 0)`.
3. BE unit tests set 28-30 and the PB fields by hand, so they exercise
a path that production never takes.
This PR makes `TabletSchema::init_from_pb` derive the three indexes from
the column names, as `append_column` and `build_current_tablet_schema`
already do. The stored values are then not needed:
- thrift `TTabletSchema` 28-30 are retired (commented out);
- proto `TabletSchemaPB` 41-43 and `TabletSchemaCloudPB` 42-44 are
reserved; `TabletSchema::to_schema_pb`, `TabletMeta` and the cloud
`pb_convert` stop writing and copying them;
- BE unit tests stop setting them. A new `TabletSchemaTest` case checks
the indexes after `init_from_pb` and after a round trip through the PB.
Compatibility: none of these ids exists on branch-4.2, branch-4.1,
branch-4.0 or branch-3.1. Meta written by an older master still loads:
protobuf keeps the reserved ids as unknown fields, and the indexes are
derived anyway. An older master BE that reads meta written by this one
gets -1, which is what it already has for every row binlog tablet
created since #65810.
### Release note
None
### Check List (For Author)
- Test:
- Build: `build.sh --fe --be` (BE Release; FE with checkstyle).
- Regression test: `row_binlog_p0` (23 suites) and `table_stream_p0`
(16 suites) pass.
- Manual test: create a row binlog table and load it; the row binlog
tablet meta no longer carries the three fields. Restart the cluster,
load again, run a binlog compaction ([0-1]...[5-5] into [0-5]) and
read through `binlog()`: every event comes back.
- Unit Test: BE, 191 tests in 19 suites, all pass: the new
`TabletSchemaTest.test_binlog_col_idx_from_pb_column_names`, and every
suite that builds a row binlog schema or touches the changed code
(`TabletSchemaTest`, `TabletMetaTest`, `PbConvert`, `RowBinlogDeriveTest`,
`RowBinlogVmergeCompactionTest`, `SegmentFlusher(Transform)FormatTest`,
`(Cloud)GroupRowset*Test`, `TestEngineStorageMigrationTask`, the
merge-on-write transform tests, ...).
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 15eb3f4 commit 26d02dc
10 files changed
Lines changed: 56 additions & 99 deletions
File tree
- be
- src
- cloud
- storage/tablet
- test
- storage
- rowset
- tablet
- transform
- testutil
- gensrc
- proto
- thrift
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
511 | 511 | | |
512 | 512 | | |
513 | 513 | | |
514 | | - | |
515 | | - | |
516 | | - | |
517 | | - | |
518 | | - | |
519 | | - | |
520 | | - | |
521 | | - | |
522 | | - | |
523 | 514 | | |
524 | 515 | | |
525 | 516 | | |
| |||
575 | 566 | | |
576 | 567 | | |
577 | 568 | | |
578 | | - | |
579 | | - | |
580 | | - | |
581 | | - | |
582 | | - | |
583 | | - | |
584 | | - | |
585 | | - | |
586 | | - | |
587 | 569 | | |
588 | 570 | | |
589 | 571 | | |
| |||
652 | 634 | | |
653 | 635 | | |
654 | 636 | | |
655 | | - | |
656 | | - | |
657 | | - | |
658 | | - | |
659 | | - | |
660 | | - | |
661 | | - | |
662 | | - | |
663 | | - | |
664 | 637 | | |
665 | 638 | | |
666 | 639 | | |
| |||
717 | 690 | | |
718 | 691 | | |
719 | 692 | | |
720 | | - | |
721 | | - | |
722 | | - | |
723 | | - | |
724 | | - | |
725 | | - | |
726 | | - | |
727 | | - | |
728 | | - | |
729 | 693 | | |
730 | 694 | | |
731 | 695 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
403 | 403 | | |
404 | 404 | | |
405 | 405 | | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
410 | | - | |
411 | | - | |
412 | | - | |
413 | | - | |
414 | | - | |
415 | 406 | | |
416 | 407 | | |
417 | 408 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1124 | 1124 | | |
1125 | 1125 | | |
1126 | 1126 | | |
1127 | | - | |
1128 | | - | |
1129 | | - | |
| 1127 | + | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
1130 | 1131 | | |
1131 | 1132 | | |
1132 | 1133 | | |
| |||
1463 | 1464 | | |
1464 | 1465 | | |
1465 | 1466 | | |
1466 | | - | |
1467 | | - | |
1468 | | - | |
1469 | 1467 | | |
1470 | 1468 | | |
1471 | 1469 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
136 | 136 | | |
137 | 137 | | |
138 | 138 | | |
139 | | - | |
140 | 139 | | |
141 | 140 | | |
142 | 141 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
937 | 937 | | |
938 | 938 | | |
939 | 939 | | |
940 | | - | |
941 | 940 | | |
942 | 941 | | |
943 | | - | |
944 | 942 | | |
945 | 943 | | |
946 | | - | |
947 | 944 | | |
948 | 945 | | |
949 | 946 | | |
| |||
3166 | 3163 | | |
3167 | 3164 | | |
3168 | 3165 | | |
3169 | | - | |
3170 | 3166 | | |
3171 | | - | |
3172 | | - | |
3173 | | - | |
3174 | 3167 | | |
3175 | 3168 | | |
3176 | 3169 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
70 | 71 | | |
71 | 72 | | |
72 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
73 | 114 | | |
74 | 115 | | |
75 | 116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | 123 | | |
129 | 124 | | |
130 | 125 | | |
| |||
167 | 162 | | |
168 | 163 | | |
169 | 164 | | |
170 | | - | |
171 | | - | |
172 | | - | |
173 | 165 | | |
174 | 166 | | |
175 | 167 | | |
| |||
283 | 275 | | |
284 | 276 | | |
285 | 277 | | |
286 | | - | |
287 | | - | |
288 | | - | |
289 | 278 | | |
290 | 279 | | |
291 | 280 | | |
| |||
485 | 474 | | |
486 | 475 | | |
487 | 476 | | |
488 | | - | |
489 | | - | |
490 | | - | |
491 | 477 | | |
492 | 478 | | |
493 | 479 | | |
| |||
684 | 670 | | |
685 | 671 | | |
686 | 672 | | |
687 | | - | |
688 | | - | |
689 | | - | |
690 | | - | |
691 | 673 | | |
692 | 674 | | |
693 | 675 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
226 | | - | |
227 | 226 | | |
228 | 227 | | |
229 | | - | |
230 | 228 | | |
231 | 229 | | |
232 | | - | |
233 | 230 | | |
234 | 231 | | |
235 | 232 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
578 | 578 | | |
579 | 579 | | |
580 | 580 | | |
581 | | - | |
582 | | - | |
583 | | - | |
584 | | - | |
585 | | - | |
586 | | - | |
587 | | - | |
588 | | - | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
589 | 585 | | |
590 | 586 | | |
591 | 587 | | |
| |||
640 | 636 | | |
641 | 637 | | |
642 | 638 | | |
643 | | - | |
644 | | - | |
645 | | - | |
646 | | - | |
647 | | - | |
648 | | - | |
649 | | - | |
650 | | - | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
651 | 643 | | |
652 | 644 | | |
653 | 645 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
57 | | - | |
58 | | - | |
59 | | - | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
| |||
0 commit comments