Repository navigation
Commit bfe46ec
authored
[fix](row binlog) make time-based incremental reads use a safe fence (#67594)
### What problem does this PR solve?
Issue Number: None
Related PR: #65850
Problem Summary:
Time-based `@incr` reads previously allowed an `endTimestamp` later than
the FE's current TSO. Such a future boundary cannot be closed by the
transaction waiting logic: a transaction may start after the request
begins but still fall before that future boundary, so the scan cannot
guarantee a complete result.
In addition, waiting only for currently running transactions is
insufficient unless the TSO boundary, transaction watermark, transaction
visibility, and table visible versions are coordinated under one
authoritative fence. Otherwise, especially on a follower FE, a
transaction can already be `VISIBLE` while the local catalog still
exposes an older visible version, causing the incremental scan to miss
rows.
This PR establishes a safe read fence for both classic and cloud modes:
1. Acquire the current TSO from the master FE.
2. Capture the transaction ID watermark after the TSO snapshot.
3. Wait for transactions at or below the watermark that involve the
scanned OLAP tables to reach a final visible or aborted state.
4. In classic mode, synchronize table publishers and wait for follower
journal replay so catalog visible versions cross the same fence.
5. In cloud mode, refresh the latest visible versions from MetaService
after the transaction wait.
6. Reject an `endTimestamp` later than the captured
`CURRENT_TSO_PHYSICAL_TIME`.
7. Convert a physical timestamp `P` to TSO boundary `(P, 0)`, preserving
half-open range semantics: `[startTimestamp, endTimestamp)`.
#### Future endTimestamp error example
Request:
```sql
SELECT id, value, __DORIS_BINLOG_OP__
FROM example_table@incr(
"startTimestamp" = "2026-09-01 00:00:00",
"endTimestamp" = "2999-01-01 00:00:00",
"incrementType" = "DETAIL"
);
```
Response:
```text
ERROR 1105 (HY000): errCode = 2, detailMessage =
endTimestamp exceeds the maximum supported time for an INCR read:
requestedEndTimestampMs=<requested epoch milliseconds>,
CURRENT_TSO_PHYSICAL_TIME=<maximum supported epoch milliseconds>
```
The maximum currently supported boundary can be obtained from:
```sql
SELECT CURRENT_TSO_PHYSICAL_TIME
FROM information_schema.tso_status;
```
### Release note
Time-based `@incr` reads now reject `endTimestamp` values after
`CURRENT_TSO_PHYSICAL_TIME`. Timestamp ranges use half-open semantics
`[startTimestamp, endTimestamp)`, with each physical timestamp mapped to
logical counter zero.
### Check List (For Author)
- Test: FE unit tests, FE and BE builds, and
`row_binlog_p0/test_binlog_changes_syntax` regression test
- Behavior changed: Yes
- Future `endTimestamp` values are rejected with the maximum supported
TSO physical time in the error message.
- Physical timestamp boundaries are interpreted as `(physicalTime, 0)`.
- Does this need documentation: No1 parent 00b08ad commit bfe46ec
15 files changed
Lines changed: 526 additions & 184 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- cloud/transaction
- nereids/rules/analysis
- qe
- service
- transaction
- tso
- test/java/org/apache/doris
- cloud/transaction
- nereids/trees/plans
- qe
- transaction
- tso
- gensrc/thrift
- regression-test/suites/row_binlog_p0
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2589 | 2589 | | |
2590 | 2590 | | |
2591 | 2591 | | |
| 2592 | + | |
| 2593 | + | |
| 2594 | + | |
| 2595 | + | |
| 2596 | + | |
| 2597 | + | |
2592 | 2598 | | |
2593 | 2599 | | |
2594 | 2600 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
711 | 711 | | |
712 | 712 | | |
713 | 713 | | |
714 | | - | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
715 | 717 | | |
716 | 718 | | |
717 | 719 | | |
718 | | - | |
| 720 | + | |
719 | 721 | | |
720 | 722 | | |
721 | 723 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
134 | 134 | | |
135 | 135 | | |
136 | 136 | | |
| 137 | + | |
137 | 138 | | |
138 | 139 | | |
139 | 140 | | |
| |||
154 | 155 | | |
155 | 156 | | |
156 | 157 | | |
| 158 | + | |
| 159 | + | |
157 | 160 | | |
158 | 161 | | |
159 | 162 | | |
| |||
3384 | 3387 | | |
3385 | 3388 | | |
3386 | 3389 | | |
| 3390 | + | |
| 3391 | + | |
| 3392 | + | |
| 3393 | + | |
| 3394 | + | |
| 3395 | + | |
| 3396 | + | |
| 3397 | + | |
| 3398 | + | |
| 3399 | + | |
| 3400 | + | |
| 3401 | + | |
| 3402 | + | |
| 3403 | + | |
| 3404 | + | |
| 3405 | + | |
| 3406 | + | |
| 3407 | + | |
| 3408 | + | |
| 3409 | + | |
| 3410 | + | |
| 3411 | + | |
| 3412 | + | |
| 3413 | + | |
| 3414 | + | |
| 3415 | + | |
| 3416 | + | |
| 3417 | + | |
| 3418 | + | |
3387 | 3419 | | |
3388 | 3420 | | |
3389 | 3421 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
928 | 928 | | |
929 | 929 | | |
930 | 930 | | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
931 | 937 | | |
932 | 938 | | |
933 | 939 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
214 | 220 | | |
215 | 221 | | |
216 | 222 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
57 | 62 | | |
58 | 63 | | |
59 | 64 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
112 | 112 | | |
113 | 113 | | |
114 | 114 | | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
115 | 124 | | |
116 | 125 | | |
117 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
623 | 623 | | |
624 | 624 | | |
625 | 625 | | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
626 | 644 | | |
627 | 645 | | |
628 | 646 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
586 | 586 | | |
587 | 587 | | |
588 | 588 | | |
589 | | - | |
590 | | - | |
| 589 | + | |
| 590 | + | |
591 | 591 | | |
592 | 592 | | |
593 | 593 | | |
| |||
0 commit comments