You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit efc929a
Browse filesBrowse the repository at this point in the historyBrowse files
### What problem does this PR solve?
Issue Number: closeapache#67093
Related PR: #xxx
Problem Summary:
Timestamp-based `@incr` reads are intended to use a left-closed,
right-open interval: `[startTimestamp, endTimestamp)`.
Previously, FE converted both timestamps to the last logical TSO of
their respective milliseconds, while BE applied `GT/LE` predicates. This
produced `(endOfStartMs, endOfEndMs]`.
Consequently, the scan excluded changes in the requested start
millisecond and included changes in the requested end millisecond. For
example, a query for `[10:00:00, 10:00:01)` excluded changes at
`10:00:00.000` and included changes at `10:00:01.000`, assigning
boundary events to the wrong window.
This PR changes BE TSO predicates to `GE/LT` and updates FE bound
construction across incremental reads, Table Streams, snapshots, time
travel, and transaction visibility waiting. Timestamp-based reads now
follow the intended half-open interval, while Table Stream consumption
and historical snapshots retain their existing inclusive committed-TSO
endpoints.
#### Scan bounds and persisted offsets
Table Stream metadata continues to store actual committed TSO values.
Only scan-facing bounds are converted:
```text
(lastConsumed, currentTso]
==
[nextTso(lastConsumed), nextTso(currentTso))
```
The conversion happens when constructing the scan view. The underlying
stream update map and offset commit paths retain the original committed
TSOs, preventing scan-bound conversion from advancing persisted offsets
or being applied again on the next consumption.
For the table below:
- `startOf(ms)` means `composeEmptyCounterTSO(ms)`, with logical counter
zero.
- `nextTso(t)` means the successor of a valid real TSO.
- `L`, `C`, and `H` are the last consumed, current committed, and
historical committed TSOs.
| Read path | Scan-facing bounds | Resulting semantics |
| --- | --- | --- |
| `@incr(startTimestamp, endTimestamp)` | `[startOf(startMs),
startOf(endMs))` | Includes the entire start millisecond and excludes
the entire end millisecond. |
| `@incr` without a start timestamp | Starts at `startOf(0)` | Reads
from the beginning up to the applicable exclusive end. |
| Incremental scan without an explicit end | Uses `nextTso(C)` when a
current committed TSO exists | Includes changes at the current committed
TSO. |
| Local and Cloud Table Stream consumption | `[nextTso(L), nextTso(C))`
| Preserves `(L, C]`. |
| First incremental consumption without a recorded lower bound | Lower
bound unset; end is `nextTso(C)` | Includes changes through `C`. |
| DUP snapshot/history scan | Upper bound `nextTso(H)` | Includes rows
committed at `H`. |
| MOW snapshot reconstruction | Base rows below `nextTso(H)`; binlog
changes starting at `nextTso(H)` | Preserves the snapshot at `H`, with
subsequent changes used to reconstruct before-images. |
| `FOR VERSION AS OF v` | Exclusive snapshot boundary `nextTso(v)` |
Keeps the requested version inclusive. |
| `FOR TIME AS OF ms` | Exclusive snapshot boundary
`nextTso(startOf(ms))` | Includes every logical TSO within the requested
millisecond. |
For MOW time travel, the base branch uses `commit_tso < boundary`, and
the binlog reconstruction branch starts at `commit_tso >= boundary`.
Both branches therefore use the same snapshot boundary.
#### Missing offsets and empty snapshot partitions
A missing bound or a negative sentinel such as `-1` is not a real
committed TSO. `toExclusiveBound` converts these values to `null`; valid
committed TSOs are converted to their successors. When a scan has no
explicit end, the planner converts the current partition TSO in the same
way and leaves the end unset if the partition has never received a real
TSO.
Snapshot partition selection also distinguishes a real consumption
baseline from an empty-partition marker. A partition that was empty when
the stream was created can have a recorded offset of `-1`; in Cloud
mode, it can additionally be marked `CONSUMED`. Neither condition alone
means that the partition contained data at the snapshot boundary.
This PR requires a positive recorded offset for such partitions to
participate in the consumed-partition snapshot path. Cloud mode also
requires the offset field to be present. This prevents snapshot
reconstruction from falling back to the partition's current TSO and
exposing rows inserted after the snapshot into a partition that was
originally empty. Incremental reads can still consume those later
changes through their normal path.
#### Visibility waiting
The non-cloud visibility waiter now uses `commitTSO < endTSO`, matching
the exclusive upper-bound representation.
- An explicit positive end timestamp `E` maps to `startOf(E)`, excluding
transactions in millisecond `E`.
- When the end is omitted, or the existing non-positive fallback
applies, query-start millisecond `M` maps to `startOf(M + 1)`. This
includes all logical counters within `M`, so relevant transactions
committed in the same millisecond are still awaited.
The Cloud visibility waiter is unchanged and continues to use its
transaction-ID watermark.
#### TSO range and sentinel handling
`Long.MAX_VALUE` is reserved as `UNBOUNDED_TSO`, representing the
latest/unbounded version. Real allocated TSOs are limited to
`MAX_REAL_TSO = Long.MAX_VALUE - 1`.
`nextTso` validates its input and returns `tso + 1`. It does not
saturate: passing `Long.MAX_VALUE` is rejected. `FOR VERSION AS OF
Long.MAX_VALUE` handles the sentinel explicitly and keeps it as the
exclusive upper bound above all real TSOs.
The TSO generator now uses `composeRealTso` to validate physical and
logical components and reject values outside the real-TSO range,
enforcing the invariant required by successor conversion.
These changes preserve the stored offset representation. They change the
interpretation of scan bounds exchanged between FE and BE, so unchanged
persisted metadata does not imply compatibility between old and new
FE/BE scan semantics.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
0 commit comments