Skip to content

Commit efc929a

Browse files
[fix](table stream) fix table stream TSO boundary semantics (apache#67480)
### What problem does this PR solve? Issue Number: close apache#67093 Related PR: #xxx Problem Summary: Timestamp-based `@incr` reads are intended to use a left-closed, right-open interval: `[startTimestamp, endTimestamp)`. Previously, FE converted both timestamps to the last logical TSO of their respective milliseconds, while BE applied `GT/LE` predicates. This produced `(endOfStartMs, endOfEndMs]`. Consequently, the scan excluded changes in the requested start millisecond and included changes in the requested end millisecond. For example, a query for `[10:00:00, 10:00:01)` excluded changes at `10:00:00.000` and included changes at `10:00:01.000`, assigning boundary events to the wrong window. This PR changes BE TSO predicates to `GE/LT` and updates FE bound construction across incremental reads, Table Streams, snapshots, time travel, and transaction visibility waiting. Timestamp-based reads now follow the intended half-open interval, while Table Stream consumption and historical snapshots retain their existing inclusive committed-TSO endpoints. #### Scan bounds and persisted offsets Table Stream metadata continues to store actual committed TSO values. Only scan-facing bounds are converted: ```text (lastConsumed, currentTso] == [nextTso(lastConsumed), nextTso(currentTso)) ``` The conversion happens when constructing the scan view. The underlying stream update map and offset commit paths retain the original committed TSOs, preventing scan-bound conversion from advancing persisted offsets or being applied again on the next consumption. For the table below: - `startOf(ms)` means `composeEmptyCounterTSO(ms)`, with logical counter zero. - `nextTso(t)` means the successor of a valid real TSO. - `L`, `C`, and `H` are the last consumed, current committed, and historical committed TSOs. | Read path | Scan-facing bounds | Resulting semantics | | --- | --- | --- | | `@incr(startTimestamp, endTimestamp)` | `[startOf(startMs), startOf(endMs))` | Includes the entire start millisecond and excludes the entire end millisecond. | | `@incr` without a start timestamp | Starts at `startOf(0)` | Reads from the beginning up to the applicable exclusive end. | | Incremental scan without an explicit end | Uses `nextTso(C)` when a current committed TSO exists | Includes changes at the current committed TSO. | | Local and Cloud Table Stream consumption | `[nextTso(L), nextTso(C))` | Preserves `(L, C]`. | | First incremental consumption without a recorded lower bound | Lower bound unset; end is `nextTso(C)` | Includes changes through `C`. | | DUP snapshot/history scan | Upper bound `nextTso(H)` | Includes rows committed at `H`. | | MOW snapshot reconstruction | Base rows below `nextTso(H)`; binlog changes starting at `nextTso(H)` | Preserves the snapshot at `H`, with subsequent changes used to reconstruct before-images. | | `FOR VERSION AS OF v` | Exclusive snapshot boundary `nextTso(v)` | Keeps the requested version inclusive. | | `FOR TIME AS OF ms` | Exclusive snapshot boundary `nextTso(startOf(ms))` | Includes every logical TSO within the requested millisecond. | For MOW time travel, the base branch uses `commit_tso < boundary`, and the binlog reconstruction branch starts at `commit_tso >= boundary`. Both branches therefore use the same snapshot boundary. #### Missing offsets and empty snapshot partitions A missing bound or a negative sentinel such as `-1` is not a real committed TSO. `toExclusiveBound` converts these values to `null`; valid committed TSOs are converted to their successors. When a scan has no explicit end, the planner converts the current partition TSO in the same way and leaves the end unset if the partition has never received a real TSO. Snapshot partition selection also distinguishes a real consumption baseline from an empty-partition marker. A partition that was empty when the stream was created can have a recorded offset of `-1`; in Cloud mode, it can additionally be marked `CONSUMED`. Neither condition alone means that the partition contained data at the snapshot boundary. This PR requires a positive recorded offset for such partitions to participate in the consumed-partition snapshot path. Cloud mode also requires the offset field to be present. This prevents snapshot reconstruction from falling back to the partition's current TSO and exposing rows inserted after the snapshot into a partition that was originally empty. Incremental reads can still consume those later changes through their normal path. #### Visibility waiting The non-cloud visibility waiter now uses `commitTSO < endTSO`, matching the exclusive upper-bound representation. - An explicit positive end timestamp `E` maps to `startOf(E)`, excluding transactions in millisecond `E`. - When the end is omitted, or the existing non-positive fallback applies, query-start millisecond `M` maps to `startOf(M + 1)`. This includes all logical counters within `M`, so relevant transactions committed in the same millisecond are still awaited. The Cloud visibility waiter is unchanged and continues to use its transaction-ID watermark. #### TSO range and sentinel handling `Long.MAX_VALUE` is reserved as `UNBOUNDED_TSO`, representing the latest/unbounded version. Real allocated TSOs are limited to `MAX_REAL_TSO = Long.MAX_VALUE - 1`. `nextTso` validates its input and returns `tso + 1`. It does not saturate: passing `Long.MAX_VALUE` is rejected. `FOR VERSION AS OF Long.MAX_VALUE` handles the sentinel explicitly and keeps it as the exclusive upper bound above all real TSOs. The TSO generator now uses `composeRealTso` to validate physical and logical components and reject values outside the real-TSO range, enforcing the invariant required by successor conversion. These changes preserve the stored offset representation. They change the interpretation of scan bounds exchanged between FE and BE, so unchanged persisted metadata does not imply compatibility between old and new FE/BE scan semantics. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [x] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [x] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
1 parent 9f3b084 commit efc929a

11 files changed

Lines changed: 261 additions & 54 deletions

File tree

‎be/src/exec/scan/olap_scanner.cpp‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -332,13 +332,14 @@ Status OlapScanner::_init_tso_predicates() {
332332

333333
const auto* tso_column = read_schema->column(tso_ordinal);
334334
const auto& tso_data_type = read_schema->data_type(tso_ordinal);
335+
// The TSO scan range is left-closed right-open [start_tso, end_tso).
335336
if (_start_tso.has_value()) {
336-
_tablet_reader_params.predicates.push_back(create_comparison_predicate<PredicateType::GT>(
337+
_tablet_reader_params.predicates.push_back(create_comparison_predicate<PredicateType::GE>(
337338
tso_ordinal, tso_column->name(), tso_data_type,
338339
Field::create_field<TYPE_BIGINT>(*_start_tso), false));
339340
}
340341
if (_end_tso.has_value()) {
341-
_tablet_reader_params.predicates.push_back(create_comparison_predicate<PredicateType::LE>(
342+
_tablet_reader_params.predicates.push_back(create_comparison_predicate<PredicateType::LT>(
342343
tso_ordinal, tso_column->name(), tso_data_type,
343344
Field::create_field<TYPE_BIGINT>(*_end_tso), false));
344345
}

‎fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStream.java‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -230,7 +230,13 @@ public boolean hasHistoricalData(long partitionId) {
230230
}
231231

232232
public boolean hasConsumedData(long partitionId) {
233-
return partitionOffset.containsKey(partitionId);
233+
// A partition that was empty at stream creation is recorded with the sentinel offset -1
234+
// (see initializeLocalOffsets); a real committed TSO is always positive (its physical part
235+
// is non-zero). So only a positive recorded offset counts as a real consumption baseline.
236+
// This keeps empty partitions out of the snapshot scan instead of letting them fall back to
237+
// the live partition TSO and leak post-snapshot rows.
238+
Long offset = partitionOffset.get(partitionId);
239+
return offset != null && offset > 0;
234240
}
235241

236242
public Pair<Long, Long> getStreamUpdate(Long partitionId) {

‎fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java‎

Lines changed: 24 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
import org.apache.doris.common.util.Util;
2828
import org.apache.doris.thrift.TColumn;
2929
import org.apache.doris.thrift.TPrimitiveType;
30+
import org.apache.doris.tso.TSOTimestamp;
3031

3132
import com.google.common.base.Preconditions;
3233
import com.google.common.collect.ImmutableList;
@@ -306,8 +307,17 @@ public List<Long> filterIncrementalPartitionIds(List<Long> partitionIds) {
306307
public List<Long> filterConsumedPartitionIds(List<Long> partitionIds) {
307308
if (hasCloudReadStates()) {
308309
return partitionIds.stream()
309-
.filter(id -> cloudReadStates.get(id).getOffsetState()
310-
== Cloud.TableStreamOffsetStatePB.TABLE_STREAM_OFFSET_CONSUMED)
310+
.filter(id -> {
311+
Cloud.TableStreamPartitionReadStatePB state = cloudReadStates.get(id);
312+
// A partition empty at stream creation is recorded as CONSUMED with the
313+
// sentinel offset -1 (CloudInternalCatalog: emptyPartition -> commit_tso=-1).
314+
// Such a partition has no real consumption baseline; exclude it so snapshot
315+
// rebuild does not fall back to the live TSO and leak post-snapshot rows.
316+
// Mirrors the non-cloud hasConsumedData() offset > 0 guard.
317+
return state.getOffsetState()
318+
== Cloud.TableStreamOffsetStatePB.TABLE_STREAM_OFFSET_CONSUMED
319+
&& state.hasOffsetTso() && state.getOffsetTso() > 0;
320+
})
311321
.collect(ImmutableList.toImmutableList());
312322
}
313323
return partitionIds.stream()
@@ -333,14 +343,24 @@ public BaseTableStream.StreamScanType getStreamScanType() {
333343
public Map<Long, Pair<Long, Long>> getPartitionOffsets(List<Long> selectedPartitionIds) {
334344
return outputUpdateMap.entrySet().stream()
335345
.filter(s -> selectedPartitionIds.contains(s.getKey()))
336-
.collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue));
346+
.collect(Collectors.toMap(Map.Entry::getKey, s -> {
347+
// Storage keeps the real committed TSO points (closed-interval semantics).
348+
// BE scans a left-closed right-open range [startTso, endTso), so convert the
349+
// bounds only in this scan-facing read view. outputUpdateMap and the offset
350+
// commit path (toOlapTableStreamUpdate) stay on the real-TSO coordinate system.
351+
Pair<Long, Long> v = s.getValue();
352+
return Pair.of(TSOTimestamp.toExclusiveBound(v.first),
353+
TSOTimestamp.toExclusiveBound(v.second));
354+
}));
337355
}
338356

339357
// get history partition offsets partitionId -> (null, historicalTimestampOffset)
340358
public Map<Long, Pair<Long, Long>> getHistoryPartitionOffsets(List<Long> selectedPartitionIds) {
341359
return outputUpdateMap.entrySet().stream()
342360
.filter(s -> selectedPartitionIds.contains(s.getKey()))
343-
.collect(Collectors.toMap(Map.Entry::getKey, s -> Pair.of(null, s.getValue().first)));
361+
// historicalTso is an inclusive upper bound; shift to the half-open exclusive end.
362+
.collect(Collectors.toMap(Map.Entry::getKey,
363+
s -> Pair.of(null, TSOTimestamp.toExclusiveBound(s.getValue().first))));
344364
}
345365

346366
public List<Long> filterNormalSnapshotPartitionIds(List<Long> partitionIds) {

‎fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java‎

Lines changed: 28 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -74,7 +74,7 @@
7474
import org.apache.doris.nereids.trees.expressions.ExprId;
7575
import org.apache.doris.nereids.trees.expressions.Expression;
7676
import org.apache.doris.nereids.trees.expressions.InPredicate;
77-
import org.apache.doris.nereids.trees.expressions.LessThanEqual;
77+
import org.apache.doris.nereids.trees.expressions.LessThan;
7878
import org.apache.doris.nereids.trees.expressions.NamedExpression;
7979
import org.apache.doris.nereids.trees.expressions.Slot;
8080
import org.apache.doris.nereids.trees.expressions.SlotReference;
@@ -502,8 +502,8 @@ private static LogicalPlan checkAndAddDeleteSignFilter(LogicalOlapScan scan, Con
502502

503503
/**
504504
* Build the time-travel (FOR VERSION/TIME AS OF) plan for an olap scan.
505-
* dup: Filter(__DORIS_COMMIT_TSO_COL__ &lt;= targetTso).
506-
* mow: base(survived rows, tso&lt;=t1) UNION ALL binlog(before-image of UPDATE_BEFORE/DELETE).
505+
* dup: Filter(__DORIS_COMMIT_TSO_COL__ &lt; targetTso), targetTso being the exclusive upper bound.
506+
* mow: base(survived rows, tso &lt; targetTso) UNION ALL binlog(before-image of UPDATE_BEFORE/DELETE).
507507
*/
508508
private LogicalPlan buildTimeTravelPlan(LogicalOlapScan scan, OlapTable olapTable,
509509
TableSnapshot snapshot, UnboundRelation unboundRelation, List<String> qualifier,
@@ -541,8 +541,9 @@ private void validateTimeTravel(OlapTable olapTable) {
541541
}
542542

543543
/**
544-
* Add Filter(__DORIS_COMMIT_TSO_COL__ &lt;= targetTso) on top of {@code child}. The tso slot is
545-
* resolved from {@code child} output; {@code child} must pass through the scan output slots.
544+
* Add Filter(__DORIS_COMMIT_TSO_COL__ &lt; targetTso) on top of {@code child}, where targetTso is
545+
* the right-open (exclusive) upper bound from resolveSnapshotTso. The tso slot is resolved from
546+
* {@code child} output; {@code child} must pass through the scan output slots.
546547
*/
547548
private LogicalPlan addCommitTsoFilter(LogicalPlan child, long targetTso, OlapTable olapTable) {
548549
Slot tsoSlot = null;
@@ -554,25 +555,35 @@ private LogicalPlan addCommitTsoFilter(LogicalPlan child, long targetTso, OlapTa
554555
}
555556
Preconditions.checkArgument(tsoSlot != null,
556557
"%s not found on table %s", Column.COMMIT_TSO_COL, olapTable.getQualifiedName());
557-
Expression conjunct = new LessThanEqual(tsoSlot, new BigIntLiteral(targetTso));
558+
Expression conjunct = new LessThan(tsoSlot, new BigIntLiteral(targetTso));
558559
return new LogicalFilter<>(ImmutableSet.of(conjunct), child);
559560
}
560561

561562
/**
562-
* Resolve a TableSnapshot to a target commit tso (inclusive upper bound).
563-
* VERSION: the literal is the tso itself. TIME: wall-clock string -&gt; ms -&gt; tso upper bound.
563+
* Resolve a TableSnapshot to the right-open (exclusive) commit-tso upper bound: the scan keeps
564+
* rows with commit_tso &lt; the returned value. VERSION: literal tso + 1 (so the literal itself is
565+
* included). TIME: successor of (requested millisecond, logical counter 0), so that exact TSO
566+
* is included but larger logical counters in the same millisecond are excluded. Used uniformly
567+
* by the dup filter, the mow union left filter and the mow union right-branch lower bound.
564568
*/
565569
private long resolveSnapshotTso(TableSnapshot snapshot) {
566570
if (snapshot.getType() == TableSnapshot.VersionType.VERSION) {
571+
long version;
567572
try {
568-
return Long.parseLong(snapshot.getValue().trim());
573+
version = Long.parseLong(snapshot.getValue().trim());
569574
} catch (NumberFormatException e) {
570575
throw new AnalysisException(
571576
"Invalid version in FOR VERSION AS OF: " + snapshot.getValue());
572577
}
578+
// UNBOUNDED_TSO (Long.MAX_VALUE) is the "latest / unbounded" sentinel: keep it as the
579+
// exclusive upper bound above every real TSO. A real version is converted to its
580+
// right-open successor so that commit_tso < result includes the requested version.
581+
return version == TSOTimestamp.UNBOUNDED_TSO
582+
? TSOTimestamp.UNBOUNDED_TSO
583+
: TSOTimestamp.nextTso(version);
573584
}
574585
long ms = OlapScanNode.parseChangeTimestamp(snapshot.getValue());
575-
return TSOTimestamp.composeFullTimestamp(ms);
586+
return TSOTimestamp.nextTso(TSOTimestamp.composePhysicalTimestamp(ms));
576587
}
577588

578589
/**
@@ -589,14 +600,16 @@ private LogicalPlan buildMowTimeTravelUnion(LogicalOlapScan baseScan, OlapTable
589600
|| ((SlotReference) slot).isVisible())
590601
.collect(Collectors.toList());
591602

592-
// left: base survived rows at t1 = delete_sign=0 AND commit_tso<=t1, projected to visible.
603+
// left: base survived rows at t1 = delete_sign=0 AND commit_tso < targetTso, projected to visible.
593604
LogicalPlan left = checkAndAddDeleteSignFilter(baseScan, ConnectContext.get(), olapTable, true);
594605
left = projectFromOriginSlots(addCommitTsoFilter(left, targetTso, olapTable), visibleOutput);
595606

596-
// right: binlog MIN_DELTA over tso>t1, keep UPDATE_BEFORE/DELETE rows (before image),
607+
// right: binlog MIN_DELTA over tso >= targetTso, keep UPDATE_BEFORE/DELETE rows (before image),
597608
// projected to the same visible schema. BE splits each change so UPDATE_BEFORE/DELETE rows
598609
// already carry the pre-change value in the (same-named) value columns.
599610
RowBinlogTableWrapper binlogTable = new RowBinlogTableWrapper(olapTable, CollectionUtils.isEmpty(partIds)
611+
// targetTso is the exclusive upper bound; the right branch reads tso >= targetTso,
612+
// seamlessly meeting the left branch's commit_tso < targetTso.
600613
? makeUniformedTimestampRangeMap(olapTable.getPartitionIds(), Pair.of(targetTso, null)) :
601614
makeUniformedTimestampRangeMap(partIds, Pair.of(targetTso, null)));
602615
RelationId binlogRelationId = cascadesContext.getStatementContext().getNextRelationId();
@@ -709,10 +722,11 @@ private StreamScanType checkChangeScanCondition(OlapTable olapTable, TableScanPa
709722

710723
private Pair<Long, Long> parseTimestampRange(TableScanParams scanParams) {
711724
Map<String, String> params = scanParams.getMapParams();
725+
// @incr reads a left-closed right-open range [startTso, endTso): BE applies GE/LT directly.
726+
// composePhysicalTimestamp maps a millisecond to its start (logical counter 0), so GE includes
727+
// the whole startMs and LT excludes the whole endMs. No +1 shift is needed here.
712728
Long startTimestamp = OlapScanNode.parseChangeTimestamp(
713729
params.getOrDefault(OlapScanNode.OLAP_START_TIMESTAMP, "0"));
714-
// BE applies start_tso < commit_tso <= end_tso. Logical zero on both boundaries
715-
// therefore gives the user-facing physical interval [startTimestamp, endTimestamp).
716730
startTimestamp = TSOTimestamp.composePhysicalTimestamp(startTimestamp);
717731
Long endTimestamp = null;
718732
if (params.containsKey((OlapScanNode.OLAP_END_TIMESTAMP))) {

‎fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java‎

Lines changed: 12 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@
100100
import org.apache.doris.thrift.TScanRangeLocation;
101101
import org.apache.doris.thrift.TScanRangeLocations;
102102
import org.apache.doris.thrift.TSortInfo;
103+
import org.apache.doris.tso.TSOTimestamp;
103104

104105
import com.google.common.annotations.VisibleForTesting;
105106
import com.google.common.base.Joiner;
@@ -581,13 +582,17 @@ private void addScanRangeLocations(Partition partition,
581582
parseBinlogScanType(scanParams, ((OlapTableWrapper) olapTable).getOriginTable());
582583
Pair<Long, Long> update = getPartitionOffset(partition.getId());
583584
if (update != null) {
585+
// push down tso range as half-open [startTso, endTso) bounds
584586
if (update.first != null) {
585587
paloRange.setStartTso(update.first);
586588
}
587-
if (update.second != null) {
588-
paloRange.setEndTso(update.second);
589-
} else {
590-
paloRange.setEndTso(partition.getTso());
589+
// No end recorded: fall back to the current committed TSO. toExclusiveBound
590+
// returns null for a partition that never got a real TSO (getTso() == -1), in
591+
// which case we leave endTso unset (no upper bound) instead of using -1.
592+
Long endTso = update.second != null
593+
? update.second : TSOTimestamp.toExclusiveBound(partition.getTso());
594+
if (endTso != null) {
595+
paloRange.setEndTso(endTso);
591596
}
592597
}
593598
if (binlogScanType != TBinlogScanType.NONE) {
@@ -1964,14 +1969,6 @@ public TableScanParams getScanParams() {
19641969
return scanParams;
19651970
}
19661971

1967-
public long getIncrementalScanEndTime() {
1968-
if (scanParams != null && scanParams.incrementalRead()
1969-
&& scanParams.getMapParams().containsKey(OLAP_END_TIMESTAMP)) {
1970-
return parseChangeTimestamp(scanParams.getMapParams().get(OLAP_END_TIMESTAMP));
1971-
}
1972-
return 0;
1973-
}
1974-
19751972
public static long parseChangeTimestamp(String ts) {
19761973
if (ts != null) {
19771974
long changeTimestamp;
@@ -1983,6 +1980,9 @@ public static long parseChangeTimestamp(String ts) {
19831980
if (changeTimestamp < 0) {
19841981
throw new ParseException("Invalid TIMESTAMP format in incr clause: " + ts);
19851982
}
1983+
if (changeTimestamp > TSOTimestamp.MAX_PHYSICAL_TIMESTAMP) {
1984+
throw new ParseException("Timestamp exceeds supported TSO range: " + ts);
1985+
}
19861986
return changeTimestamp;
19871987
}
19881988
throw new ParseException("Invalid timestamp:" + ts);

‎fe/fe-core/src/main/java/org/apache/doris/tso/TSOService.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,7 +243,7 @@ public long getTSO() {
243243
if (MetricRepo.isInit) {
244244
MetricRepo.COUNTER_TSO_CLOCK_GET_SUCCESS.increase(1L);
245245
}
246-
return TSOTimestamp.composeTimestamp(physical, logical);
246+
return TSOTimestamp.composeRealTso(physical, logical);
247247
}
248248
throw new RuntimeException("Failed to get TSO after " + maxGetTSORetryCount + " retries", lastFailure);
249249
}

0 commit comments

Comments
 (0)