Skip to content

Commit 2850519

Browse files
morningmanclaude
andcommitted
[fix](catalog) cache: give the SqlCache quiet-window gate a real wall clock so iceberg tables can cache
An iceberg table (and any query joining one) was never eligible for SqlCache. This is an availability bug, NOT a stale-read bug: staleness is guarded independently by a version-token equality check (SqlCacheContext stores getNewestUpdateVersionOrTime, NereidsSqlCacheManager compares it), which this change does not touch. Root cause: CacheAnalyzer.CacheTable.latestPartitionTime is overloaded as both the BE PCache version key AND the wall-clock input to the "quiet for N seconds" gate (now - latestPartitionTime >= cache_last_version_interval_second). For an external table it holds getNewestUpdateVersionOrTime(); hive/paimon return epoch millis so the gate works by luck, but iceberg returns last_updated_at in MICROSECONDS (~1.7e15). Math.max(now, micros) clamps now up to the micros value, so the delta is always 0 and never clears the window -> iceberg never caches, and its huge value also poisons any multi-table query it joins. Fix (connector supplies a genuine wall-clock millis, kept separate from the token): - ConnectorMvccPartitionView carries a new newestUpdateWallClockMillis alongside the monotonic marker; iceberg fills it from last_updated_at / 1000 (the connector owns its unit), while the marker stays micros for the version-token path (master parity). - PluginDrivenMvccSnapshot mirrors the field; MTMVRelatedTableIf gains a default getNewestUpdateTimeMillisForCache() returning getNewestUpdateVersionOrTime() (correct for olap/hive/paimon whose token is already millis), which PluginDrivenMvccExternalTable overrides so the range-view (iceberg) branch returns the wall-clock millis. - CacheAnalyzer gains CacheTable.latestPartitionUpdateMillis (olap: visibleVersionTime; external: the new accessor). latestPartitionTime stays the raw token (BE PCache key, full-precision staleness), and the quiet-window gate now subtracts max(latestPartitionUpdateMillis), decoupled from the token-sorted latestTable. Only iceberg's gate behavior changes (hive/paimon/olap keep the same value). The SPI field / accessor are backward compatible (overloaded ctors default to 0). Tests: - IcebergPartitionUtilsTest: the view's wall clock == micros marker / 1000. - PluginDrivenMvccExternalTableTest.testRangeViewCacheGateUsesWallClockMillisNotMarker: getNewestUpdateTimeMillisForCache returns the wall clock; the token stays the marker. - PluginTableCacheAnalyzerTest.testGateValueSourcedFromWallClockAccessor: the gate value comes from the wall-clock accessor while latestPartitionTime stays the raw token. - e2e test_iceberg_sqlcache.groovy: a quiet iceberg table becomes cacheable, and a write invalidates it (no stale results). 67 unit tests green across fe-connector-api / iceberg / hive / fe-core; builds + checkstyle pass. The probe rename that precedes this landed in 8043536a812. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
1 parent 48a2135 commit 2850519

14 files changed

Lines changed: 385 additions & 22 deletions

File tree

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccPartitionView.java‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,15 +69,23 @@ public enum Freshness {
6969
private final Freshness freshness;
7070
private final List<ConnectorMvccPartition> partitions;
7171
private final long newestUpdateMonotonicMarker;
72+
private final long newestUpdateWallClockMillis;
7273

7374
public ConnectorMvccPartitionView(Style style, Freshness freshness,
7475
List<ConnectorMvccPartition> partitions, long newestUpdateMonotonicMarker) {
76+
this(style, freshness, partitions, newestUpdateMonotonicMarker, 0L);
77+
}
78+
79+
public ConnectorMvccPartitionView(Style style, Freshness freshness,
80+
List<ConnectorMvccPartition> partitions, long newestUpdateMonotonicMarker,
81+
long newestUpdateWallClockMillis) {
7582
this.style = Objects.requireNonNull(style, "style");
7683
this.freshness = Objects.requireNonNull(freshness, "freshness");
7784
this.partitions = partitions == null
7885
? Collections.emptyList()
7986
: Collections.unmodifiableList(partitions);
8087
this.newestUpdateMonotonicMarker = newestUpdateMonotonicMarker;
88+
this.newestUpdateWallClockMillis = newestUpdateWallClockMillis;
8189
}
8290

8391
/** Returns an {@code UNPARTITIONED} view (no partitions, newest-update-time {@code 0}); the freshness
@@ -115,6 +123,19 @@ public long getNewestUpdateMonotonicMarker() {
115123
return newestUpdateMonotonicMarker;
116124
}
117125

126+
/**
127+
* The table's newest data-update time as a genuine WALL-CLOCK epoch-millis value, used only by the
128+
* SqlCache eligibility "quiet window" gate ({@code CacheAnalyzer}) — distinct from the monotonic marker
129+
* above, which stays a source-defined-scale version token. The connector normalizes to millis here (for
130+
* iceberg: {@code last_updated_at} microseconds / 1000) so the generic model never has to know a source's
131+
* unit. {@code 0} when unknown / unpartitioned. This value is NOT used for staleness (that is the token's
132+
* job via {@code getNewestUpdateVersionOrTime}); it only decides whether a table has been quiet long
133+
* enough to be worth caching, so a coarse millis value is sufficient.
134+
*/
135+
public long getNewestUpdateWallClockMillis() {
136+
return newestUpdateWallClockMillis;
137+
}
138+
118139
@Override
119140
public boolean equals(Object o) {
120141
if (this == o) {
@@ -127,19 +148,22 @@ public boolean equals(Object o) {
127148
return style == that.style
128149
&& freshness == that.freshness
129150
&& newestUpdateMonotonicMarker == that.newestUpdateMonotonicMarker
151+
&& newestUpdateWallClockMillis == that.newestUpdateWallClockMillis
130152
&& partitions.equals(that.partitions);
131153
}
132154

133155
@Override
134156
public int hashCode() {
135-
return Objects.hash(style, freshness, partitions, newestUpdateMonotonicMarker);
157+
return Objects.hash(style, freshness, partitions, newestUpdateMonotonicMarker,
158+
newestUpdateWallClockMillis);
136159
}
137160

138161
@Override
139162
public String toString() {
140163
return "ConnectorMvccPartitionView{style=" + style
141164
+ ", freshness=" + freshness
142165
+ ", partitions=" + partitions.size()
143-
+ ", newestUpdateMonotonicMarker=" + newestUpdateMonotonicMarker + "}";
166+
+ ", newestUpdateMonotonicMarker=" + newestUpdateMonotonicMarker
167+
+ ", newestUpdateWallClockMillis=" + newestUpdateWallClockMillis + "}";
144168
}
145169
}

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -602,8 +602,13 @@ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinne
602602
// partitions too). Byte-parity with master getNewestUpdateVersionOrTime: max(...).orElse(0).
603603
long newestUpdateMonotonicMarker = allByName.values().stream()
604604
.mapToLong(rb -> rb.lastUpdateTime).max().orElse(0L);
605+
// The SqlCache quiet-window gate needs a genuine wall-clock epoch-millis value; last_updated_at is an
606+
// iceberg timestamp in MICROSECONDS, so normalize to millis here (the connector owns its unit). The
607+
// monotonic marker above stays micros for the version-token path (master parity).
608+
long newestUpdateWallClockMillis = newestUpdateMonotonicMarker / 1000;
605609
return new ConnectorMvccPartitionView(ConnectorMvccPartitionView.Style.RANGE,
606-
ConnectorMvccPartitionView.Freshness.SNAPSHOT_ID, partitions, newestUpdateMonotonicMarker);
610+
ConnectorMvccPartitionView.Freshness.SNAPSHOT_ID, partitions,
611+
newestUpdateMonotonicMarker, newestUpdateWallClockMillis);
607612
}
608613

609614
/**

‎fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionUtilsTest.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -822,6 +822,12 @@ public void buildMvccPartitionViewEnumeratesRangePartitions() {
822822
// committed table has a positive value. MUTATION: mapping lastUpdateTime->0 (or orElse over no rows) -> red.
823823
Assertions.assertTrue(view.getNewestUpdateMonotonicMarker() > 0,
824824
"a committed RANGE table must report a positive newest-update-time for dictionary refresh");
825+
// The view ALSO carries a wall-clock epoch-millis for the SqlCache quiet-window gate, normalized from the
826+
// micros marker (last_updated_at is an iceberg timestamp in microseconds). MUTATION: passing the raw
827+
// micros marker as the wall clock (no /1000) -> red, which is exactly the bug that kept iceberg out of
828+
// SqlCache (a ~1.7e15 value dominating wall-clock now).
829+
Assertions.assertEquals(view.getNewestUpdateMonotonicMarker() / 1000, view.getNewestUpdateWallClockMillis(),
830+
"the wall-clock gate value must be the micros marker normalized to millis");
825831
}
826832

827833
@Test

‎fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTable.java‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -230,7 +230,8 @@ private PluginDrivenMvccSnapshot buildFromRangeView(ConnectorMvccSnapshot connec
230230
}
231231
}
232232
return new PluginDrivenMvccSnapshot(connectorSnapshot, nameToPartitionItem, nameToFreshnessValue,
233-
null, partitionType, snapshotIdFreshness, view.getNewestUpdateMonotonicMarker());
233+
null, partitionType, snapshotIdFreshness, view.getNewestUpdateMonotonicMarker(),
234+
view.getNewestUpdateWallClockMillis());
234235
}
235236

236237
/**
@@ -829,6 +830,25 @@ public long getNewestUpdateVersionOrTime() {
829830
.mapToLong(Long::longValue).filter(v -> v >= 0).max().orElse(0L);
830831
}
831832

833+
@Override
834+
public long getNewestUpdateTimeMillisForCache() {
835+
// Same branches as getNewestUpdateVersionOrTime, but the range-view (iceberg) branch returns a genuine
836+
// wall-clock epoch-millis (the connector normalizes its unit, e.g. iceberg micros/1000) instead of the
837+
// micros version token, so the SqlCache quiet-window gate can subtract it from wall-clock now without
838+
// the micros value dominating. Staleness/version comparison keeps using the token via
839+
// getNewestUpdateVersionOrTime; this value is gate-only. The last-modified (hive) and legacy (paimon)
840+
// branches already yield epoch millis, so they are unchanged.
841+
PluginDrivenMvccSnapshot pin = materializeLatest();
842+
if (pin.getConnectorSnapshot().isLastModifiedFreshness()) {
843+
return queryTableFreshness().map(ConnectorTableFreshness::getTimestampMillis).orElse(0L);
844+
}
845+
if (pin.getPartitionType() != null) {
846+
return pin.getNewestUpdateWallClockMillis();
847+
}
848+
return pin.getNameToLastModifiedMillis().values().stream()
849+
.mapToLong(Long::longValue).filter(v -> v >= 0).max().orElse(0L);
850+
}
851+
832852
@Override
833853
public boolean isValidRelatedTable() {
834854
// MTMV refresh safety gate (MTMVTask): a base table that evolved into an unsupported partitioning

‎fe/fe-core/src/main/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccSnapshot.java‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ public class PluginDrivenMvccSnapshot implements MvccSnapshot {
6161
private final PartitionType partitionType; // null => legacy LIST/UNPARTITIONED computed from isPartitionInvalid
6262
private final boolean snapshotIdFreshness; // true => getPartitionSnapshot wraps a snapshot id, else a timestamp
6363
private final long newestUpdateMonotonicMarker; // range-view table newest-update-time (dictionary refresh marker)
64+
private final long newestUpdateWallClockMillis; // range-view newest-update wall-clock millis (SqlCache gate)
6465

6566
/**
6667
* @param connectorSnapshot the scalar snapshot pin (snapshot id used for reads)
@@ -110,6 +111,24 @@ public PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot,
110111
PartitionType partitionType,
111112
boolean snapshotIdFreshness,
112113
long newestUpdateMonotonicMarker) {
114+
this(connectorSnapshot, nameToPartitionItem, nameToFreshnessValue, pinnedSchema,
115+
partitionType, snapshotIdFreshness, newestUpdateMonotonicMarker, 0L);
116+
}
117+
118+
/**
119+
* Range-view constructor that also carries the wall-clock update millis for the SqlCache gate. The
120+
* {@code newestUpdateWallClockMillis} is a genuine epoch-millis value (the connector normalizes its unit,
121+
* e.g. iceberg micros/1000); it is used ONLY by the cache eligibility quiet-window gate and never for
122+
* staleness (that stays the monotonic marker / version token).
123+
*/
124+
public PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot,
125+
Map<String, PartitionItem> nameToPartitionItem,
126+
Map<String, Long> nameToFreshnessValue,
127+
SchemaCacheValue pinnedSchema,
128+
PartitionType partitionType,
129+
boolean snapshotIdFreshness,
130+
long newestUpdateMonotonicMarker,
131+
long newestUpdateWallClockMillis) {
113132
this.connectorSnapshot = connectorSnapshot;
114133
this.nameToPartitionItem = nameToPartitionItem == null
115134
? Collections.emptyMap()
@@ -121,6 +140,7 @@ public PluginDrivenMvccSnapshot(ConnectorMvccSnapshot connectorSnapshot,
121140
this.partitionType = partitionType;
122141
this.snapshotIdFreshness = snapshotIdFreshness;
123142
this.newestUpdateMonotonicMarker = newestUpdateMonotonicMarker;
143+
this.newestUpdateWallClockMillis = newestUpdateWallClockMillis;
124144
}
125145

126146
public ConnectorMvccSnapshot getConnectorSnapshot() {
@@ -175,6 +195,15 @@ public long getNewestUpdateMonotonicMarker() {
175195
return newestUpdateMonotonicMarker;
176196
}
177197

198+
/**
199+
* The table's newest-update WALL-CLOCK epoch-millis on the range-view path — used only by the SqlCache
200+
* eligibility quiet-window gate, distinct from the monotonic marker used for staleness. {@code 0} outside
201+
* the range-view path.
202+
*/
203+
public long getNewestUpdateWallClockMillis() {
204+
return newestUpdateWallClockMillis;
205+
}
206+
178207
/**
179208
* True when at least one listed partition failed to build into a {@link PartitionItem} (its
180209
* rendered name could not be parsed), i.e. the built item map is short of the listed partition

‎fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRelatedTableIf.java‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -116,6 +116,20 @@ MTMVSnapshotIf getTableSnapshot(MTMVRefreshContext context, Optional<MvccSnapsho
116116
*/
117117
long getNewestUpdateVersionOrTime();
118118

119+
/**
120+
* The table's newest data-update time as a genuine WALL-CLOCK epoch-millis value, used ONLY by the
121+
* SqlCache eligibility "quiet window" gate ({@code CacheAnalyzer}), never for staleness. It differs from
122+
* {@link #getNewestUpdateVersionOrTime()} only when that token is not epoch-millis (iceberg returns
123+
* microseconds): the default returns the token unchanged, which is correct for every table whose token is
124+
* already epoch-millis (olap visible version, hive last-DDL millis, paimon file-creation millis), while
125+
* the plugin range-view table (iceberg) overrides it to return the connector-normalized wall-clock millis.
126+
*
127+
* @return the newest-update wall-clock epoch millis; {@code 0} for something wrong.
128+
*/
129+
default long getNewestUpdateTimeMillisForCache() {
130+
return getNewestUpdateVersionOrTime();
131+
}
132+
119133
/**
120134
* Does the current type of table allow timed triggering
121135
*

‎fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -150,7 +150,13 @@ public class CacheTable implements Comparable<CacheTable> {
150150
public TableIf table;
151151
public long latestPartitionId;
152152
public long latestPartitionVersion;
153+
// A version/order key: olap partition visible-version time (millis) or an external table's data-version
154+
// token (hive/paimon: millis; iceberg: micros). Used as the BE PCache LastVersionTime and to sort/pick
155+
// latestTable. NOT a reliable wall clock for external tables, so it is NOT used by the quiet-window gate.
153156
public long latestPartitionTime;
157+
// A genuine wall-clock epoch-millis newest-update time (olap visible-version time / external table's
158+
// connector-normalized update millis). Used ONLY by the quiet-window eligibility gate.
159+
public long latestPartitionUpdateMillis;
154160
public long partitionNum;
155161
public long sumOfPartitionNum;
156162

@@ -159,6 +165,7 @@ public CacheTable() {
159165
latestPartitionId = 0;
160166
latestPartitionVersion = 0;
161167
latestPartitionTime = 0;
168+
latestPartitionUpdateMillis = 0;
162169
partitionNum = 0;
163170
sumOfPartitionNum = 0;
164171
}
@@ -249,20 +256,27 @@ private CacheMode innerCheckCacheModeForNereids(long now) {
249256
allViewExpandStmtListStr = StringUtils.join(allViewStmtSet, "|");
250257
}
251258

259+
// The quiet-window gate must subtract a genuine wall-clock time. latestPartitionTime is only a
260+
// version/order key (an external token that is micros for iceberg), so use latestPartitionUpdateMillis
261+
// (real epoch millis for every table type) and take the newest across all scanned tables — decoupled
262+
// from latestTable, which stays token-sorted for the BE PCache version key.
263+
long newestUpdateMillis = 0;
264+
for (CacheTable cacheTable : tblTimeList) {
265+
newestUpdateMillis = Math.max(newestUpdateMillis, cacheTable.latestPartitionUpdateMillis);
266+
}
267+
252268
if (now == 0) {
253269
now = nowtime();
254270

255271
// the cloud meta service maybe has different time with fe, so we should make sure
256-
// now >= latestPartitionTime, and let regression test become stable
257-
for (CacheTable cacheTable : tblTimeList) {
258-
now = Math.max(now, cacheTable.latestPartitionTime);
259-
}
272+
// now >= the newest wall-clock update time, and let regression test become stable
273+
now = Math.max(now, newestUpdateMillis);
260274
}
261275

262276
if (enableSqlCache()
263-
&& (now - latestTable.latestPartitionTime) >= Config.cache_last_version_interval_second * 1000L) {
277+
&& (now - newestUpdateMillis) >= Config.cache_last_version_interval_second * 1000L) {
264278
if (LOG.isDebugEnabled()) {
265-
LOG.debug("Query cache time :{},{},{}", now, latestTable.latestPartitionTime,
279+
LOG.debug("Query cache time :{},{},{}", now, newestUpdateMillis,
266280
Config.cache_last_version_interval_second * 1000);
267281
}
268282

@@ -462,6 +476,9 @@ private CacheTable buildCacheTableForOlapScanNode(OlapScanNode node) {
462476
cacheTable.latestPartitionTime = partition.getVisibleVersionTime();
463477
cacheTable.latestPartitionVersion = partition.getCachedVisibleVersion();
464478
}
479+
// For olap the version time IS a wall clock, so the gate value tracks the version key.
480+
cacheTable.latestPartitionUpdateMillis =
481+
Math.max(cacheTable.latestPartitionUpdateMillis, partition.getVisibleVersionTime());
465482
}
466483
return cacheTable;
467484
}
@@ -485,8 +502,14 @@ private CacheTable buildCacheTableForExternalScanNode(ScanNode node) {
485502
cacheTable.table = tableIf;
486503
cacheTable.partitionNum = node.getSelectedPartitionNum();
487504
// Connector-agnostic data-version token (hive: max transient_lastDdlTime; iceberg/paimon: monotonic
488-
// snapshot version). Gated to MTMVRelatedTableIf tables by isExternalCacheableScanNode above.
505+
// snapshot version). Gated to MTMVRelatedTableIf tables by isExternalCacheableScanNode above. This is
506+
// the BE PCache version key; keep it as the raw token (do NOT normalize — iceberg's micros token is the
507+
// full-precision staleness key).
489508
cacheTable.latestPartitionTime = ((MTMVRelatedTableIf) tableIf).getNewestUpdateVersionOrTime();
509+
// The quiet-window gate value: a genuine wall-clock epoch-millis (iceberg normalizes its micros to
510+
// millis here; hive/paimon already return millis). Kept separate from the token above so the gate never
511+
// subtracts a non-wall-clock value.
512+
cacheTable.latestPartitionUpdateMillis = ((MTMVRelatedTableIf) tableIf).getNewestUpdateTimeMillisForCache();
490513
DatabaseIf database = tableIf.getDatabase();
491514
CatalogIf catalog = database.getCatalog();
492515
ScanTable scanTable = new ScanTable(new FullTableName(

‎fe/fe-core/src/test/java/org/apache/doris/datasource/mvcc/PluginDrivenMvccExternalTableTest.java‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1153,6 +1153,27 @@ public void testRangeViewNewestUpdateTimeUsesMonotonicMarkerNotSnapshotId() {
11531153
"the range-view path must answer the dictionary with the monotonic newest-update-time");
11541154
}
11551155

1156+
@Test
1157+
public void testRangeViewCacheGateUsesWallClockMillisNotMarker() {
1158+
// The SqlCache quiet-window gate needs a genuine wall-clock millis, distinct from the monotonic marker
1159+
// (which iceberg reports in MICROSECONDS). getNewestUpdateTimeMillisForCache must return the view's
1160+
// wall-clock value; the version token (getNewestUpdateVersionOrTime) must still be the raw marker.
1161+
// MUTATION: returning the marker here (the old conflation) makes this red — that micros value dominates
1162+
// wall-clock now in CacheAnalyzer and is exactly why iceberg never cached.
1163+
long marker = 1_700_000_000_000_000L; // micros (as iceberg reports)
1164+
long wallClock = marker / 1000; // millis (connector-normalized)
1165+
ConnectorMvccPartitionView view = new ConnectorMvccPartitionView(
1166+
ConnectorMvccPartitionView.Style.RANGE, ConnectorMvccPartitionView.Freshness.SNAPSHOT_ID,
1167+
Arrays.asList(rangePart("p20240101", "2024-01-01", "2024-01-02", FRESH_555)),
1168+
marker, wallClock);
1169+
Fixture f = Fixture.rangeView(view);
1170+
1171+
Assertions.assertEquals(wallClock, f.table.getNewestUpdateTimeMillisForCache(),
1172+
"the cache gate must use the wall-clock millis, not the micros marker");
1173+
Assertions.assertEquals(marker, f.table.getNewestUpdateVersionOrTime(),
1174+
"the version token must stay the raw monotonic marker");
1175+
}
1176+
11561177
@Test
11571178
public void testRangeViewValidRelatedTableMirrorsStyle() {
11581179
// RANGE style => valid related table; UNPARTITIONED style (the connector's eligibility gate failed) =>

0 commit comments

Comments
 (0)