You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2850519
Browse filesBrowse the repository at this point in the historyBrowse files
[fix](catalog) cache: give the SqlCache quiet-window gate a real wall clock so iceberg tables can cache
An iceberg table (and any query joining one) was never eligible for SqlCache. This
is an availability bug, NOT a stale-read bug: staleness is guarded independently by
a version-token equality check (SqlCacheContext stores getNewestUpdateVersionOrTime,
NereidsSqlCacheManager compares it), which this change does not touch.
Root cause: CacheAnalyzer.CacheTable.latestPartitionTime is overloaded as both the
BE PCache version key AND the wall-clock input to the "quiet for N seconds" gate
(now - latestPartitionTime >= cache_last_version_interval_second). For an external
table it holds getNewestUpdateVersionOrTime(); hive/paimon return epoch millis so
the gate works by luck, but iceberg returns last_updated_at in MICROSECONDS
(~1.7e15). Math.max(now, micros) clamps now up to the micros value, so the delta is
always 0 and never clears the window -> iceberg never caches, and its huge value
also poisons any multi-table query it joins.
Fix (connector supplies a genuine wall-clock millis, kept separate from the token):
- ConnectorMvccPartitionView carries a new newestUpdateWallClockMillis alongside the
monotonic marker; iceberg fills it from last_updated_at / 1000 (the connector owns
its unit), while the marker stays micros for the version-token path (master parity).
- PluginDrivenMvccSnapshot mirrors the field; MTMVRelatedTableIf gains a default
getNewestUpdateTimeMillisForCache() returning getNewestUpdateVersionOrTime() (correct
for olap/hive/paimon whose token is already millis), which PluginDrivenMvccExternalTable
overrides so the range-view (iceberg) branch returns the wall-clock millis.
- CacheAnalyzer gains CacheTable.latestPartitionUpdateMillis (olap: visibleVersionTime;
external: the new accessor). latestPartitionTime stays the raw token (BE PCache key,
full-precision staleness), and the quiet-window gate now subtracts
max(latestPartitionUpdateMillis), decoupled from the token-sorted latestTable.
Only iceberg's gate behavior changes (hive/paimon/olap keep the same value). The
SPI field / accessor are backward compatible (overloaded ctors default to 0).
Tests:
- IcebergPartitionUtilsTest: the view's wall clock == micros marker / 1000.
- PluginDrivenMvccExternalTableTest.testRangeViewCacheGateUsesWallClockMillisNotMarker:
getNewestUpdateTimeMillisForCache returns the wall clock; the token stays the marker.
- PluginTableCacheAnalyzerTest.testGateValueSourcedFromWallClockAccessor: the gate value
comes from the wall-clock accessor while latestPartitionTime stays the raw token.
- e2e test_iceberg_sqlcache.groovy: a quiet iceberg table becomes cacheable, and a
write invalidates it (no stale results).
67 unit tests green across fe-connector-api / iceberg / hive / fe-core; builds +
checkstyle pass. The probe rename that precedes this landed in 8043536a812.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01QhjyyZrkCZWJkwtoezc8Vh
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccPartitionView.java
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergPartitionUtils.java
+6-1Lines changed: 6 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -602,8 +602,13 @@ static ConnectorMvccPartitionView buildMvccPartitionView(Table table, long pinne
602
602
// partitions too). Byte-parity with master getNewestUpdateVersionOrTime: max(...).orElse(0).
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergPartitionUtilsTest.java
+6Lines changed: 6 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -822,6 +822,12 @@ public void buildMvccPartitionViewEnumeratesRangePartitions() {
822
822
// committed table has a positive value. MUTATION: mapping lastUpdateTime->0 (or orElse over no rows) -> red.
0 commit comments