Repository navigation
Commit e3be4ad
[perf](catalog) wire cross-query derived partition-view cache A into the paimon connector (S4)
Layer the generic ConnectorPartitionViewCache<V> (S2 framework) above paimon's raw
catalogOps.listPartitions round-trip so a repeated query on a partitioned paimon table
skips the derived-view BUILD (display-name rendering + null-sentinel normalization,
PaimonConnectorMetadata#collectPartitions), not just avoids re-deriving from an
already-cached remote result -- there was no such cache before. ONE typed field (unlike
iceberg's two): paimon does not override getMvccPartitionView (the generic MTMV model
falls back to its default listPartitions/LIST/timestamp path for paimon), so
listPartitions is the only enumeration hook to wrap.
- Key = (db, table, snapshotId, schemaId). snapshotId reads the SAME per-catalog
PaimonLatestSnapshotCache that beginQuerySnapshot pins queries to -- collectPartitions'
remote call is base-identifier-only (it does not apply the handle's pinned scanOptions,
unlike the scan path) so it always reflects the CURRENT catalog state; keying off
"current" (not whatever pin happens to be on the handle) keeps the cache key an honest
descriptor of what was actually computed. schemaId is pinned -1 ("unversioned" for that
axis): PaimonTableHandle carries no schemaId field, applySnapshot never threads one onto
it, and beginQuerySnapshot's own ConnectorMvccSnapshot leaves it at the builder default
(-1) for the common latest-pin path -- and collectPartitions' output does not depend on
schema version in practice (paimon partition columns are immutable post-creation).
- listPartitions: bypass cache when filter.isPresent() (mirrors the iceberg pattern; not
currently load-bearing since paimon ignores the filter value, but keeps the two paths
independent) and when the handle is unpartitioned (preserves the pre-existing "no seam
call for unpartitioned tables" contract).
- Paimon has NO iceberg.rest.session=user-style per-user cache-disabling convention (a
paimon catalog authenticates at catalog-creation time -- Kerberos UGI / HMS principal --
not per-query session identity), so the cache is constructed unconditionally on
PaimonConnector (never null there); PaimonConnectorMetadata's convenience/test ctors
still pass null (no cross-query layer) for the ~15 existing direct-construction tests.
- Invalidation routed from invalidateTable/invalidateDb/invalidateAll to the new cache
alongside the existing latestSnapshotCache/schemaAtMemo.
Tests: new PaimonConnectorMetadataPartitionViewCacheTest (7) proves same-key-hit
enumerates once, distinct-latest-snapshot re-enumerates, invalidateTable/All
re-enumerate, filter bypass, null-cache re-enumeration, and the unpartitioned
fast path touches no remote seam; PaimonConnectorCacheTest (+2) proves the cache is
always built (no session=user gate) and the 3-hook invalidation routing on the real
field. Full module suite 370/370 green (1 pre-existing skip, PaimonLiveConnectivityTest).
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_014ZngdXGBAXNDRe7jA42mAS1 parent 96d5611 commit e3be4ad
4 files changed
Lines changed: 431 additions & 2 deletions
File tree
- fe/fe-connector/fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| 27 | + | |
26 | 28 | | |
27 | 29 | | |
28 | 30 | | |
| |||
50 | 52 | | |
51 | 53 | | |
52 | 54 | | |
| 55 | + | |
53 | 56 | | |
54 | 57 | | |
55 | 58 | | |
| |||
126 | 129 | | |
127 | 130 | | |
128 | 131 | | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
129 | 145 | | |
130 | 146 | | |
131 | 147 | | |
| |||
137 | 153 | | |
138 | 154 | | |
139 | 155 | | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
140 | 159 | | |
141 | 160 | | |
142 | 161 | | |
| |||
225 | 244 | | |
226 | 245 | | |
227 | 246 | | |
228 | | - | |
| 247 | + | |
229 | 248 | | |
230 | 249 | | |
231 | 250 | | |
| |||
239 | 258 | | |
240 | 259 | | |
241 | 260 | | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
242 | 264 | | |
243 | 265 | | |
244 | 266 | | |
| |||
255 | 277 | | |
256 | 278 | | |
257 | 279 | | |
| 280 | + | |
258 | 281 | | |
259 | 282 | | |
260 | 283 | | |
261 | 284 | | |
262 | 285 | | |
263 | 286 | | |
| 287 | + | |
264 | 288 | | |
265 | 289 | | |
266 | 290 | | |
| |||
316 | 340 | | |
317 | 341 | | |
318 | 342 | | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
319 | 348 | | |
320 | 349 | | |
321 | 350 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
| 36 | + | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| |||
95 | 97 | | |
96 | 98 | | |
97 | 99 | | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
98 | 109 | | |
99 | 110 | | |
100 | 111 | | |
| |||
105 | 116 | | |
106 | 117 | | |
107 | 118 | | |
| 119 | + | |
108 | 120 | | |
109 | 121 | | |
110 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
111 | 136 | | |
112 | 137 | | |
113 | 138 | | |
114 | 139 | | |
115 | 140 | | |
116 | 141 | | |
| 142 | + | |
117 | 143 | | |
118 | 144 | | |
119 | 145 | | |
| |||
955 | 981 | | |
956 | 982 | | |
957 | 983 | | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
958 | 994 | | |
959 | 995 | | |
960 | 996 | | |
961 | 997 | | |
962 | | - | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
963 | 1035 | | |
964 | 1036 | | |
965 | 1037 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
20 | 24 | | |
21 | 25 | | |
22 | 26 | | |
23 | 27 | | |
24 | 28 | | |
| 29 | + | |
25 | 30 | | |
26 | 31 | | |
| 32 | + | |
27 | 33 | | |
28 | 34 | | |
29 | 35 | | |
| |||
83 | 89 | | |
84 | 90 | | |
85 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
86 | 152 | | |
0 commit comments