Repository navigation
Commit 222bca5
[improvement](be) Share a file scan node's reader cache across its instances
### What problem does this PR solve?
Issue Number: None
Related PR: #66399
Problem Summary:
Readers of a file scan node keep what they parse once per node in a `ShardedKVCache`: iceberg
position and equality delete files, deletion vectors, and, for a fluss union read of a primary-key
table, the keys of each bucket's log tail. Each `FileScanLocalState` built a cache of its own, so the
cache was per pipeline instance. Splits are dealt to the instances with no regard for the delete file
or bucket they share, so every instance that got a split of the same bucket or delete file parsed it
again and kept its own copy.
For a fluss union read each such parse is a JNI read of the bucket's log tail. On a 16-bucket
primary-key table whose lake has 80 splits, spread over 9 instances, `COUNT(*)` read the tails 69
times where 16 would do (`FlussUnionTailCacheHitCount` = 11 of 80 splits) and kept 6.6M keys in the
JVM heap for 1.6M distinct ones.
The fix moves the cache to `FileScanOperatorX`, created in `prepare()`, so every instance of the node
on a backend reads through one cache. This is safe to share:
- every cached value is plain data (roaring bitmaps, delete-row vectors, key blocks with an immutable
hash index) and none refers to an instance's profile, IO context or runtime state; the readers that
fill them use such objects only inside the creating function;
- the cache locks per shard and was already shared by the scanners of one instance;
- nothing erases entries, the operator outlives every local state and scanner, and the fragment
releases operators under the query memory tracker.
The shard count is what the per-instance caches had between them: the fragment's instance count
times the per-instance scanner limit, capped by the remote scan thread pool. It is the fragment's
instances and not the operator's parallelism because a serial operator has one instance that runs
the scanners of all of them (`max_scanners_concurrency`).
Measured on a Release BE against a local fluss cluster: a 16-bucket primary-key table of 30M rows,
all of them in the lake, and 1.6M keys (100K per bucket) updated since, in the log tail. Before is
master; after has this commit and the background connection close of the first commit of this
series, which takes a fixed two seconds off each of these tail reads:
- Compacted lake (80 native splits), `COUNT(*)` under union read: 21.9 s -> 1.0 s. Splits that found
the tail keys already cached 11 -> 61 of 80; keys held in the JVM heap 6.6M -> 1.6M; time in tail
reads summed over the scanners 139.8 s -> 0.9 s.
- Uncompacted lake (32 JNI merge splits), `COUNT(*)`: 13.8 s -> 1.6 s; cached 0 -> 16 of 32; keys
held 3.2M -> 1.6M.
- Union read, union read disabled and the expected values agree on `COUNT(*)`, `SUM(id)` and
`SUM(price)` for both tables, before and after.
### Release note
Scans of iceberg tables with delete files, of tables with deletion vectors, and fluss union reads of
primary-key tables no longer parse the same delete file or log tail once per pipeline instance.
### Check List (For Author)
- Test: Unit Test, Manual test
- BE UT `FileScanOperatorFlussTest.EveryInstanceOfTheNodeReadsThroughTheNodesCache` (new)
- 168 related BE unit tests pass (`FileScanOperator*`, `FileScannerV2Test.*`,
`VfileScannerExceptionTest.*`, `ScannerContextTest.*`, `*FlussUnion*`, `*IcebergReader*`)
- Manual: the benchmark above
- Behavior changed: No
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 38b689a commit 222bca5
3 files changed
Lines changed: 96 additions & 16 deletions
File tree
- be
- src/exec/operator
- test/exec/operator
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
60 | 61 | | |
61 | 62 | | |
62 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
63 | 70 | | |
64 | 71 | | |
65 | 72 | | |
| |||
107 | 114 | | |
108 | 115 | | |
109 | 116 | | |
110 | | - | |
111 | | - | |
| 117 | + | |
112 | 118 | | |
113 | 119 | | |
114 | 120 | | |
| |||
198 | 204 | | |
199 | 205 | | |
200 | 206 | | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
205 | | - | |
206 | | - | |
| 207 | + | |
207 | 208 | | |
208 | 209 | | |
209 | 210 | | |
| |||
235 | 236 | | |
236 | 237 | | |
237 | 238 | | |
238 | | - | |
| 239 | + | |
239 | 240 | | |
240 | 241 | | |
241 | 242 | | |
242 | | - | |
| 243 | + | |
243 | 244 | | |
244 | 245 | | |
245 | 246 | | |
| |||
327 | 328 | | |
328 | 329 | | |
329 | 330 | | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
330 | 339 | | |
331 | 340 | | |
332 | 341 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
89 | | - | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
94 | | - | |
95 | 89 | | |
96 | 90 | | |
97 | 91 | | |
| |||
119 | 113 | | |
120 | 114 | | |
121 | 115 | | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
122 | 121 | | |
123 | 122 | | |
124 | 123 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
20 | 23 | | |
| 24 | + | |
| 25 | + | |
21 | 26 | | |
| 27 | + | |
22 | 28 | | |
| 29 | + | |
23 | 30 | | |
| 31 | + | |
24 | 32 | | |
| 33 | + | |
| 34 | + | |
25 | 35 | | |
26 | 36 | | |
27 | 37 | | |
| |||
115 | 125 | | |
116 | 126 | | |
117 | 127 | | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
118 | 190 | | |
0 commit comments