Repository navigation
Commit a13e269
committed
refactor: make file-statistics cache keys schema-aware
File statistics are computed against a specific `file_schema`, but the
file-statistics cache was keyed only by table and path. Reading the same path
under a different schema could reuse statistics whose `column_statistics` no
longer line up, panicking during statistics projection. #22950 worked around
this by bypassing the cache entirely for anonymous explicit-schema reads, at
the cost of losing cache reuse for them.
Introduce a `SchemaFingerprint` (per-column name, type and nullability, derived
from `file_schema`) and a `FileStatisticsCacheKey { table, path, schema }`, and
key the file-statistics cache on it. Different schemas now get distinct entries
(no stale cross-schema reuse) while a repeated read of the same schema reuses
its entry, so the #22950 bypass is removed and anonymous explicit-schema reads
cache safely again.
- The fingerprint excludes field/schema metadata (cannot affect statistics) and
partition columns (their statistics are computed separately).
- Table-drop invalidation is unchanged: drop_table_entries matches on
CacheKey::table_ref(), which still returns the table, so all schema variants
for a table are removed together.
- The list-files cache continues to key on TableScopedPath.
Closes #23072.
Signed-off-by: Jiawei Zhao <[email protected]>1 parent ff677c4 commit a13e269
4 files changed
Lines changed: 164 additions & 26 deletions
File tree
- datafusion
- catalog-listing/src
- core/tests/parquet
- execution/src/cache
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
268 | 268 | | |
269 | 269 | | |
270 | 270 | | |
271 | | - | |
272 | | - | |
273 | | - | |
274 | | - | |
275 | | - | |
276 | | - | |
277 | | - | |
278 | | - | |
279 | | - | |
280 | | - | |
281 | | - | |
282 | | - | |
283 | | - | |
284 | | - | |
285 | | - | |
286 | 271 | | |
287 | 272 | | |
288 | 273 | | |
| |||
990 | 975 | | |
991 | 976 | | |
992 | 977 | | |
993 | | - | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
994 | 981 | | |
995 | | - | |
| 982 | + | |
996 | 983 | | |
997 | 984 | | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
998 | 989 | | |
999 | 990 | | |
1000 | 991 | | |
1001 | | - | |
1002 | | - | |
1003 | | - | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
1004 | 996 | | |
1005 | 997 | | |
1006 | 998 | | |
| |||
1017 | 1009 | | |
1018 | 1010 | | |
1019 | 1011 | | |
1020 | | - | |
| 1012 | + | |
1021 | 1013 | | |
1022 | | - | |
| 1014 | + | |
1023 | 1015 | | |
1024 | 1016 | | |
1025 | 1017 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
258 | | - | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
259 | 281 | | |
260 | 282 | | |
261 | 283 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
| |||
54 | 56 | | |
55 | 57 | | |
56 | 58 | | |
57 | | - | |
| 59 | + | |
58 | 60 | | |
59 | 61 | | |
60 | 62 | | |
| |||
305 | 307 | | |
306 | 308 | | |
307 | 309 | | |
308 | | - | |
| 310 | + | |
309 | 311 | | |
310 | 312 | | |
311 | 313 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
165 | 166 | | |
166 | 167 | | |
167 | 168 | | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
0 commit comments