Repository navigation
Commit b764605
committed
fix(querier): run queries with collect_statistics off
Unblocks RFC 0021 phase 2a. DataFusion 55 substitutes columns that
per-file statistics prove constant; the all-NULL case folds to a NULL
literal, which collapses `body == "…"` to a bare constant and leaves
no column reference for a pruning predicate — so RFC 0044's
body-equality queries scanned every row group instead of skipping
them. Pillar #1, caught by RFC0044.7/.8 and RFC0007.1.
Upstream: apache/datafusion#24769, fix proposed in #24770, bisected
to DataFusion #22969 (which made collect_statistics the session
default). Turning the option off restores the pruning exactly.
Both production session sites now go through one `exec::session()` so
the setting cannot drift between the query and drift paths. The
override is not purely defensive: collection also costs a footer read
per file at plan time — a cost a many-file log store pays on every
query — and Ourios derives its pruning from the RFC 0009 manifest plus
partition-directory windowing, not from DataFusion's collected file
statistics.
RFC 0021 §3.2a records the decision and names the revisit trigger; the
same three pruning tests gate it in both directions.
Verified: workspace 1448/1448, clippy pedantic zero, fmt, mdbook.
Signed-off-by: Jens Holdgaard Pedersen <[email protected]>1 parent 0ec0a87 commit b764605
4 files changed
Lines changed: 46 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
212 | 212 | | |
213 | 213 | | |
214 | 214 | | |
215 | | - | |
| 215 | + | |
216 | 216 | | |
217 | 217 | | |
218 | 218 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
39 | 61 | | |
40 | 62 | | |
41 | 63 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
| 76 | + | |
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
| |||
655 | 655 | | |
656 | 656 | | |
657 | 657 | | |
658 | | - | |
| 658 | + | |
659 | 659 | | |
660 | 660 | | |
661 | 661 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| |||
167 | 167 | | |
168 | 168 | | |
169 | 169 | | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
170 | 190 | | |
171 | 191 | | |
172 | 192 | | |
| |||
0 commit comments