Repository navigation
Commit 0da2151
authored
feat: add early emit count metric to partial aggregates (#24996)
## Which issue does this PR close?
- Closes #24995.
## Why make this change?
When DataFusion is working on a partial `GROUP BY` and memory fills up,
it sends out the work completed so far and starts collecting again.
Today, `EXPLAIN ANALYZE` does not show that this happened. This makes it
hard to tell why the partial `GROUP BY` produced more rows than
expected.
## What changed?
This PR adds an `early_emit_count` number to `EXPLAIN ANALYZE`. It goes
up once each time we send out partial results because memory is full.
If one set of partial results is sent in several smaller pieces, the
number still goes up only once.
This change does not affect query results and does not add new logs.
## How was it tested?
Tests cover each way DataFusion runs a partial `GROUP BY`. They check
that:
- the number goes up once for each memory-related send
- splitting one send into smaller pieces does not increase it more than
once
- other ways of sending partial results do not increase it
## Are there any user-facing changes?
`EXPLAIN ANALYZE` can now show `early_emit_count` for partial `GROUP BY`
steps.1 parent 4d87baf commit 0da2151
4 files changed
Lines changed: 77 additions & 10 deletions
File tree
- datafusion
- physical-plan/src/aggregates
- sqllogictest/test_files
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
163 | 163 | | |
164 | 164 | | |
165 | 165 | | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
166 | 169 | | |
167 | 170 | | |
168 | 171 | | |
| |||
389 | 392 | | |
390 | 393 | | |
391 | 394 | | |
| 395 | + | |
| 396 | + | |
392 | 397 | | |
393 | 398 | | |
394 | 399 | | |
| |||
433 | 438 | | |
434 | 439 | | |
435 | 440 | | |
| 441 | + | |
436 | 442 | | |
437 | 443 | | |
438 | 444 | | |
| |||
476 | 482 | | |
477 | 483 | | |
478 | 484 | | |
| 485 | + | |
479 | 486 | | |
480 | 487 | | |
481 | 488 | | |
| |||
1129 | 1136 | | |
1130 | 1137 | | |
1131 | 1138 | | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
1132 | 1178 | | |
1133 | 1179 | | |
1134 | 1180 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4831 | 4831 | | |
4832 | 4832 | | |
4833 | 4833 | | |
| 4834 | + | |
| 4835 | + | |
| 4836 | + | |
| 4837 | + | |
| 4838 | + | |
| 4839 | + | |
| 4840 | + | |
| 4841 | + | |
| 4842 | + | |
| 4843 | + | |
4834 | 4844 | | |
4835 | 4845 | | |
4836 | 4846 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
91 | 94 | | |
92 | 95 | | |
93 | 96 | | |
| |||
188 | 191 | | |
189 | 192 | | |
190 | 193 | | |
| 194 | + | |
| 195 | + | |
191 | 196 | | |
192 | 197 | | |
193 | 198 | | |
| |||
210 | 215 | | |
211 | 216 | | |
212 | 217 | | |
| 218 | + | |
213 | 219 | | |
214 | 220 | | |
215 | 221 | | |
| |||
315 | 321 | | |
316 | 322 | | |
317 | 323 | | |
318 | | - | |
319 | | - | |
320 | | - | |
321 | | - | |
322 | | - | |
323 | | - | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
324 | 333 | | |
325 | 334 | | |
326 | 335 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
208 | 208 | | |
209 | 209 | | |
210 | 210 | | |
211 | | - | |
212 | | - | |
213 | | - | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
214 | 214 | | |
215 | 215 | | |
216 | 216 | | |
| |||
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
| 226 | + | |
| 227 | + | |
226 | 228 | | |
227 | 229 | | |
228 | 230 | | |
| |||
0 commit comments