Repository navigation
Commit 0fcb9ab
fix: count each probe row once in HashJoinExec probe_hit_rate and avg_fanout (apache#25272)
## Which issue does this PR close?
- Closes apache#25077 .
## Rationale for this change
`EXPLAIN ANALYZE` reports `probe_hit_rate` and `avg_fanout` of
`HashJoinExec` too low whenever a probe batch produces more than
`batch_size` output rows, e.g. when the build side has many duplicate
keys. Both metrics are defined per probe row, but their counters were
updated once per output chunk.
## What changes are included in this PR?
- `probe_hit_rate` total: added only on the first chunk of each probe
batch (`state.offset == (0, None)`), the same first-chunk check that
`process_probe_batch` already uses for correlated null-aware `LeftMark`.
- `probe_hit_rate` part and `avg_fanout` total: a new
`ProcessProbeBatchState::matched_probe_idx` records the last probe index
counted so far. `count_new_matched_probe_rows` does not count a chunk's
first index again if it equals that value. Empty chunks leave it
unchanged.
- Existing state is not reused: `joined_probe_idx` is tracked after the
join filter, while these metrics are defined before it, and the lookup
`offset` can point at a probe row that has not been counted yet.
## What is the testing strategy for this PR?
Two new tests in `hash_join/exec.rs`, each run over
`hash_join_exec_configs` (batch sizes 8192/10/5/2/1, perfect hash join
on and off):
- `join_probe_metrics_count_each_probe_row_once`: duplicate build keys,
so one probe row's matches are split across chunks. Fails without this
change for batch sizes 5, 2 and 1.
- `join_probe_metrics_count_probe_row_starting_new_chunk`: unique build
keys, so chunks split between probe rows. It guards against
deduplicating by the lookup offset: comparing against `offset.0` (on
non-first chunks) instead makes this test fail for batch sizes 2 and 1,
while the first test still passes.
## Are there any user-facing changes?
`probe_hit_rate` and `avg_fanout` shown by `EXPLAIN ANALYZE` are now
correct when a probe batch is processed in several chunks. No public API
changes.1 parent a95806f commit 0fcb9ab
2 files changed
Lines changed: 175 additions & 8 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3145 | 3145 | | |
3146 | 3146 | | |
3147 | 3147 | | |
| 3148 | + | |
| 3149 | + | |
| 3150 | + | |
| 3151 | + | |
| 3152 | + | |
| 3153 | + | |
| 3154 | + | |
| 3155 | + | |
| 3156 | + | |
| 3157 | + | |
| 3158 | + | |
| 3159 | + | |
| 3160 | + | |
| 3161 | + | |
| 3162 | + | |
| 3163 | + | |
| 3164 | + | |
| 3165 | + | |
| 3166 | + | |
3148 | 3167 | | |
3149 | 3168 | | |
3150 | 3169 | | |
| |||
3191 | 3210 | | |
3192 | 3211 | | |
3193 | 3212 | | |
| 3213 | + | |
3194 | 3214 | | |
3195 | 3215 | | |
3196 | 3216 | | |
| |||
4197 | 4217 | | |
4198 | 4218 | | |
4199 | 4219 | | |
| 4220 | + | |
| 4221 | + | |
| 4222 | + | |
| 4223 | + | |
| 4224 | + | |
| 4225 | + | |
| 4226 | + | |
| 4227 | + | |
| 4228 | + | |
| 4229 | + | |
| 4230 | + | |
| 4231 | + | |
| 4232 | + | |
| 4233 | + | |
| 4234 | + | |
| 4235 | + | |
| 4236 | + | |
| 4237 | + | |
| 4238 | + | |
| 4239 | + | |
| 4240 | + | |
| 4241 | + | |
| 4242 | + | |
| 4243 | + | |
| 4244 | + | |
| 4245 | + | |
| 4246 | + | |
| 4247 | + | |
| 4248 | + | |
| 4249 | + | |
| 4250 | + | |
| 4251 | + | |
| 4252 | + | |
| 4253 | + | |
| 4254 | + | |
| 4255 | + | |
| 4256 | + | |
| 4257 | + | |
| 4258 | + | |
| 4259 | + | |
| 4260 | + | |
| 4261 | + | |
| 4262 | + | |
| 4263 | + | |
| 4264 | + | |
| 4265 | + | |
| 4266 | + | |
| 4267 | + | |
| 4268 | + | |
| 4269 | + | |
| 4270 | + | |
| 4271 | + | |
| 4272 | + | |
| 4273 | + | |
| 4274 | + | |
| 4275 | + | |
| 4276 | + | |
| 4277 | + | |
| 4278 | + | |
| 4279 | + | |
| 4280 | + | |
| 4281 | + | |
| 4282 | + | |
| 4283 | + | |
| 4284 | + | |
| 4285 | + | |
| 4286 | + | |
| 4287 | + | |
| 4288 | + | |
| 4289 | + | |
| 4290 | + | |
| 4291 | + | |
| 4292 | + | |
| 4293 | + | |
| 4294 | + | |
| 4295 | + | |
| 4296 | + | |
| 4297 | + | |
| 4298 | + | |
| 4299 | + | |
| 4300 | + | |
| 4301 | + | |
| 4302 | + | |
| 4303 | + | |
| 4304 | + | |
| 4305 | + | |
| 4306 | + | |
| 4307 | + | |
| 4308 | + | |
| 4309 | + | |
| 4310 | + | |
| 4311 | + | |
| 4312 | + | |
| 4313 | + | |
| 4314 | + | |
| 4315 | + | |
| 4316 | + | |
| 4317 | + | |
| 4318 | + | |
| 4319 | + | |
| 4320 | + | |
| 4321 | + | |
| 4322 | + | |
| 4323 | + | |
| 4324 | + | |
| 4325 | + | |
| 4326 | + | |
| 4327 | + | |
| 4328 | + | |
| 4329 | + | |
| 4330 | + | |
| 4331 | + | |
| 4332 | + | |
| 4333 | + | |
| 4334 | + | |
| 4335 | + | |
| 4336 | + | |
| 4337 | + | |
| 4338 | + | |
| 4339 | + | |
| 4340 | + | |
| 4341 | + | |
4200 | 4342 | | |
4201 | 4343 | | |
4202 | 4344 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
188 | 188 | | |
189 | 189 | | |
190 | 190 | | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
191 | 196 | | |
192 | 197 | | |
193 | 198 | | |
| |||
197 | 202 | | |
198 | 203 | | |
199 | 204 | | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
200 | 222 | | |
201 | 223 | | |
202 | 224 | | |
| |||
786 | 808 | | |
787 | 809 | | |
788 | 810 | | |
| 811 | + | |
789 | 812 | | |
790 | 813 | | |
791 | 814 | | |
| |||
803 | 826 | | |
804 | 827 | | |
805 | 828 | | |
806 | | - | |
807 | | - | |
808 | | - | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
809 | 836 | | |
810 | 837 | | |
811 | 838 | | |
| |||
890 | 917 | | |
891 | 918 | | |
892 | 919 | | |
893 | | - | |
| 920 | + | |
894 | 921 | | |
895 | 922 | | |
896 | 923 | | |
897 | | - | |
| 924 | + | |
898 | 925 | | |
899 | 926 | | |
900 | 927 | | |
901 | | - | |
902 | | - | |
903 | | - | |
| 928 | + | |
904 | 929 | | |
905 | 930 | | |
906 | 931 | | |
| |||
0 commit comments