Repository navigation
Commit e1d7fe2
authored
fix: charge accumulators' construction-time state to the memory pool in ungrouped aggregation (apache#24692)
## Which issue does this PR close?
- Closes apache#24691.
## Rationale for this change
An ungrouped aggregation can hold arbitrarily large accumulator state
that the memory pool never sees. `AggregateStream` grows its reservation
only by per-batch `size()` deltas (`size_post.saturating_sub(size_pre)`
in `aggregate_batch`), so an accumulator that allocates its retained
state in its constructor and never resizes it is charged nothing for the
stream's lifetime. The grouped path already handles this:
`GroupsAccumulatorAdapter` charges `state.size()` when it creates each
accumulator. Embedding DataFusion 54.1.0 in a Spark accelerator we
measured Spark's runtime-filter `bloom_filter_agg` (an 8 MiB bit array
zero-filled at construction, one per concurrent task) running entirely
unaccounted, so a fair-spill pool could neither fail admission nor
pressure other consumers to spill.
## What changes are included in this PR?
One charge in `AggregateStream::new`: after registering the reservation,
`try_grow` the sum of the accumulators' initial `size()`. Per-batch
deltas compose on top of it, so there is no double counting.
`test_oom`'s ungrouped arm is updated for the earlier failure point:
with its 1-byte limit the median accumulator's initial 48 bytes now fail
admission at `execute_typed`, so that arm asserts the construction-time
`ResourcesExhausted` instead of collecting first; the grouped arms are
unchanged.
## Are these changes tested?
The updated `test_oom` is the regression pin: on `main` without the fix
the ungrouped stream constructs successfully and the arm fails, with the
fix it passes. The full `datafusion-physical-plan` lib suite passes
(1784 tests).
## Are there any user-facing changes?
Yes, behavioral: an ungrouped aggregation whose accumulators'
construction-time state exceeds the memory limit now fails with
`ResourcesExhausted` at `execute` instead of silently running past the
configured limit, and memory-limited setups that relied on that
overshoot may need limits raised to what the query actually uses. No API
changes.1 parent 0f55902 commit e1d7fe2
2 files changed
Lines changed: 148 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
324 | 324 | | |
325 | 325 | | |
326 | 326 | | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
327 | 332 | | |
328 | 333 | | |
329 | 334 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3235 | 3235 | | |
3236 | 3236 | | |
3237 | 3237 | | |
3238 | | - | |
| 3238 | + | |
3239 | 3239 | | |
3240 | 3240 | | |
3241 | 3241 | | |
| |||
4011 | 4011 | | |
4012 | 4012 | | |
4013 | 4013 | | |
4014 | | - | |
| 4014 | + | |
4015 | 4015 | | |
4016 | 4016 | | |
4017 | | - | |
| 4017 | + | |
4018 | 4018 | | |
4019 | | - | |
| 4019 | + | |
| 4020 | + | |
| 4021 | + | |
| 4022 | + | |
| 4023 | + | |
| 4024 | + | |
| 4025 | + | |
| 4026 | + | |
4020 | 4027 | | |
4021 | 4028 | | |
| 4029 | + | |
4022 | 4030 | | |
| 4031 | + | |
4023 | 4032 | | |
4024 | 4033 | | |
| 4034 | + | |
4025 | 4035 | | |
| 4036 | + | |
4026 | 4037 | | |
4027 | 4038 | | |
4028 | | - | |
| 4039 | + | |
4029 | 4040 | | |
4030 | 4041 | | |
4031 | 4042 | | |
| |||
4041 | 4052 | | |
4042 | 4053 | | |
4043 | 4054 | | |
| 4055 | + | |
| 4056 | + | |
| 4057 | + | |
| 4058 | + | |
| 4059 | + | |
| 4060 | + | |
| 4061 | + | |
| 4062 | + | |
| 4063 | + | |
| 4064 | + | |
| 4065 | + | |
| 4066 | + | |
| 4067 | + | |
| 4068 | + | |
| 4069 | + | |
| 4070 | + | |
| 4071 | + | |
| 4072 | + | |
| 4073 | + | |
| 4074 | + | |
| 4075 | + | |
| 4076 | + | |
| 4077 | + | |
| 4078 | + | |
| 4079 | + | |
| 4080 | + | |
| 4081 | + | |
| 4082 | + | |
| 4083 | + | |
| 4084 | + | |
| 4085 | + | |
| 4086 | + | |
| 4087 | + | |
| 4088 | + | |
| 4089 | + | |
| 4090 | + | |
| 4091 | + | |
| 4092 | + | |
| 4093 | + | |
| 4094 | + | |
| 4095 | + | |
| 4096 | + | |
| 4097 | + | |
| 4098 | + | |
| 4099 | + | |
| 4100 | + | |
| 4101 | + | |
| 4102 | + | |
| 4103 | + | |
| 4104 | + | |
| 4105 | + | |
| 4106 | + | |
| 4107 | + | |
| 4108 | + | |
| 4109 | + | |
| 4110 | + | |
| 4111 | + | |
| 4112 | + | |
| 4113 | + | |
| 4114 | + | |
| 4115 | + | |
| 4116 | + | |
| 4117 | + | |
| 4118 | + | |
| 4119 | + | |
| 4120 | + | |
| 4121 | + | |
| 4122 | + | |
| 4123 | + | |
| 4124 | + | |
| 4125 | + | |
| 4126 | + | |
| 4127 | + | |
| 4128 | + | |
| 4129 | + | |
| 4130 | + | |
| 4131 | + | |
| 4132 | + | |
| 4133 | + | |
| 4134 | + | |
| 4135 | + | |
| 4136 | + | |
| 4137 | + | |
| 4138 | + | |
| 4139 | + | |
| 4140 | + | |
| 4141 | + | |
| 4142 | + | |
| 4143 | + | |
| 4144 | + | |
| 4145 | + | |
| 4146 | + | |
| 4147 | + | |
| 4148 | + | |
| 4149 | + | |
| 4150 | + | |
| 4151 | + | |
| 4152 | + | |
| 4153 | + | |
| 4154 | + | |
| 4155 | + | |
| 4156 | + | |
| 4157 | + | |
| 4158 | + | |
| 4159 | + | |
| 4160 | + | |
| 4161 | + | |
| 4162 | + | |
| 4163 | + | |
| 4164 | + | |
| 4165 | + | |
| 4166 | + | |
| 4167 | + | |
| 4168 | + | |
| 4169 | + | |
| 4170 | + | |
| 4171 | + | |
| 4172 | + | |
| 4173 | + | |
| 4174 | + | |
| 4175 | + | |
| 4176 | + | |
| 4177 | + | |
| 4178 | + | |
| 4179 | + | |
| 4180 | + | |
| 4181 | + | |
4044 | 4182 | | |
4045 | 4183 | | |
4046 | 4184 | | |
| |||
0 commit comments