Repository navigation
Commit 7a58afe
committed
feat(proto): plumb PhysicalProtoConverterExtension into try_encode_expr / try_decode_expr
Closes apache#22920.
apache#21807 introduced the `DynamicFilterPhysicalExpr` dedup pipeline so that
identical references on the wire reconstruct to one shared `Arc<Inner>`
on the decode side via `expression_id` cache keys. The pipeline works
end-to-end for plans built from upstream types, and apache#22011 hooked it
through Sort/Aggregate/HashJoin plan codecs.
However, the pipeline stops at the `PhysicalExtensionCodec` boundary.
Downstream users with custom file sources or plan extension types
encode/decode their nested `PhysicalExprNode` fields via
`try_encode_expr` / `try_decode_expr`, which did not receive the active
`PhysicalProtoConverterExtension`. The only entry points available
inside such a codec were the free `serialize_physical_expr` /
`parse_physical_expr` helpers, hardwired to
`DefaultPhysicalProtoConverter`. So any expression inside an extension
codec's serialized blob got `expr_id = None` on the wire, and the
deduplicating deserializer's cache never hit for refs that crossed an
extension boundary -- even when an outer `DeduplicatingProtoConverter`
was in effect on the rest of the plan. Two references to the same
`DynamicFilterPhysicalExpr` (one inside an extension expr, one outside)
reconstructed as distinct `Inner` allocations, so heap-max updates from
a TopK SortExec failed to propagate to a FileScan predicate carried by
a custom file source.
This matches the existing plan-level pattern (`try_encode` and
`try_decode` already take a `proto_converter` parameter, added when the
plan codec was wired into the converter pipeline). This change extends
the expr-level methods the same way:
fn try_decode_expr(
&self,
buf: &[u8],
inputs: &[Arc<dyn PhysicalExpr>],
proto_converter: &dyn PhysicalProtoConverterExtension, // new
) -> Result<Arc<dyn PhysicalExpr>>;
fn try_encode_expr(
&self,
node: &Arc<dyn PhysicalExpr>,
buf: &mut Vec<u8>,
proto_converter: &dyn PhysicalProtoConverterExtension, // new
) -> Result<()>;
The two internal call sites in `serialize_physical_expr_with_converter`
and `parse_physical_expr_with_converter` are updated to thread the
active converter through. Downstream codecs can now route nested
expression encode/decode through `proto_converter.physical_expr_to_proto`
/ `proto_converter.proto_to_physical_expr`, picking up dedup
automatically.
This is a breaking change for downstream codecs that override
`try_encode_expr` / `try_decode_expr` -- they will need to add the
new parameter (and may pass `_proto_converter` if they don't carry
nested expressions). Codecs that only override `try_encode` /
`try_decode` (plan-level) are unaffected. Wire format is unchanged
(`PhysicalExprNode.expr_id` is unchanged); the fix is purely on the
Rust trait API.
Test: a new `extension_codec_expr_participates_in_deduplication`
proto-roundtrip test constructs a `BinaryExpr` whose left operand is a
bare `DynamicFilterPhysicalExpr` and right operand is a custom
`WrapperExpr` whose extension codec embeds the same dynamic filter
inside its serialized blob. After a `DeduplicatingProtoConverter`
roundtrip, an `update()` applied to the bare-side decoded filter is
observed by `current()` on the wrapped-side decoded filter, proving
both refs back the same `Inner`. Without the converter-aware extension
codec, this test fails because the two refs end up with distinct
`Inner` allocations.1 parent b8998c7 commit 7a58afe
4 files changed
Lines changed: 262 additions & 3 deletions
File tree
- datafusion/proto
- src/physical_plan
- tests/cases
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
412 | 412 | | |
413 | 413 | | |
414 | 414 | | |
415 | | - | |
416 | | - | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
417 | 420 | | |
418 | 421 | | |
419 | 422 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3940 | 3940 | | |
3941 | 3941 | | |
3942 | 3942 | | |
| 3943 | + | |
| 3944 | + | |
| 3945 | + | |
| 3946 | + | |
| 3947 | + | |
| 3948 | + | |
| 3949 | + | |
| 3950 | + | |
| 3951 | + | |
3943 | 3952 | | |
3944 | 3953 | | |
3945 | 3954 | | |
3946 | 3955 | | |
| 3956 | + | |
3947 | 3957 | | |
3948 | 3958 | | |
3949 | 3959 | | |
3950 | 3960 | | |
| 3961 | + | |
| 3962 | + | |
| 3963 | + | |
| 3964 | + | |
| 3965 | + | |
| 3966 | + | |
| 3967 | + | |
3951 | 3968 | | |
3952 | 3969 | | |
3953 | 3970 | | |
3954 | 3971 | | |
| 3972 | + | |
3955 | 3973 | | |
3956 | 3974 | | |
3957 | 3975 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
365 | 365 | | |
366 | 366 | | |
367 | 367 | | |
368 | | - | |
| 368 | + | |
369 | 369 | | |
370 | 370 | | |
371 | 371 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1161 | 1161 | | |
1162 | 1162 | | |
1163 | 1163 | | |
| 1164 | + | |
1164 | 1165 | | |
1165 | 1166 | | |
1166 | 1167 | | |
| |||
1175 | 1176 | | |
1176 | 1177 | | |
1177 | 1178 | | |
| 1179 | + | |
1178 | 1180 | | |
1179 | 1181 | | |
1180 | 1182 | | |
| |||
4127 | 4129 | | |
4128 | 4130 | | |
4129 | 4131 | | |
| 4132 | + | |
| 4133 | + | |
| 4134 | + | |
| 4135 | + | |
| 4136 | + | |
| 4137 | + | |
| 4138 | + | |
| 4139 | + | |
| 4140 | + | |
| 4141 | + | |
| 4142 | + | |
| 4143 | + | |
| 4144 | + | |
| 4145 | + | |
| 4146 | + | |
| 4147 | + | |
| 4148 | + | |
| 4149 | + | |
| 4150 | + | |
| 4151 | + | |
| 4152 | + | |
| 4153 | + | |
| 4154 | + | |
| 4155 | + | |
| 4156 | + | |
| 4157 | + | |
| 4158 | + | |
| 4159 | + | |
| 4160 | + | |
| 4161 | + | |
| 4162 | + | |
| 4163 | + | |
| 4164 | + | |
| 4165 | + | |
| 4166 | + | |
| 4167 | + | |
| 4168 | + | |
| 4169 | + | |
| 4170 | + | |
| 4171 | + | |
| 4172 | + | |
| 4173 | + | |
| 4174 | + | |
| 4175 | + | |
| 4176 | + | |
| 4177 | + | |
| 4178 | + | |
| 4179 | + | |
| 4180 | + | |
| 4181 | + | |
| 4182 | + | |
| 4183 | + | |
| 4184 | + | |
| 4185 | + | |
| 4186 | + | |
| 4187 | + | |
| 4188 | + | |
| 4189 | + | |
| 4190 | + | |
| 4191 | + | |
| 4192 | + | |
| 4193 | + | |
| 4194 | + | |
| 4195 | + | |
| 4196 | + | |
| 4197 | + | |
| 4198 | + | |
| 4199 | + | |
| 4200 | + | |
| 4201 | + | |
| 4202 | + | |
| 4203 | + | |
| 4204 | + | |
| 4205 | + | |
| 4206 | + | |
| 4207 | + | |
| 4208 | + | |
| 4209 | + | |
| 4210 | + | |
| 4211 | + | |
| 4212 | + | |
| 4213 | + | |
| 4214 | + | |
| 4215 | + | |
| 4216 | + | |
| 4217 | + | |
| 4218 | + | |
| 4219 | + | |
| 4220 | + | |
| 4221 | + | |
| 4222 | + | |
| 4223 | + | |
| 4224 | + | |
| 4225 | + | |
| 4226 | + | |
| 4227 | + | |
| 4228 | + | |
| 4229 | + | |
| 4230 | + | |
| 4231 | + | |
| 4232 | + | |
| 4233 | + | |
| 4234 | + | |
| 4235 | + | |
| 4236 | + | |
| 4237 | + | |
| 4238 | + | |
| 4239 | + | |
| 4240 | + | |
| 4241 | + | |
| 4242 | + | |
| 4243 | + | |
| 4244 | + | |
| 4245 | + | |
| 4246 | + | |
| 4247 | + | |
| 4248 | + | |
| 4249 | + | |
| 4250 | + | |
| 4251 | + | |
| 4252 | + | |
| 4253 | + | |
| 4254 | + | |
| 4255 | + | |
| 4256 | + | |
| 4257 | + | |
| 4258 | + | |
| 4259 | + | |
| 4260 | + | |
| 4261 | + | |
| 4262 | + | |
| 4263 | + | |
| 4264 | + | |
| 4265 | + | |
| 4266 | + | |
| 4267 | + | |
| 4268 | + | |
| 4269 | + | |
| 4270 | + | |
| 4271 | + | |
| 4272 | + | |
| 4273 | + | |
| 4274 | + | |
| 4275 | + | |
| 4276 | + | |
| 4277 | + | |
| 4278 | + | |
| 4279 | + | |
| 4280 | + | |
| 4281 | + | |
| 4282 | + | |
| 4283 | + | |
| 4284 | + | |
| 4285 | + | |
| 4286 | + | |
| 4287 | + | |
| 4288 | + | |
| 4289 | + | |
| 4290 | + | |
| 4291 | + | |
| 4292 | + | |
| 4293 | + | |
| 4294 | + | |
| 4295 | + | |
| 4296 | + | |
| 4297 | + | |
| 4298 | + | |
| 4299 | + | |
| 4300 | + | |
| 4301 | + | |
| 4302 | + | |
| 4303 | + | |
| 4304 | + | |
| 4305 | + | |
| 4306 | + | |
| 4307 | + | |
| 4308 | + | |
| 4309 | + | |
| 4310 | + | |
| 4311 | + | |
| 4312 | + | |
| 4313 | + | |
| 4314 | + | |
| 4315 | + | |
| 4316 | + | |
| 4317 | + | |
| 4318 | + | |
| 4319 | + | |
| 4320 | + | |
| 4321 | + | |
| 4322 | + | |
| 4323 | + | |
| 4324 | + | |
| 4325 | + | |
| 4326 | + | |
| 4327 | + | |
| 4328 | + | |
| 4329 | + | |
| 4330 | + | |
| 4331 | + | |
| 4332 | + | |
| 4333 | + | |
| 4334 | + | |
| 4335 | + | |
| 4336 | + | |
| 4337 | + | |
| 4338 | + | |
| 4339 | + | |
| 4340 | + | |
| 4341 | + | |
| 4342 | + | |
| 4343 | + | |
| 4344 | + | |
| 4345 | + | |
| 4346 | + | |
| 4347 | + | |
| 4348 | + | |
| 4349 | + | |
| 4350 | + | |
| 4351 | + | |
| 4352 | + | |
| 4353 | + | |
| 4354 | + | |
| 4355 | + | |
| 4356 | + | |
| 4357 | + | |
| 4358 | + | |
| 4359 | + | |
| 4360 | + | |
| 4361 | + | |
| 4362 | + | |
| 4363 | + | |
| 4364 | + | |
| 4365 | + | |
| 4366 | + | |
| 4367 | + | |
0 commit comments