You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Codegen dispatcher: guard the closure-serialize step so a non-serializable tree falls back cleanly instead of throwing at plan time #5573
Every other failure mode in that method degrades cleanly — the disabled-config check, CometBatchKernelCodegen.canHandle, the literal-payload conversion, the data-arg conversion and the return-type conversion all call withFallbackReason and return None, which the caller treats as a clean Spark fallback. The serialize call is the one step that can throw instead, and a NotSerializableException there escapes during planning, which is a much worse failure than falling back.
Today's callers are all curated expression types where this is very unlikely to bite. It matters because of where the sweep wants to go next: routing an unrecognized StaticInvoke or Invoke through the dispatcher as a catch-all hands it arbitrary trees, including ones holding references the closure serializer will refuse. So this should land before that change.
The fix is small — wrap the serialize in a try / catch, call withFallbackReason with the exception message and return None, matching the shape of the checks around it. A unit test can force it with an expression capturing a deliberately non-serializable object.
Activity