You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Native Delta scan: derive the task injection key from a stable scan identity #5943
common.source in the Delta contrib's scan builders (CometDeltaNativeScan.scala) is derived from simpleStringWithNodeId(), which reads Spark's explain-local id map and can yield (unknown) during ordinary planning. Two independently converted scans of the same table with identical common fields but different partition selections can then receive the same injection key, and if both contribute to one collection scope the map merge keeps only one payload. Exchange boundaries isolate the common self-join shapes today, so this is a key-contract concern rather than an observed wrong answer.
Describe the potential solution
Use the original physical scan's SparkPlan.id in both builders, for example s"${scanExec.nodeName} (${scanExec.id})", and add a regression that clears the explain map, converts two scans with disjoint selected files, and asserts distinct keys, with an equivalent-scan control that keeps sameResult and the semantic hashes unchanged. The key stays excluded from semantic equality.
Additional context
Raised in the #5365 review (item 1). Filed as a follow-up so the contrib can land and the change gets its own review.
Activity
dwsmith1983 commentedon Oct 9, 2026
take