Skip to content

Native Delta scan: derive the task injection key from a stable scan identity #5943

Closed

Description

@dwsmith1983

What is the problem the feature request solves?

common.source in the Delta contrib's scan builders (CometDeltaNativeScan.scala) is derived from simpleStringWithNodeId(), which reads Spark's explain-local id map and can yield (unknown) during ordinary planning. Two independently converted scans of the same table with identical common fields but different partition selections can then receive the same injection key, and if both contribute to one collection scope the map merge keeps only one payload. Exchange boundaries isolate the common self-join shapes today, so this is a key-contract concern rather than an observed wrong answer.

Describe the potential solution

Use the original physical scan's SparkPlan.id in both builders, for example s"${scanExec.nodeName} (${scanExec.id})", and add a regression that clears the explain map, converts two scans with disjoint selected files, and asserts distinct keys, with an equivalent-scan control that keeps sameResult and the semantic hashes unchanged. The key stays excluded from semantic equality.

Additional context

Raised in the #5365 review (item 1). Filed as a follow-up so the contrib can land and the change gets its own review.

Activity

  1. dwsmith1983 commented on Oct 9, 2026

    @dwsmith1983
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:scanParquet scan / data readingenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions