You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
PlanDataInjector does N x M canInject calls per operator tree #4530
PlanDataInjector.injectPlanData walks every operator in the tree against every registered injector:
for (injector <- injectors if injector.canInject(op)) { ... }
That is N operators x M injectors canInject calls, even though most operators in any tree are non-scan and match no injector.
Proposed fix
Add def opStructCase: Operator.OpStructCase to the PlanDataInjector trait and build a Map[Operator.OpStructCase, PlanDataInjector]. Look up by op.getOpStructCase (O(1)) and then run a single canInject confirm (which may still inspect detail fields). Non-scan operators skip the iteration entirely. Pure performance change -- no behavior difference.
Relationship to the Delta integration
This is an independent micro-optimization, not Delta-specific. It is being extracted out of the in-progress Delta Lake contrib integration branch (which registers a third injector) purely to keep that PR focused; there is no functional dependency. A PR will follow shortly.
Activity
perf: O(1) PlanDataInjector lookup by op kind (apache#4535)
perf: O(1) PlanDataInjector lookup by op kind (apache#4535)
perf: O(1) PlanDataInjector lookup by op kind (apache#4535)