Repository navigation
Commit 3fa608d
authored
fix(amber): reject $i in settings the plan is built from (apache#8750)
### What changes were proposed in this PR?
Root cause: inside a loop block, a loop-variable reference (apache#8639) is
written into the operator's setting when the iteration's state arrives.
The compiler, however, builds the operator's output schema and its
partitioning earlier, when the workflow compiles, from the reference's
placeholder. A reference in a property the plan is built from therefore
never reaches the schema or the partitioning.
```
Before: Loop Start (a = "x") -> Projection (alias $a) -> Loop End
output column named "$a", null in every row
After: compile error: Projection cannot refer to loop variables (/attributes/0/alias -> $a):
its output schema or partitioning is built from that property when the workflow is
compiled, before the loop runs
```
| Piece | What it does |
|---|---|
| `@NoLoopVariable` | Marks a descriptor property that can't hold a loop
variable because the plan is built from it: an output column's name or
type, or what an input is partitioned on. `LogicalOp`'s documentation
states the rule for operator authors. |
| `StateReferencing.noLoopVariableProperties` /
`noLoopVariableReferences` | Name the marked properties by their JSON
names, and find the references under them at any depth. Both go through
Jackson's bean introspection, so renamed JSON names such as `"result
attribute"` resolve. |
| `OperatorMetadataGenerator` | Puts `"noLoopVariable": true` on each
marked property in the operator's JSON schema, from the same helper, so
the panel (apache#8640) can use the same list. |
| `WorkflowCompiler` | Inside a loop block, a reference under a marked
property is a compile error, reported before the physical plan is built
so it is the operator's first error. Outside every block nothing
changes: a `$a` alias is still the literal column name. |
| `ExecutionReconfigurationService` | Modifying an operator that refers
to loop variables while the workflow runs is refused: `Limit refers to
loop variables, so it can't be changed while the workflow runs.` Its
rebuilt executor would never have the variables written into it. |
Marked properties:
| Operator | Properties |
|---|---|
| Projection | drop option, attributes |
| Dictionary Matcher, Unnest String | result attribute |
| Aggregate | aggregations, group-by keys |
| File Scan From Input | attribute name, file-name column option (the
attribute name is declared by the text-source trait, so File Scan and
Text Input carry the mark too; as sources they are never inside a loop
block) |
| Type Casting | cast units |
| Hash Join, Interval Join | the join attributes |
| Sort Partitions | sort attribute, domain min and max |
### Any related issues, documentation, discussions?
Closes apache#8749
Related to apache#8635 and apache#8639.
### How was this PR tested?
New tests, each run red against the pre-change code first:
| Spec | Covers |
|---|---|
| `StateReferenceModuleSpec` | `noLoopVariableProperties` names each
marked property by its JSON name, a trait's included;
`noLoopVariableReferences` picks the references under a marked property
whether it is renamed, escaped, nested or declared by a trait, and
nothing from a descriptor with no marked property or no reference. |
| `OperatorMetadataGeneratorSpec` | The schema marks exactly the
annotated properties (renamed ones under their JSON names) and no
others, and for every operator the marked set equals
`noLoopVariableProperties`. |
| `WorkflowCompilerSpec` | Inside a loop block, a reference in each
marked property is a compile error, and it comes before the operator's
other errors. A reference in a property the executor reads, on an
operator whose plan is built from other properties, still binds. Outside
every block, a `$a` Projection alias stays the literal column name. |
| `ExecutionReconfigurationServiceSpec` | An edit that makes an operator
inside a loop block refer to a loop variable is refused, with the
message above. An operator inside a block that refers to no loop
variable, and a `$x` literal outside every block, can still be modified.
|
```
sbt "WorkflowCore/testOnly org.apache.texera.amber.core.state.* org.apache.texera.amber.core.executor.*" # 75 passed
sbt "WorkflowCompiler/test" # 83 passed
sbt "WorkflowOperator/testOnly org.apache.texera.amber.operator.metadata.* ...projection.* ...aggregate.* \
...sortPartitions.* ...dictionary.* ...unneststring.* ...typecasting.* ...hashJoin.* ...intervalJoin.*" # 467 passed
sbt "WorkflowExecutionService/testOnly org.apache.texera.web.service.ExecutionReconfigurationServiceSpec" # 18 passed
sbt "scalafmtCheckAll" "<WorkflowCore|WorkflowOperator|WorkflowCompiler|WorkflowExecutionService>/scalafixAll --check" # clean
```
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)1 parent 05d48eb commit 3fa608d
20 files changed
Lines changed: 537 additions & 23 deletions
File tree
- amber/src
- main/scala/org/apache/texera/web/service
- test/scala/org/apache/texera/web/service
- common
- workflow-compiler/src
- main/scala/org/apache/texera/common/compiler
- test/scala/org/apache/texera/common/compiler
- workflow-core/src
- main/scala/org/apache/texera/amber/core/state
- test/scala/org/apache/texera/amber/core/state
- workflow-operator/src
- main/scala/org/apache/texera/amber/operator
- aggregate
- dictionary
- hashJoin
- intervalJoin
- metadata
- projection
- sortPartitions
- source/scan
- file
- text
- typecasting
- unneststring
- test/scala/org/apache/texera/amber/operator/metadata
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
27 | 29 | | |
28 | 30 | | |
29 | 31 | | |
| 32 | + | |
| 33 | + | |
30 | 34 | | |
31 | 35 | | |
32 | 36 | | |
| |||
60 | 64 | | |
61 | 65 | | |
62 | 66 | | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
69 | 73 | | |
70 | | - | |
71 | | - | |
72 | | - | |
73 | | - | |
74 | | - | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
75 | 81 | | |
76 | | - | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
77 | 91 | | |
78 | 92 | | |
79 | 93 | | |
80 | 94 | | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
81 | 108 | | |
82 | 109 | | |
83 | 110 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| 40 | + | |
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
| |||
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
51 | 57 | | |
| 58 | + | |
52 | 59 | | |
53 | 60 | | |
54 | 61 | | |
55 | | - | |
| 62 | + | |
56 | 63 | | |
57 | 64 | | |
58 | 65 | | |
| |||
470 | 477 | | |
471 | 478 | | |
472 | 479 | | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
473 | 573 | | |
474 | 574 | | |
475 | 575 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
33 | | - | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
34 | 37 | | |
35 | 38 | | |
36 | 39 | | |
| |||
82 | 85 | | |
83 | 86 | | |
84 | 87 | | |
85 | | - | |
86 | | - | |
87 | | - | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
88 | 93 | | |
89 | 94 | | |
90 | 95 | | |
91 | 96 | | |
92 | 97 | | |
93 | 98 | | |
94 | | - | |
| 99 | + | |
| 100 | + | |
95 | 101 | | |
96 | 102 | | |
97 | 103 | | |
| |||
101 | 107 | | |
102 | 108 | | |
103 | 109 | | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
104 | 129 | | |
105 | 130 | | |
106 | 131 | | |
| |||
0 commit comments