Repository navigation
Commit 8687c2a
authored
feat(frontend): accept $variable references in loop blocks (apache#8640)
### What changes were proposed in this PR?
Inside a control block (`Loop Start ... Loop End`), any primitive
property of an operator (string, integer, number, boolean) can now hold
a loop-variable reference such as `$i`. The frontend sends the literal
string `"$i"` unchanged. The backend marks it as a placeholder when it
compiles the plan and binds it when the iteration's state message
reaches the worker (apache#8639). Operators outside every block behave as
before.
```
Loop Start (i = 0) -> Limit(limit: "$i") -> Loop End
^ panel offers $i; canvas valid; sent as "$i"
rename i to j in the Loop Start -> Limit warns "$i is not a variable of an enclosing block; ..."; Run stays enabled
delete a link upstream -> Limit outside the block -> "must be integer"
disable the Loop End -> the same: the backend compiles and runs only enabled operators
```
| Piece | What it does |
|---|---|
| Block membership (`enclosingLoopStarts`, `loop-block.util.ts`) |
Worked out in the frontend from the graph as it stands: every enabled
operator, valid or not, and the links between them, which is the plan
the backend compiles and runs (`WorkflowGraph.getSubDAG`). It is a port
of the backend's `LogicalPlan.enclosingLoopStarts` (apache#8639), with the
same treatment of a Loop Start that no Loop End closes, nested and
sequential blocks, branches, and outermost-first order.
`LogicalPlanSpec` and `loop-block.util.spec.ts` both run one shared
fixture (`common/workflow-compiler/src/test/resources/loop-blocks.json`:
named cases plus random graphs, with the backend's answers), so the two
cannot drift apart, and a change to the fixture runs the frontend stack
too (`labeler.yml`). An unfinished operator elsewhere on the canvas no
longer keeps an operator in a block it has left. |
| Loop variables (`extractLoopVariables`, `loopVariablesInScope`) |
Reads the variables from each enclosing Loop Start's `initialization`
code, following Python's module-level binding: assignments at any depth
outside a `def`/`class` body, for-loop targets, one-line compound bodies
such as `try: k = 1`, and names an import binds (`from math import pi`,
`import numpy as np`, relative modules, parenthesized lists). |
| Property panel | Inside a block, each plain primitive field becomes a
text box that offers `$name` for every variable in scope. So does a
field whose accepted values depend on another field of its row (value
rules, as on the sklearn trainers' hyperparameter values): its rules
still judge every value that is not a reference, and the text box
suggests the values they accept after the variables. Enums, arrays and
custom widgets keep their own controls, and so does a property the
backend marks `"noLoopVariable": true` (apache#8750: settings the plan is
built from, such as Projection's columns), together with everything
nested under it. A `$name` that no enclosing block declares gets a
warning under the field; the operator stays valid, and the run itself
reports the name if no Loop Start sets it. |
| Canvas validation | Inside a block, the schema `type` error is waived
exactly where a reference sits in a primitive-typed property.
`loopScopeChangeStream` emits on every edit that can change an
operator's scope: an operator or link added or deleted, an operator
disabled or enabled, or a Loop Start's properties changed. On each one,
validation re-validates every operator that holds a reference, so the
Run button follows edits two hops away. |
| Open panel | Rebuilt on `loopScopeChangeStream` when the operator's
scope changed. |
| Shared editing (`updateYTypeFromObject`) | A string now replaces a
number or boolean instead of throwing, which storing `"$i"` over a
number needs. |
Panel behavior for a field inside a block:
| Field type | Typed text | Stored value |
|---|---|---|
| integer | `7` | `7` (number) |
| integer / number / boolean | `$i` | `"$i"` |
| number | `0.5`, typed one key at a time | `0.5`; partial text such as
`7.` stays text until complete |
| boolean | `true` / `FALSE` | `true` / `false`; other text is flagged |
| string | `hello` / `$i` | `"hello"` / `"$i"` |
| value rules (e.g. `C`, which must be > 0) | `$i` / `1.0` / `-1` |
`"$i"` / `"1.0"` / `"-1"`, flagged by the rules |
| any | `$foo` (undeclared) | `"$foo"`, with a warning; the operator
stays valid |
**Related.** apache#8639 (on `main`) binds the references and computes the
blocks. apache#8750 (on `main`) adds the `noLoopVariable` marks.
**Screenshots.** The Limit outside the loop keeps the number field it
has on `main`. The Limit inside the loop gets a text box that suggests
the loop's variables and flags a name the loop does not declare. The
third shot predates the change of that flag to a warning, which leaves
Run enabled.
| Before: outside a loop (same as `main`) | After: inside a loop |
After: an undeclared name |
|---|---|---|
| 
| 
| 
|
The workflow in the shots: `Text Input -> Loop Start (i = 0) -> Limit
($i) -> Loop End`, plus a second Limit fed straight from Text Input,
outside the loop.

### Any related issues, documentation, discussions?
Closes apache#8636
### How was this PR tested?
New and updated Vitest specs, run from `frontend/`:
| Spec | Covers |
|---|---|
| `LogicalPlanSpec` (backend) | `enclosingLoopStarts` on every case and
random graph of the shared fixture, outermost first. The fixture's Loop
Start and Loop End are read by the backend's JSON type names, so a
renamed type fails here. The walk-based check on named cases and 400
random DAGs stays. |
| `loop-block.util.spec.ts` (new) | The frontend's port on every case
and random graph of the same fixture, and on every operator type it
names. A graph with a cycle, which the backend rejects but the canvas
allows, ends and counts as outside every block. A link to an operator it
was not given is ignored. |
| `loop-variable.util.spec.ts` | Assignment forms; if/else, try/except,
while and for bodies; for targets; one-line compound bodies; import
forms (`from ... import`, aliases, relative modules, parenthesized
lists, `import a.b`, several on one line; `*` binds nothing); def/class
bodies and with/except-as names excluded; Python keywords never taken
for names. `loopVariablesInScope`: outermost first without duplicates,
undefined outside every block, a disabled operator and its links left
out. `loopScopeChangeStream` emits on adds, deletes, disables, enables
and Loop Start property changes, and on nothing else. |
| `loop-variable-field.util.spec.ts` | Complete vs partial numerals
(`7.`, `-0.`, `1e-` kept as text); the parser sets the control without
redrawing the box; the string `""` default; the boolean validator
rejects `maybe`. On a field with value rules: a reference passes the
rules, every other value is still judged by them, the re-validation hook
survives, and `props` stays the one object `templateOptions` names. An
unknown name gives a warning, not a validation error.
`inheritNoLoopVariable` carries a mark to every schema under it, through
each keyword that holds a schema and through `$ref`s, without marking a
definition an unmarked property shares. |
| `loop-variable-input.component.spec.ts` (new) | Rendered through a
real formly form: typing `1.05`, `-0.5` and `7.5` one key at a time
stores the right number and keeps the box as typed; `$i` and integers;
formly attributes reach the input. An unknown name shows the warning.
With value rules: the suggestions add the accepted values of the row's
parameter (empty box, typed prefix in either case, `$` prefix), and a
changed parameter re-judges the value. |
| `validation-workflow.service.spec.ts` | The waiver applies only to
primitive types. An unknown name leaves the operator valid. Canvas state
follows each edit at once: a block opened or closed, and an operator
moved out of its block and back while another operator is invalid. Edits
two hops away are covered too: an upstream link deleted, a block closed
by a link between two other operators, an operator inserted into the
body, and a Loop End disabled and enabled again. Each of these four
fails if validation stops listening to `loopScopeChangeStream`. |
| `operator-property-edit-frame.component.spec.ts` | Primitive fields
converted inside a block; file picker, enum and array keep their
controls; the form rebuilds when a link, a disabled or enabled operator,
or the Loop Start's variables change its scope, and only then. The open
operator leaves its block and comes back while another operator on the
canvas is invalid. A value-rules field takes `$i` inside a block and
keeps its own control outside; with a schema enum or a custom widget it
keeps its control inside too. A `noLoopVariable` property, and a field
nested under one, keeps its control inside a block. Typing `$` into a
number field inside a block keeps the form as it is. |
| `shared-editing.interface.spec.ts` | A string replaces a number in a
Y.Map instead of throwing. |
Each new behavior spec was checked red against the pre-fix code before
the final run.
```
yarn ng test --watch=false --include=<the 7 loop and shared-editing specs> # 488 passed, 1 skipped (existing it.skip)
yarn ng test --watch=false # 240 files, 6815 passed, 1 skipped (existing it.skip)
yarn exec tsc --noEmit -p src/tsconfig.app.json (and src/tsconfig.spec.json) # clean
yarn exec ngc -p src/tsconfig.app.json --noEmit # AOT + strictTemplates clean
yarn prettier-eslint --list-different <changed files> && yarn eslint <changed files> # clean
sbt "WorkflowCompiler/testOnly *LogicalPlanSpec" # 27 passed
sbt scalafmtCheckAll "WorkflowCompiler/scalafixAll --check" # clean
```
`yarn ng build` did not complete locally, because the worktree's
`node_modules` is a symlink; CI's frontend build passes on macOS, Ubuntu
and Windows.
An earlier revision was checked by hand on a local deployment with apache#8639
and apache#8642 (the test branch in apache#8645): the loop workflow runs, and the
panel behaves as in the screenshots. The changes since (membership from
the whole graph, imports, the warning, `noLoopVariable`) are covered by
the specs above. Manual steps for the reviewer (Chrome, desktop): link
`Loop Start -> Limit -> Loop End` and set the Loop Start initialization
to `i = 0`. Open `Limit`: its integer field is a text box, typing `$`
offers `$i`, and choosing it keeps the operator valid. Typing `$foo`
shows a warning, "$foo is not a variable of an enclosing block; ...",
and Run stays enabled. Delete a link so `Limit` leaves the block, or
disable the Loop End: the field is a number box again and `$i` is a type
error.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5.1, Claude Opus 5.5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)1 parent f674649 commit 8687c2a
24 files changed
Lines changed: 3835 additions & 28 deletions
File tree
- .github
- common/workflow-compiler/src
- main/scala/org/apache/texera/common/compiler/model
- test
- resources
- scala/org/apache/texera/common/compiler
- frontend/src/app
- common/formly
- workspace
- component
- loop-variable-input
- property-editor/operator-property-edit-frame
- service
- operator-metadata
- validation
- workflow-graph/model
- types
- util
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
21 | 26 | | |
22 | 27 | | |
23 | 28 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
93 | 93 | | |
94 | 94 | | |
95 | 95 | | |
96 | | - | |
97 | | - | |
98 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
| |||
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
34 | 34 | | |
35 | 35 | | |
36 | 36 | | |
| 37 | + | |
37 | 38 | | |
38 | 39 | | |
39 | 40 | | |
| |||
282 | 283 | | |
283 | 284 | | |
284 | 285 | | |
285 | | - | |
| 286 | + | |
286 | 287 | | |
287 | 288 | | |
288 | 289 | | |
| |||
291 | 292 | | |
292 | 293 | | |
293 | 294 | | |
294 | | - | |
| 295 | + | |
295 | 296 | | |
296 | 297 | | |
297 | 298 | | |
298 | 299 | | |
299 | 300 | | |
300 | 301 | | |
301 | 302 | | |
302 | | - | |
| 303 | + | |
303 | 304 | | |
304 | | - | |
305 | | - | |
| 305 | + | |
| 306 | + | |
306 | 307 | | |
307 | 308 | | |
308 | 309 | | |
309 | 310 | | |
310 | 311 | | |
311 | 312 | | |
312 | | - | |
313 | | - | |
314 | | - | |
315 | | - | |
316 | | - | |
| 313 | + | |
| 314 | + | |
317 | 315 | | |
318 | | - | |
| 316 | + | |
319 | 317 | | |
320 | 318 | | |
321 | 319 | | |
| |||
425 | 423 | | |
426 | 424 | | |
427 | 425 | | |
428 | | - | |
| 426 | + | |
429 | 427 | | |
430 | 428 | | |
431 | 429 | | |
| |||
440 | 438 | | |
441 | 439 | | |
442 | 440 | | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
443 | 503 | | |
444 | 504 | | |
445 | 505 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
| 110 | + | |
110 | 111 | | |
111 | 112 | | |
112 | 113 | | |
| |||
325 | 326 | | |
326 | 327 | | |
327 | 328 | | |
| 329 | + | |
328 | 330 | | |
329 | 331 | | |
330 | 332 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
| 43 | + | |
42 | 44 | | |
43 | 45 | | |
44 | 46 | | |
| |||
89 | 91 | | |
90 | 92 | | |
91 | 93 | | |
| 94 | + | |
92 | 95 | | |
93 | 96 | | |
94 | 97 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
0 commit comments