Skip to content

Commit d7ce3a6

Browse files
authored
feat(amber): bind $i loop-variable references in operators (apache#8639)
### What changes were proposed in this PR? Inside a loop block (LoopStart ... LoopEnd), a property of a JVM operator may now hold `$i`, a reference to the loop variable `i`. This works for string and numeric/boolean properties alike; the frontend (apache#8640) sends the literal `"$i"`, and the operator's own code does not change. The reference is handled in three places: - **Parse.** When `"$i"` cannot be converted to a numeric or boolean property's type, a Jackson problem handler (`StateReferenceModule`) puts in a typed placeholder (0 / 0.0 / false) and records the property's JSON pointer in a hidden `stateReferences` sidecar on `LogicalOp`. Strings are left alone. - **Compile.** The compiler works out which operators sit inside a loop block (`LogicalPlan.enclosingLoopStarts`: each operator's Loop Starts, from folds over the plan's topological order). Inside a block it also records every whole-string `"$i"` literal. Outside every block, a typed reference is a compile error. - **Worker.** The operator's own executor is built and opened as usual, with the placeholders in its setting: the descriptor its constructor parses. The worker registers every state message on the executor right before `processState` (`OperatorExecutor.registerState` and `state`, the JVM counterpart of Python's `self.state` in apache#8642). Until the operator first sees data, each referenced variable that a message carries is written into that setting, in place. Right before the first tuple, or the end of input, the worker checks that every reference has a value, and the setting is not written after that. Copies of one loop's state that disagree are an error; an inner loop's value shadows an outer one's (the worker passes each message's nesting depth). **Which settings can hold `$i`** (inside a loop block) | Setting | Holds `$i`? | What the user sees | |---|---|---| | Text read while the operator processes rows (a Filter's value or column, ...) | Yes | The iteration's value, written into the setting before the operator's first row | | Number, integer or decimal (Limit's `limit`, Interval Join's constant, ...) | Yes | The iteration's value, converted to the setting's type. Before the run, the compiler sees the placeholder `0` / `0.0` | | True/false | Yes | Same as a number, with the placeholder `false` | | A setting the compiled plan is built from: an output column's name or type, or what an input is partitioned on (Projection's columns and Drop Option, Unnest String's and Dictionary Matcher's result column, Aggregate, join keys, Type Casting, Sort Partitions) | No | With apache#8750, a compile error: `Projection cannot refer to loop variables (/isDrop -> $drop): its output schema or partitioning is built from that property when the workflow is compiled, before the loop runs`. With this PR alone, a wrong result or the operator's own error (Projection's missing-column message) | | Any setting of an operator that runs generated or user code: Python UDFs (code editor included), sklearn, Python charts, Sort | No | Compile error: `Sort cannot refer to loop variables yet (/attributes/0/attribute -> $col): its code is generated before the loop runs` | | A dropdown of fixed choices (an enum, e.g. a Filter's condition) | No; the panel (apache#8640) doesn't offer `$i` there | If it is sent anyway (e.g. through `SyncExecutionResource`), the workflow fails to parse with Jackson's `not one of the values accepted for Enum class` | **Other cases** (replaces the current Case / Behaviour table) | Case | Behaviour | |---|---| | Outside every loop block | A text `"$AAPL"` stays a literal, as on main. A number or true/false `"$n"` is a compile error: `Limit refers to loop variables (/limit -> $n), but is not inside a loop block` | | An input the operator reads first is fed, even by a single link, from outside one of the operator's loop blocks: from outside the loop, from an outer loop's body, or from a sibling block | Compile error: `Interval Join refers to loop variables (/constant -> $i), but its input 'left table' is fed from outside the loop block, so tuples from there can arrive before the loop state`. An input read only after one that carries the state (a join's probe side) may come from anywhere | | A name no enclosing Loop Start assigns (the panel flags it; `SyncExecutionResource` doesn't) | Fails at the operator's first row, or at its finish if it gets no rows: `property /limit refers to loop variable n, but no state message carried it` | | A value that doesn't fit the setting | Fails at the same point: `property /limit refers to loop variable n, but its value abc does not fit it: ...` | | Nested loops that name the same variable | The inner loop's value wins | | Copies of one loop's state that reach an operator disagree (e.g. a UDF changed `i` on one branch, and the branches meet) | Fails: `loop variable i got two different values in one iteration, 1 and 2: a loop's variables must not change inside its body`. On a single path there is only one copy, so the operators after the UDF get its changed value | | Editing the operator while the workflow runs | Refused, with apache#8750 | Which operators are inside a block is decided on the plan the compiler receives. That plan has no disabled operators, and for run-to-operator it keeps only the target's upstream. The panel decides on the whole canvas. **The operator's lifecycle is unchanged.** The value is set in the operator's setting when the state arrives, so the operator runs exactly as it does outside a loop: ``` worker starts state messages arrive first tuple executor built and opened, registered on the executor (state), and every reference has a value, or the operator placeholders in its setting each variable written into the setting, fails naming the missing ones; in place (a later message wins) the setting is not written after this ``` `open()` runs at worker start, and `produceStateOnStart` and `processState` run on the operator itself, with what they return used. Three executors used to read their setting before a state can arrive, and now read it at first use: Reservoir Sampling in its constructor, and Split and Dictionary Matcher in `open()`. Every other JVM executor already reads its setting while it processes data. ### Any related issues, documentation, discussions? Closes apache#8635 Lands together with apache#8750, which makes `$i` in a setting the compiled plan is built from a compile error. Frontend: apache#8640 (merges after this PR). Python operators: apache#8642 (independent). User-facing overview: apache#8643 ### How was this PR tested? | Suite | Result | |---|---| | WorkflowCore: `org.apache.texera.amber.core.state.*` and `core.executor.*`, incl. `StateReferenceModuleSpec`, `StateReferenceBindingSpec`, `CoreExecutorReflectionSpec` | 71/71 | | WorkflowCompiler full `test`, incl. `LogicalPlanSpec` (loop-block membership and enclosing Loop Starts, checked against a port of the panel's walk on every case and 400 random plans) and the loop-variable cases of `WorkflowCompilerSpec` (lenient and strict) | 80/80 | | WorkflowOperator: Reservoir Sampling, Split, Dictionary Matcher and metadata specs | 260/260 | | workflow-compiling-service `test` | 55/55 | | amber: `Test/compile`, `DataProcessorSpec`, `DPThreadSpec`, the worker's promise handlers incl. `EndChannelHandlerSpec`, `ComputingUnitMasterSpec` | 116/116 | | Lint | `scalafixAll --check` and `scalafmtCheckAll` on all five touched sbt projects | ``` sbt "project WorkflowCore" "testOnly *StateReferenceModuleSpec *StateReferenceBindingSpec *CoreExecutorReflectionSpec" sbt "project WorkflowCompiler" test ``` What the tests pin: - **End to end in unit form.** The frontend's `{"limit":"$n","operatorType":"Limit"}` is compiled inside `TextInput -> LoopStart -> Limit -> LoopEnd` and built through `ExecFactory`. After the state `{n: 2}`, exactly 2 of 5 tuples pass. - **Parse handovers.** Typed placeholders get the right pointers whether the object is parsed directly, with the type id first, in the middle or last, or nested in another document. - **Compile rules.** Outside every block, Filter `"$AAPL"` stays a literal and Limit `"$n"` is a compile error. Inside a block, a Scala-built Filter `"$i"` gets its sidecar, while Sort `"$col"` is a compile error, and so is an operator with a reference whose read-first input has a link from outside one of its loop blocks (outside the loop, an outer loop's body, a sibling block, or one outside link next to an inside one). - **Worker.** The executor is built and opened at worker start. `produceStateOnStart` and `processState` run on the operator itself, and what `processState` returns is emitted. Each state message writes the variables into the setting in place, into every property that names them, top-level and nested pointers alike, keeping the object the executor holds; an outer `{i: 1}` followed by an inner `{i: 5}` leaves 5. A tuple before any state fails, naming only the missing variables, and the setting is not written after the first tuple; an operator with no rows fails the same way at its finish. Copies of one loop's state with different values fail, equal ones pass, and a deeper loop's value replaces an outer one's in either arrival order. Values are coerced to the property's type, and a null, a list or a map for a string property is refused. - **Executors.** Reservoir Sampling, Split and Dictionary Matcher use the value written after construction and `open()`. - **Mappers.** Every Dropwizard application that parses workflows registers the module through `JSONUtils.registerWorkflowModules`; `WorkflowCompilingServiceRunSpec` and `ComputingUnitMasterSpec` parse a typed `"$n"` through each bootstrap mapper. `LoopIntegrationSpec` gains three `@IntegrationTest` runs, which CI's `amber-integration` job runs: - a number setting: `Limit($n)` with n = 1, 2, 3 gives 1 + 2 + 3 = 6 rows; - nested loops that both name `i`: `Filter(line = $i)` in the inner body passes all 2 x 2 rows only with the inner `i`; - a text setting on two branches meeting at a Union before `Filter(line = $i)`: 2 copies x 3 iterations = 6 rows (a constant or a `$i` that never rebinds gives 2). **Run by hand.** On a local deployment of this PR together with apache#8642 and apache#8640 (the test branch in apache#8645), a workflow `Text Input (1..6) -> Loop Start (i = 0, output table) -> Limit ($i) -> Loop End (i += 1 while i < len(table))` runs to completion. Each iteration binds `$i` to its own `i`, so Loop End collects 0 + 1 + 2 + 3 + 4 + 5 = 15 rows: `1`, then `1, 2`, then `1, 2, 3`, and so on. The Limit fed straight from Text Input, outside the loop, keeps its constant 3. | The Limit's property inside the loop | Loop End's result after the run | |---|---| | ![Limit inside the loop holding $i](https://raw.githubusercontent.com/aglinxinyuan/texera/390db38157491c992db75f25dc703bad05b4ab65/panel-after-inside-loop.png) | ![Loop End result: 15 rows, first page 1, 1, 2, 1, 2](https://raw.githubusercontent.com/aglinxinyuan/texera/ca12ba98464825cb6d16ba1ba90316fac37ca6ca/loop-end-result-15-rows.png) | ![The completed run: canvas, Loop End's update and condition, and its 15-row result](https://raw.githubusercontent.com/aglinxinyuan/texera/ca12ba98464825cb6d16ba1ba90316fac37ca6ca/loop-run-binds-i-per-iteration.png) ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Fable 5.1, Claude Opus 5.5) 🤖 Generated with [Claude Code](https://claude.com/claude-code)
1 parent b7993fb commit d7ce3a6

31 files changed

Lines changed: 2379 additions & 38 deletions

File tree

‎amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/DataProcessor.scala‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,9 @@ class DataProcessor(
7171

7272
@transient var executor: OperatorExecutor = _
7373

74+
/** The executor whose loop-variable references `bindStateReferences` has bound. */
75+
@transient private var executorWithBoundReferences: OperatorExecutor = _
76+
7477
def initTimerService(adaptiveBatchingMonitor: WorkerTimerService): Unit = {
7578
this.adaptiveBatchingMonitor = adaptiveBatchingMonitor
7679
}
@@ -97,6 +100,16 @@ class DataProcessor(
97100
def collectStatistics(): WorkerStatistics =
98101
statisticsManager.getStatistics(executor)
99102

103+
/**
104+
* `OperatorExecutor.bindStateReferences`, before the executor first sees data or finishes; after
105+
* that a reference comparison per tuple. An executor installed in its place binds its own.
106+
*/
107+
def bindStateReferences(): Unit =
108+
if (executorWithBoundReferences ne executor) {
109+
executor.bindStateReferences()
110+
executorWithBoundReferences = executor
111+
}
112+
100113
/**
101114
* process currentInputTuple through executor logic.
102115
* this function is only called by the DP thread.
@@ -105,6 +118,8 @@ class DataProcessor(
105118
try {
106119
val portIdentity: PortIdentity =
107120
this.inputGateway.getChannel(inputManager.currentChannelId).getPortId
121+
// The executor sees data: its setting must have its loop variables by now.
122+
bindStateReferences()
108123
outputManager.outputIterator.setTupleOutput(
109124
executor.processTupleMultiPort(
110125
tuple,
@@ -128,6 +143,8 @@ class DataProcessor(
128143
loopStartId: String
129144
): Unit = {
130145
try {
146+
// Before processState, which then sees the loop variables written into the setting.
147+
executor.registerState(state, loopCounter)
131148
val outputState = executor.processState(state, port)
132149
if (outputState.isDefined) {
133150
// Carry the incoming loop envelope through unchanged: loop operators

‎amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers/EndChannelHandler.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,8 @@ trait EndChannelHandler {
4141
dp.inputManager.getPort(portId).completed = true
4242
dp.inputManager.initBatch(channelId, Array.empty)
4343
try {
44+
// The executor finishes: a setting that refers to loop variables must have them by now.
45+
dp.bindStateReferences()
4446
val outputState = dp.executor.produceStateOnFinish(portId.id)
4547
if (outputState.isDefined) {
4648
// Operator-ORIGINATED boundary state, so no LoopStart stamp

‎amber/src/main/scala/org/apache/texera/web/ComputingUnitMaster.scala‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@
1919

2020
package org.apache.texera.web
2121

22-
import com.fasterxml.jackson.module.scala.DefaultScalaModule
2322
import com.typesafe.scalalogging.LazyLogging
2423
import io.dropwizard.Configuration
2524
import io.dropwizard.configuration.{EnvironmentVariableSubstitutor, SubstitutingSourceProvider}
@@ -40,7 +39,7 @@ import org.apache.texera.amber.engine.common.client.AmberClient
4039
import org.apache.texera.amber.engine.common.storage.SequentialRecordStorage
4140
import org.apache.texera.amber.engine.common.{AmberRuntime, Utils}
4241
import org.apache.texera.amber.util.JSONUtils.objectMapper
43-
import org.apache.texera.amber.util.ObjectMapperUtils
42+
import org.apache.texera.amber.util.{JSONUtils, ObjectMapperUtils}
4443
import org.apache.commons.jcs3.access.exception.InvalidArgumentException
4544
import org.apache.texera.auth.SessionUser
4645
import org.apache.texera.dao.SqlServer
@@ -134,8 +133,8 @@ class ComputingUnitMaster extends io.dropwizard.Application[Configuration] with
134133
classOf[PveWebsocketResource]
135134
)
136135
)
137-
// register scala module to dropwizard default object mapper
138-
bootstrap.getObjectMapper.registerModule(DefaultScalaModule)
136+
// register the modules that parse workflows (SyncExecutionResource) to dropwizard's mapper
137+
JSONUtils.registerWorkflowModules(bootstrap.getObjectMapper)
139138
}
140139

141140
override def run(configuration: Configuration, environment: Environment): Unit = {

‎amber/src/test/integration/org/apache/texera/amber/engine/e2e/LoopIntegrationSpec.scala‎

Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,12 +41,19 @@ import org.apache.texera.amber.engine.e2e.TestUtils.{
4141
workflowContext
4242
}
4343
import org.apache.texera.amber.operator.LogicalOp
44+
import org.apache.texera.amber.operator.filter.{
45+
ComparisonType,
46+
FilterPredicate,
47+
SpecializedFilterOpDesc
48+
}
4449
import org.apache.texera.amber.operator.limit.LimitOpDesc
4550
import org.apache.texera.amber.operator.loop.{LoopEndOpDesc, LoopStartOpDesc}
4651
import org.apache.texera.amber.operator.sleep.SleepOpDesc
4752
import org.apache.texera.amber.operator.source.scan.text.TextInputSourceOpDesc
4853
import org.apache.texera.amber.operator.udf.python.PythonUDFOpDescV2
54+
import org.apache.texera.amber.operator.union.UnionOpDesc
4955
import org.apache.texera.amber.tags.IntegrationTest
56+
import org.apache.texera.amber.util.JSONUtils.objectMapper
5057
import org.apache.texera.common.compiler.model.LogicalLink
5158
import org.scalatest.flatspec.AnyFlatSpecLike
5259
import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach, Outcome, Retries}
@@ -188,6 +195,17 @@ class LoopIntegrationSpec
188195
op
189196
}
190197

198+
/** A Limit whose Int `limit` is `"$name"`, parsed as the editor's JSON is, into a placeholder. */
199+
private def limitReferringTo(name: String): LogicalOp =
200+
objectMapper.readValue(s"""{"operatorType":"Limit","limit":"$$$name"}""", classOf[LogicalOp])
201+
202+
/** `attribute = value`; a `$name` value refers to a loop variable (bound at the worker). */
203+
private def filterEquals(attribute: String, value: String): SpecializedFilterOpDesc = {
204+
val op = new SpecializedFilterOpDesc()
205+
op.predicates = List(new FilterPredicate(attribute, ComparisonType.EQUAL_TO, value))
206+
op
207+
}
208+
191209
// NOTE: a JavaUDF (runtime-compiled Java) case cannot run in this suite.
192210
// The integration tests run forkless under sbt (fork := false), where
193211
// `javax.tools` sees only the sbt launcher on `java.class.path` -- the
@@ -317,6 +335,91 @@ class LoopIntegrationSpec
317335
)
318336
}
319337

338+
it should "bind a $n loop-variable reference in a JVM operator's number setting on every iteration" in {
339+
// TextInput(3 rows) -> LoopStart(n = 1 / table) -> Limit($n)
340+
// -> LoopEnd(n += 1 / n <= len(table)).
341+
// Iteration n passes n of the 3 rows: 1 + 2 + 3 = 6 at the LoopEnd. The placeholder 0 would
342+
// give 0, and an n that never rebinds to the new value 3.
343+
val src = textInput("a\nb\nc")
344+
val start = loopStart("n = 1", "table")
345+
val mid = limitReferringTo("n")
346+
val end = loopEnd("n += 1", "n <= len(table)")
347+
val materialized = runAndGetMaterializedRowCounts(
348+
List(src, start, mid, end),
349+
List(link(src, start), link(start, mid), link(mid, end))
350+
)
351+
val endRows = materialized.getOrElse(end.operatorIdentifier, -1L)
352+
assert(
353+
endRows == 6,
354+
s"Limit($$n) must pass n rows on iterations n = 1, 2, 3: expected 6, got $endRows " +
355+
s"(all: $materialized)"
356+
)
357+
}
358+
359+
it should "bind the inner loop's variable where it shadows the outer loop's of the same name" in {
360+
// TextInput("0","1") -> OuterStart(i = 0 / table) -> InnerStart(i = 0 / table.iloc[i])
361+
// -> Filter(line = $i) -> InnerEnd -> OuterEnd.
362+
// Both loops name their variable i. On each first inner iteration the Filter receives both
363+
// loops' states, the outer one's a loop further out (loopCounter 1), and must take the inner
364+
// i: inner iteration i emits the row "i", so all 2 x 2 rows pass. The outer i on the second
365+
// outer iteration (1, against the row "0") would drop one, and treating the two as one loop's
366+
// conflicting values would fail the run.
367+
val src = textInput("0\n1")
368+
val outerStart = loopStart("i = 0", "table")
369+
val innerStart = loopStart("i = 0", "table.iloc[i]")
370+
val mid = filterEquals("line", "$i")
371+
val innerEnd = loopEnd("i += 1", "i < len(table)")
372+
val outerEnd = loopEnd("i += 1", "i < len(table)")
373+
val materialized = runAndGetMaterializedRowCounts(
374+
List(src, outerStart, innerStart, mid, innerEnd, outerEnd),
375+
List(
376+
link(src, outerStart),
377+
link(outerStart, innerStart),
378+
link(innerStart, mid),
379+
link(mid, innerEnd),
380+
link(innerEnd, outerEnd)
381+
)
382+
)
383+
val outerRows = materialized.getOrElse(outerEnd.operatorIdentifier, -1L)
384+
assert(
385+
outerRows == 4,
386+
s"Filter(line = $$i) must bind the inner i on all 4 inner iterations: expected 4, got " +
387+
s"$outerRows (all: $materialized)"
388+
)
389+
}
390+
391+
it should "bind a $i reference after two branches of the loop body meet again" in {
392+
// TextInput("0","1","2") -> LoopStart(i = 0 / table.iloc[i]) -> Limit(10) and Sleep(0)
393+
// -> Union -> Filter(line = $i) -> LoopEnd.
394+
// The Filter receives the iteration's state once per branch, and the copies agree; the row
395+
// "i" arrives once per branch and passes against i: 2 rows on each of 3 iterations.
396+
val src = textInput("0\n1\n2")
397+
val start = loopStart("i = 0", "table.iloc[i]")
398+
val left = limit(10)
399+
val right = sleep(0)
400+
val union = new UnionOpDesc()
401+
val mid = filterEquals("line", "$i")
402+
val end = loopEnd("i += 1", "i < len(table)")
403+
val materialized = runAndGetMaterializedRowCounts(
404+
List(src, start, left, right, union, mid, end),
405+
List(
406+
link(src, start),
407+
link(start, left),
408+
link(start, right),
409+
link(left, union),
410+
link(right, union),
411+
link(union, mid),
412+
link(mid, end)
413+
)
414+
)
415+
val endRows = materialized.getOrElse(end.operatorIdentifier, -1L)
416+
assert(
417+
endRows == 6,
418+
s"Filter(line = $$i) after the branches meet must pass both copies of each iteration's " +
419+
s"row: expected 6, got $endRows (all: $materialized)"
420+
)
421+
}
422+
320423
it should "run a nested loop whose inner body chains multiple JVM operators" in {
321424
// TextInput -> OuterStart -> InnerStart -> Limit -> Sleep(0) -> InnerEnd -> OuterEnd.
322425
//

0 commit comments

Comments
 (0)