Skip to content

Commit 97de6cc

Browse files
committed
feat(amber): bind $K loop-variable references in operators
A property of an operator inside a loop block (LoopStart ... LoopEnd) may name a loop variable as `$K`. The frontend keeps sending the literal string whatever the property's type; the backend binds it in three steps. Parse. LogicalOp is now a StateReferencing descriptor, with a `stateReferences` sidecar (JSON pointer -> variable name) that the property panel never shows. StateReferenceModule, registered on JSONUtils.objectMapper and on the workflow-compiling-service mapper, adds a Jackson problem handler: when a whole-string `$name` cannot be converted to an integral, floating-point or boolean property, it returns the placeholder 0 / 0.0 / false and records the value's pointer, relative to the operator, in the operator's sidecar. A thin wrapper around each concrete StateReferencing deserializer opens that recording frame. An object that a polymorphic type deserializer hands over mid-way is first copied into a buffer of its own, because Jackson replays the fields before a late type id from a buffer whose parse context copies the first field's value, not the object. String properties keep the literal, and any other target (an enum, ...) keeps Jackson's ordinary error. Compile. WorkflowCompiler works out which operators lie inside a loop block (LoopBlockMembership, the frontend's block walk) and makes the sidecar final before the physical plan serializes it into the executor's descString. Inside a block it adds every whole-string `$name` of the operator's JSON. Outside every block a string such as "$AAPL" stays the literal it is on main, and a typed placeholder is a compile error naming the property and the variable. Inside a block, an operator whose executor runs generated code, or with an input fed from outside the block, is a compile error too. Worker. ExecFactory hands out a LateBoundExecutor when the descString's sidecar is not empty. It merges the values of the state messages it receives, a later one winning so that an inner loop's variable shadows an outer one, and forwards each message. Once every referenced variable has arrived it patches the placeholders, coerced to their JSON type, builds the real executor through the normal factory path and opens it; every later call goes to that executor, which is never rebound. A tuple before then is an error naming the missing variables. Closes apache#8635
1 parent 6defa2c commit 97de6cc

16 files changed

Lines changed: 1727 additions & 6 deletions

File tree

‎amber/src/test/integration/org/apache/texera/amber/engine/e2e/LoopIntegrationSpec.scala‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,11 @@ import org.apache.texera.amber.engine.e2e.TestUtils.{
4141
workflowContext
4242
}
4343
import org.apache.texera.amber.operator.LogicalOp
44+
import org.apache.texera.amber.operator.filter.{
45+
ComparisonType,
46+
FilterPredicate,
47+
SpecializedFilterOpDesc
48+
}
4449
import org.apache.texera.amber.operator.limit.LimitOpDesc
4550
import org.apache.texera.amber.operator.loop.{LoopEndOpDesc, LoopStartOpDesc}
4651
import org.apache.texera.amber.operator.sleep.SleepOpDesc
@@ -188,6 +193,13 @@ class LoopIntegrationSpec
188193
op
189194
}
190195

196+
/** `attribute = value`; a `$name` value refers to a loop variable (bound at the worker). */
197+
private def filterEquals(attribute: String, value: String): SpecializedFilterOpDesc = {
198+
val op = new SpecializedFilterOpDesc()
199+
op.predicates = List(new FilterPredicate(attribute, ComparisonType.EQUAL_TO, value))
200+
op
201+
}
202+
191203
// NOTE: a JavaUDF (runtime-compiled Java) case cannot run in this suite.
192204
// The integration tests run forkless under sbt (fork := false), where
193205
// `javax.tools` sees only the sbt launcher on `java.class.path` -- the
@@ -317,6 +329,33 @@ class LoopIntegrationSpec
317329
)
318330
}
319331

332+
it should "bind a $i loop-variable reference in a JVM operator's property on every iteration" in {
333+
// TextInput("0","1","2") -> LoopStart(i = 0 / table.iloc[i]) -> Filter(line = $i) -> LoopEnd.
334+
//
335+
// The Filter's predicate value is the literal string "$i". The Filter sits inside the
336+
// block, so the compiler records "$i" in its sidecar, and ExecFactory hands the
337+
// worker a LateBoundExecutor, which binds "$i" to the iteration's `i` when the LoopStart's
338+
// state message arrives -- the region's input reader replays states before tuples -- and
339+
// only then instantiates the real Filter. Iteration i emits the single row
340+
// "i" and the Filter compares it against i, so every iteration's row passes and the
341+
// LoopEnd accumulates 3 rows. A constant "0" instead of "$i" would pass only the first
342+
// iteration's row and give 1; a reference that never rebinds to the new i would too.
343+
val src = textInput("0\n1\n2")
344+
val start = loopStart("i = 0", "table.iloc[i]")
345+
val mid = filterEquals("line", "$i")
346+
val end = loopEnd("i += 1", "i < len(table)")
347+
val materialized = runAndGetMaterializedRowCounts(
348+
List(src, start, mid, end),
349+
List(link(src, start), link(start, mid), link(mid, end))
350+
)
351+
val endRows = materialized.getOrElse(end.operatorIdentifier, -1L)
352+
assert(
353+
endRows == 3,
354+
s"Filter(line = $$i) must pass its own iteration's row on all 3 iterations: " +
355+
s"expected 3, got $endRows (all: $materialized)"
356+
)
357+
}
358+
320359
it should "run a nested loop whose inner body chains multiple JVM operators" in {
321360
// TextInput -> OuterStart -> InnerStart -> Limit -> Sleep(0) -> InnerEnd -> OuterEnd.
322361
//
Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.texera.common.compiler
21+
22+
import org.apache.texera.amber.core.virtualidentity.OperatorIdentity
23+
import org.apache.texera.amber.operator.LogicalOp
24+
import org.apache.texera.amber.operator.loop.{LoopEndOpDesc, LoopStartOpDesc}
25+
import org.apache.texera.common.compiler.model.LogicalPlan
26+
27+
import scala.collection.mutable
28+
29+
/**
30+
* Which operators of a logical plan lie inside a loop block (LoopStart ... LoopEnd).
31+
*
32+
* An operator is inside a block when it lies on a path LoopStart -> ... -> operator -> ... ->
33+
* LoopEnd whose two ends match. The control operators are not inside their own block, but an
34+
* inner block's LoopStart and LoopEnd are inside the outer one. Blocks in sequence
35+
* (S1 -> a -> E1 -> x -> S2 -> b -> E2) do not leak into each other: walking downstream from x,
36+
* S2 opens a block that E2 closes again, so only a LoopEnd met with no block open closes the
37+
* operator's own; the upstream walk to the LoopStart is the mirror image.
38+
*
39+
* This is the frontend's `getEnclosingLoopStarts` (loop-block.util.ts), which decides where the
40+
* property panel offers `$K`, so both sides agree on which operators a loop variable reaches.
41+
* Pure: it reads only the plan's operators and links, never its DAG, so it also ends on a graph
42+
* with a cycle through a LoopStart or LoopEnd (see `reachesUnmatched`).
43+
*/
44+
object LoopBlockMembership {
45+
46+
private sealed trait Control
47+
private case object Start extends Control
48+
private case object End extends Control
49+
50+
private def controlOf(op: LogicalOp): Option[Control] =
51+
op match {
52+
case _: LoopStartOpDesc => Some(Start)
53+
case _: LoopEndOpDesc => Some(End)
54+
case _ => None
55+
}
56+
57+
/** The operators of `plan` that lie inside some loop block. */
58+
def operatorsInsideLoopBlocks(plan: LogicalPlan): Set[OperatorIdentity] = {
59+
val controls: Map[OperatorIdentity, Control] =
60+
plan.operators.flatMap(op => controlOf(op).map(op.operatorIdentifier -> _)).toMap
61+
if (!controls.values.exists(_ == Start) || !controls.values.exists(_ == End)) {
62+
// No block can close: nearly every plan takes this exit.
63+
Set.empty
64+
} else {
65+
val downstream = plan.links.groupMap(_.fromOpId)(_.toOpId)
66+
val upstream = plan.links.groupMap(_.toOpId)(_.fromOpId)
67+
plan.operators
68+
.map(_.operatorIdentifier)
69+
.filter { id =>
70+
val control = controls.get(id)
71+
// Downstream a LoopStart opens a nested block and a LoopEnd closes one; the operator's
72+
// own block is closed by a LoopEnd met with none open. A LoopStart starts the walk
73+
// with its own block open. Upstream it is the mirror image.
74+
reachesUnmatched(id, downstream, controls, Start, End, open(control, Start)) &&
75+
reachesUnmatched(id, upstream, controls, End, Start, open(control, End))
76+
}
77+
.toSet
78+
}
79+
}
80+
81+
private def open(control: Option[Control], opener: Control): Int =
82+
if (control.contains(opener)) 1 else 0
83+
84+
/**
85+
* Whether a breadth-first walk from `origin` over `adjacency` meets a `closes` operator with
86+
* no block open, counting the blocks opened (`opens`) and closed along each path.
87+
*
88+
* A state is a (node, open count) pair, and the open count is capped at the number of
89+
* `opens` operators plus the blocks open at the start. A path without repeated nodes passes
90+
* each opener at most once, so on a DAG the count never exceeds the cap and no answer
91+
* changes; on a cycle through an opener, where every lap would open one more block, the cap
92+
* keeps the states finite (nodes x (cap + 1)) and the walk ends.
93+
*/
94+
private def reachesUnmatched(
95+
origin: OperatorIdentity,
96+
adjacency: Map[OperatorIdentity, List[OperatorIdentity]],
97+
controls: Map[OperatorIdentity, Control],
98+
opens: Control,
99+
closes: Control,
100+
initiallyOpen: Int
101+
): Boolean = {
102+
val maxOpen = controls.values.count(_ == opens) + initiallyOpen
103+
val visited = mutable.Set((origin, initiallyOpen))
104+
val queue = mutable.Queue((origin, initiallyOpen))
105+
def visit(state: (OperatorIdentity, Int)): Unit =
106+
if (visited.add(state)) queue.enqueue(state)
107+
var found = false
108+
while (queue.nonEmpty && !found) {
109+
val (node, openCount) = queue.dequeue()
110+
val next = adjacency.getOrElse(node, Nil).iterator
111+
while (next.hasNext && !found) {
112+
val neighbor = next.next()
113+
controls.get(neighbor) match {
114+
case Some(`closes`) if openCount == 0 => found = true
115+
case Some(`closes`) => visit((neighbor, openCount - 1))
116+
case Some(`opens`) =>
117+
// Only a lap around a cycle exceeds the cap: see above.
118+
if (openCount < maxOpen) visit((neighbor, openCount + 1))
119+
case _ => visit((neighbor, openCount))
120+
}
121+
}
122+
}
123+
found
124+
}
125+
}

‎common/workflow-compiler/src/main/scala/org/apache/texera/common/compiler/WorkflowCompiler.scala‎

Lines changed: 124 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,13 +19,18 @@
1919

2020
package org.apache.texera.common.compiler
2121

22+
import com.fasterxml.jackson.databind.node.ObjectNode
2223
import com.google.protobuf.timestamp.Timestamp
2324
import com.typesafe.scalalogging.{LazyLogging, Logger}
2425
import org.apache.texera.common.compiler.WorkflowCompiler.{
2526
collectOutputSchemaFromPhysicalPlan,
26-
convertErrorListToWorkflowFatalErrorMap
27+
convertErrorListToWorkflowFatalErrorMap,
28+
normalizeStateReferences,
29+
unbindableStateReferences
2730
}
28-
import org.apache.texera.common.compiler.model.{LogicalPlan, LogicalPlanPojo}
31+
import org.apache.texera.common.compiler.model.{LogicalLink, LogicalPlan, LogicalPlanPojo}
32+
import org.apache.texera.amber.core.executor.OpExecWithCode
33+
import org.apache.texera.amber.core.state.StateReferencing.literalReferences
2934
import org.apache.texera.amber.core.tuple.Schema
3035
import org.apache.texera.amber.core.virtualidentity.OperatorIdentity
3136
import org.apache.texera.amber.core.workflow.{
@@ -37,6 +42,9 @@ import org.apache.texera.amber.core.workflow.{
3742
}
3843
import org.apache.texera.amber.core.workflowruntimestate.FatalErrorType.COMPILATION_ERROR
3944
import org.apache.texera.amber.core.workflowruntimestate.WorkflowFatalError
45+
import org.apache.texera.amber.operator.LogicalOp
46+
import org.apache.texera.amber.operator.loop.LoopStartOpDesc
47+
import org.apache.texera.amber.util.JSONUtils.objectMapper
4048
import org.apache.texera.amber.util.StackTraceUtils.getStackTraceWithAllCauses
4149

4250
import java.time.Instant
@@ -72,6 +80,109 @@ object WorkflowCompiler {
7280
opIdToError.toMap
7381
}
7482

83+
/**
84+
* Make `logicalOp`'s `stateReferences` sidecar final, now that its place in the plan is known.
85+
* This runs before the operator's physical plan is built: the descriptor serializes the sidecar
86+
* into its executor's descString there, and the worker binds exactly what it names.
87+
*
88+
* Inside a loop block the sidecar also gets every whole-string `$name` value of the operator's
89+
* JSON (`StateReferencing.literalReferences`), next to the typed placeholders the parse put in;
90+
* an entry whose pointer names no value of that JSON has nothing to bind and is dropped. An
91+
* input fed from outside the block is then an error (`inputsAheadOfLoopState`).
92+
*
93+
* Outside every block a string such as "$AAPL" stays the literal it was before loop variables
94+
* existed, and the sidecar holds only typed placeholders (the user wrote "$n" for an Int),
95+
* which nothing would ever bind: each is an error naming the property and the variable. They
96+
* stay in the sidecar, so compiling the same operator again reports them again.
97+
*/
98+
def normalizeStateReferences(
99+
logicalOp: LogicalOp,
100+
plan: LogicalPlan,
101+
insideLoopBlocks: Set[OperatorIdentity]
102+
): Option[Throwable] =
103+
if (insideLoopBlocks.contains(logicalOp.operatorIdentifier)) {
104+
val tree = objectMapper.valueToTree[ObjectNode](logicalOp)
105+
logicalOp.stateReferences = (literalReferences(tree) ++ logicalOp.stateReferences).filter {
106+
case (pointer, _) => tree.at(pointer).isValueNode
107+
}
108+
inputsAheadOfLoopState(logicalOp, plan, insideLoopBlocks)
109+
} else {
110+
Option.when(logicalOp.stateReferences.nonEmpty) {
111+
val name = logicalOp.operatorInfo.userFriendlyName
112+
new IllegalArgumentException(
113+
logicalOp.stateReferences.toSeq.sorted
114+
.map {
115+
case (pointer, variable) =>
116+
s"property $pointer refers to loop variable $variable, " +
117+
s"but $name is not inside a loop block"
118+
}
119+
.mkString("; ")
120+
)
121+
}
122+
}
123+
124+
/**
125+
* An operator inside a loop block binds its references once the loop state has arrived (see
126+
* `LateBoundExecutor`), so a tuple that reaches it earlier cannot be processed. The state comes
127+
* ahead of the tuples on a link from a LoopStart or from another operator inside a block, but
128+
* not on one from outside every block. An input with such a link is an error, unless the
129+
* operator reads it only after an input whose links all carry the state (a join's probe input).
130+
*/
131+
private def inputsAheadOfLoopState(
132+
logicalOp: LogicalOp,
133+
plan: LogicalPlan,
134+
insideLoopBlocks: Set[OperatorIdentity]
135+
): Option[Throwable] =
136+
if (logicalOp.stateReferences.isEmpty) {
137+
None
138+
} else {
139+
val loopStarts =
140+
plan.operators.collect { case start: LoopStartOpDesc => start.operatorIdentifier }.toSet
141+
def carriesState(link: LogicalLink): Boolean =
142+
insideLoopBlocks.contains(link.fromOpId) || loopStarts.contains(link.fromOpId)
143+
val linksByPort = plan.getUpstreamLinks(logicalOp.operatorIdentifier).groupBy(_.toPortId)
144+
def carriesOnlyState(port: PortIdentity): Boolean =
145+
linksByPort.get(port).exists(_.forall(carriesState))
146+
val exposed = logicalOp.operatorInfo.inputPorts.filter { input =>
147+
linksByPort.get(input.id).exists(!_.forall(carriesState)) &&
148+
!input.dependencies.exists(carriesOnlyState)
149+
}
150+
Option.when(exposed.nonEmpty) {
151+
val inputs = exposed.map { input =>
152+
if (input.displayName.nonEmpty) s"input '${input.displayName}'"
153+
else s"input port ${input.id.id}"
154+
}
155+
new IllegalArgumentException(
156+
s"${logicalOp.operatorInfo.userFriendlyName} refers to loop variables " +
157+
s"(${formatReferences(logicalOp.stateReferences)}), but its ${inputs.mkString(" and ")} " +
158+
s"${if (exposed.size == 1) "is" else "are"} fed from outside the loop block, so " +
159+
"tuples from there can arrive before the loop state"
160+
)
161+
}
162+
}
163+
164+
/**
165+
* A reference is bound by the late-bound executor that `ExecFactory` builds from the
166+
* descriptor's JSON. An operator whose executor runs code instead -- a UDF, or a descriptor
167+
* that generates Python from its properties, such as the sklearn operators -- never takes that
168+
* path: its code is built from the placeholders before the loop runs, and nothing binds them.
169+
*/
170+
def unbindableStateReferences(logicalOp: LogicalOp, subPlan: PhysicalPlan): Option[Throwable] =
171+
Option.when(
172+
logicalOp.stateReferences.nonEmpty &&
173+
subPlan.operators.exists(_.opExecInitInfo.isInstanceOf[OpExecWithCode])
174+
) {
175+
new UnsupportedOperationException(
176+
s"${logicalOp.operatorInfo.userFriendlyName} cannot refer to loop variables yet " +
177+
s"(${formatReferences(logicalOp.stateReferences)}): its code is generated before " +
178+
"the loop runs"
179+
)
180+
}
181+
182+
/** A descriptor's references as the compile errors name them: `/pointer -> $name, ...`. */
183+
private def formatReferences(references: Map[String, String]): String =
184+
references.toSeq.sorted.map { case (pointer, name) => s"$pointer -> $$$name" }.mkString(", ")
185+
75186
private def collectOutputSchemaFromPhysicalPlan(
76187
physicalPlan: PhysicalPlan,
77188
errorList: ArrayBuffer[(OperatorIdentity, Throwable)]
@@ -145,13 +256,24 @@ class WorkflowCompiler(
145256
(terminalLogicalOps ++ logicalOpsToViewResult.map(OperatorIdentity(_))).toSet
146257
var physicalPlan = PhysicalPlan(operators = Set.empty, links = Set.empty)
147258
val outputPortsNeedingStorage: mutable.HashSet[GlobalPortIdentity] = mutable.HashSet()
259+
val insideLoopBlocks = LoopBlockMembership.operatorsInsideLoopBlocks(logicalPlan)
148260

149261
logicalPlan.getTopologicalOpIds.asScala.foreach(logicalOpId =>
150262
Try {
151263
val logicalOp = logicalPlan.getOperator(logicalOpId)
152264
val upstreamLinks = logicalPlan.getUpstreamLinks(logicalOp.operatorIdentifier)
153265

266+
def report(error: Throwable): Unit =
267+
errorList match {
268+
case Some(list) => list.append((logicalOpId, error))
269+
case None => throw error
270+
}
271+
// Before the physical plan: the descriptor serializes its sidecar into its descString.
272+
normalizeStateReferences(logicalOp, logicalPlan, insideLoopBlocks).foreach(report)
154273
val subPlan = logicalOp.getPhysicalPlan(context.workflowId, context.executionId)
274+
// Before the code-generation check below: the first error per operator is reported, and
275+
// code generated from a placeholder may fail on it with a less specific message.
276+
unbindableStateReferences(logicalOp, subPlan).foreach(report)
155277
subPlan
156278
.topologicalIterator()
157279
.map(subPlan.getOperator)

0 commit comments

Comments
 (0)