Skip to content

Commit 75c85aa

Browse files
kz930claude
andauthored
feat(workflow-operator): export the first five operators, and serve the script (apache#8502)
### What changes were proposed in this PR? Five operators implement `StandaloneCodeGenerator`, which is what turns the translator from a mechanism into something that produces a script that runs: Distinct, Filter, Limit, Projection and Union. Four of them read a single input; Union reads a variadic port, so it names the whole list of upstreams rather than a fixed count, which a fixed count gets wrong in both directions. The endpoint the editor calls comes with them, because what it is worth testing on is a script that runs rather than one made of placeholders. It takes a plan, compiles it first so a scan source can read the schema off the file it points at, hands the translator the output schemas that gives, and returns the script. A failed compile is logged and translation goes on without them, so a workflow whose file is not chosen yet still exports. `pyStringLiteral` comes with them too. A generator has to write a column name into the source it emits, and writing the quotes by hand lets any quote, backslash or newline in the name close the literal early and change, or break, the emitted program. apache#8327 is the trait and the translator, and nothing that uses them. ### Any related issues, documentation, discussions? Part of apache#8325, 2 of 27; that issue lists the set in order. Closes apache#8501, the task this change is the whole of. ### How was this PR tested? Each operator asserts the block it emits in its own spec. `PythonTemplateBuilderApiSpec` covers what `pyStringLiteral` escapes, including the NUL that Python refuses to compile anywhere in a source file. `WorkflowToPythonResourceSpec` drives the endpoint over a plan built from these operators and reads the script back. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 4d7fd49 commit 75c85aa

16 files changed

Lines changed: 554 additions & 12 deletions

File tree

‎common/pybuilder/src/main/scala/org/apache/texera/amber/pybuilder/PythonTemplateBuilder.scala‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,32 @@ object PythonTemplateBuilder {
209209
def wrapWithPythonDecoderExpr(text: String): String =
210210
s"self.decode_python_template('$text')"
211211

212+
/**
213+
* Render `text` as a Python double-quoted string literal, quotes included.
214+
*
215+
* For generators that emit standalone Python source rather than an operator
216+
* for the runtime: they cannot use the decode expression (it needs the
217+
* operator's `decode_python_template`, and it is deliberately rejected inside
218+
* quotes), so they need the value as a *literal*. Writing `"$value"` by hand
219+
* instead lets any quote, backslash or newline in the value close the literal
220+
* early and change — or break — the emitted program.
221+
*
222+
* Escapes exactly what can end a double-quoted single-line literal, plus NUL:
223+
* Python refuses to compile source that holds one anywhere, so a column name
224+
* carrying it would break the whole script rather than only this literal.
225+
*/
226+
def pyStringLiteral(text: String): String = {
227+
val escaped = Option(text)
228+
.getOrElse("")
229+
.replace("\\", "\\\\")
230+
.replace("\"", "\\\"")
231+
.replace("\r", "\\r")
232+
.replace("\n", "\\n")
233+
.replace("\t", "\\t")
234+
.replace(0.toChar.toString, "\\x00")
235+
"\"" + escaped + "\""
236+
}
237+
212238
sealed trait RenderMode extends Product with Serializable
213239
object RenderMode {
214240
case object Plain extends RenderMode

‎common/pybuilder/src/test/scala/org/apache/texera/amber/pybuilder/PythonTemplateBuilderApiSpec.scala‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -244,4 +244,33 @@ class PythonTemplateBuilderApiSpec extends AnyFunSuite {
244244
test("hasUnclosedQuote: three opening single quotes count as unclosed") {
245245
assert(PythonLexerUtils.hasUnclosedQuote("'''abc"))
246246
}
247+
248+
// -------- pyStringLiteral --------
249+
250+
// Every character that can end a double-quoted single-line literal, since one that
251+
// slips through does not fail here but changes the emitted program.
252+
test("pyStringLiteral: quotes the value and escapes what would close the literal") {
253+
assert(PythonTemplateBuilder.pyStringLiteral("plain") == "\"plain\"")
254+
assert(PythonTemplateBuilder.pyStringLiteral("say \"hi\"") == "\"say \\\"hi\\\"\"")
255+
assert(PythonTemplateBuilder.pyStringLiteral("a\\b") == "\"a\\\\b\"")
256+
assert(PythonTemplateBuilder.pyStringLiteral("one\ntwo") == "\"one\\ntwo\"")
257+
assert(PythonTemplateBuilder.pyStringLiteral("a\tb") == "\"a\\tb\"")
258+
assert(PythonTemplateBuilder.pyStringLiteral("a\rb") == "\"a\\rb\"")
259+
}
260+
261+
// A column name arrives from JSON and can be absent; an empty literal is a value the
262+
// emitted program can carry, where `null` would reach it as the four letters.
263+
test("pyStringLiteral: renders a null as the empty literal") {
264+
assert(PythonTemplateBuilder.pyStringLiteral(null) == "\"\"")
265+
}
266+
267+
// NUL is the one character that a literal cannot carry verbatim: Python refuses to
268+
// compile a source file holding one, so it takes the whole script down rather than
269+
// this value alone.
270+
test("pyStringLiteral: escapes a NUL rather than emitting it") {
271+
val literal = PythonTemplateBuilder.pyStringLiteral("a" + 0.toChar + "b")
272+
assert(literal == "\"a\\x00b\"")
273+
assert(!literal.contains(0.toChar))
274+
}
275+
247276
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/distinct/DistinctOpDesc.scala‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@ package org.apache.texera.amber.operator.distinct
2222
import org.apache.texera.amber.core.executor.OpExecWithClassName
2323
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2424
import org.apache.texera.amber.core.workflow.{HashPartition, InputPort, OutputPort, PhysicalOp}
25-
import org.apache.texera.amber.operator.LogicalOp
25+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator}
2626
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2727

28-
class DistinctOpDesc extends LogicalOp {
28+
class DistinctOpDesc extends LogicalOp with StandaloneCodeGenerator {
2929

3030
override def getPhysicalOp(
3131
workflowId: WorkflowIdentity,
@@ -54,4 +54,9 @@ class DistinctOpDesc extends LogicalOp {
5454
outputPorts = List(OutputPort(blocking = true))
5555
)
5656

57+
override def generateStandaloneCode(): String = {
58+
// JVM op uses LinkedHashSet to preserve first-occurrence order;
59+
// pandas drop_duplicates does the same by default.
60+
"out1df = in1df.drop_duplicates(ignore_index=True)"
61+
}
5762
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala‎

Lines changed: 42 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,12 @@ import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import org.apache.texera.amber.core.executor.OpExecWithClassName
2424
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2525
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalOp}
26+
import org.apache.texera.amber.operator.StandaloneCodeGenerator
2627
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
28+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
2729
import org.apache.texera.amber.util.JSONUtils.objectMapper
2830

29-
class SpecializedFilterOpDesc extends FilterOpDesc {
31+
class SpecializedFilterOpDesc extends FilterOpDesc with StandaloneCodeGenerator {
3032

3133
@JsonProperty(value = "predicates", required = true)
3234
@JsonPropertyDescription("multiple predicates in OR")
@@ -60,4 +62,43 @@ class SpecializedFilterOpDesc extends FilterOpDesc {
6062
supportReconfiguration = true
6163
)
6264
}
65+
66+
override def generateStandaloneCode(): String = {
67+
// No predicate keeps no row: the executor's filter is `predicates.exists`,
68+
// which answers false on an empty list. Passing the frame through would be
69+
// the opposite answer.
70+
if (predicates.isEmpty) return "out1df = in1df.iloc[0:0].copy()"
71+
val conditions = predicates.map { p =>
72+
val colLit = pyStringLiteral(p.attribute)
73+
p.condition match {
74+
case ComparisonType.IS_NULL => s"""(in1df[$colLit].isna())"""
75+
case ComparisonType.IS_NOT_NULL => s"""(in1df[$colLit].notna())"""
76+
case other =>
77+
val op = other.getName // returns "=", ">=", "<", etc. (see ComparisonType.java)
78+
val pyOp = if (op == "=") "==" else op
79+
// notna mirrors FilterPredicate, which answers false for every condition
80+
// but IS_NULL / IS_NOT_NULL once the field is null. Only `!=` needs it —
81+
// pandas answers True there, where every other operator answers False —
82+
// but guarding all of them keeps the one rule visible in one place.
83+
s"""(in1df[$colLit].notna() & (in1df[$colLit] $pyOp ${coerceValue(p.value)}))"""
84+
}
85+
}
86+
s"out1df = in1df[${conditions.mkString(" | ")}].reset_index(drop=True)"
87+
}
88+
89+
// Try numeric coercion so generated code compares column values against the right type.
90+
// Strings that don't parse fall through to a quoted string literal.
91+
private def coerceValue(raw: String): String = {
92+
try {
93+
raw.toInt.toString
94+
} catch {
95+
case _: NumberFormatException =>
96+
try {
97+
raw.toDouble.toString
98+
} catch {
99+
case _: NumberFormatException =>
100+
pyStringLiteral(raw)
101+
}
102+
}
103+
}
63104
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/limit/LimitOpDesc.scala‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,12 +25,12 @@ import org.apache.texera.amber.core.executor.OpExecWithClassName
2525
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2626
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalOp}
2727
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
28-
import org.apache.texera.amber.operator.{LogicalOp, StateTransferFunc}
28+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator, StateTransferFunc}
2929
import org.apache.texera.amber.util.JSONUtils.objectMapper
3030

3131
import scala.util.{Success, Try}
3232

33-
class LimitOpDesc extends LogicalOp {
33+
class LimitOpDesc extends LogicalOp with StandaloneCodeGenerator {
3434

3535
@JsonProperty(required = true)
3636
@JsonSchemaTitle("Limit")
@@ -80,4 +80,11 @@ class LimitOpDesc extends LogicalOp {
8080
}
8181
Success(newPhysicalOp, Some(stateTransferFunc))
8282
}
83+
84+
override def generateStandaloneCode(): String = {
85+
// Clamped, because the two sides read a negative limit differently: the
86+
// executor's `count < limit` is false from the first tuple and emits
87+
// nothing, while pandas' head(-n) drops only the last n rows.
88+
s"out1df = in1df.head(${math.max(0, limit)}).reset_index(drop=True)"
89+
}
8390
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/projection/ProjectionOpDesc.scala‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,17 +26,23 @@ import org.apache.texera.amber.core.tuple.Schema
2626
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2727
import org.apache.texera.amber.core.workflow.PhysicalOp.oneToOnePhysicalOp
2828
import org.apache.texera.amber.core.workflow._
29+
import org.apache.texera.amber.operator.StandaloneCodeGenerator
2930
import org.apache.texera.amber.operator.map.MapOpDesc
3031
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
32+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
3133
import org.apache.texera.amber.util.JSONUtils.objectMapper
3234

33-
class ProjectionOpDesc extends MapOpDesc {
35+
class ProjectionOpDesc extends MapOpDesc with StandaloneCodeGenerator {
3436

3537
@JsonProperty(required = true, defaultValue = "false")
3638
@JsonSchemaTitle("Drop Option")
3739
@JsonPropertyDescription("check to drop the selected attributes")
3840
var isDrop: Boolean = false
3941

42+
// Named explicitly, without `required`: the form already asks for these and must go
43+
// on accepting an empty list, but a field carrying no annotation is invisible to
44+
// anything reading the operator's config by reflection.
45+
@JsonProperty
4046
var attributes: List[AttributeUnit] = List()
4147

4248
override def getPhysicalOp(
@@ -98,4 +104,32 @@ class ProjectionOpDesc extends MapOpDesc {
98104
outputPorts = List(OutputPort())
99105
)
100106
}
107+
108+
override def generateStandaloneCode(): String = {
109+
val units = Option(attributes).getOrElse(List.empty)
110+
// The engine refuses an empty selection, so the script says so too. Passing
111+
// the frame through would hand back data where a run would have stopped.
112+
if (units.isEmpty)
113+
return """raise ValueError("Please select at least one attribute to project.")"""
114+
115+
if (isDrop) {
116+
// Drop mode ignores aliases (matches ProjectionOpExec).
117+
val cols = units.map(u => pyStringLiteral(u.getOriginalAttribute)).mkString("[", ", ", "]")
118+
s"out1df = in1df.drop(columns=$cols)"
119+
} else {
120+
val originals =
121+
units.map(u => pyStringLiteral(u.getOriginalAttribute)).mkString("[", ", ", "]")
122+
// AttributeUnit.getAlias returns originalAttribute when alias is blank,
123+
// so an explicit rename is only needed when they differ.
124+
val renames = units
125+
.filter(u => u.getAlias != u.getOriginalAttribute)
126+
.map(u => s"""${pyStringLiteral(u.getOriginalAttribute)}: ${pyStringLiteral(u.getAlias)}""")
127+
if (renames.isEmpty) {
128+
s"out1df = in1df[$originals].copy()"
129+
} else {
130+
val renameMap = renames.mkString("{", ", ", "}")
131+
s"out1df = in1df[$originals].rename(columns=$renameMap)"
132+
}
133+
}
134+
}
101135
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/union/UnionOpDesc.scala‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,10 +22,10 @@ package org.apache.texera.amber.operator.union
2222
import org.apache.texera.amber.core.executor.OpExecWithClassName
2323
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2424
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalOp}
25-
import org.apache.texera.amber.operator.LogicalOp
2625
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
26+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator}
2727

28-
class UnionOpDesc extends LogicalOp {
28+
class UnionOpDesc extends LogicalOp with StandaloneCodeGenerator {
2929

3030
override def getPhysicalOp(
3131
workflowId: WorkflowIdentity,
@@ -50,4 +50,11 @@ class UnionOpDesc extends LogicalOp {
5050
inputPorts = List(InputPort()),
5151
outputPorts = List(OutputPort())
5252
)
53+
54+
// UNION ALL: UnionOpExec passes tuples through without dedup. The port is
55+
// variadic, so the code names the whole list of upstreams rather than a fixed
56+
// two — naming two dropped a third and left the second unbound when only one
57+
// was drawn.
58+
override def generateStandaloneCode(): String =
59+
"out1df = pd.concat(inAlldf, ignore_index=True)"
5360
}

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/distinct/DistinctOpDescSpec.scala‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,4 +106,12 @@ class DistinctOpDescSpec extends AnyFlatSpec with Matchers {
106106
val b = new DistinctOpDesc
107107
a.operatorIdentifier should not equal b.operatorIdentifier
108108
}
109+
110+
// The JVM operator keeps the first occurrence of a duplicate, which is what
111+
// drop_duplicates does by default, so the emitted line says nothing about order.
112+
"DistinctOpDesc.generateStandaloneCode" should "drop duplicates in place" in {
113+
(new DistinctOpDesc).generateStandaloneCode() shouldBe
114+
"out1df = in1df.drop_duplicates(ignore_index=True)"
115+
}
116+
109117
}

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDescSpec.scala‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,4 +70,22 @@ class SpecializedFilterOpDescSpec extends AnyFlatSpec with Matchers {
7070
restored shouldBe a[SpecializedFilterOpDesc]
7171
restored.asInstanceOf[SpecializedFilterOpDesc].predicates shouldBe empty
7272
}
73+
74+
// A null answers false for every condition but IS_NULL / IS_NOT_NULL, which pandas
75+
// does not do on its own for `!=`, so the emitted condition carries the guard.
76+
"SpecializedFilterOpDesc.generateStandaloneCode" should "emit one condition per predicate" in {
77+
val d = new SpecializedFilterOpDesc
78+
d.predicates = List(new FilterPredicate("age", ComparisonType.GREATER_THAN, "18"))
79+
val code = d.generateStandaloneCode()
80+
code should include("in1df[\"age\"]")
81+
code should include("out1df")
82+
}
83+
84+
// The executor filters on `predicates.exists`, which answers false on an empty
85+
// list, so no predicate keeps no row. The columns survive; the rows do not.
86+
it should "keep no row when there is no predicate" in {
87+
(new SpecializedFilterOpDesc).generateStandaloneCode() shouldBe
88+
"out1df = in1df.iloc[0:0].copy()"
89+
}
90+
7391
}

‎common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/limit/LimitOpDescSpec.scala‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,4 +94,21 @@ class LimitOpDescSpec extends AnyFlatSpec with Matchers {
9494
transfer(oldExec, newExec)
9595
newExec.count shouldBe 3
9696
}
97+
98+
// The index is reset because the operator hands its downstream a fresh table
99+
// rather than a view of the one it read.
100+
"LimitOpDesc.generateStandaloneCode" should "take the first N rows" in {
101+
val d = new LimitOpDesc
102+
d.limit = 3
103+
d.generateStandaloneCode() shouldBe "out1df = in1df.head(3).reset_index(drop=True)"
104+
}
105+
106+
// head(-1) would drop the last row and keep the rest, where the executor's
107+
// `count < limit` is false from the first tuple and keeps nothing.
108+
it should "keep nothing when the limit is negative" in {
109+
val d = new LimitOpDesc
110+
d.limit = -1
111+
d.generateStandaloneCode() shouldBe "out1df = in1df.head(0).reset_index(drop=True)"
112+
}
113+
97114
}

0 commit comments

Comments
 (0)