You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 75c85aa
Browse filesBrowse the repository at this point in the historyBrowse files
feat(workflow-operator): export the first five operators, and serve the script (apache#8502)
### What changes were proposed in this PR?
Five operators implement `StandaloneCodeGenerator`, which is what turns
the translator from a mechanism into something that produces a script
that runs: Distinct, Filter, Limit, Projection and Union. Four of them
read a single input; Union reads a variadic port, so it names the whole
list of upstreams rather than a fixed count, which a fixed count gets
wrong in both directions.
The endpoint the editor calls comes with them, because what it is worth
testing on is a script that runs rather than one made of placeholders.
It takes a plan, compiles it first so a scan source can read the schema
off the file it points at, hands the translator the output schemas that
gives, and returns the script. A failed compile is logged and
translation goes on without them, so a workflow whose file is not chosen
yet still exports.
`pyStringLiteral` comes with them too. A generator has to write a column
name into the source it emits, and writing the quotes by hand lets any
quote, backslash or newline in the name close the literal early and
change, or break, the emitted program.
apache#8327 is the trait and the translator, and nothing that uses them.
### Any related issues, documentation, discussions?
Part of apache#8325, 2 of 27; that issue lists the set in order.
Closesapache#8501, the task this change is the whole of.
### How was this PR tested?
Each operator asserts the block it emits in its own spec.
`PythonTemplateBuilderApiSpec` covers what `pyStringLiteral` escapes,
including the NUL that Python refuses to compile anywhere in a source
file. `WorkflowToPythonResourceSpec` drives the endpoint over a plan
built from these operators and reads the script back.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDesc.scala
Copy file name to clipboardExpand all lines: common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/projection/ProjectionOpDesc.scala
Copy file name to clipboardExpand all lines: common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/distinct/DistinctOpDescSpec.scala
+8Lines changed: 8 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -106,4 +106,12 @@ class DistinctOpDescSpec extends AnyFlatSpec with Matchers {
106
106
valb=newDistinctOpDesc
107
107
a.operatorIdentifier should not equal b.operatorIdentifier
108
108
}
109
+
110
+
// The JVM operator keeps the first occurrence of a duplicate, which is what
111
+
// drop_duplicates does by default, so the emitted line says nothing about order.
112
+
"DistinctOpDesc.generateStandaloneCode" should "drop duplicates in place" in {
Copy file name to clipboardExpand all lines: common/workflow-operator/src/test/scala/org/apache/texera/amber/operator/filter/SpecializedFilterOpDescSpec.scala
+18Lines changed: 18 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -70,4 +70,22 @@ class SpecializedFilterOpDescSpec extends AnyFlatSpec with Matchers {
0 commit comments