Skip to content

Commit 2f0089f

Browse files
kz930claude
andauthored
feat(workflow-operator): export the join and set operators as Python (apache#8506)
### What changes were proposed in this PR? Six operators implement `StandaloneCodeGenerator`: the three joins and the three set operations. What they have in common is that they combine or compare tables rather than read what is inside a row. apache#8340 keeps the operators that work on text. Aggregate and the sort family follow in apache#8709, and the samplers and control flow in apache#8710. Review then found two places where the block answered a question the engine answers differently. The hash join asked pandas for a suffix, which appends once and then refuses the duplicate it just made, and let an outer join's holes cost an integer column its type. Running the engine on those cases found the platform out of step with itself. HashJoin declared a right column's new name by checking only the left names, while the executor also avoids the other right names. So a right `x` beside a right `x#@1` was declared as a second `x#@1`, and one value was lost. An unmatched right row in an outer join kept its raw names and landed in the left column. Both now go through one renaming rule in `JoinUtils`. The shared table then found one more. In a right or full outer join on keys both sides name alike, the engine writes an unmatched right row's key into the left key column, and the export left it empty. The engine had its own half: an unmatched right key named like a left payload column landed in that payload. The export now fills the key, and the engine leaves it out of the payload. The set operations matched rows the way pandas does. The engine's HashSet compares rows with Scala's `==`, under which NaN equals nothing, and hashes them by each double's bits, which tell 0.0 from -0.0. The export now selects rows by a key that follows both rules. A probe worker that received no build rows could not name an unmatched right row's columns, because it read the left names off the first build row. This happens when the left input is empty or no left key hashes to that worker. The runtime now records each input port's declared schema on the executor before the port finishes (`OperatorExecutor.inputSchemas`, filled in `EndChannelHandler`), and the probe reads the left names from it. This is the Scala counterpart of what apache#8766 does for Python operators. ### Any related issues, documentation, discussions? Part of apache#8325, step 11a; apache#8709 and apache#8710 follow. That issue lists the set in order. The set operations and joins keyed on a null or a NaN need the nullable dtypes apache#8704 reads an Arrow file into. Without it a null and a NaN share a slot and both behave as they did before, so this PR is no worse on its own and only complete alongside that one. Closes apache#8505, the task this change is the whole of. ### How was this PR tested? Each operator asserts the block it emits in its own spec. The interval join reads its keys off the merged frame rather than writing over an input column that shares a temporary's name. The join cases the review named run in the verify tables of apache#8363, which put the same input through the engine and through the script and compare the two. What those tables cannot reach stays here as executed tests: a null kept apart from a NaN in the set operations and the join key. `HashJoinProbeOpExecSpec` pins the platform fixes. Each set operation's spec runs its native executor and the export over NaN, signed-zero, finite and null rows. `HashJoinProbeOpExecSpec` covers an empty build side, and `EndChannelHandlerSpec` checks that the schema is in place before the operator finishes the port. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 379f9a7 commit 2f0089f

18 files changed

Lines changed: 1409 additions & 57 deletions

File tree

‎amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers/EndChannelHandler.scala‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,8 @@ trait EndChannelHandler {
4040
val portId = dp.inputGateway.getChannel(channelId).getPortId
4141
dp.inputManager.getPort(portId).completed = true
4242
dp.inputManager.initBatch(channelId, Array.empty)
43+
// Tell the executor this port's schema, even if the port got no rows.
44+
dp.executor.inputSchemas(portId.id) = dp.inputManager.getPort(portId).schema
4345
try {
4446
// The executor finishes: a setting that refers to loop variables must have them by now.
4547
dp.bindStateReferences()

‎amber/src/test/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers/EndChannelHandlerSpec.scala‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,6 +252,18 @@ class EndChannelHandlerSpec extends AnyFlatSpec {
252252
assert(executor.finishOutputPorts.toList == List(currentPortId.id))
253253
}
254254

255+
it should "give the operator the finished port's declared schema before it finishes the port" in {
256+
// A HashJoin probe worker that received no build rows names its unmatched right rows from
257+
// this, so it has to be in place when the operator is asked to finish, not after.
258+
val executor = new RecordingExecutor()
259+
val fixture = new Fixture(executor, otherInputPortCompleted = false)
260+
261+
fixture.endChannel()
262+
263+
assert(executor.schemasAtFinish.toList == List(Map(currentPortId.id -> schema)))
264+
assert(!executor.inputSchemas.contains(otherPortId.id))
265+
}
266+
255267
it should "emit the boundary state the operator produced on finish, with no loop envelope" in {
256268
val state = State(Map("boundary" -> "end"))
257269
val fixture = new Fixture(new RecordingExecutor(onFinishState = _ => Some(state)))
@@ -498,8 +510,12 @@ object EndChannelHandlerSpec {
498510
/** Ports handed to `onFinish`, i.e. what `onFinishMultiPort` forwarded. */
499511
val finishOutputPorts: ArrayBuffer[Int] = ArrayBuffer()
500512

513+
/** `inputSchemas` as the first finish callback saw it. */
514+
val schemasAtFinish: ArrayBuffer[Map[Int, Schema]] = ArrayBuffer()
515+
501516
override def produceStateOnFinish(port: Int): Option[State] = {
502517
statePorts += port
518+
schemasAtFinish += inputSchemas.toMap
503519
onFinishState(port)
504520
}
505521

‎common/workflow-core/src/main/scala/org/apache/texera/amber/core/executor/OperatorExecutor.scala‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,9 +21,11 @@ package org.apache.texera.amber.core.executor
2121

2222
import com.google.common.collect.MapMaker
2323
import org.apache.texera.amber.core.state.State
24-
import org.apache.texera.amber.core.tuple.{Tuple, TupleLike}
24+
import org.apache.texera.amber.core.tuple.{Schema, Tuple, TupleLike}
2525
import org.apache.texera.amber.core.workflow.PortIdentity
2626

27+
import scala.collection.mutable
28+
2729
/**
2830
* Inside a loop block a setting may refer to a loop variable (`$i`): the worker writes each
2931
* iteration's value into the settings object the executor holds, in place (`registerState`). For
@@ -91,6 +93,17 @@ trait OperatorExecutor {
9193
*/
9294
final def bindStateReferences(): Unit =
9395
Option(OperatorExecutor.bindings.get(this)).foreach(_.bind())
96+
97+
/**
98+
* What each finished input port was declared to carry, keyed by port index
99+
* and written by the runtime before `onFinish` for that port.
100+
*
101+
* The tuples say this too, so an operator only needs to ask when there were
102+
* none: a port can finish having carried no rows, and its schema is then the
103+
* only record of what its columns were.
104+
*/
105+
final def inputSchemas: mutable.Map[Int, Schema] =
106+
OperatorExecutor.inputSchemas.computeIfAbsent(this, _ => mutable.Map())
94107
}
95108

96109
object OperatorExecutor {
@@ -100,4 +113,6 @@ object OperatorExecutor {
100113
private val states = new MapMaker().weakKeys().makeMap[OperatorExecutor, State]()
101114
private[executor] val bindings =
102115
new MapMaker().weakKeys().makeMap[OperatorExecutor, StateReferenceBinding]()
116+
private val inputSchemas =
117+
new MapMaker().weakKeys().makeMap[OperatorExecutor, mutable.Map[Int, Schema]]()
103118
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/SamplingHelpers.scala‎

Lines changed: 4 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -25,18 +25,12 @@ package org.apache.texera.amber.operator
2525
object SamplingHelpers {
2626

2727
/**
28-
* A Python transcription of `java.util.Random`, for operators whose executor
29-
* draws from one.
30-
*
31-
* A sampler decides per row whether to keep it, so which rows survive is
32-
* fixed by the exact sequence the generator produces. Seeding Python's
33-
* `random` or numpy's with the engine's seed selects a different set, and
34-
* the script would then report a different sample than the workflow it came
35-
* from. Only the same generator gives the same rows.
28+
* `_TexeraJavaRandom`: `java.util.Random` in Python, so a sampler keeps the rows the engine
29+
* keeps. Python's or numpy's generator with the same seed picks other rows. `next_int`
30+
* rejects a draw where Java's int sum overflows, which Python's integers never do.
3631
*/
3732
val JavaRandom: String =
38-
"""# java.util.Random, transcribed so sampling matches the engine.
39-
|class _TexeraJavaRandom:
33+
"""class _TexeraJavaRandom:
4034
| _MASK = (1 << 48) - 1
4135
| _MULTIPLIER = 0x5DEECE66D
4236
| _ADDEND = 0xB
@@ -60,8 +54,6 @@ object SamplingHelpers {
6054
| while True:
6155
| bits = self._next(31)
6256
| value = bits % bound
63-
| # Java rejects a draw by letting this sum overflow an int.
64-
| # Python would carry it and never reject.
6557
| probe = bits - value + (bound - 1)
6658
| if ((probe + (1 << 31)) % (1 << 32)) - (1 << 31) >= 0:
6759
| return value""".stripMargin

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/cartesianProduct/CartesianProductOpDesc.scala‎

Lines changed: 26 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,11 @@ import org.apache.texera.amber.core.executor.OpExecWithClassName
2323
import org.apache.texera.amber.core.tuple.{Attribute, Schema}
2424
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2525
import org.apache.texera.amber.core.workflow._
26-
import org.apache.texera.amber.operator.LogicalOp
26+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator}
2727
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2828

29-
class CartesianProductOpDesc extends LogicalOp {
29+
class CartesianProductOpDesc extends LogicalOp with StandaloneCodeGenerator {
30+
3031
override def getPhysicalOp(
3132
workflowId: WorkflowIdentity,
3233
executionId: ExecutionIdentity
@@ -103,4 +104,27 @@ class CartesianProductOpDesc extends LogicalOp {
103104
),
104105
outputPorts = List(OutputPort())
105106
)
107+
108+
// Schema mirrors SchemaPropagationFunc: left columns kept as-is, each right
109+
// column renamed by repeatedly appending "#@1" while the candidate name
110+
// collides with any left column OR any other right column's ORIGINAL name.
111+
// The renamed-name table is recomputed at runtime from the actual DataFrame
112+
// columns. Known divergence: row order — pandas cross-merge varies right
113+
// fastest (L1R1, L1R2, L2R1, L2R2); the JVM op buffers left and emits per
114+
// arriving right tuple (L1R1, L2R1, L1R2, L2R2). Cartesian product is set-
115+
// semantically order-agnostic, so this is acceptable.
116+
override def generateStandaloneCode(): String = {
117+
"""_left_cols = list(in1df.columns)
118+
|_right_cols = list(in2df.columns)
119+
|_left_set = set(_left_cols)
120+
|_right_set = set(_right_cols)
121+
|_rename = {}
122+
|for _col in _right_cols:
123+
| _new = _col
124+
| _others = _right_set - {_col}
125+
| while _new in _left_set or _new in _others:
126+
| _new = _new + "#@1"
127+
| _rename[_col] = _new
128+
|out1df = in1df.merge(in2df.rename(columns=_rename), how="cross").reset_index(drop=True)""".stripMargin
129+
}
106130
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/hashJoin/HashJoinOpDesc.scala‎

Lines changed: 141 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -23,20 +23,21 @@ import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchemaTitle}
2424
import org.apache.texera.amber.core.executor.OpExecWithClassName
2525
import org.apache.texera.amber.core.state.NoLoopVariable
26-
import org.apache.texera.amber.core.tuple.{Attribute, Schema}
26+
import org.apache.texera.amber.core.tuple.{Attribute, AttributeType, Schema}
2727
import org.apache.texera.amber.core.virtualidentity.{
2828
ExecutionIdentity,
2929
PhysicalOpIdentity,
3030
WorkflowIdentity
3131
}
3232
import org.apache.texera.amber.core.workflow._
33-
import org.apache.texera.amber.operator.LogicalOp
33+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator, StandaloneHelpers}
3434
import org.apache.texera.amber.operator.hashJoin.HashJoinOpDesc.HASH_JOIN_INTERNAL_KEY_NAME
3535
import org.apache.texera.amber.operator.metadata.annotations.{
3636
AutofillAttributeName,
3737
AutofillAttributeNameOnPort1
3838
}
3939
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
40+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
4041
import org.apache.texera.amber.util.JSONUtils.objectMapper
4142

4243
object HashJoinOpDesc {
@@ -54,8 +55,7 @@ object HashJoinOpDesc {
5455
}
5556
}
5657
""")
57-
class HashJoinOpDesc[K] extends LogicalOp {
58-
58+
class HashJoinOpDesc[K] extends LogicalOp with StandaloneCodeGenerator {
5959
@JsonProperty(required = true)
6060
@JsonSchemaTitle("Left Input Attribute")
6161
@JsonPropertyDescription("attribute to be joined on the Left Input")
@@ -158,19 +158,19 @@ class HashJoinOpDesc[K] extends LogicalOp {
158158
buildSchema.getAttributes.filterNot(_.getName == HASH_JOIN_INTERNAL_KEY_NAME)
159159
val leftAttributeNames = leftAttributes.map(_.getName).toSet
160160

161-
// Filter and rename attributes from the probe schema to avoid conflicts
161+
// Filter and rename attributes from the probe schema to avoid conflicts,
162+
// by the rule the executor names its fields with. Checking the left
163+
// side alone declared two columns under one name when the right side
164+
// already carried the suffixed one, and the row, matched to the schema
165+
// by name, lost the value the executor had put under the other name.
166+
val rightNames = probeSchema.getAttributeNames.filterNot(_ == probeAttributeName)
162167
val rightAttributes = probeSchema.getAttributes
163168
.filterNot(_.getName == probeAttributeName)
164169
.map { attr =>
165-
var newName = attr.getName
166-
while (leftAttributeNames.contains(newName)) {
167-
val suffixIndex = """#@(\d+)$""".r
168-
.findFirstMatchIn(newName)
169-
.map(_.group(1).toInt + 1)
170-
.getOrElse(1)
171-
newName = s"${attr.getName}#@$suffixIndex"
172-
}
173-
new Attribute(newName, attr.getType)
170+
new Attribute(
171+
JoinUtils.renamed(attr.getName, leftAttributeNames.toSeq, rightNames),
172+
attr.getType
173+
)
174174
}
175175

176176
// Combine left and right attributes into a new schema
@@ -203,4 +203,131 @@ class HashJoinOpDesc[K] extends LogicalOp {
203203
),
204204
outputPorts = List(OutputPort())
205205
)
206+
207+
// Equi-join: the probe key is dropped and a colliding right column takes
208+
// "#@1", as JoinUtils names it. Rows come out in the merge's order rather
209+
// than the engine's.
210+
/** Only the declared type can say which columns an outer join widened, so
211+
* without a schema the widening stands.
212+
*/
213+
override def generateStandaloneCode(inputSchemas: Map[PortIdentity, Schema]): String = {
214+
val integral = (a: Attribute) =>
215+
a.getType == AttributeType.INTEGER || a.getType == AttributeType.LONG
216+
val declaredIntegers = (port: PortIdentity) =>
217+
inputSchemas.get(port).map(_.getAttributes.filter(integral).map(_.getName)).getOrElse(List())
218+
// Each side keeps its own list, under the names that side's frame carries,
219+
// so a column the merge renames is still the column its own schema typed.
220+
if (joinType == JoinType.INNER) generateStandaloneCode()
221+
else
222+
generateStandaloneCode(
223+
declaredIntegers(operatorInfo.inputPorts.head.id),
224+
declaredIntegers(operatorInfo.inputPorts.last.id)
225+
)
226+
}
227+
228+
override def generateStandaloneCode(): String =
229+
generateStandaloneCode(List(), List())
230+
231+
override def standaloneHelpers(): Seq[String] = Seq(StandaloneHelpers.AttributeCasts)
232+
233+
private def generateStandaloneCode(
234+
leftIntegerColumns: List[String],
235+
rightIntegerColumns: List[String]
236+
): String = {
237+
val buildKeyLit = objectMapper.writeValueAsString(buildAttributeName)
238+
val probeKeyLit = objectMapper.writeValueAsString(probeAttributeName)
239+
val how = joinType match {
240+
case JoinType.INNER => "inner"
241+
case JoinType.LEFT_OUTER => "left"
242+
case JoinType.RIGHT_OUTER => "right"
243+
case JoinType.FULL_OUTER => "outer"
244+
}
245+
// An unmatched row leaves a hole, and a hole costs a pandas integer column
246+
// its type: int64 becomes float64, which rounds every value past 2^53
247+
// before anything can put the type back. The engine writes a null and
248+
// leaves the column INTEGER, so widen to the integer dtype that holds a
249+
// hole before the merge digs one.
250+
val widen = leftIntegerColumns.nonEmpty || rightIntegerColumns.nonEmpty
251+
val namesLit = (names: List[String]) => names.map(pyStringLiteral).mkString("[", ", ", "]")
252+
val widening =
253+
if (!widen) ""
254+
else
255+
s"""_left_ints = {_c: "Int64" for _c in ${namesLit(leftIntegerColumns)} if _c in in1df.columns}
256+
|_right_ints = {_c: "Int64" for _c in ${namesLit(rightIntegerColumns)} if _c in in2df.columns}
257+
|""".stripMargin
258+
val leftFrame = if (widen) "in1df.astype(_left_ints)" else "in1df"
259+
// Cast before the rename, because the names above are the ones the right
260+
// input declared.
261+
val rightFrame =
262+
if (widen) "in2df.astype(_right_ints).rename(columns=_rename)"
263+
else "in2df.rename(columns=_rename)"
264+
// HashJoinProbeOpExec's rename, written out: append "#@1" until the name is
265+
// free. pandas' `suffixes` appends once and then refuses the duplicate it
266+
// just made.
267+
//
268+
// JoinUtils sets the probe key aside before it renames anything, so a
269+
// payload column never has to step around the key that is about to leave.
270+
// The key itself still rides along, and it only has to land where nothing
271+
// else already sits.
272+
//
273+
// The merge joins on codes and not on the keys. The probe looks each key up
274+
// in a HashMap under Scala's ==, where a NaN finds nothing, not even another
275+
// NaN, while 0.0 meets -0.0 and a null meets a null; merge pairs NaN with
276+
// NaN. `_texera_key_codes` from StandaloneHelpers.AttributeCasts gives each
277+
// NaN a code of its own.
278+
val merge =
279+
s"""_left_cols = set(in1df.columns)
280+
|_right_cols = list(in2df.columns)
281+
|_payload = [_c for _c in _right_cols if _c != $probeKeyLit]
282+
|_payload_set = set(_payload)
283+
|_rename = {}
284+
|for _col in _payload:
285+
| _new = _col
286+
| _others = _payload_set - {_col}
287+
| while _new in _left_cols or _new in _others:
288+
| _new = _new + "#@1"
289+
| _rename[_col] = _new
290+
|_probe_key = $probeKeyLit
291+
|_taken = _left_cols | set(_rename.values())
292+
|while _probe_key in _taken:
293+
| _probe_key = _probe_key + "#@1"
294+
|if _probe_key != $probeKeyLit:
295+
| _rename[$probeKeyLit] = _probe_key
296+
|_join_key = "_texera_join_key"
297+
|while _join_key in _taken or _join_key == _probe_key:
298+
| _join_key = _join_key + "_"
299+
|_left_codes, _right_codes = _texera_key_codes(in1df[[$buildKeyLit]], in2df[[$probeKeyLit]])
300+
|""".stripMargin + widening +
301+
s"""out1df = $leftFrame.assign(**{_join_key: _left_codes}).merge(
302+
| $rightFrame.assign(**{_join_key: _right_codes}),
303+
| how=${pyStringLiteral(how)},
304+
| on=_join_key,
305+
|).drop(columns=[_join_key])""".stripMargin
306+
// `_probe_key` is whatever the rename settled on, so this drops the right
307+
// column rather than a left one that happened to share its name. Asked of
308+
// the name the rename produced and not of the two the operator was given:
309+
// when both sides name the key alike the rename still moves the right one
310+
// aside, and a test on the operator's own two names read that as one shared
311+
// column and kept the copy, where the engine emits the key once.
312+
//
313+
// Where both sides name the key alike, the engine's unmatched right row
314+
// fills the left key column with its own key. The merge left that column
315+
// empty and put the key in the renamed right one, which is about to go.
316+
// A matched row holds the same value in both, so filling the holes changes
317+
// only the rows the right side alone brought.
318+
val rightRowsKept = joinType == JoinType.RIGHT_OUTER || joinType == JoinType.FULL_OUTER
319+
val fillKey =
320+
if (rightRowsKept && buildAttributeName == probeAttributeName)
321+
s"""out1df[$buildKeyLit] = out1df[$buildKeyLit].where(
322+
| out1df[$buildKeyLit].notna(), out1df[_probe_key]
323+
|)
324+
|""".stripMargin
325+
else ""
326+
val tail =
327+
fillKey +
328+
s"""if _probe_key != $buildKeyLit:
329+
| out1df = out1df.drop(columns=[_probe_key])
330+
|out1df = out1df.reset_index(drop=True)""".stripMargin
331+
s"$merge\n$tail"
332+
}
206333
}

0 commit comments

Comments
 (0)