Skip to content

Commit 5db1ca0

Browse files
kz930claude
andauthored
feat(workflow-operator): export the samplers, Split and the control-flow operators as Python (apache#8710)
### What changes were proposed in this PR? The two samplers, Split, and If, Dummy and Sleep implement `StandaloneCodeGenerator`. The samplers need more than a pandas expression. A sampler decides per row whether to keep it, so which rows survive is fixed by the exact sequence `java.util.Random` produces; seeding Python's own generator with the same number selects a different set, and the script would then report a different sample than the workflow it came from. `SamplingHelpers` transcribes the generator, and the three operators that draw from one, the two samplers and Split, emit it once per script. If cannot receive the condition the workflow sets upstream, since that only exists at run time. The script takes the True branch as before, and now says so on stderr. The message names the variable to set before the step to take the False branch. Review then found two places where the transcribed generator answered a question the engine answers differently. It carried its rejection sum as a Python integer where Java lets an int overflow, and never refused the bound of zero a reservoir of nothing hands it. The edge values the shared table now carries found one more. The reservoir sampler rebuilt its output from the rows' values, so pandas inferred each column again and a timestamp past 2262 came back as an object column. It now takes the rows from the input by position. ### Any related issues, documentation, discussions? Part of apache#8325, step 11c; that issue lists the set in order. Closes apache#8708, the task this change is the whole of. ### How was this PR tested? Each operator asserts the block it emits in its own spec. What the verify tables cannot reach stays here as executed tests: the samplers' draws against `java.util.Random`, a reservoir of nothing, and the warning If prints when nothing sets its condition. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 4f36c15 commit 5db1ca0

14 files changed

Lines changed: 575 additions & 12 deletions

File tree

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
package org.apache.texera.amber.operator
21+
22+
/** The Python a sampler's standalone code needs, emitted once per script via
23+
* [[StandaloneCodeGenerator.standaloneHelpers]].
24+
*/
25+
object SamplingHelpers {
26+
27+
/**
28+
* A Python transcription of `java.util.Random`, for operators whose executor
29+
* draws from one.
30+
*
31+
* A sampler decides per row whether to keep it, so which rows survive is
32+
* fixed by the exact sequence the generator produces. Seeding Python's
33+
* `random` or numpy's with the engine's seed selects a different set, and
34+
* the script would then report a different sample than the workflow it came
35+
* from. Only the same generator gives the same rows.
36+
*/
37+
val JavaRandom: String =
38+
"""# java.util.Random, transcribed so sampling matches the engine.
39+
|class _TexeraJavaRandom:
40+
| _MASK = (1 << 48) - 1
41+
| _MULTIPLIER = 0x5DEECE66D
42+
| _ADDEND = 0xB
43+
|
44+
| def __init__(self, seed):
45+
| self._seed = (seed ^ self._MULTIPLIER) & self._MASK
46+
|
47+
| def _next(self, bits):
48+
| self._seed = (self._seed * self._MULTIPLIER + self._ADDEND) & self._MASK
49+
| value = self._seed >> (48 - bits)
50+
| return value - (1 << 32) if value >= (1 << 31) else value
51+
|
52+
| def next_double(self):
53+
| return ((self._next(26) << 27) + self._next(27)) * (2.0 ** -53)
54+
|
55+
| def next_int(self, bound):
56+
| if bound <= 0:
57+
| raise ValueError("bound must be positive")
58+
| if bound & (-bound) == bound:
59+
| return (bound * self._next(31)) >> 31
60+
| while True:
61+
| bits = self._next(31)
62+
| value = bits % bound
63+
| # Java rejects a draw by letting this sum overflow an int.
64+
| # Python would carry it and never reject.
65+
| probe = bits - value + (bound - 1)
66+
| if ((probe + (1 << 31)) % (1 << 32)) - (1 << 31) >= 0:
67+
| return value""".stripMargin
68+
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/dummy/DummyOpDesc.scala‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,9 +23,14 @@ import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
2424
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PortIdentity}
2525
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
26-
import org.apache.texera.amber.operator.{LogicalOp, PortDescription, PortDescriptor}
26+
import org.apache.texera.amber.operator.{
27+
LogicalOp,
28+
PortDescription,
29+
PortDescriptor,
30+
StandaloneCodeGenerator
31+
}
2732

28-
class DummyOpDesc extends LogicalOp with PortDescriptor {
33+
class DummyOpDesc extends LogicalOp with PortDescriptor with StandaloneCodeGenerator {
2934

3035
@JsonProperty
3136
@JsonSchemaTitle("Description")
@@ -66,4 +71,12 @@ class DummyOpDesc extends LogicalOp with PortDescriptor {
6671
allowPortCustomization = true
6772
)
6873
}
74+
75+
override def generateStandaloneCode(): String = {
76+
// Placeholder operator: pass the first input through to the output.
77+
// Multi-port configurations don't fully translate under the current
78+
// single-output placeholder scheme; downstream of extra ports would
79+
// alias the same variable.
80+
"out1df = in1df"
81+
}
6982
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/ifStatement/IfOpDesc.scala‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,12 @@ import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
2424
import org.apache.texera.amber.core.executor.OpExecWithClassName
2525
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2626
import org.apache.texera.amber.core.workflow._
27-
import org.apache.texera.amber.operator.LogicalOp
27+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator}
2828
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
29+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
2930
import org.apache.texera.amber.util.JSONUtils.objectMapper
3031

31-
class IfOpDesc extends LogicalOp {
32+
class IfOpDesc extends LogicalOp with StandaloneCodeGenerator {
3233
@JsonProperty(required = true)
3334
@JsonSchemaTitle("Condition State")
3435
@JsonPropertyDescription("name of the state variable to evaluate")
@@ -72,4 +73,33 @@ class IfOpDesc extends LogicalOp {
7273
),
7374
outputPorts = List(OutputPort(PortIdentity(), "False"), OutputPort(PortIdentity(1), "True"))
7475
)
76+
77+
// The engine flips the active output port when a State message arrives on the
78+
// Condition port. A script has no State channel, so the condition is read from
79+
// a global named after it, true when nothing set it, which is the port the
80+
// engine starts on. The Condition input carries no rows to read either way.
81+
//
82+
// Taking True silently would report the True route for a workflow whose
83+
// condition says otherwise, so an unset switch says which route it took and
84+
// how to take the other.
85+
override def standaloneImports(): Seq[String] = Seq("import sys")
86+
87+
override def generateStandaloneCode(): String = {
88+
val globalName = "_texera_if_" + Option(conditionName).getOrElse("")
89+
val globalLit = pyStringLiteral(globalName)
90+
s"""if $globalLit not in globals():
91+
| print(
92+
| "If: the script cannot receive the condition the workflow sets upstream, "
93+
| "so it takes the True branch. To take the False branch, set "
94+
| + $globalLit + " = False before this step.",
95+
| file=sys.stderr,
96+
| )
97+
|_texera_if_cond = bool(globals().get($globalLit, True))
98+
|if _texera_if_cond:
99+
| out2df = in2df.copy()
100+
| out1df = in2df.iloc[0:0].copy()
101+
|else:
102+
| out1df = in2df.copy()
103+
| out2df = in2df.iloc[0:0].copy()""".stripMargin
104+
}
75105
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/randomksampling/RandomKSamplingOpDesc.scala‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,11 +23,12 @@ import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import org.apache.texera.amber.core.executor.OpExecWithClassName
2424
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2525
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalOp}
26+
import org.apache.texera.amber.operator.{SamplingHelpers, StandaloneCodeGenerator}
2627
import org.apache.texera.amber.operator.filter.FilterOpDesc
2728
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2829
import org.apache.texera.amber.util.JSONUtils.objectMapper
2930

30-
class RandomKSamplingOpDesc extends FilterOpDesc {
31+
class RandomKSamplingOpDesc extends FilterOpDesc with StandaloneCodeGenerator {
3132

3233
@JsonProperty(value = "random k sample percentage", required = true)
3334
@JsonPropertyDescription("random k sampling with given percentage")
@@ -60,4 +61,23 @@ class RandomKSamplingOpDesc extends FilterOpDesc {
6061
outputPorts = List(OutputPort()),
6162
supportReconfiguration = true
6263
)
64+
65+
override def standaloneHelpers(): Seq[String] = Seq(SamplingHelpers.JavaRandom)
66+
67+
// A per-row Bernoulli filter drawn from the same generator, so a rerun keeps
68+
// the exact rows rather than merely the same proportion of them.
69+
//
70+
// The engine seeds it with the worker count, which makes the surviving rows a
71+
// property of the deployment. One process states the one seed it can and so
72+
// agrees with a single-worker run; a wider one splits the input and samples
73+
// each share from the start of the sequence, which no seed here reproduces.
74+
override def generateStandaloneCode(): String = {
75+
val p = percentage / 100.0
76+
s"""_texera_rks_rng = _TexeraJavaRandom(1)
77+
|_texera_rks_mask = pd.Series(
78+
| [$p >= _texera_rks_rng.next_double() for _ in range(len(in1df))],
79+
| index=in1df.index, dtype=bool
80+
|)
81+
|out1df = in1df[_texera_rks_mask].reset_index(drop=True)""".stripMargin
82+
}
6383
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/reservoirsampling/ReservoirSamplingOpDesc.scala‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,11 +23,11 @@ import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import org.apache.texera.amber.core.executor.OpExecWithClassName
2424
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2525
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalOp}
26-
import org.apache.texera.amber.operator.LogicalOp
26+
import org.apache.texera.amber.operator.{LogicalOp, SamplingHelpers, StandaloneCodeGenerator}
2727
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2828
import org.apache.texera.amber.util.JSONUtils.objectMapper
2929

30-
class ReservoirSamplingOpDesc extends LogicalOp {
30+
class ReservoirSamplingOpDesc extends LogicalOp with StandaloneCodeGenerator {
3131

3232
@JsonProperty(value = "number of item sampled in reservoir sampling", required = true)
3333
@JsonPropertyDescription("reservoir sampling with k items being kept randomly")
@@ -60,4 +60,32 @@ class ReservoirSamplingOpDesc extends LogicalOp {
6060
outputPorts = List(OutputPort())
6161
)
6262
}
63+
64+
override def standaloneHelpers(): Seq[String] = Seq(SamplingHelpers.JavaRandom)
65+
66+
// The executor runs Algorithm R (Vitter): fill the reservoir with the first k
67+
// tuples, then for tuple m+1 (m >= k) draw i = rand.nextInt(m), uniform in
68+
// [0, m), and replace reservoir[i] iff i < k.
69+
//
70+
// Its generator is seeded with the worker count, so the rows agree with a
71+
// single-worker run and not with a wider one. See RandomKSamplingOpDesc for
72+
// why no seed closes that gap.
73+
//
74+
// The reservoir holds row positions, and the rows are taken from the input
75+
// by them, so every column keeps its dtype. A frame rebuilt from the rows'
76+
// values infers each one again, and a timestamp past pandas' nanosecond
77+
// range came back as an object column.
78+
override def generateStandaloneCode(): String = {
79+
s"""_texera_rs_rng = _TexeraJavaRandom(1)
80+
|_texera_rs_k = $k
81+
|_texera_rs_reservoir = []
82+
|for _texera_rs_n in range(len(in1df)):
83+
| if _texera_rs_n < _texera_rs_k:
84+
| _texera_rs_reservoir.append(_texera_rs_n)
85+
| else:
86+
| _texera_rs_i = _texera_rs_rng.next_int(_texera_rs_n)
87+
| if _texera_rs_i < _texera_rs_k:
88+
| _texera_rs_reservoir[_texera_rs_i] = _texera_rs_n
89+
|out1df = in1df.iloc[_texera_rs_reservoir].reset_index(drop=True)""".stripMargin
90+
}
6391
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/sleep/SleepOpDesc.scala‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,11 +24,11 @@ import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
2424
import org.apache.texera.amber.core.executor.OpExecWithClassName
2525
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2626
import org.apache.texera.amber.core.workflow.{InputPort, OutputPort, PhysicalOp}
27-
import org.apache.texera.amber.operator.LogicalOp
27+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator}
2828
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2929
import org.apache.texera.amber.util.JSONUtils.objectMapper
3030

31-
class SleepOpDesc extends LogicalOp {
31+
class SleepOpDesc extends LogicalOp with StandaloneCodeGenerator {
3232

3333
@JsonProperty(required = true)
3434
@JsonSchemaTitle("Sleep Time (seconds)")
@@ -63,4 +63,10 @@ class SleepOpDesc extends LogicalOp {
6363
inputPorts = List(InputPort()),
6464
outputPorts = List(OutputPort())
6565
)
66+
67+
override def generateStandaloneCode(): String = {
68+
// JVM op sleeps between each tuple; pandas operates in batch, not row-by-row,
69+
// so per-row sleep has no equivalent. Translate as a passthrough.
70+
"out1df = in1df"
71+
}
6672
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/split/SplitOpDesc.scala‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,12 @@ import com.kjetland.jackson.jsonSchema.annotations.{
2929
import org.apache.texera.amber.core.executor.OpExecWithClassName
3030
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
3131
import org.apache.texera.amber.core.workflow._
32-
import org.apache.texera.amber.operator.LogicalOp
32+
import org.apache.texera.amber.operator.{LogicalOp, SamplingHelpers, StandaloneCodeGenerator}
3333
import org.apache.texera.amber.operator.metadata.annotations.HideAnnotation
3434
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
3535
import org.apache.texera.amber.util.JSONUtils.objectMapper
3636

37-
class SplitOpDesc extends LogicalOp {
37+
class SplitOpDesc extends LogicalOp with StandaloneCodeGenerator {
3838

3939
@JsonSchemaTitle("Split Percentage")
4040
@JsonProperty(defaultValue = "80")
@@ -99,4 +99,25 @@ class SplitOpDesc extends LogicalOp {
9999
)
100100
}
101101

102+
override def standaloneHelpers(): Seq[String] = Seq(SamplingHelpers.JavaRandom)
103+
104+
override def generateStandaloneCode(): String = {
105+
// The executor sends each tuple to the upper port iff nextInt(100) < k,
106+
// drawing from one generator per run. Reproducing that draw keeps the same
107+
// rows on the same side; the mask drives both ports, so `out1df` takes the
108+
// upper k% and `out2df` the remainder and the two partition the input.
109+
//
110+
// With "Auto-Generate Seed" the executor seeds from the clock, so that run
111+
// is not reproducible by anything, itself included — the script then seeds
112+
// from its own clock, matching the intent rather than a particular run.
113+
val seedExpr = if (random) "int(_texera_time.time() * 1000)" else seed.toString
114+
val timeImport = if (random) "import time as _texera_time\n" else ""
115+
s"""${timeImport}_texera_split_rng = _TexeraJavaRandom($seedExpr)
116+
|_texera_split_mask = pd.Series(
117+
| [_texera_split_rng.next_int(100) < $k for _ in range(len(in1df))],
118+
| index=in1df.index, dtype=bool
119+
|)
120+
|out1df = in1df[_texera_split_mask].reset_index(drop=True)
121+
|out2df = in1df[~_texera_split_mask].reset_index(drop=True)""".stripMargin
122+
}
102123
}

0 commit comments

Comments
 (0)