Skip to content

Commit 959939d

Browse files
committed
test: bind number and true/false loop variables in generated Python operators
Inside a control block, a number or true/false property of an operator whose Scala descriptor generates Python, such as Radar Chart's Fill Opacity or Radar Plot's Show Legend, is written into the code before the loop runs, so the compiler rejects a reference such as $o there today. These tests pin how such a reference is read when the code runs: - PythonOperatorDescriptorSpec and WorkflowCompilerSpec: the code is generated from probe values in the reference's place, and where the value lands as it is, the code calls self.loop_variable_value(name, kind) instead: 'integer' for an int property, 'long' for a long one. A value the descriptor checks or adjusts (Histogram2D's bins, Hugging Face's limits, Figure Factory Table's Row Height and Font Size), one inside a text, a comment or a name, one the code computes with, never holds or reads when the operator starts, and one that decides the output schema stay compile errors that name the property and why, never a probe value the user did not type. The compile probes the code once and decides every reason from that binding: the code's reason before the schema's, and a schema reason only once every input has a schema. - test_operator.py: loop_variable_value reads the values bound at the operator's first tuple and converts them as StateReferenceBinding does: trimmed as String.trim, then new BigDecimal(text) .longValueExact() (within 32 bits for an int property, as Jackson writes it), Double.parseDouble, or true/false by equalsIgnoreCase. The parity tables are generated on JDK 17 through the compiled State.fromJson and StateReferenceBinding.coerce, and through a Limit's int property for the 32-bit bound. Binding fails for a required value its property cannot take, as the JVM's binding does. - test_executor_manager.py: binding requires the values the generated code reads, by name and kind. - LoopIntegrationSpec: Radar Chart's opacity and Radar Plot's legend follow the loop variable on every iteration; each legend is matched to its iteration by the plot's trace count. Before the next commit the Scala specs do not compile (there is no loopVariableBinding), the new Python tests fail with AttributeError or on the values they read, and the two new LoopIntegrationSpec cases fail to compile their workflow.
1 parent 8f83171 commit 959939d

5 files changed

Lines changed: 1666 additions & 20 deletions

File tree

‎amber/src/test/integration/org/apache/texera/amber/engine/e2e/LoopIntegrationSpec.scala‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -457,6 +457,86 @@ class LoopIntegrationSpec
457457
)
458458
}
459459

460+
/** Run the loop workflow and return the charts (html-content) the Loop End `end` materialized. */
461+
private def runAndGetCharts(
462+
operators: List[LogicalOp],
463+
links: List[LogicalLink],
464+
end: LogicalOp
465+
): List[String] =
466+
runWorkflowAndReadResults(
467+
system,
468+
buildWorkflow(operators, links, materializedContext()),
469+
List(end.operatorIdentifier),
470+
_.get().toList,
471+
Duration.fromSeconds(90)
472+
).getOrElse(end.operatorIdentifier, Nil).map(_.getField[String]("html-content"))
473+
474+
/** Each number or true/false value of `key` in a chart's figure JSON, in order. */
475+
private def figureValues(chart: String, key: String): List[String] =
476+
s""""$key":([0-9A-Za-z.+-]+)""".r.findAllMatchIn(chart).map(_.group(1)).toList
477+
478+
it should "read a $o loop-variable reference in a Python-generated operator's number property on every iteration" in {
479+
// Sales CSV -> LoopStart(o = 0.25 / table) -> Radar Chart(fill opacity = $o)
480+
// -> LoopEnd(o += 0.25 / o <= 0.75).
481+
//
482+
// The chart's Python is generated before the loop runs, from probe values in the opacity's
483+
// place. It writes the opacity into the code as it is, so inside the block the compiler had
484+
// the code read it from the iteration's state message instead
485+
// (self.loop_variable_value('o', 'number')). Each iteration draws one chart, every trace of
486+
// which carries that iteration's o, and the LoopEnd accumulates the 3 charts. The opacity the
487+
// code was generated with would draw every chart alike; an o that never rebinds to the new
488+
// value would repeat one.
489+
val src = TestOperators.smallCsvScanOpDesc()
490+
val start = loopStart("o = 0.25", "table")
491+
val chart = objectMapper.readValue(
492+
"""{"operatorType":"RadarChart","nameColumn":"Region",
493+
|"valueColumns":["Unit Price","Unit Cost"],"fillOpacity":"$o"}""".stripMargin,
494+
classOf[LogicalOp]
495+
)
496+
val end = loopEnd("o += 0.25", "o <= 0.75")
497+
val opacities = runAndGetCharts(
498+
List(src, start, chart, end),
499+
List(link(src, start), link(start, chart), link(chart, end)),
500+
end
501+
).map(figureValues(_, "opacity").distinct).sortBy(_.mkString)
502+
assert(
503+
opacities == List(List("0.25"), List("0.5"), List("0.75")),
504+
s"each iteration's chart must use its own o as the opacity: got $opacities"
505+
)
506+
}
507+
508+
it should "read a $legend loop-variable reference in a Python-generated operator's true/false property on every iteration" in {
509+
// Sales CSV -> LoopStart(i = 0, legend = True / table.head(i + 1))
510+
// -> Radar Plot(show legend = $legend) -> LoopEnd(i += 1, legend = not legend / i < 2).
511+
//
512+
// The plot's Python holds show_legend = True or False, the property as Python writes it; inside
513+
// the block the code reads it from the iteration's state message instead
514+
// (self.loop_variable_value('legend', 'boolean')). Iteration i plots i + 1 rows, one trace
515+
// each, so a plot's trace count (its "theta" lists; plotly's template names the trace type
516+
// once more) tells which iteration drew it: the first iteration's plot shows its legend and
517+
// the second's does not. The placeholder false would hide both legends, a legend that never
518+
// rebinds to the new value would show both, and a value read the wrong way round would swap
519+
// them.
520+
val src = TestOperators.smallCsvScanOpDesc()
521+
val start = loopStart("i = 0\nlegend = True", "table.head(i + 1)")
522+
val plot = objectMapper.readValue(
523+
"""{"operatorType":"RadarPlot","selectedAttributes":["Unit Price","Unit Cost"],
524+
|"linePattern":"solid","showLegend":"$legend"}""".stripMargin,
525+
classOf[LogicalOp]
526+
)
527+
val end = loopEnd("i += 1\nlegend = not legend", "i < 2")
528+
val legends = runAndGetCharts(
529+
List(src, start, plot, end),
530+
List(link(src, start), link(start, plot), link(plot, end)),
531+
end
532+
).map(chart => "\"theta\":".r.findAllMatchIn(chart).size -> figureValues(chart, "showlegend"))
533+
.sortBy(_._1)
534+
assert(
535+
legends == List(1 -> List("true"), 2 -> List("false")),
536+
s"each iteration's plot, by its trace count, must use its own legend: got $legends"
537+
)
538+
}
539+
460540
it should "bind a $n loop-variable reference in a JVM operator's number setting on every iteration" in {
461541
// TextInput(3 rows) -> LoopStart(n = 1 / table) -> Limit($n)
462542
// -> LoopEnd(n += 1 / n <= len(table)).

‎amber/src/test/python/core/architecture/managers/test_executor_manager.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,21 @@ def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]
7171
}
7272
"""
7373

74+
# Code generated where numeric and boolean properties refer to loop variables
75+
# too: an integer property twice, a number one, and a text one.
76+
GENERATED_VALUE_LOOKUPS_CODE = """
77+
from pytexera import *
78+
79+
class ProcessTableOperator(UDFTableOperator):
80+
def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]:
81+
yield {
82+
"x": self.loop_variable_value('n', 'integer'),
83+
"y": self.loop_variable_value('n', 'integer'),
84+
"opacity": self.loop_variable_value('alpha', 'number'),
85+
"title": self.loop_variable_text('t'),
86+
}
87+
"""
88+
7489
# The same reads in a generated source operator, which binds before it produces.
7590
GENERATED_SOURCE_LOOKUPS_CODE = """
7691
from pytexera import *
@@ -364,6 +379,30 @@ def test_binding_requires_the_loop_variables_the_code_reads(
364379
executor.register_state(State({"t": "z"}))
365380
assert executor.loop_variable_text("t") == "x"
366381

382+
def test_binding_requires_the_loop_variable_values_the_code_reads(
383+
self, executor_manager
384+
):
385+
# The numeric and boolean properties' references too: each once, by
386+
# name and kind, after the text ones.
387+
executor_manager.initialize_executor(
388+
code=GENERATED_VALUE_LOOKUPS_CODE, is_source=False, language="python"
389+
)
390+
executor = executor_manager.executor
391+
executor.register_state(State({"i": 1}))
392+
with pytest.raises(RuntimeError) as exc_info:
393+
executor.bind_loop_variables()
394+
assert str(exc_info.value) == "; ".join(
395+
f"the operator refers to loop variable ${name}, but no state message "
396+
"carried it before the operator's first tuple or the end of an input"
397+
for name in ("t", "alpha", "n")
398+
)
399+
executor.register_state(State({"t": "x", "alpha": 0.5, "n": 3000000000}))
400+
with pytest.raises(RuntimeError, match="does not fit its property"):
401+
executor.bind_loop_variables()
402+
executor.register_state(State({"n": 3}))
403+
executor.bind_loop_variables()
404+
assert executor.loop_variable_value("n", "integer") == 3
405+
367406
def test_tmp_dir_is_reclaimed_when_manager_is_abandoned(self):
368407
# `fs` reclaimed the tmp directory on GC (FS.__del__ -> TempFS.close()
369408
# -> clean()). A manager dropped without close() must still not leak,

0 commit comments

Comments
 (0)