Skip to content

Commit 61ef208

Browse files
kz930claude
andauthored
feat(visualization): export the tables, ranges and embedded documents (apache#8366)
### What changes were proposed in this PR? Nine of them, and what they have in common is that the output is not a plot drawn from numbers: a table rendered as a figure, a range a reader moves, an image or an HTML document passed through, and the waterfall, which reads as a chart but is built by accumulating rows rather than plotting them. The waterfall plots every row and appends the total as a bar of its own, rather than consuming the last row to make one. One of these also changes what the engine does. URL Visualizer wrote the cell straight into the iframe's `src`, so a quote in it closed the attribute and left the rest of the cell standing as attributes of the iframe. Both paths escape it now, the same five characters in the same order. Image Visualizer's exported script reads a string as ready base64 only where the column was declared BINARY, which is where the executor's own `b64encode` would have accepted it. ### Any related issues, documentation, discussions? Part of apache#8325, 15 of 27; that issue lists the set in order. It needs the Plotly mixin from apache#8365, so it does not compile until that lands. Closes apache#7974. Closes apache#8418, the task this change is the whole of. ### How was this PR tested? Each operator asserts the block it emits in its own spec. All nine have also been run through the engine and through their generated scripts, on every configuration their schemas offer, on a branch that carries this change together with the verification runner from apache#8362, and the two answers agreed for each of the nine. The rows these operators add to that runner arrive with it rather than as whole new files here. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 217cbfe commit 61ef208

15 files changed

Lines changed: 984 additions & 45 deletions

File tree

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/ImageViz/ImageVisualizerOpDesc.scala‎

Lines changed: 60 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,16 +22,19 @@ package org.apache.texera.amber.operator.visualization.ImageViz
2222
import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
2424
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
25-
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext
25+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.{
26+
PythonTemplateBuilderStringContext,
27+
pyStringLiteral
28+
}
2629
import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString
2730
import org.apache.texera.amber.core.workflow.PortIdentity
28-
import org.apache.texera.amber.operator.PythonOperatorDescriptor
31+
import org.apache.texera.amber.operator.{PythonOperatorDescriptor, StandaloneCodeGenerator}
2932
import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName
3033
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
3134
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder
3235

3336
import javax.validation.constraints.NotNull
34-
class ImageVisualizerOpDesc extends PythonOperatorDescriptor {
37+
class ImageVisualizerOpDesc extends PythonOperatorDescriptor with StandaloneCodeGenerator {
3538

3639
@JsonProperty(required = true)
3740
@JsonSchemaTitle("image content column")
@@ -98,4 +101,58 @@ class ImageVisualizerOpDesc extends PythonOperatorDescriptor {
98101
finalCode.encode
99102
}
100103

104+
// Output is an HTML visualization, not a tabular DataFrame.
105+
// The translator skips it in the leaf-DataFrame print block.
106+
override def producesDataFrame(): Boolean = false
107+
108+
override def generateStandaloneCode(): String = generateStandaloneCode(Map.empty)
109+
110+
/** The executor hands `b64encode` whatever the column holds, so a column of
111+
* text ends in the reason page: `b64encode` refuses a `str`. The exported
112+
* script reads the same column out of JSONL, where a BINARY column arrives as
113+
* the base64 text of its bytes, which is already what the tag wants.
114+
*
115+
* So the text is read as an image only where the column was declared BINARY.
116+
* A column of any other type falls through to the same `b64encode` call and
117+
* fails there, which is the page the run showed.
118+
*/
119+
override def generateStandaloneCode(inputSchemas: Map[PortIdentity, Schema]): String = {
120+
val carriesBytes = inputSchemas
121+
.get(operatorInfo.inputPorts.head.id)
122+
.flatMap(_.getAttributes.find(_.getName == binaryContent))
123+
.forall(_.getType == AttributeType.BINARY)
124+
val textIsTheImage =
125+
if (carriesBytes) "isinstance(binary_image_data, str)" else "False"
126+
s"""import base64
127+
|
128+
|LT = chr(60)
129+
|GT = chr(62)
130+
|
131+
|def encode_image_to_html(binary_image_data):
132+
| try:
133+
| if $textIsTheImage:
134+
| encoded_image_str = binary_image_data
135+
| else:
136+
| encoded_image_data = base64.b64encode(binary_image_data)
137+
| encoded_image_str = encoded_image_data.decode("utf-8")
138+
| return (
139+
| f'{LT}img src="data:image;base64,{encoded_image_str}" alt="Image" '
140+
| f'style="max-width: 100vw; max-height: 90vh; width: auto; height: auto;"{GT}'
141+
| )
142+
| except Exception:
143+
| return (
144+
| f'{LT}h1{GT}Image is not available.{LT}/h1{GT}'
145+
| f'{LT}p{GT}Reason: Binary input is not valid{LT}/p{GT}'
146+
| )
147+
|
148+
|all_images_html = f"{LT}div{GT}" + "".join(
149+
| encode_image_to_html(binary_image_data)
150+
| for binary_image_data in in1df[${pyStringLiteral(binaryContent)}]
151+
|) + f"{LT}/div{GT}"
152+
|
153+
|with open(outputHtml, "w", encoding="utf-8") as output:
154+
| output.write(all_images_html)
155+
|print("Image visualizer saved to " + outputHtml)""".stripMargin
156+
}
157+
101158
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/figureFactoryTable/FigureFactoryTableOpDesc.scala‎

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,15 +22,18 @@ package org.apache.texera.amber.operator.visualization.figureFactoryTable
2222
import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2323
import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchemaTitle}
2424
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
25-
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext
25+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.{
26+
PythonTemplateBuilderStringContext,
27+
pyStringLiteral
28+
}
2629
import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString
2730
import org.apache.texera.amber.core.workflow.PortIdentity
28-
import org.apache.texera.amber.operator.PythonOperatorDescriptor
31+
import org.apache.texera.amber.operator.{PythonOperatorDescriptor, StandaloneCodeGenerator}
2932
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
3033
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder
3134

3235
import javax.validation.constraints.{DecimalMin, NotEmpty}
33-
class FigureFactoryTableOpDesc extends PythonOperatorDescriptor {
36+
class FigureFactoryTableOpDesc extends PythonOperatorDescriptor with StandaloneCodeGenerator {
3437

3538
@JsonProperty(required = false)
3639
@JsonSchemaTitle("Font Size")
@@ -44,7 +47,8 @@ class FigureFactoryTableOpDesc extends PythonOperatorDescriptor {
4447
@JsonPropertyDescription("Font color of the Figure Factory Table")
4548
@JsonSchemaInject(json = """
4649
{
47-
"pattern": "^\\s*$|^\\s*#(?:\\s*[0-9a-fA-F]){3}(?:(?:\\s*[0-9a-fA-F]){3})?\\s*$|^\\s*(?:[rR]\\s*[gG]\\s*[bB]|[hH]\\s*[sS]\\s*[lL]|[hH]\\s*[sS]\\s*[vV])(?:\\s*[aA])?\\s*\\(\\s*(?:\\s*[0-9.])+(?:\\s*%)?(?:\\s*,(?:\\s*[0-9.])+(?:\\s*%)?){2,3}\\s*\\)\\s*$|^\\s*[vV]\\s*[aA]\\s*[rR]\\s*\\(\\s*-\\s*-[^)]*\\)\\s*$|^\\s*[a-zA-Z][a-zA-Z\\s]*$"
50+
"pattern": "^\\s*$|^\\s*#(?:\\s*[0-9a-fA-F]){3}(?:(?:\\s*[0-9a-fA-F]){3})?\\s*$|^\\s*(?:[rR]\\s*[gG]\\s*[bB]|[hH]\\s*[sS]\\s*[lL]|[hH]\\s*[sS]\\s*[vV])(?:\\s*[aA])?\\s*\\(\\s*(?:\\s*[0-9.])+(?:\\s*%)?(?:\\s*,(?:\\s*[0-9.])+(?:\\s*%)?){2,3}\\s*\\)\\s*$|^\\s*[vV]\\s*[aA]\\s*[rR]\\s*\\(\\s*-\\s*-[^)]*\\)\\s*$|^\\s*[a-zA-Z][a-zA-Z\\s]*$",
51+
"examples": ["red"]
4852
}
4953
""")
5054
var fontColor: EncodableString = "#000000"
@@ -147,4 +151,39 @@ class FigureFactoryTableOpDesc extends PythonOperatorDescriptor {
147151
.add("html-content", AttributeType.STRING)
148152
Map(operatorInfo.outputPorts.head.id -> outputSchema)
149153
}
154+
155+
override def producesDataFrame(): Boolean = false
156+
157+
override def generateStandaloneCode(): String = {
158+
val attributes = columns.map(c => pyStringLiteral(c.attributeName)).mkString(", ")
159+
s"""import plotly.figure_factory as ff
160+
|
161+
|def render_error(error_msg):
162+
| return "<h1>Figure Factory Table is not available.</h1><p>Reason is: {}</p>".format(
163+
| error_msg
164+
| )
165+
|
166+
|if in1df.empty:
167+
| with open(outputHtml, "w", encoding="utf-8") as output:
168+
| output.write(render_error("input table is empty."))
169+
|else:
170+
| table = in1df.dropna(subset=[$attributes])
171+
| if table.empty:
172+
| with open(outputHtml, "w", encoding="utf-8") as output:
173+
| output.write(render_error("value column contains only non-positive numbers or nulls."))
174+
| else:
175+
| filtered_table = table[[$attributes]]
176+
| headers = filtered_table.columns.tolist()
177+
| cell_values = [filtered_table[col].tolist() for col in headers]
178+
| data = [headers] + list(map(list, zip(*cell_values)))
179+
| fig = ff.create_table(data, height_constant=$rowHeight, font_colors=[${pyStringLiteral(
180+
fontColor
181+
)}])
182+
| for i in range(len(fig.layout.annotations)):
183+
| fig.layout.annotations[i].font.size = $fontSize
184+
| fig.update_layout(margin=dict(l=0, r=0, b=0, t=0))
185+
| fig.write_json(outputJson)
186+
| fig.write_html(outputHtml)
187+
| print("Figure factory table saved to " + outputHtml)""".stripMargin
188+
}
150189
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/htmlviz/HtmlVizOpDesc.scala‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,10 @@ import org.apache.texera.amber.core.executor.OpExecWithClassName
2525
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
2626
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2727
import org.apache.texera.amber.core.workflow.{PhysicalOp, SchemaPropagationFunc}
28-
import org.apache.texera.amber.operator.LogicalOp
29-
import org.apache.texera.amber.operator.metadata.annotations.AutofillAttributeName
28+
import org.apache.texera.amber.operator.{LogicalOp, StandaloneCodeGenerator}
29+
import org.apache.texera.amber.operator.metadata.annotations.{AutofillAttributeName, SampleColumn}
3030
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
31+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
3132
import org.apache.texera.amber.util.JSONUtils.objectMapper
3233

3334
import javax.validation.constraints.NotNull
@@ -36,10 +37,11 @@ import javax.validation.constraints.NotNull
3637
* HTML Visualization operator to render any given HTML code
3738
* This is the description of the operator
3839
*/
39-
class HtmlVizOpDesc extends LogicalOp {
40+
class HtmlVizOpDesc extends LogicalOp with StandaloneCodeGenerator {
4041
@JsonProperty(required = true)
4142
@JsonSchemaTitle("HTML content")
4243
@AutofillAttributeName
44+
@SampleColumn("short_text")
4345
@NotNull(message = "HTML content cannot be empty")
4446
var htmlContentAttrName: String = ""
4547

@@ -74,4 +76,10 @@ class HtmlVizOpDesc extends LogicalOp {
7476
OperatorGroupConstants.VISUALIZATION_MEDIA_GROUP
7577
)
7678

79+
// Mirrors HtmlVizOpExec: emit one row per input row whose single
80+
// "html-content" column is the value of htmlContentAttrName, passed through
81+
// unconverted (the exec does not coerce either).
82+
override def generateStandaloneCode(): String =
83+
s"""out1df = pd.DataFrame({"html-content": in1df[${pyStringLiteral(htmlContentAttrName)}]})"""
84+
7785
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/nestedTable/NestedTableOpDesc.scala‎

Lines changed: 79 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,15 +22,16 @@ import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
2222
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
2323
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext
2424
import org.apache.texera.amber.core.workflow.PortIdentity
25-
import org.apache.texera.amber.operator.PythonOperatorDescriptor
25+
import org.apache.texera.amber.operator.{PythonOperatorDescriptor, StandaloneCodeGenerator}
2626
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2727
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder
28+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
2829

2930
import java.util
3031
import javax.validation.constraints.NotEmpty
3132
import scala.jdk.CollectionConverters.ListHasAsScala
3233

33-
class NestedTableOpDesc extends PythonOperatorDescriptor {
34+
class NestedTableOpDesc extends PythonOperatorDescriptor with StandaloneCodeGenerator {
3435

3536
@JsonPropertyDescription(
3637
"List of columns to include in the nested table chart and their subgroup"
@@ -137,4 +138,80 @@ class NestedTableOpDesc extends PythonOperatorDescriptor {
137138
|"""
138139
finalcode.encode
139140
}
141+
142+
override def producesDataFrame(): Boolean = false
143+
144+
override def generateStandaloneCode(): String = {
145+
val sortedColumns = includedColumns.asScala.sortBy(_.attributeGroup)
146+
147+
val multiIndexTuples = sortedColumns
148+
.map { config =>
149+
val name =
150+
if (config.newName != null && config.newName.nonEmpty) config.newName
151+
else config.originalName
152+
s"""(${pyStringLiteral(config.attributeGroup)}, ${pyStringLiteral(name)})"""
153+
}
154+
.mkString(",\n ")
155+
156+
val rowValues = sortedColumns
157+
.map(config => s"""row[${pyStringLiteral(config.originalName)}]""")
158+
.mkString(", ")
159+
160+
s"""import pandas as pd
161+
|import numpy as np
162+
|
163+
|def render_error(error_msg):
164+
| return '''<h1>Nested Table is not available.</h1>
165+
| <p>Reason is: {} </p>
166+
| '''.format(error_msg)
167+
|
168+
|if in1df.empty:
169+
| with open(outputHtml, "w", encoding="utf-8") as output:
170+
| output.write(render_error("input table is empty."))
171+
|else:
172+
| columns = pd.MultiIndex.from_tuples([
173+
| $multiIndexTuples
174+
| ])
175+
|
176+
| data = []
177+
| for _, row in in1df.iterrows():
178+
| data.append([
179+
| $rowValues
180+
| ])
181+
|
182+
| df = pd.DataFrame(data, columns=columns)
183+
|
184+
| styles = [
185+
| {'selector': 'th', 'props': [('background-color', '#f2f2f2'),
186+
| ('color', 'black'),
187+
| ('font-weight', 'bold'),
188+
| ('border', '1px solid #ddd'),
189+
| ('padding', '8px'),
190+
| ('text-align', 'center')]},
191+
| {'selector': 'td', 'props': [('border', '1px solid #ddd'),
192+
| ('padding', '8px'),
193+
| ('text-align', 'center')]},
194+
| {'selector': 'caption', 'props': [('caption-side', 'top'),
195+
| ('font-size', '16pt'),
196+
| ('font-weight', 'bold'),
197+
| ('text-align', 'left'),
198+
| ('padding', '10px')]},
199+
| {'selector': '.row_heading', 'props': [('text-align', 'left'),
200+
| ('font-weight', 'normal')]},
201+
| {'selector': '.blank.level0', 'props': [('display', 'none')]}
202+
| ]
203+
|
204+
| styled_table = (
205+
| df.style
206+
| .set_table_styles(styles)
207+
| .format(precision=2, na_rep="")
208+
| .set_table_attributes('class="dataframe"')
209+
| .hide(axis="index")
210+
| )
211+
|
212+
| html = styled_table.to_html()
213+
| with open(outputHtml, "w", encoding="utf-8") as output:
214+
| output.write(html)
215+
| print("Nested table saved to " + outputHtml)""".stripMargin
216+
}
140217
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/visualization/parallelCoordinatesPlot/ParallelCoordinatesPlotOpDesc.scala‎

Lines changed: 40 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,14 +24,18 @@ import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchema
2424
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
2525
import org.apache.texera.amber.core.workflow.PortIdentity
2626
import org.apache.texera.amber.operator.PythonOperatorDescriptor
27+
import org.apache.texera.amber.operator.visualization.PlotlyStandaloneCode
2728
import org.apache.texera.amber.operator.metadata.annotations.{
2829
AutofillAttributeName,
2930
AutofillAttributeNameList
3031
}
3132
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
3233
import org.apache.texera.amber.pybuilder.PyStringTypes.EncodableString
3334
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder
34-
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.PythonTemplateBuilderStringContext
35+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.{
36+
PythonTemplateBuilderStringContext,
37+
pyStringLiteral
38+
}
3539

3640
import javax.validation.constraints.{NotNull, Size}
3741

@@ -50,7 +54,7 @@ import javax.validation.constraints.{NotNull, Size}
5054
}
5155
}
5256
""")
53-
class ParallelCoordinatesPlotOpDesc extends PythonOperatorDescriptor {
57+
class ParallelCoordinatesPlotOpDesc extends PythonOperatorDescriptor with PlotlyStandaloneCode {
5458

5559
@JsonProperty(value = "dimensions", required = true)
5660
@JsonSchemaTitle("Dimensions")
@@ -134,4 +138,38 @@ class ParallelCoordinatesPlotOpDesc extends PythonOperatorDescriptor {
134138
|"""
135139
finalcode.encode
136140
}
141+
142+
override def producesDataFrame(): Boolean = false
143+
144+
override def generateStandaloneCode(): String = {
145+
val dimCols = dimensions.map(pyStringLiteral).mkString("[", ", ", "]")
146+
val colorLit = pyStringLiteral(color)
147+
val colorFilter =
148+
if (color != null && color.nonEmpty) s""" & (in1df[$colorLit].notnull())""" else ""
149+
val colorArg =
150+
if (color != null && color.nonEmpty) s""", color=$colorLit""" else ""
151+
152+
s"""def render_error(error_msg):
153+
| return '''<h1>Parallel coordinates plot is not available.</h1>
154+
| <p>Reason is: {} </p>
155+
| '''.format(error_msg)
156+
|
157+
|if in1df.empty:
158+
| with open(outputHtml, "w", encoding="utf-8") as output:
159+
| output.write(render_error("Input table is empty."))
160+
|else:
161+
| table = in1df[in1df[$dimCols].notnull().all(axis=1)$colorFilter].copy()
162+
| if table.empty:
163+
| with open(outputHtml, "w", encoding="utf-8") as output:
164+
| output.write(render_error("No valid rows after filtering."))
165+
| else:
166+
| fig = px.parallel_coordinates(
167+
| table,
168+
| dimensions=$dimCols$colorArg
169+
| )
170+
| fig.write_json(outputJson)
171+
| fig.write_html(outputHtml)
172+
| print("Parallel coordinates plot saved to " + outputHtml)""".stripMargin
173+
}
174+
137175
}

0 commit comments

Comments
 (0)