Skip to content

Commit 241b504

Browse files
kz930claude
andauthored
feat(workflow-operator): export the line sources and the URL fetcher as Python (apache#8705)
### What changes were proposed in this PR? File Scan, the file-scan source, Text Input and URL Fetcher implement `StandaloneCodeGenerator`. Each slices the raw lines before parsing, as the engine does, so a line outside the window is never converted. A boolean line is read as `parseField` reads one, where 1 is true and other text ends the run. A timestamp line is kept as text and parsed as a column with the helper apache#8597 adds, as the CSV readers parse theirs. Every file is read as bytes and decoded the way the engine's reader decodes it. Bytes the charset cannot hold become U+FFFD, and UTF-16 with no byte-order mark reads as big-endian. Python's codecs refuse both, so a file read under the wrong Encoding used to end the script while the engine read on. URL Fetcher is reported as unverifiable rather than exported blind. It reads a live URL, so two runs are not required to agree and a comparison would only measure the network. File Scan takes its filenames from an input port, so the verification feeds it a table that names the files to read, as it feeds a transform. That lands with apache#8364 and apache#8363. ### Any related issues, documentation, discussions? Part of apache#8325, step 12c; that issue lists the set in order. It needs apache#8341 for the shared source code and apache#8597 for the timestamp helpers, so both merge first. Closes apache#8703, the task this change is the whole of. One engine defect was found while matching these readers, and it is fixed here rather than worked around: the export alone could not close it. apache#8598, where a file scan extracting an archive could not also include the filename, because the line-by-line branch emitted a one-field row against the two-column schema its own flag declares. A second, apache#8596, where the Encoding field never reached the executor, was fixed upstream by apache#8605, and this branch takes that fix. Closes apache#8598. ### How was this PR tested? Each operator asserts the block it emits in its own spec. Where a difference is one of behaviour rather than of text, the spec runs the engine's own executor and the generated script over the same input and compares what each produced: the boolean parsing rules and the entries read out of an archive. Once the verification lands, every operator is also run both ways on every configuration its schema offers; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) --------- Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
1 parent f68ae31 commit 241b504

10 files changed

Lines changed: 1007 additions & 27 deletions

File tree

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/fetcher/URLFetcherOpDesc.scala‎

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,22 +20,33 @@
2020
package org.apache.texera.amber.operator.source.fetcher
2121

2222
import com.fasterxml.jackson.annotation.{JsonProperty, JsonPropertyDescription}
23-
import com.kjetland.jackson.jsonSchema.annotations.JsonSchemaTitle
23+
import com.kjetland.jackson.jsonSchema.annotations.{JsonSchemaInject, JsonSchemaTitle}
2424
import org.apache.texera.amber.core.executor.OpExecWithClassName
2525
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
2626
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
2727
import org.apache.texera.amber.core.workflow.{OutputPort, PhysicalOp, SchemaPropagationFunc}
28+
import org.apache.texera.amber.operator.StandaloneCodeGenerator
2829
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
2930
import org.apache.texera.amber.operator.source.SourceOperatorDescriptor
3031
import org.apache.texera.amber.util.JSONUtils.objectMapper
3132

32-
class URLFetcherOpDesc extends SourceOperatorDescriptor {
33+
class URLFetcherOpDesc extends SourceOperatorDescriptor with StandaloneCodeGenerator {
3334

35+
// No `pattern`: the reader is `java.net.URL`, which asks only that the value carry
36+
// a scheme its JVM has a handler for. That is not something a regex can state --
37+
// one excluding `www.example.com` would still pass `htp://x`, so it would advertise
38+
// a validation the field does not have. `examples` offers a realistic value without
39+
// claiming to constrain anything.
3440
@JsonProperty(required = true)
3541
@JsonSchemaTitle("URL")
3642
@JsonPropertyDescription(
3743
"Only accepts standard URL format"
3844
)
45+
@JsonSchemaInject(json = """
46+
{
47+
"examples": ["https://example.com"]
48+
}
49+
""")
3950
var url: String = _
4051

4152
@JsonProperty(required = true)
@@ -87,4 +98,38 @@ class URLFetcherOpDesc extends SourceOperatorDescriptor {
8798
outputPorts = List(OutputPort())
8899
)
89100

101+
// The generated snippet uses `urllib.request`, which the translator's shared
102+
// imports don't include. Following the per-operator convention (e.g. Split
103+
// emits `import numpy as np`), the code block prepends its own import so the
104+
// generated script is self-contained.
105+
override def generateStandaloneCode(): String = {
106+
val urlLiteral = objectMapper.writeValueAsString(url)
107+
val isUtf8 = decodingMethod == DecodingMethod.UTF_8
108+
// IOUtils.toString decodes through a reader that substitutes U+FFFD for a malformed
109+
// byte, so the executor returns a string for any body at all. Python's decode raises
110+
// instead, which turned a response the executor reads into a failed export.
111+
val valueExpr =
112+
if (isUtf8) """_content.decode("utf-8", errors="replace")""" else "_content"
113+
val buf = scala.collection.mutable.ArrayBuffer[String]()
114+
buf += "import http.client"
115+
buf += "import urllib.request"
116+
buf += s"_url = $urlLiteral"
117+
// Catch the fetch failures, not everything: the executor guards only the fetch, so
118+
// a value with no scheme stops it, and `except Exception` here would swallow that
119+
// and hand back a row instead. The two are told apart by type -- a fetch raises
120+
// OSError (URLError, HTTPError, TimeoutError) or HTTPException on a malformed
121+
// response, a missing scheme raises ValueError.
122+
// Still divergent, and not fixable this way: a scheme that merely is not RECOGNISED
123+
// ("htp://x") stops the executor but reaches Python as URLError, and the two
124+
// languages do not recognise the same schemes anyway -- Java takes `mailto:`,
125+
// urlopen does not -- so no list of schemes is right on both sides.
126+
buf += "try:"
127+
buf += " with urllib.request.urlopen(_url) as _resp:"
128+
buf += " _content = _resp.read()"
129+
buf += "except (OSError, http.client.HTTPException):"
130+
buf += """ _content = f"Fetch failed for URL: {_url}".encode("utf-8")"""
131+
buf += s"""out1df = pd.DataFrame({"URL content": [$valueExpr]})"""
132+
buf.mkString("\n")
133+
}
134+
90135
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanOpDesc.scala‎

Lines changed: 134 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,18 @@ import org.apache.texera.amber.core.workflow.{
3131
PhysicalOp,
3232
SchemaPropagationFunc
3333
}
34+
import org.apache.texera.amber.operator.StandaloneCodeGenerator
3435
import org.apache.texera.amber.operator.metadata.{OperatorGroupConstants, OperatorInfo}
3536
import org.apache.texera.amber.operator.source.SourceOperatorDescriptor
36-
import org.apache.texera.amber.operator.source.scan.FileDecodingMethod
37+
import org.apache.texera.amber.operator.source.scan.{FileAttributeType, FileDecodingMethod}
3738
import org.apache.texera.amber.operator.source.scan.text.TextSourceOpDesc
39+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
3840
import org.apache.texera.amber.util.JSONUtils.objectMapper
3941

40-
class FileScanOpDesc extends SourceOperatorDescriptor with TextSourceOpDesc {
42+
class FileScanOpDesc
43+
extends SourceOperatorDescriptor
44+
with TextSourceOpDesc
45+
with StandaloneCodeGenerator {
4146
@JsonProperty(defaultValue = "UTF_8", required = true)
4247
@JsonSchemaTitle("Encoding")
4348
var fileEncoding: FileDecodingMethod = FileDecodingMethod.UTF_8
@@ -88,4 +93,131 @@ class FileScanOpDesc extends SourceOperatorDescriptor with TextSourceOpDesc {
8893
inputPorts = List(InputPort(displayName = "Filename")),
8994
outputPorts = List(OutputPort())
9095
)
96+
97+
override def generateStandaloneCode(): String = {
98+
val col = attributeName
99+
val enc = fileEncoding.toString.replace("_", "-").toLowerCase
100+
val buf = scala.collection.mutable.ArrayBuffer[String]()
101+
102+
val isBinary =
103+
attributeType == FileAttributeType.BINARY || attributeType == FileAttributeType.LARGE_BINARY
104+
val encLit = pyStringLiteral(enc)
105+
106+
// The executor takes the row's first String field, not its first column, so a row that
107+
// carries an id ahead of the path still finds the path. Reading column 0 opened the id.
108+
// A row with no string at all makes the executor's `.get` throw, so this raises too
109+
// rather than quietly skipping the row.
110+
buf += "def _texera_file_name(row):"
111+
buf += " for _v in row:"
112+
buf += " if isinstance(_v, str):"
113+
buf += " return _v"
114+
buf += """ raise ValueError(f"no file name in row: {row!r}")"""
115+
buf += ""
116+
buf += "_rows = []"
117+
buf += "for _fn in (_texera_file_name(r) for r in in1df.itertuples(index=False)):"
118+
119+
// Every file opens binary and is decoded by `_texera_decode`, which reads the
120+
// bytes the way the engine's reader does (see that helper). A whole file keeps
121+
// its text as decoded, so a CRLF stays a CRLF. Lines go through a StringIO in
122+
// universal-newline mode, which ends a line where `open(..., "r")` does, and so
123+
// where the engine does.
124+
val readWholeOf = if (isBinary) "_f.read()" else s"_texera_decode(_f.read(), $encLit)"
125+
val linesOf = s"io.StringIO(_texera_decode(_f.read(), $encLit), newline=None)"
126+
127+
// With extract on the engine reads the files INSIDE the archive, one tuple
128+
// per entry, and skips the entries macOS adds. `zipfile` is the reader that
129+
// matches: the platform casts what it opened to a ZipArchiveInputStream, so
130+
// an archive that is not a zip fails on both sides. A directory entry is
131+
// read, not skipped, there and here alike — it yields nothing.
132+
val (indent, nameExpr, readWhole, lines) =
133+
if (extract) {
134+
buf += " with zipfile.ZipFile(_fn) as _z:"
135+
buf += " for _name in _z.namelist():"
136+
buf += """ if _name.startswith("__MACOSX"):"""
137+
buf += " continue"
138+
buf += " with _z.open(_name) as _f:"
139+
(" " * 16, "_name", readWholeOf, linesOf)
140+
} else {
141+
buf += " with open(_fn, \"rb\") as _f:"
142+
(" " * 8, "_fn", readWholeOf, linesOf)
143+
}
144+
145+
// Whatever the flag says, as the platform now reads it: every row carries the
146+
// name of the file its value came from, a line's as much as a whole file's.
147+
// See FileScanUtils.createTuplesFromFile.
148+
val emitFilename = outputFileName
149+
150+
if (attributeType.isSingle) {
151+
if (emitFilename) buf += s"${indent}_rows.append(($nameExpr, $readWhole))"
152+
else buf += s"${indent}_rows.append($readWhole)"
153+
} else {
154+
val castExpr = attributeType match {
155+
case FileAttributeType.INTEGER => "_texera_parse_int(l, 32)"
156+
case FileAttributeType.LONG => "_texera_parse_int(l, 64)"
157+
case FileAttributeType.DOUBLE => "_texera_java_double(l)"
158+
case FileAttributeType.BOOLEAN => TextSourceOpDesc.BooleanParserCall
159+
case FileAttributeType.TIMESTAMP => "l.rstrip()"
160+
case _ => """l.rstrip("\n")"""
161+
}
162+
// The slice applies to the raw lines, as the engine drops and takes
163+
// before parsing: a line outside the window is never converted, so an
164+
// unparseable one there costs nothing. Taking after dropping also keeps
165+
// a large limit from overflowing the end index. The window is per entry,
166+
// as the engine's is: it drops and takes inside the flatMap over entries.
167+
val linesExpr =
168+
if (fileScanOffset.isEmpty && fileScanLimit.isEmpty) lines
169+
else {
170+
val dropped =
171+
fileScanOffset
172+
.filter(_ > 0)
173+
.fold(s"$lines.readlines()")(o => s"$lines.readlines()[$o:]")
174+
fileScanLimit.fold(dropped)(l => s"$dropped[:${l.max(0)}]")
175+
}
176+
val row = if (emitFilename) s"($nameExpr, $castExpr)" else castExpr
177+
buf += s"${indent}_rows.extend($row for l in $linesExpr)"
178+
}
179+
180+
val colLit = pyStringLiteral(col)
181+
if (emitFilename) {
182+
buf += s"""out1df = pd.DataFrame(_rows, columns=["filename", $colLit])"""
183+
} else {
184+
buf += s"""out1df = pd.DataFrame({$colLit: _rows})"""
185+
}
186+
buf ++= TextSourceOpDesc.parseTimestampColumn(attributeType, colLit)
187+
188+
buf.mkString("\n")
189+
}
190+
191+
override def standaloneHelpers(): Seq[String] = {
192+
val isBinary =
193+
attributeType == FileAttributeType.BINARY || attributeType == FileAttributeType.LARGE_BINARY
194+
TextSourceOpDesc.lineHelpers(attributeType) ++
195+
(if (isBinary) Seq.empty else Seq(FileScanOpDesc.DecodeHelper))
196+
}
197+
198+
override def standaloneImports(): Seq[String] = {
199+
val readsLines =
200+
!attributeType.isSingle && attributeType != FileAttributeType.BINARY &&
201+
attributeType != FileAttributeType.LARGE_BINARY
202+
(if (readsLines) Seq("import io") else Seq.empty) ++
203+
(if (extract) Seq("import zipfile") else Seq.empty)
204+
}
205+
}
206+
207+
object FileScanOpDesc {
208+
209+
/** The engine decodes through Java's InputStreamReader, which puts U+FFFD in
210+
* place of any bytes it cannot decode rather than failing, and reads UTF-16
211+
* with no byte-order mark as big-endian. Python's codecs refuse both, so a file
212+
* read under the wrong encoding ended the script where the engine read on.
213+
*/
214+
val DecodeHelper: String =
215+
"""def _texera_decode(raw, encoding):
216+
| if encoding == "utf-16":
217+
| if raw[:2] == b"\xfe\xff":
218+
| return raw[2:].decode("utf-16-be", "replace")
219+
| if raw[:2] == b"\xff\xfe":
220+
| return raw[2:].decode("utf-16-le", "replace")
221+
| return raw.decode("utf-16-be", "replace")
222+
| return raw.decode(encoding, "replace")""".stripMargin
91223
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/file/FileScanSourceOpDesc.scala‎

Lines changed: 114 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,23 @@ import org.apache.texera.amber.core.executor.OpExecWithClassName
2929
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
3030
import org.apache.texera.amber.core.virtualidentity.{ExecutionIdentity, WorkflowIdentity}
3131
import org.apache.texera.amber.core.workflow.{PhysicalOp, SchemaPropagationFunc}
32+
import org.apache.texera.amber.operator.StandaloneCodeGenerator
33+
import org.apache.texera.amber.operator.StandaloneCodeGenerator.SourceFilePlaceholder
3234
import org.apache.texera.amber.operator.metadata.annotations.HideAnnotation
3335
import org.apache.texera.amber.operator.source.scan.text.TextSourceOpDesc
34-
import org.apache.texera.amber.operator.source.scan.{FileDecodingMethod, ScanSourceOpDesc}
36+
import org.apache.texera.amber.operator.source.scan.{
37+
FileAttributeType,
38+
FileDecodingMethod,
39+
ScanSourceOpDesc
40+
}
41+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
3542
import org.apache.texera.amber.util.JSONUtils.objectMapper
3643

3744
@JsonIgnoreProperties(value = Array("limit", "offset", "fileEncoding"))
38-
class FileScanSourceOpDesc extends ScanSourceOpDesc with TextSourceOpDesc {
45+
class FileScanSourceOpDesc
46+
extends ScanSourceOpDesc
47+
with TextSourceOpDesc
48+
with StandaloneCodeGenerator {
3949
@JsonProperty(defaultValue = "UTF_8", required = true)
4050
@JsonSchemaTitle("Encoding")
4151
@JsonSchemaInject(
@@ -68,6 +78,108 @@ class FileScanSourceOpDesc extends ScanSourceOpDesc with TextSourceOpDesc {
6878

6979
fileTypeName = Option("")
7080

81+
override def standaloneSourcePath(): Option[String] = fileName
82+
83+
override def generateStandaloneCode(): String = {
84+
val col = attributeName
85+
// `encoding` is the charset the panel offers, and the one the executor now
86+
// decodes with.
87+
val enc = encoding.toString.replace("_", "-").toLowerCase
88+
val colLit = pyStringLiteral(col)
89+
val encLit = pyStringLiteral(enc)
90+
val buf = scala.collection.mutable.ArrayBuffer[String]()
91+
92+
val isBinary =
93+
attributeType == FileAttributeType.BINARY || attributeType == FileAttributeType.LARGE_BINARY
94+
95+
val castExpr = attributeType match {
96+
case FileAttributeType.INTEGER => "_texera_parse_int(l, 32)"
97+
case FileAttributeType.LONG => "_texera_parse_int(l, 64)"
98+
case FileAttributeType.DOUBLE => "_texera_java_double(l)"
99+
case FileAttributeType.BOOLEAN => TextSourceOpDesc.BooleanParserCall
100+
case FileAttributeType.TIMESTAMP => "l.rstrip()"
101+
case _ => """l.rstrip("\n")"""
102+
}
103+
104+
// The slice applies to the raw lines, as the engine drops and takes before
105+
// parsing: a line outside the window is never converted, so an unparseable
106+
// one there costs nothing. Taking after dropping also keeps a large limit
107+
// from overflowing the end index. With extract on, the window is per entry,
108+
// as the engine's is: it drops and takes inside the flatMap over entries.
109+
def windowed(lines: String): String =
110+
if (fileScanOffset.isEmpty && fileScanLimit.isEmpty) lines
111+
else {
112+
val dropped =
113+
fileScanOffset.filter(_ > 0).fold(s"$lines.readlines()")(o => s"$lines.readlines()[$o:]")
114+
fileScanLimit.fold(dropped)(l => s"$dropped[:${l.max(0)}]")
115+
}
116+
117+
// Every file and entry opens binary and is decoded by the shared
118+
// `_texera_decode`, so UTF-16 without a byte-order mark reads as big-endian,
119+
// as the engine reads it. A whole file keeps its CRLF; lines go through a
120+
// universal-newline StringIO, which ends a line where the engine does.
121+
val readWholeOf = if (isBinary) "_f.read()" else s"_texera_decode(_f.read(), $encLit)"
122+
val linesOf = s"io.StringIO(_texera_decode(_f.read(), $encLit), newline=None)"
123+
124+
// Whatever the flag says, as the platform now reads it: every row carries
125+
// the name of the file its value came from, a line's as much as a whole
126+
// file's. See FileScanUtils.createTuplesFromFile.
127+
val emitFilename = outputFileName
128+
129+
if (extract) {
130+
// The engine reads the files INSIDE the archive, one tuple per entry, and
131+
// skips the entries macOS adds. `zipfile` is the reader that matches: the
132+
// platform casts what it opened to a ZipArchiveInputStream, so an archive
133+
// that is not a zip fails on both sides. A directory entry is read, not
134+
// skipped, there and here alike — it yields nothing. The filename column
135+
// carries the ENTRY's name, which is what the engine puts there.
136+
buf += "_rows = []"
137+
buf += s"with zipfile.ZipFile($SourceFilePlaceholder) as _z:"
138+
buf += " for _name in _z.namelist():"
139+
buf += """ if _name.startswith("__MACOSX"):"""
140+
buf += " continue"
141+
buf += " with _z.open(_name) as _f:"
142+
if (attributeType.isSingle) {
143+
if (emitFilename) buf += s" _rows.append((_name, $readWholeOf))"
144+
else buf += s" _rows.append($readWholeOf)"
145+
} else {
146+
val linesExpr = windowed(linesOf)
147+
val row = if (emitFilename) s"(_name, $castExpr)" else castExpr
148+
buf += s" _rows.extend($row for l in $linesExpr)"
149+
}
150+
if (emitFilename) buf += s"""out1df = pd.DataFrame(_rows, columns=["filename", $colLit])"""
151+
else buf += s"""out1df = pd.DataFrame({$colLit: _rows})"""
152+
} else if (attributeType.isSingle) {
153+
val dfCols =
154+
if (emitFilename) s"""{"filename": $SourceFilePlaceholder, $colLit: [$readWholeOf]}"""
155+
else s"""{$colLit: [$readWholeOf]}"""
156+
buf += s"""with open($SourceFilePlaceholder, "rb") as _f:"""
157+
buf += s""" out1df = pd.DataFrame($dfCols)"""
158+
} else {
159+
val linesExpr = windowed(linesOf)
160+
val dfCols =
161+
if (emitFilename)
162+
s"""{"filename": $SourceFilePlaceholder, $colLit: [$castExpr for l in $linesExpr]}"""
163+
else s"""{$colLit: [$castExpr for l in $linesExpr]}"""
164+
buf += s"""with open($SourceFilePlaceholder, "rb") as _f:"""
165+
buf += s""" out1df = pd.DataFrame($dfCols)"""
166+
}
167+
buf ++= TextSourceOpDesc.parseTimestampColumn(attributeType, colLit)
168+
169+
buf.mkString("\n")
170+
}
171+
172+
override def standaloneHelpers(): Seq[String] = {
173+
val isBinary =
174+
attributeType == FileAttributeType.BINARY || attributeType == FileAttributeType.LARGE_BINARY
175+
TextSourceOpDesc.lineHelpers(attributeType) ++
176+
(if (isBinary) Seq.empty else Seq(FileScanOpDesc.DecodeHelper))
177+
}
178+
179+
override def standaloneImports(): Seq[String] =
180+
(if (attributeType.isSingle) Seq.empty else Seq("import io")) ++
181+
(if (extract) Seq("import zipfile") else Seq.empty)
182+
71183
override def getPhysicalOp(
72184
workflowId: WorkflowIdentity,
73185
executionId: ExecutionIdentity

0 commit comments

Comments
 (0)