Skip to content

Commit 8dcdf05

Browse files
kz930claude
andauthored
feat(workflow-operator): export the CSV sources as Python (apache#8341)
### What changes were proposed in this PR? The three CSV source operators (CSV, CSV Old and Parallel CSV) implement `StandaloneCodeGenerator`, so an exported script starts from the same data the workflow did rather than from a variable the reader has to fill in. They also carry what every source after them uses: the placeholder a source writes where its file is named, and the parse that reads a timestamp column from text. A source is the one place where the script cannot simply repeat what the operator does. The engine resolves a dataset through Texera's storage and hands the operator a URI; a script has no such resolver, so it reads the file from its own directory. Each source writes a placeholder where its file should be named and declares the path it reads, and the translator assigns the names once it can see the whole plan: the last segment of the path where that is free, and a numbered form of it where two sources read different files whose paths end the same way. The last segment rather than a parse of the whole string, because the resolver percent-encodes the file-relative parts but leaves the repository and version names as the user typed them, and a dataset version with a space in its name makes `new URI` throw before any code is generated. That is what makes an exported script portable, and it is equally its one precondition: the data has to sit beside the script, under the names the script states. A timestamp is read as the engine reads one. The engine hands each field to DateParserUtils, so a row states its own format, any year reads, an offset is moved to the local zone, and the reading is cut to the millisecond. pandas' date parsing does none of that. So each CSV reader reads the column as text and converts it with the helper apache#8597 adds. The end of an exported scan window is counted in Long: two Ints the operator accepts can add up past what an Int holds, and the slice came out negative. Two of the changes made in review are deliberately not the literal instruction, because the three CSV readers do not agree with each other about a blank field. Old CSV sets `keep_default_na=False` but names no missing value at all: scala-csv hands a blank back as the empty string rather than as a null, so the `na_values` the main CSV source needs would null a value this reader keeps. Parallel CSV asks for such a column as text rather than as a nullable integer: its schema is read with scala-csv, where one blank types the whole column as text, while its executor nulls the blank and parses the rest as that text. Each fix follows the reader it belongs to rather than the one next to it. ### Any related issues, documentation, discussions? Part of apache#8325, step 12a; apache#8702 and apache#8703 follow. That issue lists the set in order. It needs apache#8327 for the trait, so it does not compile until that lands, and the rows these operators add to the verification runner follow with the harness rather than as whole new files here. It also needs apache#8597, whose helpers the timestamp reading calls, so apache#8597 merges first. Closes apache#8415, the task this change is the whole of. One engine defect was found while matching these readers, and it is fixed here rather than worked around: the export alone could not close it. apache#8602, where a scan window of no rows left the type inference nothing to read, so CSV declared a schema of no columns and CSV Old threw. Its JSONL half follows in apache#8702, which closes it. ### How was this PR tested? Each operator asserts the block it emits in its own spec. Where a difference is one of behaviour rather than of text, the spec runs the engine's own executor and the generated script over the same file and compares what each produced: the row window the parallel reader ignores, and the nullable long each reader has to keep exact. Once the verification lands, every operator is also run both ways on every configuration its schema offers; this branch is cut from main and does not carry that machinery, so those runs are not on this diff's CI. ### Was this PR authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 5) --------- Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
1 parent 4bad5e4 commit 8dcdf05

11 files changed

Lines changed: 1180 additions & 37 deletions

File tree

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/StandaloneCodeGenerator.scala‎

Lines changed: 130 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ package org.apache.texera.amber.operator
2121

2222
import org.apache.texera.amber.core.tuple.{AttributeType, Schema}
2323
import org.apache.texera.amber.core.workflow.PortIdentity
24+
import org.apache.texera.amber.pybuilder.PythonTemplateBuilder.pyStringLiteral
2425

2526
import java.net.URLDecoder
2627
import java.nio.charset.StandardCharsets
@@ -67,21 +68,34 @@ trait StandaloneCodeGenerator {
6768
}
6869

6970
/**
70-
* The file's own name, for a script that reads it from its own directory
71-
* rather than through Texera's resolved URI.
71+
* The file this operator reads, as Texera resolved it, or None for one that
72+
* reads no file.
73+
*
74+
* The script cannot open a resolved URI, so the body writes
75+
* [[StandaloneCodeGenerator.SourceFilePlaceholder]] where the file should be
76+
* named and the translator puts a name there. The name has to come from the
77+
* plan: two sources reading different files whose paths end in the same
78+
* segment both asked for `data.csv`, and the script read one of them twice.
79+
*/
80+
def standaloneSourcePath(): Option[String] = None
81+
82+
/**
83+
* The name to offer for [[standaloneSourcePath]], before the plan has had a
84+
* chance to say whether another source already wants it.
7285
*
7386
* Taken from the last path segment instead of by parsing the whole string as a
7487
* URI: the resolver percent-encodes the file-relative segments but leaves the
7588
* repository and version names as the user typed them, so a dataset version
7689
* called `v3 - with long text` makes `new URI` throw on the space and no code
7790
* is generated at all.
7891
*/
79-
protected def sourceBasename(rawPath: String): String = {
80-
val segment = rawPath.split("/").lastOption.getOrElse("")
81-
// Percent-decoding only, matching what `URI.getPath` used to return here: form
82-
// decoding would also turn a literal `+` in a file name into a space.
83-
URLDecoder.decode(segment.replace("+", "%2B"), StandardCharsets.UTF_8)
84-
}
92+
final def standaloneSourceName(): Option[String] =
93+
standaloneSourcePath().map { rawPath =>
94+
val segment = rawPath.split("/").lastOption.getOrElse("")
95+
// Percent-decoding only, matching what `URI.getPath` used to return here: form
96+
// decoding would also turn a literal `+` in a file name into a space.
97+
URLDecoder.decode(segment.replace("+", "%2B"), StandardCharsets.UTF_8)
98+
}
8599

86100
def producesDataFrame(): Boolean = true
87101

@@ -110,3 +124,111 @@ trait StandaloneCodeGenerator {
110124
*/
111125
def standaloneImports(): Seq[String] = Seq.empty
112126
}
127+
128+
object StandaloneCodeGenerator {
129+
130+
/**
131+
* What a source writes where the file it reads should be named.
132+
*
133+
* A bare identifier rather than a string literal, because the translator only
134+
* rewrites the code parts of a body and leaves literals and comments alone.
135+
*/
136+
val SourceFilePlaceholder: String = "sourceFile"
137+
138+
/** Python that gives an empty read the columns and types its schema declares.
139+
*
140+
* With no row there is nothing to infer a type from, so pandas leaves every
141+
* column an object, and a JSONL read of no lines has no columns at all. The
142+
* executor still declares the schema it inferred from the file, so the next
143+
* step looks for those columns in those types. A read with rows is left alone.
144+
*/
145+
def typeAnEmptyRead(frame: String, schema: Schema): String = {
146+
val names = schema.getAttributes.map(a => pyStringLiteral(a.getName))
147+
val dtypes = schema.getAttributes.flatMap { a =>
148+
emptyDtypes.get(a.getType).map(d => s"""${pyStringLiteral(a.getName)}: "$d"""")
149+
}
150+
s"""if $frame.empty:
151+
| $frame = $frame.reindex(columns=[${names.mkString(", ")}]).astype({${dtypes
152+
.mkString(", ")}})""".stripMargin
153+
}
154+
155+
/** Python that gives a read the schema's column names and reads each column the way the
156+
* engine does, with every name and its type stated once, in a `_schema` dict.
157+
*
158+
* A TIMESTAMP column goes through `_texera_text_to_timestamp` and a DOUBLE one through
159+
* `_texera_read_doubles`, both from [[StandaloneHelpers.AttributeCasts]], which the caller
160+
* declares. The loop names only the readers the schema needs, so a table with no such
161+
* column leaves the helper out of the script. An empty read takes the declared types.
162+
*/
163+
def readAsDeclared(frame: String, schema: Schema): String = {
164+
val attributes = schema.getAttributes
165+
val entries = attributes.map { a =>
166+
s"""${pyStringLiteral(a.getName)}: "${emptyDtypes.getOrElse(a.getType, "object")}""""
167+
}
168+
val readers = Seq(
169+
AttributeType.TIMESTAMP -> "_texera_text_to_timestamp",
170+
AttributeType.DOUBLE -> "_texera_read_doubles"
171+
).filter { case (t, _) => attributes.exists(_.getType == t) }
172+
val branches = readers.zipWithIndex.map {
173+
case ((t, reader), i) =>
174+
s""" ${if (i == 0) "if" else "elif"} _t == "${emptyDtypes(t)}":
175+
| $frame[_c] = $reader($frame[_c])""".stripMargin
176+
}
177+
val loop =
178+
if (branches.isEmpty) Seq.empty
179+
else Seq("for _c, _t in _schema.items():") ++ branches
180+
(Seq(s"_schema = {${entries.mkString(", ")}}", s"$frame.columns = list(_schema)") ++ loop ++
181+
Seq(s"if $frame.empty:", s" $frame = $frame.astype(_schema)")).mkString("\n")
182+
}
183+
184+
/** Python that reads each TIMESTAMP column of `schema`, held as text, the way
185+
* the engine reads it, with `_texera_text_to_timestamp` from
186+
* [[StandaloneHelpers.AttributeCasts]], which the caller declares.
187+
*
188+
* pandas parses a date column into nanoseconds, which end in 2262, and left a
189+
* later year as text. The engine hands each field to DateParserUtils, so a
190+
* row states its own format, any year reads, and the reading is cut to the
191+
* millisecond.
192+
*/
193+
def parseTimestamps(frame: String, schema: Schema): String =
194+
readEach(frame, schema, AttributeType.TIMESTAMP, "_texera_text_to_timestamp")
195+
196+
/** Python that reads each DOUBLE column of `schema` that pandas left as text
197+
* the way the engine reads it, with `_texera_read_doubles` from
198+
* [[StandaloneHelpers.AttributeCasts]], which the caller declares.
199+
*
200+
* The engine reads a field with `toDouble`, so "NaN", "2d" and "0x1p3" are
201+
* numbers to it, and the column was inferred DOUBLE. pandas' float parser
202+
* takes none of them and kept the whole column as strings.
203+
*/
204+
def parseDoubles(frame: String, schema: Schema): String =
205+
readEach(frame, schema, AttributeType.DOUBLE, "_texera_read_doubles")
206+
207+
// Python that passes each column of `attributeType` through `reader`: one line
208+
// for a single column, and a loop over the names for several, so a wide table
209+
// does not spend a line per column.
210+
private def readEach(
211+
frame: String,
212+
schema: Schema,
213+
attributeType: AttributeType,
214+
reader: String
215+
): String =
216+
schema.getAttributes
217+
.filter(_.getType == attributeType)
218+
.map(a => pyStringLiteral(a.getName)) match {
219+
case Seq() => ""
220+
case Seq(name) => s"$frame[$name] = $reader($frame[$name])"
221+
case names =>
222+
s"""for _c in [${names.mkString(", ")}]:
223+
| $frame[_c] = $reader($frame[_c])""".stripMargin
224+
}
225+
226+
private val emptyDtypes: Map[AttributeType, String] = Map(
227+
AttributeType.INTEGER -> "Int32",
228+
AttributeType.LONG -> "Int64",
229+
AttributeType.DOUBLE -> "float64",
230+
AttributeType.BOOLEAN -> "boolean",
231+
AttributeType.TIMESTAMP -> "datetime64[us]",
232+
AttributeType.STRING -> "object"
233+
)
234+
}

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/StandaloneHelpers.scala‎

Lines changed: 89 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -40,22 +40,31 @@ object StandaloneHelpers {
4040
* Refusing is still part of the contract where the engine refuses: text with
4141
* no leading digits raises, and a script that quietly wrote NaN instead
4242
* would report an answer the run it was exported from never reached.
43+
*
44+
* - `_texera_java_trim`: `String.trim`, dropping only the characters up to U+0020.
45+
* - `_texera_parse_int`: `Integer.parseInt` or `Long.parseLong`, a sign and digits in range.
46+
* - `_texera_java_double`: `Double.parseDouble`, with a d/f suffix and hex, refusing the rest.
47+
* - `_texera_double_key`: `Double.compareTo`'s order, -0.0 below 0.0 and NaN above all.
48+
* - `_texera_key_codes`: one code per row's key under Scala's ==, shared across the frames
49+
* given, where a null meets a null, 0.0 meets -0.0 and each NaN stands alone. With
50+
* `signed`, a double is keyed with its sign, as the set operators' `HashSet[Tuple]`
51+
* tells 0.0 from -0.0 and a double from an integer.
52+
* - `_texera_read_doubles`: a DOUBLE column pandas left as text, read the engine's way.
4353
*/
4454
val AttributeCasts: String =
4555
"""# AttributeTypeUtils, transcribed so a cast answers as the engine does.
4656
|def _texera_cast_boolean(x):
4757
| # toBoolean first, then `toInt == 1`: "0" and "2" are both false.
4858
| if isinstance(x, str):
49-
| text = x.strip()
59+
| text = _texera_java_trim(x)
5060
| lowered = text.lower()
5161
| if lowered == "true":
5262
| return True
5363
| if lowered == "false":
5464
| return False
55-
| return int(text) == 1
65+
| return _texera_parse_int(text, 32) == 1
5666
| return x != 0
5767
|
58-
|
5968
|def _texera_parse_number(text):
6069
| # java.text.NumberFormat for Locale.US, which a cast reaches through
6170
| # `parseField(force = true)`. Lenient where Python's int is not: it stops
@@ -81,15 +90,13 @@ object StandaloneHelpers {
8190
| return value, False
8291
| return float(value), True
8392
|
84-
|
8593
|def _texera_long_value(value, is_double):
8694
| # Number.longValue(): a Double truncates toward zero and saturates at
8795
| # long's bounds, where a Long is already itself.
8896
| if is_double:
8997
| return max(-(2 ** 63), min(2 ** 63 - 1, int(value)))
9098
| return value
9199
|
92-
|
93100
|def _texera_cast_integral(x):
94101
| # The LONG target. Scala's toLong takes no decimal point of its own, so
95102
| # a fraction only ever arrives already parsed.
@@ -101,7 +108,6 @@ object StandaloneHelpers {
101108
| return _texera_long_value(x, True)
102109
| return int(x)
103110
|
104-
|
105111
|def _texera_cast_int32(x, wrap):
106112
| # The two 32-bit narrowings differ: Long.toInt keeps the low bits, so
107113
| # 2147483648 comes back as -2147483648, while Double.toInt saturates.
@@ -121,7 +127,6 @@ object StandaloneHelpers {
121127
| # int() first: Double.intValue truncates toward zero before it clamps.
122128
| return max(-2147483648, min(2147483647, int(value)))
123129
|
124-
|
125130
|def _texera_epoch_millis_to_timestamp(s):
126131
| # `new Timestamp(long)` reads MILLISECONDS where pd.to_datetime defaults
127132
| # to nanoseconds, and renders in the JVM's default zone, so leaving the
@@ -137,12 +142,85 @@ object StandaloneHelpers {
137142
| .dt.tz_localize(None)
138143
| )
139144
|
140-
|
141145
|def _texera_cast_double(x):
142-
| if isinstance(x, str):
143-
| return float(x.strip())
144-
| return float(x)
146+
| return _texera_java_double(x) if isinstance(x, str) else float(x)
145147
|
148+
|def _texera_java_trim(text):
149+
| return text.strip("".join(map(chr, range(33))))
150+
|
151+
|def _texera_parse_int(text, bits):
152+
| import re
153+
|
154+
| text = _texera_java_trim(text)
155+
| bound = 2 ** (bits - 1)
156+
| if not re.fullmatch(r"[+-]?\d+", text) or not -bound <= int(text) < bound:
157+
| raise ValueError("For input string: " + repr(text))
158+
| return int(text)
159+
|
160+
|def _texera_java_double(text):
161+
| import re
162+
|
163+
| text = _texera_java_trim(text)
164+
| if not re.fullmatch(
165+
| r"[+-]?(NaN|Infinity|((\d+\.?\d*|\.\d+)([eE][+-]?\d+)?"
166+
| r"|0[xX]([0-9a-fA-F]+\.?|[0-9a-fA-F]*\.[0-9a-fA-F]+)[pP][+-]?\d+)[fFdD]?)",
167+
| text,
168+
| re.ASCII,
169+
| ):
170+
| raise ValueError("For input string: " + repr(text))
171+
| if text[-1] in "dDfF":
172+
| text = text[:-1]
173+
| if "x" not in text.lower():
174+
| return float(text)
175+
| try:
176+
| return float.fromhex(text)
177+
| except OverflowError:
178+
| return float("-inf") if text.startswith("-") else float("inf")
179+
|
180+
|def _texera_double_key(x):
181+
| import math
182+
|
183+
| if math.isnan(x):
184+
| return (1, 0.0, 0.0)
185+
| return (0, x, math.copysign(1.0, x))
186+
|
187+
|def _texera_key_codes(*frames, signed=False):
188+
| import math
189+
|
190+
| codes = {}
191+
|
192+
| def key(v, missing):
193+
| if missing:
194+
| return None
195+
| if isinstance(v, float):
196+
| if math.isnan(v):
197+
| return object()
198+
| if signed:
199+
| return (v, math.copysign(1.0, v))
200+
| return v
201+
|
202+
| def code(df):
203+
| out = []
204+
| for values, missing in zip(df.itertuples(index=False, name=None), df.isna().to_numpy()):
205+
| row = tuple(key(v, m) for v, m in zip(values, missing))
206+
| out.append(codes.setdefault(row, len(codes)))
207+
| return pd.Series(out, index=df.index, dtype="int64")
208+
|
209+
| return [code(df) for df in frames]
210+
|
211+
|def _texera_read_doubles(s):
212+
| import numpy as np
213+
|
214+
| if pd.api.types.is_numeric_dtype(s):
215+
| return s
216+
| holes = s.isna().to_numpy()
217+
| values = np.array(
218+
| [np.nan if hole else _texera_java_double(x) for x, hole in zip(s, holes)],
219+
| dtype="float64",
220+
| )
221+
| if np.isnan(values[~holes]).any():
222+
| values = pd.arrays.FloatingArray(values, holes)
223+
| return pd.Series(values, index=s.index, name=s.name)
146224
|
147225
|def _texera_cast_string(s):
148226
| # `toString` on the field, so the COLUMN's type decides the text and

‎common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/source/scan/ScanSourceOpDesc.scala‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,9 +82,15 @@ abstract class ScanSourceOpDesc extends SourceOperatorDescriptor {
8282

8383
/** Rows actually used for type inference: INFER_READ_LIMIT, capped by `windowLimit`
8484
* when smaller.
85+
*
86+
* A window of no rows is still a window on this file, and the file's columns do
87+
* not depend on how many of its rows were asked for, so a Limit of 0 samples as
88+
* if no limit were set. Capping by it left nothing to infer from, and the
89+
* operator declared a schema of no columns at all.
8590
*/
8691
@JsonIgnore
87-
def inferSampleSize: Int = windowLimit.getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
92+
def inferSampleSize: Int =
93+
windowLimit.filter(_ > 0).getOrElse(INFER_READ_LIMIT).min(INFER_READ_LIMIT)
8894

8995
override def sourceSchema(): Schema = null
9096

0 commit comments

Comments
 (0)