Repository navigation
Commit 8dcdf05
feat(workflow-operator): export the CSV sources as Python (apache#8341)
### What changes were proposed in this PR?
The three CSV source operators (CSV, CSV Old and Parallel CSV) implement
`StandaloneCodeGenerator`, so an exported script starts from the same
data the workflow did rather than from a variable the reader has to fill
in. They also carry what every source after them uses: the placeholder a
source writes where its file is named, and the parse that reads a
timestamp column from text.
A source is the one place where the script cannot simply repeat what the
operator does. The engine resolves a dataset through Texera's storage
and hands the operator a URI; a script has no such resolver, so it reads
the file from its own directory. Each source writes a placeholder where
its file should be named and declares the path it reads, and the
translator assigns the names once it can see the whole plan: the last
segment of the path where that is free, and a numbered form of it where
two sources read different files whose paths end the same way. The last
segment rather than a parse of the whole string, because the resolver
percent-encodes the file-relative parts but leaves the repository and
version names as the user typed them, and a dataset version with a space
in its name makes `new URI` throw before any code is generated. That is
what makes an exported script portable, and it is equally its one
precondition: the data has to sit beside the script, under the names the
script states.
A timestamp is read as the engine reads one. The engine hands each field
to DateParserUtils, so a row states its own format, any year reads, an
offset is moved to the local zone, and the reading is cut to the
millisecond. pandas' date parsing does none of that. So each CSV reader
reads the column as text and converts it with the helper apache#8597 adds.
The end of an exported scan window is counted in Long: two Ints the
operator accepts can add up past what an Int holds, and the slice came
out negative.
Two of the changes made in review are deliberately not the literal
instruction, because the three CSV readers do not agree with each other
about a blank field. Old CSV sets `keep_default_na=False` but names no
missing value at all: scala-csv hands a blank back as the empty string
rather than as a null, so the `na_values` the main CSV source needs
would null a value this reader keeps. Parallel CSV asks for such a
column as text rather than as a nullable integer: its schema is read
with scala-csv, where one blank types the whole column as text, while
its executor nulls the blank and parses the rest as that text. Each fix
follows the reader it belongs to rather than the one next to it.
### Any related issues, documentation, discussions?
Part of apache#8325, step 12a; apache#8702 and apache#8703 follow. That issue lists the
set in order. It needs apache#8327 for the trait, so it does not compile until
that lands, and the rows these operators add to the verification runner
follow with the harness rather than as whole new files here. It also
needs apache#8597, whose helpers the timestamp reading calls, so apache#8597 merges
first.
Closes apache#8415, the task this change is the whole of.
One engine defect was found while matching these readers, and it is
fixed here rather than worked around: the export alone could not close
it. apache#8602, where a scan window of no rows left the type inference
nothing to read, so CSV declared a schema of no columns and CSV Old
threw. Its JSONL half follows in apache#8702, which closes it.
### How was this PR tested?
Each operator asserts the block it emits in its own spec. Where a
difference is one of behaviour rather than of text, the spec runs the
engine's own executor and the generated script over the same file and
compares what each produced: the row window the parallel reader ignores,
and the nullable long each reader has to keep exact. Once the
verification lands, every operator is also run both ways on every
configuration its schema offers; this branch is cut from main and does
not carry that machinery, so those runs are not on this diff's CI.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>1 parent 4bad5e4 commit 8dcdf05
11 files changed
Lines changed: 1180 additions & 37 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator
- source/scan
- csvOld
- csv
- test/scala/org/apache/texera/amber/operator/source/scan
- csvOld
- csv
- workflow-compiling-service/src
- main/scala/org/apache/texera/amber/translator
- test/scala/org/apache/texera/amber/translator
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
67 | 68 | | |
68 | 69 | | |
69 | 70 | | |
70 | | - | |
71 | | - | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
72 | 85 | | |
73 | 86 | | |
74 | 87 | | |
75 | 88 | | |
76 | 89 | | |
77 | 90 | | |
78 | 91 | | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | | - | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
85 | 99 | | |
86 | 100 | | |
87 | 101 | | |
| |||
110 | 124 | | |
111 | 125 | | |
112 | 126 | | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
43 | 53 | | |
44 | 54 | | |
45 | 55 | | |
46 | 56 | | |
47 | 57 | | |
48 | 58 | | |
49 | | - | |
| 59 | + | |
50 | 60 | | |
51 | 61 | | |
52 | 62 | | |
53 | 63 | | |
54 | 64 | | |
55 | | - | |
| 65 | + | |
56 | 66 | | |
57 | 67 | | |
58 | | - | |
59 | 68 | | |
60 | 69 | | |
61 | 70 | | |
| |||
81 | 90 | | |
82 | 91 | | |
83 | 92 | | |
84 | | - | |
85 | 93 | | |
86 | 94 | | |
87 | 95 | | |
88 | 96 | | |
89 | 97 | | |
90 | 98 | | |
91 | 99 | | |
92 | | - | |
93 | 100 | | |
94 | 101 | | |
95 | 102 | | |
| |||
101 | 108 | | |
102 | 109 | | |
103 | 110 | | |
104 | | - | |
105 | 111 | | |
106 | 112 | | |
107 | 113 | | |
| |||
121 | 127 | | |
122 | 128 | | |
123 | 129 | | |
124 | | - | |
125 | 130 | | |
126 | 131 | | |
127 | 132 | | |
| |||
137 | 142 | | |
138 | 143 | | |
139 | 144 | | |
140 | | - | |
141 | 145 | | |
142 | | - | |
143 | | - | |
144 | | - | |
| 146 | + | |
145 | 147 | | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
146 | 224 | | |
147 | 225 | | |
148 | 226 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
82 | 82 | | |
83 | 83 | | |
84 | 84 | | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
85 | 90 | | |
86 | 91 | | |
87 | | - | |
| 92 | + | |
| 93 | + | |
88 | 94 | | |
89 | 95 | | |
90 | 96 | | |
| |||
0 commit comments