Repository navigation
Commit 9730c86
authored
fix(workflow-operator): auto-rename empty CSV column headers (apache#5282)
### What changes were proposed in this PR?
A CSV with an empty column header (e.g. a trailing comma `id,name,,age`)
produced an `Attribute` with an empty name. After apache#3295 every output
port writes via `IcebergTableWriter → Parquet`, where Avro rejects empty
names with `IllegalArgumentException: Empty name` at flush time — losing
the operator port's entire result.
Rename blank header positions to `column-<index>` (the convention used
by pandas, Spark, R, and DuckDB) in all three CSV scan operators:
`CSVScanSourceOpDesc`, `ParallelCSVScanSourceOpDesc`, and
`CSVOldScanSourceOpDesc`. The issue named the first two; the legacy
`csvOld` variant is still registered in `LogicalOp` and had the same
latent bug.
Note: `ParallelCSVScanSourceOpDesc` is currently commented out of
`LogicalOp`'s operator registry (so it is not reachable from the UI),
but it is fixed here for consistency and so the bug does not resurface
if it is re-enabled for experiments.
### Any related issues, documentation, discussions?
Closes apache#5279.
### How was this PR tested?
Added three cases to `CSVScanSourceOpDescSpec` — one per operator —
feeding a CSV with an empty third header (`id,name,,age`) and asserting
the inferred schema is `["id", "name", "column-3", "age"]`.
`WorkflowOperator` suite is green (13 passed); scalafmt clean.
### Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (claude-opus-4-7)1 parent 5cc75bc commit 9730c86
4 files changed
Lines changed: 65 additions & 5 deletions
File tree
- common/workflow-operator/src
- main/scala/org/apache/texera/amber/operator/source/scan
- csvOld
- csv
- test/scala/org/apache/texera/amber/operator/source/scan/csv
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
124 | 128 | | |
125 | 129 | | |
126 | 130 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
107 | | - | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
108 | 111 | | |
109 | 112 | | |
110 | | - | |
| 113 | + | |
111 | 114 | | |
112 | 115 | | |
113 | 116 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
102 | 102 | | |
103 | 103 | | |
104 | 104 | | |
105 | | - | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
106 | 109 | | |
107 | 110 | | |
108 | | - | |
| 111 | + | |
109 | 112 | | |
110 | 113 | | |
111 | 114 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
| 29 | + | |
29 | 30 | | |
30 | 31 | | |
31 | 32 | | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
32 | 36 | | |
33 | 37 | | |
34 | 38 | | |
| |||
38 | 42 | | |
39 | 43 | | |
40 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
41 | 57 | | |
42 | 58 | | |
43 | 59 | | |
| |||
160 | 176 | | |
161 | 177 | | |
162 | 178 | | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
163 | 213 | | |
0 commit comments