Repository navigation
Commit 97e1c1c
authored
fix(proto): preserve CSV/JSON scan options on roundtrip (apache#24233)
## Which issue does this PR close?
- Closes apache#24180.
## Rationale for this change
Physical-plan protobuf serialization does not preserve several CSV and
JSON scan options. Custom CSV terminators, JSON newline-delimited mode,
and file compression therefore revert to their defaults after a
roundtrip, which can cause the
decoded plan to read the file incorrectly.
## What changes are included in this PR?
- Add `terminator` to `CsvScanExecNode`.
- Add `newline_delimited` to `JsonScanExecNode`.
- Add `file_compression_type` to the shared `FileScanExecConf`.
- Serialize and restore these options in CSV and JSON scans.
## Are these changes tested?
Yes. Extended the CSV and JSON physical-plan roundtrip tests to cover
custom terminators, non-newline-delimited JSON, compression, and
backward-compatible defaults.
## Are there any user-facing changes?
CSV and JSON scans now preserve their format and compression options
across protobuf roundtrips. The protobuf changes are additive and
backward compatible. `JsonSource` also gains an `is_newline_delimited`
getter.1 parent f10331f commit 97e1c1c
10 files changed
Lines changed: 317 additions & 28 deletions
File tree
- datafusion
- datasource-csv/src
- datasource-json/src
- datasource/src/file_scan_config
- proto-models
- proto
- src/generated
- proto
- src/physical_plan
- tests/cases/plans
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
356 | 356 | | |
357 | 357 | | |
358 | 358 | | |
| 359 | + | |
359 | 360 | | |
360 | 361 | | |
361 | 362 | | |
| |||
576 | 577 | | |
577 | 578 | | |
578 | 579 | | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
579 | 591 | | |
580 | 592 | | |
581 | 593 | | |
582 | 594 | | |
583 | | - | |
| 595 | + | |
584 | 596 | | |
585 | 597 | | |
586 | 598 | | |
587 | 599 | | |
588 | 600 | | |
589 | | - | |
590 | | - | |
591 | | - | |
592 | | - | |
| 601 | + | |
593 | 602 | | |
594 | 603 | | |
595 | 604 | | |
| |||
617 | 626 | | |
618 | 627 | | |
619 | 628 | | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
620 | 634 | | |
621 | 635 | | |
622 | 636 | | |
| |||
632 | 646 | | |
633 | 647 | | |
634 | 648 | | |
635 | | - | |
| 649 | + | |
| 650 | + | |
636 | 651 | | |
637 | 652 | | |
638 | | - | |
639 | | - | |
640 | | - | |
641 | | - | |
642 | | - | |
643 | | - | |
644 | | - | |
| 653 | + | |
645 | 654 | | |
646 | 655 | | |
647 | 656 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
162 | 167 | | |
163 | 168 | | |
164 | 169 | | |
| |||
254 | 259 | | |
255 | 260 | | |
256 | 261 | | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
257 | 267 | | |
258 | 268 | | |
259 | 269 | | |
| |||
265 | 275 | | |
266 | 276 | | |
267 | 277 | | |
268 | | - | |
| 278 | + | |
269 | 279 | | |
270 | 280 | | |
271 | 281 | | |
| |||
289 | 299 | | |
290 | 300 | | |
291 | 301 | | |
292 | | - | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
293 | 306 | | |
294 | 307 | | |
295 | 308 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
31 | | - | |
| 30 | + | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
| 41 | + | |
41 | 42 | | |
42 | 43 | | |
43 | 44 | | |
| |||
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
| 50 | + | |
49 | 51 | | |
50 | 52 | | |
51 | 53 | | |
| 54 | + | |
52 | 55 | | |
53 | 56 | | |
54 | 57 | | |
| |||
57 | 60 | | |
58 | 61 | | |
59 | 62 | | |
60 | | - | |
61 | | - | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
62 | 66 | | |
63 | 67 | | |
64 | 68 | | |
| |||
114 | 118 | | |
115 | 119 | | |
116 | 120 | | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
117 | 128 | | |
118 | 129 | | |
119 | 130 | | |
| |||
131 | 142 | | |
132 | 143 | | |
133 | 144 | | |
| 145 | + | |
134 | 146 | | |
135 | 147 | | |
136 | 148 | | |
137 | 149 | | |
138 | 150 | | |
139 | 151 | | |
140 | 152 | | |
141 | | - | |
| 153 | + | |
| 154 | + | |
142 | 155 | | |
143 | 156 | | |
144 | 157 | | |
| |||
194 | 207 | | |
195 | 208 | | |
196 | 209 | | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
197 | 223 | | |
198 | 224 | | |
199 | 225 | | |
| |||
226 | 252 | | |
227 | 253 | | |
228 | 254 | | |
229 | | - | |
| 255 | + | |
| 256 | + | |
230 | 257 | | |
231 | 258 | | |
232 | 259 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1265 | 1265 | | |
1266 | 1266 | | |
1267 | 1267 | | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
1268 | 1271 | | |
1269 | 1272 | | |
1270 | 1273 | | |
| |||
1291 | 1294 | | |
1292 | 1295 | | |
1293 | 1296 | | |
| 1297 | + | |
| 1298 | + | |
1294 | 1299 | | |
1295 | 1300 | | |
1296 | 1301 | | |
1297 | 1302 | | |
| 1303 | + | |
| 1304 | + | |
1298 | 1305 | | |
1299 | 1306 | | |
1300 | 1307 | | |
| |||
0 commit comments