Repository navigation
Commit c1cb715
authored
refactor(proto): migrate CsvSource serde (apache#24177)
## Which issue does this PR close?
- Part of apache#23516.
## Rationale for this change
Part of epic apache#23494. Moves `CsvSource` protobuf serialization from the
central dispatch into the source implementation.
## What changes are included in this PR?
Add protobuf serialization and deserialization to `CsvSource`, including
the CSV format options represented by the existing wire format.
Repoint the live decode arm to `CsvSource::try_from_proto` and remove
the old central encode arm. Keep `try_into_csv_scan_physical_plan` as a
deprecated compatibility wrapper that delegates to the new
implementation.
The protobuf wire format remains unchanged.
## Are these changes tested?
Yes. Added `roundtrip_csv_scan_preserves_format_options`, covering
header, delimiter, quote, escape, comment, multiline values, and
truncated rows.
## Are there any user-facing changes?
The existing `PhysicalPlanNodeExt` method remains available as a
deprecated compatibility wrapper. There is no immediate API removal or
wire-format change.1 parent 038bfa2 commit c1cb715
5 files changed
Lines changed: 211 additions & 113 deletions
File tree
- datafusion
- datasource-csv
- src
- proto
- src
- physical_plan
- tests/cases
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
308 | 308 | | |
309 | 309 | | |
310 | 310 | | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
311 | 354 | | |
312 | 355 | | |
313 | 356 | | |
| |||
501 | 544 | | |
502 | 545 | | |
503 | 546 | | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
| 604 | + | |
| 605 | + | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | 18 | | |
40 | 19 | | |
41 | 20 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
27 | 26 | | |
28 | 27 | | |
29 | 28 | | |
30 | 29 | | |
31 | 30 | | |
32 | | - | |
33 | | - | |
| 31 | + | |
34 | 32 | | |
35 | 33 | | |
36 | 34 | | |
| |||
94 | 92 | | |
95 | 93 | | |
96 | 94 | | |
97 | | - | |
98 | 95 | | |
99 | 96 | | |
100 | 97 | | |
| |||
126 | 123 | | |
127 | 124 | | |
128 | 125 | | |
| 126 | + | |
129 | 127 | | |
130 | 128 | | |
131 | 129 | | |
| |||
1084 | 1082 | | |
1085 | 1083 | | |
1086 | 1084 | | |
1087 | | - | |
1088 | | - | |
| 1085 | + | |
| 1086 | + | |
1089 | 1087 | | |
1090 | 1088 | | |
1091 | 1089 | | |
| |||
1340 | 1338 | | |
1341 | 1339 | | |
1342 | 1340 | | |
| 1341 | + | |
| 1342 | + | |
| 1343 | + | |
| 1344 | + | |
1343 | 1345 | | |
1344 | 1346 | | |
1345 | 1347 | | |
1346 | 1348 | | |
1347 | 1349 | | |
1348 | 1350 | | |
1349 | | - | |
1350 | | - | |
1351 | | - | |
1352 | | - | |
1353 | | - | |
1354 | | - | |
1355 | | - | |
1356 | | - | |
1357 | | - | |
1358 | | - | |
1359 | | - | |
1360 | | - | |
1361 | | - | |
1362 | | - | |
1363 | | - | |
1364 | | - | |
1365 | | - | |
1366 | | - | |
1367 | | - | |
1368 | | - | |
1369 | | - | |
1370 | | - | |
1371 | | - | |
1372 | | - | |
1373 | | - | |
1374 | | - | |
1375 | | - | |
1376 | | - | |
1377 | | - | |
1378 | | - | |
1379 | | - | |
1380 | | - | |
1381 | | - | |
1382 | | - | |
1383 | | - | |
1384 | | - | |
1385 | | - | |
1386 | | - | |
| 1351 | + | |
| 1352 | + | |
| 1353 | + | |
| 1354 | + | |
1387 | 1355 | | |
1388 | 1356 | | |
1389 | | - | |
1390 | | - | |
1391 | | - | |
1392 | | - | |
1393 | | - | |
| 1357 | + | |
| 1358 | + | |
| 1359 | + | |
1394 | 1360 | | |
1395 | 1361 | | |
1396 | 1362 | | |
| |||
2561 | 2527 | | |
2562 | 2528 | | |
2563 | 2529 | | |
2564 | | - | |
2565 | | - | |
2566 | | - | |
2567 | | - | |
2568 | | - | |
2569 | | - | |
2570 | | - | |
2571 | | - | |
2572 | | - | |
2573 | | - | |
2574 | | - | |
2575 | | - | |
2576 | | - | |
2577 | | - | |
2578 | | - | |
2579 | | - | |
2580 | | - | |
2581 | | - | |
2582 | | - | |
2583 | | - | |
2584 | | - | |
2585 | | - | |
2586 | | - | |
2587 | | - | |
2588 | | - | |
2589 | | - | |
2590 | | - | |
2591 | | - | |
2592 | | - | |
2593 | | - | |
2594 | | - | |
2595 | | - | |
2596 | | - | |
2597 | | - | |
2598 | | - | |
2599 | | - | |
2600 | | - | |
2601 | | - | |
2602 | | - | |
2603 | | - | |
2604 | | - | |
2605 | 2530 | | |
2606 | 2531 | | |
2607 | 2532 | | |
| |||
0 commit comments