Repository navigation
Commit f956fd1
authored
fix(proto): preserve Arrow IPC stream format (apache#24224)
## Which issue does this PR close?
- Closes apache#24196.
## Rationale for this change
`ArrowSource` distinguishes between Arrow IPC file and stream formats,
but this format was not serialized. As a result, a stream scan
round-tripped through protobuf as a file scan, selecting the wrong
opener and potentially allowing unsupported range-based repartitioning.
## What changes are included in this PR?
- Add an `ArrowIpcFormat` discriminator to `ArrowScanExecNode`.
- Serialize and restore the `ArrowSource` IPC format.
- Decode payloads without the new field as file format, preserving the
previous behavior.
- Regenerate the prost and pbjson models.
- Add regression coverage for file, stream, and older payloads without
the format field.
## Are these changes tested?
Yes:
- `cargo test -p datafusion-proto --test proto_integration
roundtrip_arrow`
- `cargo test -p datafusion-proto --test proto_integration
arrow_scan_without_format_field_decodes_as_file_format`
- `cargo fmt --all`
- Targeted all-feature clippy for `datafusion-datasource-arrow` and
`datafusion-proto`
## Are there any user-facing changes?
Arrow IPC stream scans now preserve their format across protobuf round
trips.
The binary protobuf change is additive. Older payloads continue to
decode as file format. Stream preservation requires both producer and
consumer to include this change because older versions do not carry or
read the discriminator. The generated Rust `ArrowScanExecNode` struct
gains a `format` field.1 parent b9399dc commit f956fd1
5 files changed
Lines changed: 256 additions & 9 deletions
File tree
- datafusion
- datasource-arrow/src
- proto-models
- proto
- src/generated
- proto/tests/cases/plans
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
403 | 403 | | |
404 | 404 | | |
405 | 405 | | |
406 | | - | |
407 | | - | |
408 | | - | |
409 | | - | |
| 406 | + | |
| 407 | + | |
410 | 408 | | |
411 | 409 | | |
412 | 410 | | |
| |||
416 | 414 | | |
417 | 415 | | |
418 | 416 | | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
419 | 422 | | |
420 | 423 | | |
421 | 424 | | |
422 | 425 | | |
| 426 | + | |
423 | 427 | | |
424 | 428 | | |
425 | 429 | | |
| |||
430 | 434 | | |
431 | 435 | | |
432 | 436 | | |
433 | | - | |
434 | | - | |
| 437 | + | |
| 438 | + | |
435 | 439 | | |
436 | 440 | | |
437 | 441 | | |
| |||
455 | 459 | | |
456 | 460 | | |
457 | 461 | | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
458 | 469 | | |
459 | | - | |
460 | | - | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
461 | 477 | | |
462 | 478 | | |
463 | 479 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1296 | 1296 | | |
1297 | 1297 | | |
1298 | 1298 | | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
1299 | 1308 | | |
1300 | 1309 | | |
| 1310 | + | |
1301 | 1311 | | |
1302 | 1312 | | |
1303 | 1313 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1445 | 1445 | | |
1446 | 1446 | | |
1447 | 1447 | | |
| 1448 | + | |
| 1449 | + | |
| 1450 | + | |
| 1451 | + | |
| 1452 | + | |
| 1453 | + | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
| 1470 | + | |
| 1471 | + | |
| 1472 | + | |
| 1473 | + | |
| 1474 | + | |
| 1475 | + | |
| 1476 | + | |
| 1477 | + | |
| 1478 | + | |
| 1479 | + | |
| 1480 | + | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
| 1486 | + | |
| 1487 | + | |
| 1488 | + | |
| 1489 | + | |
| 1490 | + | |
| 1491 | + | |
| 1492 | + | |
| 1493 | + | |
| 1494 | + | |
| 1495 | + | |
| 1496 | + | |
| 1497 | + | |
| 1498 | + | |
| 1499 | + | |
| 1500 | + | |
| 1501 | + | |
| 1502 | + | |
| 1503 | + | |
| 1504 | + | |
| 1505 | + | |
| 1506 | + | |
| 1507 | + | |
| 1508 | + | |
| 1509 | + | |
| 1510 | + | |
| 1511 | + | |
| 1512 | + | |
| 1513 | + | |
| 1514 | + | |
| 1515 | + | |
| 1516 | + | |
| 1517 | + | |
| 1518 | + | |
1448 | 1519 | | |
1449 | 1520 | | |
1450 | 1521 | | |
| |||
1456 | 1527 | | |
1457 | 1528 | | |
1458 | 1529 | | |
| 1530 | + | |
| 1531 | + | |
| 1532 | + | |
1459 | 1533 | | |
1460 | 1534 | | |
1461 | 1535 | | |
1462 | 1536 | | |
| 1537 | + | |
| 1538 | + | |
| 1539 | + | |
| 1540 | + | |
| 1541 | + | |
1463 | 1542 | | |
1464 | 1543 | | |
1465 | 1544 | | |
| |||
1472 | 1551 | | |
1473 | 1552 | | |
1474 | 1553 | | |
| 1554 | + | |
1475 | 1555 | | |
1476 | 1556 | | |
1477 | 1557 | | |
1478 | 1558 | | |
1479 | 1559 | | |
| 1560 | + | |
1480 | 1561 | | |
1481 | 1562 | | |
1482 | 1563 | | |
| |||
1499 | 1580 | | |
1500 | 1581 | | |
1501 | 1582 | | |
| 1583 | + | |
1502 | 1584 | | |
1503 | 1585 | | |
1504 | 1586 | | |
| |||
1519 | 1601 | | |
1520 | 1602 | | |
1521 | 1603 | | |
| 1604 | + | |
1522 | 1605 | | |
1523 | 1606 | | |
1524 | 1607 | | |
| |||
1527 | 1610 | | |
1528 | 1611 | | |
1529 | 1612 | | |
| 1613 | + | |
| 1614 | + | |
| 1615 | + | |
| 1616 | + | |
| 1617 | + | |
| 1618 | + | |
1530 | 1619 | | |
1531 | 1620 | | |
1532 | 1621 | | |
1533 | 1622 | | |
| 1623 | + | |
1534 | 1624 | | |
1535 | 1625 | | |
1536 | 1626 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1996 | 1996 | | |
1997 | 1997 | | |
1998 | 1998 | | |
| 1999 | + | |
| 2000 | + | |
1999 | 2001 | | |
2000 | 2002 | | |
2001 | 2003 | | |
| |||
2794 | 2796 | | |
2795 | 2797 | | |
2796 | 2798 | | |
| 2799 | + | |
| 2800 | + | |
| 2801 | + | |
| 2802 | + | |
| 2803 | + | |
| 2804 | + | |
| 2805 | + | |
| 2806 | + | |
| 2807 | + | |
| 2808 | + | |
| 2809 | + | |
| 2810 | + | |
| 2811 | + | |
| 2812 | + | |
| 2813 | + | |
| 2814 | + | |
| 2815 | + | |
| 2816 | + | |
| 2817 | + | |
| 2818 | + | |
| 2819 | + | |
| 2820 | + | |
| 2821 | + | |
| 2822 | + | |
| 2823 | + | |
| 2824 | + | |
| 2825 | + | |
| 2826 | + | |
| 2827 | + | |
| 2828 | + | |
2797 | 2829 | | |
2798 | 2830 | | |
2799 | 2831 | | |
| |||
0 commit comments