Repository navigation
Commit 9752c34
authored
### Rationale for this change
Add support for LargeBinaryType, BinaryViewType, LargeStringType, and StringViewType in NumPy to Arrow conversion.
### What changes are included in this PR?
Adds new `Visit` methods in `NumPyConverter` for `LargeStringType`, `StringViewType`, `LargeBinaryType`, and `BinaryViewType`. These are built on top of new templated `VisitString` and `VisitBinary` methods.
Also adds the Arrow -> NumPy type map for the large binary types.
### Are these changes tested?
New test added showing a string and binary array over 2 GiB is valid and still a single chunk. Also added the new Arrow to NumPy maps to a schema test.
### Are there any user-facing changes?
Adds support for converting NumPy string/binary lists to large binary types.
* Closes: #35289
* GitHub Issue: #35289
Authored-by: Adam Binford <[email protected]>
Signed-off-by: AlenkaF <[email protected]>
1 parent 30adc91 commit 9752c34
4 files changed
Lines changed: 155 additions & 50 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
| 228 | + | |
| 229 | + | |
228 | 230 | | |
229 | 231 | | |
230 | 232 | | |
| 233 | + | |
| 234 | + | |
231 | 235 | | |
232 | 236 | | |
233 | 237 | | |
| |||
285 | 289 | | |
286 | 290 | | |
287 | 291 | | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
288 | 298 | | |
289 | 299 | | |
290 | 300 | | |
| |||
554 | 564 | | |
555 | 565 | | |
556 | 566 | | |
557 | | - | |
558 | | - | |
559 | | - | |
| 567 | + | |
| 568 | + | |
560 | 569 | | |
561 | 570 | | |
562 | | - | |
| 571 | + | |
563 | 572 | | |
564 | 573 | | |
565 | 574 | | |
566 | 575 | | |
567 | | - | |
| 576 | + | |
568 | 577 | | |
569 | 578 | | |
570 | 579 | | |
571 | 580 | | |
572 | 581 | | |
573 | 582 | | |
574 | | - | |
| 583 | + | |
575 | 584 | | |
576 | 585 | | |
577 | 586 | | |
| |||
584 | 593 | | |
585 | 594 | | |
586 | 595 | | |
| 596 | + | |
| 597 | + | |
| 598 | + | |
| 599 | + | |
| 600 | + | |
| 601 | + | |
| 602 | + | |
| 603 | + | |
587 | 604 | | |
588 | 605 | | |
589 | 606 | | |
| |||
592 | 609 | | |
593 | 610 | | |
594 | 611 | | |
| 612 | + | |
| 613 | + | |
| 614 | + | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
595 | 632 | | |
596 | 633 | | |
597 | 634 | | |
| |||
631 | 668 | | |
632 | 669 | | |
633 | 670 | | |
634 | | - | |
635 | | - | |
| 671 | + | |
| 672 | + | |
636 | 673 | | |
637 | 674 | | |
638 | 675 | | |
| |||
660 | 697 | | |
661 | 698 | | |
662 | 699 | | |
663 | | - | |
664 | | - | |
665 | | - | |
666 | | - | |
667 | | - | |
| 700 | + | |
| 701 | + | |
668 | 702 | | |
669 | 703 | | |
670 | 704 | | |
| |||
708 | 742 | | |
709 | 743 | | |
710 | 744 | | |
711 | | - | |
| 745 | + | |
712 | 746 | | |
713 | 747 | | |
714 | 748 | | |
715 | 749 | | |
716 | 750 | | |
717 | 751 | | |
718 | 752 | | |
719 | | - | |
| 753 | + | |
720 | 754 | | |
721 | 755 | | |
722 | 756 | | |
723 | 757 | | |
724 | 758 | | |
725 | 759 | | |
726 | 760 | | |
727 | | - | |
| 761 | + | |
728 | 762 | | |
729 | 763 | | |
730 | 764 | | |
| |||
737 | 771 | | |
738 | 772 | | |
739 | 773 | | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
740 | 784 | | |
741 | 785 | | |
742 | 786 | | |
| |||
745 | 789 | | |
746 | 790 | | |
747 | 791 | | |
| 792 | + | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
| 803 | + | |
| 804 | + | |
| 805 | + | |
| 806 | + | |
| 807 | + | |
| 808 | + | |
| 809 | + | |
| 810 | + | |
| 811 | + | |
| 812 | + | |
| 813 | + | |
| 814 | + | |
| 815 | + | |
| 816 | + | |
| 817 | + | |
748 | 818 | | |
749 | 819 | | |
750 | 820 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2498 | 2498 | | |
2499 | 2499 | | |
2500 | 2500 | | |
2501 | | - | |
| 2501 | + | |
| 2502 | + | |
| 2503 | + | |
| 2504 | + | |
| 2505 | + | |
| 2506 | + | |
| 2507 | + | |
| 2508 | + | |
| 2509 | + | |
2502 | 2510 | | |
2503 | 2511 | | |
2504 | | - | |
2505 | | - | |
2506 | | - | |
| 2512 | + | |
| 2513 | + | |
| 2514 | + | |
2507 | 2515 | | |
2508 | 2516 | | |
2509 | 2517 | | |
2510 | | - | |
2511 | | - | |
| 2518 | + | |
| 2519 | + | |
2512 | 2520 | | |
2513 | 2521 | | |
2514 | 2522 | | |
2515 | 2523 | | |
2516 | 2524 | | |
2517 | | - | |
| 2525 | + | |
2518 | 2526 | | |
2519 | 2527 | | |
2520 | | - | |
| 2528 | + | |
2521 | 2529 | | |
2522 | 2530 | | |
2523 | 2531 | | |
2524 | 2532 | | |
2525 | | - | |
2526 | | - | |
| 2533 | + | |
| 2534 | + | |
2527 | 2535 | | |
2528 | 2536 | | |
2529 | 2537 | | |
| |||
2643 | 2651 | | |
2644 | 2652 | | |
2645 | 2653 | | |
2646 | | - | |
| 2654 | + | |
| 2655 | + | |
| 2656 | + | |
| 2657 | + | |
| 2658 | + | |
| 2659 | + | |
| 2660 | + | |
| 2661 | + | |
| 2662 | + | |
2647 | 2663 | | |
2648 | 2664 | | |
2649 | 2665 | | |
2650 | 2666 | | |
2651 | 2667 | | |
2652 | | - | |
2653 | | - | |
2654 | | - | |
| 2668 | + | |
| 2669 | + | |
| 2670 | + | |
2655 | 2671 | | |
2656 | 2672 | | |
2657 | 2673 | | |
2658 | | - | |
2659 | | - | |
| 2674 | + | |
| 2675 | + | |
2660 | 2676 | | |
2661 | 2677 | | |
2662 | 2678 | | |
2663 | 2679 | | |
2664 | 2680 | | |
2665 | | - | |
| 2681 | + | |
2666 | 2682 | | |
2667 | 2683 | | |
2668 | | - | |
| 2684 | + | |
2669 | 2685 | | |
2670 | 2686 | | |
2671 | 2687 | | |
2672 | 2688 | | |
2673 | | - | |
2674 | | - | |
| 2689 | + | |
| 2690 | + | |
2675 | 2691 | | |
2676 | 2692 | | |
2677 | 2693 | | |
| |||
3268 | 3284 | | |
3269 | 3285 | | |
3270 | 3286 | | |
3271 | | - | |
3272 | | - | |
| 3287 | + | |
| 3288 | + | |
| 3289 | + | |
3273 | 3290 | | |
3274 | 3291 | | |
3275 | 3292 | | |
| |||
3286 | 3303 | | |
3287 | 3304 | | |
3288 | 3305 | | |
3289 | | - | |
3290 | | - | |
| 3306 | + | |
| 3307 | + | |
| 3308 | + | |
| 3309 | + | |
3291 | 3310 | | |
3292 | | - | |
| 3311 | + | |
3293 | 3312 | | |
3294 | 3313 | | |
3295 | | - | |
3296 | 3314 | | |
| 3315 | + | |
| 3316 | + | |
| 3317 | + | |
| 3318 | + | |
| 3319 | + | |
| 3320 | + | |
| 3321 | + | |
| 3322 | + | |
| 3323 | + | |
3297 | 3324 | | |
3298 | | - | |
3299 | | - | |
| 3325 | + | |
| 3326 | + | |
3300 | 3327 | | |
3301 | | - | |
3302 | | - | |
3303 | | - | |
3304 | | - | |
3305 | | - | |
3306 | | - | |
| 3328 | + | |
| 3329 | + | |
| 3330 | + | |
| 3331 | + | |
| 3332 | + | |
| 3333 | + | |
3307 | 3334 | | |
3308 | 3335 | | |
3309 | 3336 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
75 | 75 | | |
76 | 76 | | |
77 | 77 | | |
| 78 | + | |
| 79 | + | |
78 | 80 | | |
79 | 81 | | |
| 82 | + | |
| 83 | + | |
80 | 84 | | |
81 | 85 | | |
82 | 86 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
| 72 | + | |
| 73 | + | |
72 | 74 | | |
73 | 75 | | |
| 76 | + | |
| 77 | + | |
74 | 78 | | |
75 | 79 | | |
76 | 80 | | |
| |||
0 commit comments