Repository navigation
Commit 407ab7a
authored
fix: count the largest capacity seen for a shared buffer address (apache#27097)
## Which issue does this PR close?
- Closes apache#27069.
## Rationale for this change
A sort over batches that came through the Arrow C Data Interface as
zero-copy slices of one larger batch panics, in release builds too:
```
assertion failed: reservation.size() + shared.size() >= reserved_bytes
```
This is a regression in 56 (it comes from the shared-buffer accounting
added in apache#25820). Apache DataFusion Comet hits it on TPC-DS q67a, where
the sort input arrives from the JVM through FFI.
`RecordBatchMemoryCounter` identifies a buffer by its start address and
sizes it with `Buffer::capacity()`, keeping the first capacity it sees
for an address. A buffer from `Buffer::from_custom_allocation`, which is
what `arrow::ffi::from_ffi` builds, reports the length it was imported
with rather than the size of the allocation. Slices of one batch that
are exported and imported separately therefore share a start address but
report different capacities. When a smaller one is counted first, the
shared input size from `sort_run_sizes` ends up below a later run's own
`get_record_batch_memory_size`, and the check in `sort_run` fails.
`RecordBatchMemoryTracker` has the same problem in its uncount path. It
adds the capacity of the buffer that brings an address's count from 0 to
1, but subtracts the capacity of whichever buffer brings it back to 0.
If that one is larger, `memory_usage` underflows, and TopK's
`checked_sub(..).unwrap()` on its batch store size panics.
## What changes are included in this PR?
- `BufferIdSet` (used by the counter) stores the capacity counted for
each address, both inline and in the overflow map. `insert` returns the
bytes not counted yet: the full capacity for a new address, the growth
when a larger capacity shows up at a known address, and 0 otherwise.
- `BufferIdMap` (used by the tracker) stores the counted capacity next
to each reference count. `increment` returns the bytes added the same
way, and `decrement` returns the recorded capacity when the count
reaches 0, so the tracker always releases exactly what it added.
- The inline addresses stay in their own contiguous array and the
capacities go in a parallel one, so the lookup scan still only touches
addresses.
- `ExternalSorter::sort_run` returns an internal error instead of
panicking when either of its reservation checks fails, so a future
accounting mismatch fails the query rather than aborting the process,
which for Comet is the JVM.
`record_batch_memory` against `main`, as medians of 8 alternating rounds
on an M3 Max: a one-column batch goes from 8.5 ns to 12.8 ns because the
counter is larger to construct (7.8 ns to 13.1 ns on its own),
`shared_slices/16` is 8–11% slower, `count_uncount/16` and
`count_uncount/64` are up to 6% slower, and the rest are within ±4% or
within the variation between rounds.
Buffers that agree on capacity at a shared address, which covers every
buffer not built with `Buffer::from_custom_allocation`, are counted
exactly as before.
One behavior to note: the tracker keeps an address charged at the
largest capacity seen until every buffer at that address is released,
even after the largest one is released. The allocation is at least that
large and stays alive while any slice of it is held, so the overestimate
is safe, and it keeps count and uncount symmetric.
## What is the testing strategy for this PR?
- `sort_runs_sharing_address_with_different_capacities` in
`sorts/sort.rs` sorts two runs over one custom allocation that declare
16 and 1024 values, with `sort_in_place_threshold_bytes` set to 0 so
each batch stays its own run. Without the fix, the check above fails,
now as an internal error rather than a panic.
- `test_record_batch_memory_counter_counts_largest_capacity_at_address`
in `utils/memory.rs` counts two arrays over one allocation in both
orders, with the inline set and after promotion to the overflow map, and
checks the bytes each call returns and the total.
- `test_uncount_releases_largest_capacity_at_address` counts the smaller
array and then the larger one, releases them in that order, and checks
that the tracker returns to its starting usage, inline and after
promotion.
## Are there any user-facing changes?
No. Sorting FFI-imported sliced batches no longer panics, and if the
sort's reservation check fails again, the query fails with an internal
error instead of a panic. Memory accounting for buffers that share an
address and disagree on capacity is now at least as large as before. No
public API changes.1 parent 25beec8 commit 407ab7a
2 files changed
Lines changed: 230 additions & 66 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5 | 5 | | |
6 | 6 | | |
7 | 7 | | |
| 8 | + | |
8 | 9 | | |
9 | 10 | | |
10 | 11 | | |
| |||
226 | 227 | | |
227 | 228 | | |
228 | 229 | | |
229 | | - | |
230 | | - | |
231 | | - | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
232 | 233 | | |
233 | 234 | | |
234 | 235 | | |
| |||
237 | 238 | | |
238 | 239 | | |
239 | 240 | | |
240 | | - | |
241 | | - | |
| 241 | + | |
| 242 | + | |
242 | 243 | | |
243 | 244 | | |
244 | 245 | | |
| |||
373 | 374 | | |
374 | 375 | | |
375 | 376 | | |
376 | | - | |
377 | | - | |
378 | | - | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
379 | 380 | | |
380 | 381 | | |
381 | 382 | | |
| |||
384 | 385 | | |
385 | 386 | | |
386 | 387 | | |
387 | | - | |
388 | | - | |
389 | | - | |
| 388 | + | |
390 | 389 | | |
391 | 390 | | |
392 | 391 | | |
| |||
604 | 603 | | |
605 | 604 | | |
606 | 605 | | |
607 | | - | |
| 606 | + | |
| 607 | + | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
608 | 611 | | |
609 | 612 | | |
610 | 613 | | |
| 614 | + | |
| 615 | + | |
611 | 616 | | |
612 | | - | |
| 617 | + | |
613 | 618 | | |
614 | 619 | | |
615 | 620 | | |
616 | 621 | | |
617 | 622 | | |
618 | 623 | | |
| 624 | + | |
619 | 625 | | |
620 | 626 | | |
621 | 627 | | |
622 | 628 | | |
623 | 629 | | |
624 | 630 | | |
625 | 631 | | |
626 | | - | |
| 632 | + | |
| 633 | + | |
627 | 634 | | |
628 | | - | |
| 635 | + | |
| 636 | + | |
629 | 637 | | |
630 | 638 | | |
631 | | - | |
632 | | - | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
633 | 643 | | |
634 | 644 | | |
635 | 645 | | |
636 | 646 | | |
| 647 | + | |
637 | 648 | | |
638 | | - | |
| 649 | + | |
639 | 650 | | |
640 | 651 | | |
641 | | - | |
642 | | - | |
643 | | - | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
644 | 655 | | |
645 | | - | |
| 656 | + | |
646 | 657 | | |
647 | 658 | | |
648 | 659 | | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
649 | 668 | | |
650 | 669 | | |
651 | | - | |
652 | | - | |
653 | | - | |
654 | | - | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
655 | 675 | | |
656 | | - | |
657 | | - | |
| 676 | + | |
| 677 | + | |
658 | 678 | | |
659 | | - | |
660 | | - | |
| 679 | + | |
661 | 680 | | |
662 | 681 | | |
663 | 682 | | |
664 | 683 | | |
665 | 684 | | |
666 | 685 | | |
| 686 | + | |
| 687 | + | |
667 | 688 | | |
668 | 689 | | |
669 | 690 | | |
670 | | - | |
671 | | - | |
| 691 | + | |
| 692 | + | |
672 | 693 | | |
673 | 694 | | |
674 | 695 | | |
675 | 696 | | |
676 | 697 | | |
677 | 698 | | |
678 | 699 | | |
| 700 | + | |
679 | 701 | | |
680 | 702 | | |
681 | 703 | | |
682 | 704 | | |
683 | 705 | | |
684 | 706 | | |
685 | 707 | | |
686 | | - | |
687 | | - | |
| 708 | + | |
| 709 | + | |
688 | 710 | | |
689 | | - | |
| 711 | + | |
690 | 712 | | |
691 | | - | |
| 713 | + | |
692 | 714 | | |
693 | | - | |
| 715 | + | |
694 | 716 | | |
695 | 717 | | |
696 | 718 | | |
697 | | - | |
698 | | - | |
699 | | - | |
700 | | - | |
701 | | - | |
702 | | - | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
703 | 724 | | |
704 | 725 | | |
705 | 726 | | |
706 | 727 | | |
707 | 728 | | |
708 | 729 | | |
| 730 | + | |
709 | 731 | | |
710 | | - | |
| 732 | + | |
711 | 733 | | |
712 | 734 | | |
713 | 735 | | |
714 | 736 | | |
715 | 737 | | |
716 | 738 | | |
717 | | - | |
| 739 | + | |
718 | 740 | | |
719 | 741 | | |
720 | | - | |
| 742 | + | |
721 | 743 | | |
722 | | - | |
| 744 | + | |
723 | 745 | | |
724 | 746 | | |
725 | | - | |
726 | | - | |
727 | | - | |
728 | | - | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
729 | 750 | | |
730 | | - | |
731 | | - | |
732 | | - | |
733 | | - | |
734 | | - | |
735 | | - | |
736 | | - | |
737 | | - | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
738 | 757 | | |
739 | 758 | | |
740 | 759 | | |
| |||
743 | 762 | | |
744 | 763 | | |
745 | 764 | | |
| 765 | + | |
746 | 766 | | |
747 | 767 | | |
748 | 768 | | |
749 | 769 | | |
750 | 770 | | |
| 771 | + | |
751 | 772 | | |
752 | 773 | | |
753 | | - | |
| 774 | + | |
754 | 775 | | |
755 | | - | |
| 776 | + | |
756 | 777 | | |
757 | 778 | | |
758 | 779 | | |
759 | | - | |
| 780 | + | |
760 | 781 | | |
761 | 782 | | |
762 | 783 | | |
| |||
1229 | 1250 | | |
1230 | 1251 | | |
1231 | 1252 | | |
| 1253 | + | |
| 1254 | + | |
| 1255 | + | |
| 1256 | + | |
| 1257 | + | |
| 1258 | + | |
| 1259 | + | |
| 1260 | + | |
| 1261 | + | |
| 1262 | + | |
| 1263 | + | |
| 1264 | + | |
| 1265 | + | |
| 1266 | + | |
| 1267 | + | |
| 1268 | + | |
| 1269 | + | |
| 1270 | + | |
| 1271 | + | |
| 1272 | + | |
| 1273 | + | |
| 1274 | + | |
| 1275 | + | |
| 1276 | + | |
| 1277 | + | |
| 1278 | + | |
| 1279 | + | |
| 1280 | + | |
| 1281 | + | |
| 1282 | + | |
| 1283 | + | |
| 1284 | + | |
| 1285 | + | |
| 1286 | + | |
| 1287 | + | |
| 1288 | + | |
| 1289 | + | |
| 1290 | + | |
| 1291 | + | |
| 1292 | + | |
| 1293 | + | |
| 1294 | + | |
| 1295 | + | |
| 1296 | + | |
| 1297 | + | |
| 1298 | + | |
| 1299 | + | |
| 1300 | + | |
| 1301 | + | |
| 1302 | + | |
| 1303 | + | |
| 1304 | + | |
| 1305 | + | |
| 1306 | + | |
| 1307 | + | |
| 1308 | + | |
| 1309 | + | |
| 1310 | + | |
| 1311 | + | |
| 1312 | + | |
| 1313 | + | |
| 1314 | + | |
| 1315 | + | |
| 1316 | + | |
| 1317 | + | |
| 1318 | + | |
| 1319 | + | |
| 1320 | + | |
| 1321 | + | |
| 1322 | + | |
| 1323 | + | |
| 1324 | + | |
| 1325 | + | |
| 1326 | + | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
| 1339 | + | |
1232 | 1340 | | |
1233 | 1341 | | |
1234 | 1342 | | |
| |||
0 commit comments