You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3837ac9
Browse filesBrowse the repository at this point in the historyBrowse files
/// Reconstruct `globally_sorted_batches` to organize the payload buffers of each
469
-
/// `StringViewArray` in sequential order by calling `gc()` on them.
470
-
///
471
-
/// Note this is a workaround until <https://github.com/apache/arrow-rs/issues/7185> is
472
-
/// available
473
-
///
474
-
/// # Rationale
475
-
/// After (merge-based) sorting, all batches will be sorted into a single run,
476
-
/// but physically this sorted run is chunked into many small batches. For
477
-
/// `StringViewArray`s inside each sorted run, their inner buffers are not
478
-
/// re-constructed by default, leading to non-sequential payload locations
479
-
/// (permutated by `interleave()` Arrow kernel). A single payload buffer might
480
-
/// be shared by multiple `RecordBatch`es.
481
-
/// When writing each batch to disk, the writer has to write all referenced buffers,
482
-
/// because they have to be read back one by one to reduce memory usage. This
483
-
/// causes extra disk reads and writes, and potentially execution failure.
484
-
///
485
-
/// # Example
486
-
/// Before sorting:
487
-
/// batch1 -> buffer1
488
-
/// batch2 -> buffer2
489
-
///
490
-
/// sorted_batch1 -> buffer1
491
-
/// -> buffer2
492
-
/// sorted_batch2 -> buffer1
493
-
/// -> buffer2
494
-
///
495
-
/// Then when spilling each batch, the writer has to write all referenced buffers
0 commit comments