Repository navigation
Open
Labels
Description
Describe the bug
We use the ipc format to transfer recordbatch between nodes, and then we find that using lz4 compression or no compression will cause the result returned by the get_array_memory_size() method of the recordbatch after transmission to be particularly large. And use zstd compression the result is smaller.
To Reproduce
check this repo https://github.com/haohuaijin/ipc-bug, or the code is below, the arrow version is 52.2.0
use std::sync::Arc;
use arrow::array::{ArrayRef, RecordBatch, StringArray};
use arrow_schema::{DataType, Field, Schema};
use arrow::ipc::{
writer::{FileWriter, IpcWriteOptions},
CompressionType,
};
use std::io::Cursor;
fn main() {
let batch = generate_record_batch(100, 100);
println!(
"num_rows: {}, original batch size: {:?}",
batch.num_rows(),
batch.get_array_memory_size(),
);
let ipc_options = IpcWriteOptions::default();
let ipc_options = ipc_options
.try_with_compression(Some(CompressionType::LZ4_FRAME))
.unwrap();
let buf = Vec::new();
let mut writer = FileWriter::try_new_with_options(buf, &batch.schema(), ipc_options).unwrap();
writer.write(&batch).unwrap();
let hits_buf = writer.into_inner().unwrap();
println!("ipc buffer size: {:?}", hits_buf.len());
// read from ipc
let buf = Cursor::new(hits_buf);
let reader = arrow::ipc::reader::FileReader::try_new(buf, None).unwrap();
let batches = reader.into_iter().map(|r| r.unwrap()).collect::<Vec<_>>();
println!(
"num rows: {}, after decode batch size: {}, batches len: {}",
batches.iter().map(|b| b.num_rows()).sum::<usize>(),
batches
.iter()
.map(|b| b.get_array_memory_size())
.sum::<usize>(),
batches.len()
);
}
pub fn generate_record_batch(columns: usize, rows: usize) -> RecordBatch {
let mut fields = Vec::with_capacity(columns);
for i in 0..columns {
fields.push(Field::new(&format!("column_{}", i), DataType::Utf8, false));
}
let schema = Arc::new(Schema::new(fields));
let mut arrays: Vec<ArrayRef> = Vec::with_capacity(columns);
for i in 0..columns {
let column_data: Vec<String> = (0..rows).map(|j| format!("row_{}_col_{}", j, i)).collect();
let array = StringArray::from(column_data);
arrays.push(Arc::new(array) as ArrayRef);
}
RecordBatch::try_new(schema, arrays).unwrap()
}the output with LZ4_FRAME
num_rows: 100, original batch size: 261600 <-- (255KB)
ipc buffer size: 112446
num rows: 100, after decode batch size: 10392800, batches len: 1 <-- (10MB)
the output with ZSTD
num_rows: 100, original batch size: 261600
ipc buffer size: 73406
num rows: 100, after decode batch size: 180400, batches len: 1 <-- (176KB)
the output whtiout compression
num_rows: 100, original batch size: 261600
ipc buffer size: 200766
num rows: 100, after decode batch size: 38181600, batches len: 1 <-- (36MB)
Expected behavior
decoded recordbatch size should be similar regardless what compression type was used during encoding.
Additional context
Reacted by Andrew Lamb
Activity
haohuaijin commentedon Sep 10, 2024
Hi @alamb, the use case is that we use DataFusion for distributed search and Arrow Flight for data transmission. When dealing with large tables (e.g., 1000 columns), DataFusion reports "resource exhausted" errors, even with a small amount of data(e.g., a few MB). After investigation, we found that the issue stems from DataFusion's use of
get_array_memory_size()to calculate the size of a RecordBatch. When using LZ4 compression or no compression, the size returned byget_array_memory_size()after decoding is significantly large, leading to the "resource exhausted" errors.you can check the above code. When changing RecordBatch to 1000 columns and the LZ4 method, the decode data size is almost 1GB(return by
get_array_memory_size), and the original data size is only 2.4 MB. (note: arrow-flight use ipc to encode and decode)alamb commentedon Sep 11, 2024
Hi @haohuaijin -- the symptoms you describe certainly sounds non ideal
I wonder if the problem is with the reporting of memory used (
get_array_memory_size) or if that much memory is really being allocated by the IPC decoder.For example, if something in the IPC decoder was using 1MB buffers by default, then 1000 columns * 1MB buffers would result in 1GB of memory used.
So I agree this sounds like a bug, but my guess is that it is a real memory allocation issue rather than a reporting issue (but I haven't confirmed this)
Perhaps you can use something like https://github.com/KDE/heaptrack to track the actual allocations
haohuaijin commentedon Sep 19, 2024
Sorry for the delay, @alamb. Recently, I have had some travels.
I set the paraments of
generate_record_batchto(1000, 100)and usecargo build --releaseto compile https://github.com/haohuaijin/ipc-bug and runheaptrack ./target/release/ipc-bugto get the data.Then, I used heaptrack to visualize the data and get the following pictures.




the file in https://github.com/haohuaijin/ipc-bug/blob/main/heaptrack.ipc-bug.1013097.zst.
haohuaijin commentedon Sep 22, 2024
Hi @alamb, I think I found the reason after reading the code.
While decoding the
RecordBatch, we first construct anArrayReader. Then, we call thenext_buffermethod inArrayReaderto get theBufferneeded to construct eachArray. Innext_buffer, we useslice_with_lengthto obtain a slice of the totalBuffer.The code shows that the
datainBufferis shared among allBufferinstances. Whenget_array_memory_size()is called on aRecordBatch, it callsget_array_memory_size()for eachArray. EachArraythen callscapacity()on eachBuffer. Sincecapacity()refers to the total capacity of thedata(the size ofRecordBatch) rather than the capacity used by thisArray, this has causedget_array_memory_size()to be very large(especially when there are many fields).arrow-rs/arrow-ipc/src/reader.rs
Lines 562 to 569 in d05cf6d
arrow-rs/arrow-ipc/src/reader.rs
Lines 404 to 406 in d05cf6d
arrow-rs/arrow-ipc/src/reader.rs
Lines 51 to 63 in d05cf6d
arrow-rs/arrow-buffer/src/buffer/immutable.rs
Lines 223 to 237 in d05cf6d
arrow-rs/arrow-buffer/src/buffer/immutable.rs
Lines 166 to 168 in d05cf6d
alamb commentedon Sep 24, 2024
See related discussion: #6439
Jefffrey commentedon Aug 27, 2026
related:
get_array_memory_sizereturns incorrect size if underlying buffers are shared #5969RecordBatch.get_array_memory_size()#6607