Bug(arrow-row): calling convert_raw function cause "offset overflow" panic

Describe the bug

Datafusion Table Info:
Having a http_url column, which DataType is Utf8, and it has a lot of distinct values.
Datafusion SQL

select http_url from tab group http_url

Panic Info

Reason
GroupValuesRows in Datafusion stores rows using Rows, and this bug may be triggered when calling emit function.

arrow-rs/arrow-row/src/variable.rs

Lines 217 to 226 in 49e714d

    
           let values_capacity = rows.iter().map(|row| decoded_len(row, options)).sum(); 
        
           let mut offsets = BufferBuilder::<I>::new(len + 1); 
        
           offsets.append(I::zero()); 
        
           let mut values = MutableBuffer::new(values_capacity); 
        
           for row in rows { 
        
               let offset = decode_blocks(row, options, |b| values.extend_from_slice(b)); 
        
               *row = &row[offset..]; 
        
               offsets.append(I::from_usize(values.len()).expect("offset overflow")) 
        
           }

In the extreme case, if I append two Utf8 values which size are large(len1 + len2 > i32::MAX) into Rows twice, and then call convert_rows, which should also trigger the bug. 🤔

To Reproduce

Expected behavior

Additional context

In newer versions of DataFusion I would expect that grouping on a string column would not use row format, but instead usse the special GroupValuesBytes:

https://github.com/apache/datafusion/blob/7db4213b71ed9e914c5a4f16954abfa20b091ae3/datafusion/physical-plan/src/aggregates/group_values/bytes.rs#L27

The bug you describe certainly can happen if there are large numbers of distinct large strings in a multi-column group 🤔

FWIW in general offset overflows do yield panics in arrow-rs, the additional plumbing for error handling what is almost always an unrecoverable error has been hard to justify, although I suspect in this case it could be made into an ArrowError without it being a breaking change.

Edit: I've updated this to be an enhancement, panics are not a bug

The bug you describe certainly can happen if there are large numbers of distinct large strings in a multi-column group 🤔

Yes, this problem was first discovered in the case of a group by multi-column.

FWIW in general offset overflows do yield panics in arrow-rs, the additional plumbing for error handling what is almost always an unrecoverable error has been hard to justify, although I suspect in this case it could be made into an ArrowError without it being a breaking change.

Edit: I've updated this to be an enhancement, panics are not a bug

Do you mean we need to add a new function like the following?

pub fn try_decode_binary<I: OffsetSizeTrait>(
    rows: &mut [&[u8]],
    options: SortOptions,
) -> Result<GenericBinaryArray<I>, ArrowError> {
    ...
}

Do you mean we need to add a new function like the following?

Maybe you could add a check in your code (or in datafusion 🤔 ) on the size of the string buffer and make a new record batch if they exceed 2GB or something. This might be related: apache/datafusion#9562

Making a single array with more than 2GB of string data is likely to be non ideal in a bunch of ways

Navigation Menu

Repository navigation

Bug(arrow-row): calling `convert_raw` function cause "offset overflow" panic #6112

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	let values_capacity = rows.iter().map(\|row\| decoded_len(row, options)).sum();
	let mut offsets = BufferBuilder::<I>::new(len + 1);
	offsets.append(I::zero());
	let mut values = MutableBuffer::new(values_capacity);

	for row in rows {
	let offset = decode_blocks(row, options, \|b\| values.extend_from_slice(b));
	*row = &row[offset..];
	offsets.append(I::from_usize(values.len()).expect("offset overflow"))
	}

Bug(arrow-row): calling convert_raw function cause "offset overflow" panic #6112

Description

Activity

alamb commented on Jul 25, 2024

alamb commented on Jul 25, 2024

tustvold commented on Jul 25, 2024

JasonLi-cn commented on Jul 25, 2024

JasonLi-cn commented on Jul 26, 2024

alamb commented on Jul 26, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Bug(arrow-row): calling `convert_raw` function cause "offset overflow" panic #6112