Is there a reason to change from // to ///?

@jorgecarleitao if Buffer always has Bytes<u8>, I think this alone doesn't address one of the motivations of doing this; being that we'd like to know what boundary to slice buffers in when offsetting arrays.

This would be when using Buffer::offset and Buffer::length on nested arrays.

I agree.

However, we can't make Buffer<T> because ArrayData force us to use an homogeneous type across all buffers. We can't drop ArrayData because FFI, IPC, CSV, JSON, equality and array/transform depend on it.

This PR was a stop-gap.

Th target design for me is in the proposal. Specifically, for primitive arrays, it would look like this

But because of what I wrote (a lot of dependencies on ArrayData), it requires a large (like +10k lines large) refactor of this crate.

Is it worth still pursuing adding more information to Buffer while working separately on the proposal? I'm blocked on some parquet work that I'm doing, because it depends on being able to correctly slice nested arrays.

I think that any work that unblocks you is worth pursuing :)

A path could be:

1. Make Buffer support both bit slices and byte slices:

• Add a new offset_bits to Buffer.
• Expose a method Buffer::slice_bitmap that increments offset_bits (and panics if called with offset != 0).
• Make Buffer::slice panic if called with offset_bits != 0.

These will allow us to slice Buffer when it is either holder of bits and a holder of bytes (we just need to use either API independently). This is required because the Boolean array's values are in bits (and thus part of ArrayData::buffers).

2. Implement a method

pub fn slice(&self, offset: usize, length: usize) -> Self

on every array, that calls slice or slice_bits depending on what the Buffer represents (and also fix the incorrect slicing of arrays with offsets).

3. make Array::slice call the concrete implementations

4. Make all calls of get_bit and the like to use Buffer::offset_bits to compute slot nullability and/or values.

Note that I have not tested this and thus this may not work. I also do not know the consequences to IPC and FFI. :/

I am more confident that the proposal works because it passes IPC tests (IPC read and write, little and big endian ;)).