GH-20408: [Go] Implement Encode and Decode functions for REE - #34534

Does this apply input.Offset? That makes it a bit too different from the other buffers pointers in this struct.

one complexity of binary arrays is that the offset values are always absolute offsets into the raw data. So in order for everything to work correctly, the slice for the raw values (inputValues) can't be shifted by the offset. In addition, you can't easily shift the validity bitmap in a predictable way because it's a bitmap and you aren't guaranteed the offset is easily divisible by 8. So those can't utilize the offset built in.

However, exec.GetSpanOffsets already accounts for the offset when creating the typed slice of offsets allowing the simple change of just not needing to apply the offset to the readOffset in the loop for var binary arrays.

TIL. Thanks.

the same is also true for exec.GetSpanValues

The difference between GetSpanValues and GetSpanOffsets is that GetSpanOffsets adds the expected +1 to the length that offsets should have. so len(exec.GetSpanOffsets(....)) == span.Len+1

Also, exec.GetSpanOffsets doesn't do the nil check that GetSpanValues does.

inputValueOffset could be just inputOffset here, right? It refers to the logical offset of the parent array. (?)

• de.inputOffset is the logical offset of the parent array
• de.inputPhysicalOffset is the physical offset into the values and runs children arrays
• de.inputValueOffset is the internal offset of the values child array

For example, let's say you have the array [1, 2, 3, 4, 5]. Then let's take a slice of that, [2, 3, 4] and use that as our values for a run-end-encoded array. Finally let's add a logical offset to the REE array.

To make up some runs, that gives us the final REE array of:

• Parent: logical offset = 20, len = 30
  • Runs: offset = 0, len = 3, [15, 30, 50]
  • Values: offset = 1, len = 3, [1, 2, 3, 4, 5]

To get the proper values for the runs, we first find the physical offset (1) from the logical offset. Then when we index into the values child, we have to also apply the child's offset (1). Giving us the first value of 3 (index = 2). This of course also applies to determining the bitmap validity for nulls.

Does that make sense? This is also tested in the unit tests.

Oh right. I encapsulated much of this in a forward iterator in the C++ version and had forgotten there are 3 offset variables to address a physical value sometimes :)

yea, Go doesn't really have iterators so there's just the merged runs iterator thing, but that's designed for iterating across 2 REE arrays simultaneously and getting the runs that line up rather than just iterating a single REE array. Maybe one day Go will provide a way to hook into the for ... := range construction for custom types lol

so there's just the merged runs iterator thing, but that's designed for iterating across 2 REE arrays simultaneously

I took that thigk apart in the C++ version and made a similar class that takes a single ArraySpan and allows iterating on the runs. It's not completely C++ iterator compatible, so I think a similar thing could be achieved with Go. Perhaps Go doesn't inline as aggressively and it wouldn't be as efficient.