Repository navigation
[Parquet] Prototype ALP encoding #8748
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The parquet community is considering adding new encodings to the format.
One proposed encoding is:
Describe the solution you'd like
I would like someone(s) to create a branch of arrow-rs 's parquet reader and implement this proposed encoding in the context of parquet to see how it would work.
This would involve likely
- Adding a new encoding to the metadata: source link
- Add the appropriate encoder/decoders in parquet (existing code)
- Add some basic tests showing data could be round tripped with these encodings: examples of similar tets here
- Add some benchmarks: examples of similar benchmarks
Here are some existing crates
- https://crates.io/crates/alp
- https://crates.io/crates/vortex-alp (maybe this code)
Describe alternatives you've considered
Additional context
Reacted by Liam Bao
Metadata
Metadata
Assignees
Labels
No labels
Activity
alamb commentedon Oct 30, 2025
FYI @liamzwbao we were speaking about this
alamb commentedon Dec 8, 2025
Here is a C/C++ implementation from @prtkgaur:
alamb commentedon Jan 14, 2026
If anyone is interested, there is now a draft spec that we could try to implement
alamb commentedon Feb 4, 2026
Update here is that the ALP spec is pretty close to being finalized in the C++ side - the current version of the spec is this document: https://docs.google.com/document/d/1xz2cudDpN2Y1ImFcTXh15s-3fPtD_aWt/edit (I haven't translated it into markdown yet)
I think this ticket is ready for someone to try and implement it in Rust
In parallel, we are also trying to get some example files to test with
As an aside, @julienledem noted in the Parquet sync call today that he had good luck with Claude code translating the C/C++ implementation and the spec into a working java implementation. Maybe we can use such an approach with the Rust implementation as well
sdf-jkl commentedon Feb 4, 2026
I'd like to give this a go
sdf-jkl commentedon Feb 4, 2026
take
alamb commentedon Feb 5, 2026
BTW I started working on generating example Parquet files that are encoded using ALP using the C++ implementation. See here
It doesn't have the entire setup, but this file should be enough to verify the plumbing is working:
single_f64_ALP.zip
To begin I suggest
With that framework in place, we can begin working on the actual decoding algorithm
Polepoint commentedon May 1, 2026
Hi @sdf-jkl @alamb , any update on the remaining work for this PR?
It looks like the latest review comments and cargo fmt were addressed on Mar
12. Is there anything still blocking review or moving this out of draft?
alamb commentedon May 3, 2026
Hi @Polepoint
My personal focus recently has been on reviewing the proposed spec, see:
Next I hope to review the work that @sdf-jkl has done in #9372
If you are interested in helping move the project along, perhaps you could help review / test the implementation in #9372 -- for example, does it work / get the same answers compared to the data in apache/parquet-testing#100 ?
alamb commentedon Aug 4, 2026
Since the ALP encoding has been adopted by parquet, closing this "prototype" issue and filed a ticket to track the actual implementaiton
alamb commentedon Aug 4, 2026
nice work @sdf-jkl
[Parquet] ALP encoder/decoder support (#9372)
[Parquet] ALP encoder/decoder support (apache#9372)
[Parquet] ALP encoder/decoder support (apache#9372)