GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification by prtkgaur · Pull Request #557 · apache/parquet-format

prtkgaur · 2026-03-11T04:03:57Z

Add the encoding specification for ALP (encoding value 10) to Encodings.md. ALP compresses FLOAT and DOUBLE columns by converting values to integers via decimal scaling, then applying Frame of Reference encoding and bit-packing. Values that cannot be losslessly round-tripped are stored as exceptions.

Closes [Proposal] Add ALP encoding support in parquet file format #533

See rendered preview here: https://github.com/prtkgaur/parquet-format/blob/alpEncoding/Encodings.md#adaptive-lossless-floating-point-alp--10

The spec covers:

Page layout: 7-byte header, offset array, compressed vectors
Vector format: AlpInfo, ForInfo, packed values, exception data
Encoding math: two-step multiplication for cross-language consistency
Parameter selection, exception detection, and decoding steps

Based on the paper "ALP: Adaptive Lossless floating-Point Compression" (Afroozeh and Boncz, SIGMOD 2024). Wire format matches the C++ Arrow and Java parquet-java implementations.

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

emkornfield · 2026-03-24T23:24:34Z

+
+##### Header (7 bytes)
+
+All multi-byte values are little-endian.


Suggested change

All multi-byte values are little-endian.

All multi-byte values are stored in little-endian order.

emkornfield · 2026-03-24T23:25:47Z

+|--------|-------|------|------|-------------|
+| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) |
+| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) |
+| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) |


emkornfield · 2026-03-24T23:26:59Z

+
+| Offset | Field | Size | Type | Description |
+|--------|-------|------|------|-------------|
+| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) |


we think ALP-RD fits in well here? I forget what the extension point is and why we were OK keeping this field, but not version.

changing the code to read ALP's newer laid out bits would be difficult to apply across all implementations. but AlpRD is a completely newer implementations.

Added a note to the field description.

emkornfield · 2026-03-24T23:27:36Z

+|--------|-------|------|------|-------------|
+| 0 | compression_mode | 1 byte | uint8 | Compression mode (must be 0 = ALP) |
+| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) |
+| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) |


emkornfield · 2026-03-24T23:28:58Z

+**Note:** The number of elements per vector and the packed data size are NOT stored
+in the header. They are derived:
+* Elements per vector: `vector_size` for all vectors except the last, which may be smaller.
+* Packed data size: `ceil(num_elements_in_vector * bit_width / 8)`.


bit_width isn't in the header either, so it is a little strange to call ths out here?

I also hihglighted this as confusing when reading the spec. I recommend removing this sentence as the packed data size is covered in the "Vector Format" section below

comment regarding packed data size has been removed.

emkornfield · 2026-03-24T23:30:01Z

+
+**Note:** The number of elements per vector and the packed data size are NOT stored
+in the header. They are derived:
+* Elements per vector: `vector_size` for all vectors except the last, which may be smaller.


This seems like a little bit of a strange callout since it is covered on line 457 explicitly. and log_vector_size is stored in the header?

I agree this is redundant -- I think leaving the note is valuable clarification

Note: The number of elements per vector and the packed data size are NOT stored
in the header, they are derived

However the other items are not

Removed the packed data size bullet, kept the elements-per-vector clarification.

emkornfield · 2026-03-24T23:31:06Z

+values. Each offset gives the byte position of the corresponding vector's data,
+measured from the start of the offset array itself.
+
+The first offset equals `num_vectors * 4` (pointing just past the offset array).


Suggested change

The first offset equals `num_vectors * 4` (pointing just past the offset array).

The first offset always equals `num_vectors * 4` (pointing just past the offset array).

Lets be explicitly here that we don't support padding.

emkornfield · 2026-03-24T23:33:04Z

+Data section sizes:
+| Section             | Size Formula                | Description                  |
+|---------------------|-----------------------------|------------------------------|
+| PackedValues        | ceil(N * bit\_width / 8)    | Bit-packed delta values      |


emkornfield · 2026-03-24T23:35:09Z

+|---------------------|-----------------------------|------------------------------|
+| PackedValues        | ceil(N * bit\_width / 8)    | Bit-packed delta values      |
+| ExceptionPositions  | num\_exceptions * 2 bytes   | uint16 indices of exceptions |
+| ExceptionValues     | num\_exceptions * sizeof(T) | Original float/double values |


emkornfield · 2026-03-24T23:37:48Z

+
+The FOR-encoded deltas, bit-packed into `ceil(num_elements_in_vector * bit_width / 8)` bytes.
+Values are packed from the least significant bit of each byte to the most significant bit,
+in groups of 8 values, using the same bit-packing order as the


Where does the group of 8 values come in? Wouldn't this messup the number of bytes math?

Good catch — removed 'groups of 8' phrasing. Now simply references the same LSB-first packing order as RLE/Bit-Packing Hybrid.

(I think RleBitPackHybrid was in my mind at that point :) )

emkornfield · 2026-03-24T23:38:55Z

+The encoding uses two separate multiplications (not a single multiplication by
+`10^(e-f)`, and not division) to ensure that implementations produce identical
+floating-point rounding across languages. The powers of 10 MUST be stored as
+precomputed floating-point constants (i.e., literal values like `1e-3f`), not


Why can't they be precomputed at runtime?

You're right — this shouldn't mandate literals vs runtime computation. Reworded to require that encoder and decoder use identical power-of-10 values, without prescribing how they're obtained.

emkornfield · 2026-03-24T23:39:39Z

+| Type   | Magic Number                      | Formula                          |
+|--------|-----------------------------------|----------------------------------|
+| FLOAT  | 2^22 + 2^23 = 12,582,912         | `(int)((value + magic) - magic)` |
+| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(long)((value + magic) - magic)` |


emkornfield · 2026-03-24T23:39:46Z

+
+| Type   | Magic Number                      | Formula                          |
+|--------|-----------------------------------|----------------------------------|
+| FLOAT  | 2^22 + 2^23 = 12,582,912         | `(int)((value + magic) - magic)` |


emkornfield · 2026-03-24T23:40:13Z

+```
+-------------------------------------------------------------------+
+|                                                                   |
+|   encoded = round( value  *  10^e  *  10^(-f) )                  |


Suggested change

| encoded = round( value * 10^e * 10^(-f) ) |

| encoded = fast_round( value * 10^e * 10^(-f) ) |

alamb

Thank you @prtkgaur and @emkornfield -- I started going through this proposal as well. I haven't made it through the entire thing, but I left commands on what I have made it through first.

@prtkgaur how would you like to address feedback on this PR? Would you like to process the comments? I would also be happy to make a PR with proposed edits to your branch if that would be better. Please just let me know

alamb · 2026-04-28T13:25:41Z

+ALP works by converting floating-point values to integers using decimal scaling,
+then applying Frame of Reference (FOR) encoding and bit-packing. Values that
+cannot be losslessly converted are stored as exceptions. The encoding achieves
+high compression for decimal-like floating-point data (e.g., monetary values,
+sensor readings) while remaining fully lossless.


You have more summary at the end of the encoding, but I think a few more sentences in the intro would help people understand this more easily

Suggested change

ALP works by converting floating-point values to integers using decimal scaling,

then applying Frame of Reference (FOR) encoding and bit-packing. Values that

cannot be losslessly converted are stored as exceptions. The encoding achieves

high compression for decimal-like floating-point data (e.g., monetary values,

sensor readings) while remaining fully lossless.

ALP works by converting floating-point values to integers using decimal scaling,

then applying Frame of Reference (FOR) encoding and bit-packing. Values that

cannot be losslessly converted are stored as exceptions. The encoding achieves

high compression for decimal-like floating-point data (e.g., monetary values,

sensor readings) while remaining fully lossless. Values do not depend on

each other, which enables quick random access and parallel encode/decode.

Agreed — added sentence about value independence enabling random access and parallel encode/decode.

alamb · 2026-04-29T12:59:59Z

+
+**Note:** The number of elements per vector and the packed data size are NOT stored
+in the header. They are derived:
+* Elements per vector: `vector_size` for all vectors except the last, which may be smaller.


I agree this is redundant -- I think leaving the note is valuable clarification

Note: The number of elements per vector and the packed data size are NOT stored
in the header, they are derived

However the other items are not

alamb · 2026-04-29T13:00:59Z

+**Note:** The number of elements per vector and the packed data size are NOT stored
+in the header. They are derived:
+* Elements per vector: `vector_size` for all vectors except the last, which may be smaller.
+* Packed data size: `ceil(num_elements_in_vector * bit_width / 8)`.


I also hihglighted this as confusing when reading the spec. I recommend removing this sentence as the packed data size is covered in the "Vector Format" section below

Incorporate review comments from emkornfield and alamb on PR apache#557:

alamb · 2026-04-30T18:35:19Z

I expect to have a second round of feedback tomorrow

alamb

Ok, I made it through the spec again. I think it is really quite close

Let me know what you think about the comments and what you think abotu the idea of starting to draft a Blog post.

alamb · 2026-05-01T14:47:25Z

-| ------------------------------------- | -------------- |
-| [Bit-packed (Deprecated)](#BITPACKED) | BIT_PACKED = 4 |
-
+| Encoding | ID | Supported Types |


Something seems strange to me about this chart. The version on main

parquet-format/Encodings.md

Lines 28 to 40 in 96edf77

### Supported Encodings

For details on current implementation status, see the [Implementation Status](https://parquet.apache.org/docs/file-format/implementationstatus/#encodings) page.

| Encoding type | Encoding enum | Supported Types |

| ------------------------------------------------ | --------------------------------------------------------- | ------------------------------------------------- |

| [Plain](#PLAIN) | PLAIN = 0 | All Physical Types |

| [Dictionary Encoding](#DICTIONARY) | PLAIN_DICTIONARY = 2 (Deprecated) <br> RLE_DICTIONARY = 8 | All Physical Types |

| [Run Length Encoding / Bit-Packing Hybrid](#RLE) | RLE = 3 | BOOLEAN, Dictionary Indices |

| [Delta Encoding](#DELTAENC) | DELTA_BINARY_PACKED = 5 | INT32, INT64 |

| [Delta-length byte array](#DELTALENGTH) | DELTA_LENGTH_BYTE_ARRAY = 6 | BYTE_ARRAY |

| [Delta Strings](#DELTASTRING) | DELTA_BYTE_ARRAY = 7 | BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY |

| [Byte Stream Split](#BYTESTREAMSPLIT) | BYTE_STREAM_SPLIT = 9 | INT32, INT64, FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY |

Doesn't appear in this document.

I would expect this PR that adds the new ALP encoding to add a new entry to the existing table, rather than add an entirely new table.

Maybe we need to merge up from main to this branch (I am happy to do this and I think I have the permissions to push to your branch, but I don't want to mess things up on you)

You're right, this branch was forked before the summary table was added in #550. Rebased onto upstream/master and added ALP as a row to the existing table.

alamb · 2026-05-01T14:54:00Z

+
+##### Fast Rounding
+
+The rounding function uses a "magic number" technique for branchless rounding:


I think it would help to use consistent terminology here to above.

Suggested change

The rounding function uses a "magic number" technique for branchless rounding:

The `fast_round` function uses a "magic number" technique for branchless rounding.

`fast_round(value)` is defined as follows:

Done. Updated to use fast_round consistently and added "is defined as follows".

alamb · 2026-05-01T15:01:03Z

+| FLOAT  | 2^22 + 2^23 = 12,582,912         | `(int32_t)((value + magic) - magic)` |
+| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` |
+
+For negative values, the signs are reversed: `(int32_t)((value - magic) + magic)` for FLOAT, `(int64_t)((value - magic) + magic)` for DOUBLE.


I don't understand this sentence. The C++ implementation does not seem to change sign order from what I can tell:

https://github.com/apache/arrow/pull/48345/changes#diff-f9ab708cab94060b4067fff0a6739e9c3751b450422115663b2bd0badfcc748bR487-R490

/// \brief Convert a float to an int without rounding static inline auto FastRound(T n) -> SignedExactType { n = n + Constants::kMagicNumber - Constants::kMagicNumber; return static_cast<SignedExactType>(n); }

Also the "Fast Rounding" section of the ALP paper doesn't mention sign reversal that I can find

You're right -- the sign reversal was incorrect. The C++ implementation uses a single formula for both positive and negative values (n + magic - magic). Removed the paragraph.

alamb · 2026-05-01T15:06:23Z

+(the encoded integer of the first non-exception value, or 0 if all values
+are exceptions) before FOR encoding. This keeps the FOR range tight.
+
+##### Frame of Reference and Bit-Packing


This section has a worked example of FOR / bitpacking so perhaps we could write it like

Suggested change

##### Frame of Reference and Bit-Packing

##### Example: Frame of Reference and Bit-Packing

alamb · 2026-05-01T15:06:37Z

+
+##### Frame of Reference and Bit-Packing
+
+After decimal encoding and exception substitution:


Suggested change

After decimal encoding and exception substitution:

Given the following data after decimal encoding and exception substitution:

alamb · 2026-05-01T15:10:34Z

+           + sum(vector_bytes for each vector)   // all vectors
+```
+
+#### Constants Reference


This appears to be C/++ implementation details -- I am not sure it adds any value to the spec

alamb · 2026-05-01T15:11:19Z

+encoded size. However, ALP and Byte Stream Split can be complementary: ALP
+exploits decimal structure while Byte Stream Split exploits byte-level correlation.
+
+#### Size Calculations


I don't think we need to add add to the Parquet spec -- it isn't required to implement the encoder/decoder for ALP.

alamb · 2026-05-01T15:12:17Z

+converted are stored separately as *exceptions*. The encoding achieves high
+compression for decimal-like floating-point data (e.g., monetary values, sensor
+readings) while remaining fully lossless. Each value is encoded independently,
+enabling random access to individual vectors and parallel encode/decode.


Suggested change

enabling random access to individual vectors and parallel encode/decode.

enabling random access to individual values and parallel encode/decode.

alamb · 2026-05-01T15:21:46Z

+floating-point rounding across languages. Implementations must ensure that the
+encoder and decoder use identical power-of-10 values for a given exponent.


I don't understand what "use identical power-of-10 values" means. ALL encoders and decoders must to use the exact same values (and floating point arithmetic) as I understand it.

Yes, what you said. Reworded to: "All implementations MUST use the exact same floating-point arithmetic and power-of-10 constants to guarantee cross-language interoperability."

alamb · 2026-05-01T15:26:06Z

+To avoid the cost of exhaustive search on every vector, implementations
+SHOULD use sampling to select up to 5 candidate (exponent, factor)
+combinations (the "encoding preset") at the start of each column chunk.
+Each vector then searches only those 5 candidates.


I think SHOULD is too strong a word here. Maybe we could soften it to a suggestion

Suggested change

To avoid the cost of exhaustive search on every vector, implementations

SHOULD use sampling to select up to 5 candidate (exponent, factor)

combinations (the "encoding preset") at the start of each column chunk.

Each vector then searches only those 5 candidates.

To avoid the cost of exhaustive search on every vector, implementations

can use a sampling approach. One such approach, described in the paper, is to

select up to 5 candidate (exponent, factor) combinations (the "encoding preset")

at the start of each column chunk, and when encoding each vector,

test each of the 5 candidates for the fewest exceptions.

pitrou · 2026-05-05T20:34:34Z

+readings) while remaining fully lossless. Each value is encoded independently,
+enabling random access to individual vectors and parallel encode/decode.
+
+#### Overview


The algorithm description is so long that I think it should be moved to a separate file that we would link to here. In this file we would just keeping the description that is before the Overview.

Yes this makes sense. I too thought it became long and having a separate file would be good.
Let me take a stab at it after I address the above comments and get approval. Else the comment threads on github will get lost.

Maybe similar to how it is done for bloom filter: https://github.com/apache/parquet-format/blob/master/BloomFilter.md

I also think it would be ok to if we moved the content to a separate file as a follow on PR (after the spec change is approved) as in my mind moving the content to a separate file does not affect the content of the spec

It seems we should do this as a follow on PR. I can do so after this PR is merged

Got it. Let's address it in a follow up PR

Filed a ticket to track

[ALP] Move ALP description into its own file (rather than inlined in Encoding.md) #602

PR to move the content:

GH-602: Move ALP encoding specification to AlpEncoding.md #608

Add the encoding specification for ALP (encoding value 10) to Encodings.md. ALP compresses FLOAT and DOUBLE columns by converting values to integers via decimal scaling, then applying Frame of Reference encoding and bit-packing. Values that cannot be losslessly round-tripped are stored as exceptions. The spec covers: - Page layout: 7-byte header, offset array, compressed vectors - Vector format: AlpInfo, ForInfo, packed values, exception data - Encoding math: two-step multiplication for cross-language consistency - Parameter selection, exception detection, and decoding steps Based on the paper "ALP: Adaptive Lossless floating-Point Compression" (Afroozeh and Boncz, SIGMOD 2024). Wire format matches the C++ Arrow and Java parquet-java implementations.

Incorporate review comments from emkornfield and alamb on PR apache#557:

wgtmac · 2026-07-22T02:18:29Z

 | [Delta-length byte array](#DELTALENGTH)          | DELTA_LENGTH_BYTE_ARRAY = 6                               | BYTE_ARRAY                                        |
 | [Delta Strings](#DELTASTRING)                    | DELTA_BYTE_ARRAY = 7                                      | BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY                  |
 | [Byte Stream Split](#BYTESTREAMSPLIT)            | BYTE_STREAM_SPLIT = 9                                     | INT32, INT64, FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY |
+| [ALP](#ALP)                                      | ALP = 10                                                  | FLOAT, DOUBLE                                     |


I've raised the same question in the google doc earlier. FP16 and FP8 have fewer bits so the benefit may not be that significant compared to regular float and double. But I agree that we can discuss these in the future to not expand and thus block current spec design.

wgtmac · 2026-07-22T02:26:56Z

+converted are stored separately as *exceptions*. The encoding achieves high
+compression for decimal-like floating-point data (e.g., monetary values, sensor
+readings) while remaining fully lossless. Each value is encoded independently,
+enabling random access to individual values and parallel encode/decode.


Suggested change

enabling random access to individual values and parallel encode/decode.

enabling random access to individual values and parallel encoding/decoding.

wgtmac · 2026-07-22T02:55:44Z

+-------------+------+------+-----+---------+----------+----------+-----+----------+
+```
+
+The compression pipeline below describes *one* way to produce a conforming


IIUC, the informative compression description should be move to a separate doc later in the followup PR because here we just need necessary description for decoding a ALP-encoded page (i.e. the Page Layout section).

cc @alamb

Ack. Makes sense. We can do this in a follow up PR

tracking in

[ALP] Move ALP description into its own file (rather than inlined in Encoding.md) #602

wgtmac · 2026-07-22T03:03:39Z

+ Byte:    0              1               2              3    4    5    6
+       +----------------+---------------+--------------+----+----+----+----+


Suggested change

Byte: 0 1 2 3 4 5 6

+----------------+---------------+--------------+----+----+----+----+

Byte: 0 1 2 3 4 5 6

+----------------+---------------+--------------+----+----+----+----+

Let's adjust the alignment here.

wgtmac · 2026-07-22T03:21:04Z

+|--------|-------|------|------|-------------|
+| 0 | compression_mode | 1 byte | uint8 | Compression mode (0 = ALP). Reserved for future variants (e.g., ALP-RD). |
+| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) |
+| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in the inclusive range \[3, 15\]. Recommended default: 10 (vector size 1024) |


If the minimum value is 3, why not defining it as log_vector_size_minus_three so it can start with 0? Not a strong opinion since this anyway consume a byte which is large enough.

I'd like to keep log_vector_size, maps directly to 1 << log_vector_size.

wgtmac · 2026-07-22T03:22:47Z

+                              v
+    +----------------------------------------------------------+
+    |  1. CHOOSE PARAMETERS                                    |
+    |     Select (exponent, factor) pair for this vector       |


Suggested change

| Select (exponent, factor) pair for this vector |

| Select (exponent, factor) pair for this array |

Be consistent with Input: float/double array above.

wgtmac · 2026-07-22T03:33:51Z

+Each subsequent offset equals the previous offset plus the stored size of the
+previous vector. No padding is inserted between vectors.
+
+A vector's absolute byte position within the page is


This is incorrect when pages are compressed or have rep/def levels. I would suggest removing this sentence. If we want to emphasize random access of ALP, perhaps we should replace page_data_start with a more precise start point after decompressed.

wgtmac · 2026-07-22T03:41:23Z

+| 0 | compression_mode | 1 byte | uint8 | Compression mode (0 = ALP). Reserved for future variants (e.g., ALP-RD). |
+| 1 | integer_encoding | 1 byte | uint8 | Integer encoding (must be 0 = FOR + bit-packing) |
+| 2 | log_vector_size | 1 byte | uint8 | log2(vector\_size). Must be in the inclusive range \[3, 15\]. Recommended default: 10 (vector size 1024) |
+| 3 | num_elements | 4 bytes | int32 | Total number of floating-point values in the page |


wgtmac · 2026-07-22T03:44:13Z

+values. Each offset gives the byte position of the corresponding vector's data,
+measured from the start of the offset array itself.
+
+The first offset always equals `num_vectors * 4` (pointing just past the offset array).


Why num_vectors * 4 but not num_vectors * 4 + 7 to include the header? Usually the whole buffer is stored including header, so implementations need to manually add 7 each time to get an absolute offset to each vector.

I don't see any particular reason for one or the other. Given that we have used the num_vectors * 4 in the prototype implementation I think it is ok to leave this as is

wgtmac · 2026-07-22T05:18:45Z

+
+This encoding is adapted from the paper
+["ALP: Adaptive Lossless floating-Point Compression"](https://dl.acm.org/doi/10.1145/3626717)
+by Afroozeh and Boncz (SIGMOD 2024).


It seems that the paper has THREE authors but we have only listed TWO.

wgtmac · 2026-07-22T06:14:48Z

+    +----------------------------------------------------------+
+    |  3. FRAME OF REFERENCE (FOR)                             |
+    |     min_val = min(encoded[:])                            |
+    |     delta[i] = encoded[i] - min_val                      |


We need to clarify that delta[i] is an unsigned integer to avoid signed overflow.

Thanks. Updated.

State explicitly that the magic-number add/subtract are floating-point operations (only the final cast is integer), and that the constants are written as integers merely because they are exact. Add a MUST requiring FLOAT fast_round to be evaluated in single precision and DOUBLE in double precision, since a promotion of the FLOAT path to double drops the value into a binade with ULP below 1.0, defeating the rounding entirely.

The 1.5*2^n midpoint constant exists to make the magic-number rounding branchless, covering a symmetric range in a single formula. Since this spec branches on sign, the midpoint is the wrong choice: with a branch, 2^n doubles the usable domain (to +/-2^23 for FLOAT and +/-2^52 for DOUBLE) and makes the branch actually necessary, whereas with the midpoint the branch was redundant. Update the constants, the domain description, and the prose (the technique is no longer branchless).

Resolve open review comments on PR apache#557 (apacheGH-533): - parquet.thrift: add missing Encoding.ALP = 10 (thread 50). The enum value was never added; prose and the summary table said "ALP = 10" but the machine-readable contract lacked it. - Fix paper authorship to three authors: Afroozeh, Kuffo, Boncz (62). - "For each data page..." intro, drop misleading "page-level header" (16/49). - num_elements is the non-null value count (56). - Correct vector offset base for compressed / rep-def pages: use alp_data_start (decoded page data) instead of page_data_start+7 (55). - Note Vector Format applies to compression_mode=0 / integer_encoding=0 (59). - Label vector header vs data section; forward-ref num_exceptions/bit_width (60/61). - Clarify deltas are non-negative/unsigned with no sign extension (64). - Exceptions stored as raw IEEE 754 bits, MUST NOT canonicalize NaN (63). - Header byte-diagram alignment and "parallel encoding/decoding" wording (49/52). Leaves the contested fast_round sign-branching (threads 31/47/48) untouched pending reviewer consensus.

Follow-up to review comments on PR apache#557 (apacheGH-533): - Pipeline step 1 wording: "for this array" to match the diagram's "Input: float/double array" (thread 54). - Clarify FOR deltas are computed and stored as unsigned integers, explicitly to avoid signed overflow when max-min exceeds the encoded type's signed maximum; no sign extension on unpack (thread 64). - Annotate the Vector Format diagram with a bracket row marking the vector header vs the data section (thread 61).

alamb · 2026-07-29T06:56:48Z

SInce we have had a successful vote, I suggest we merge this PR and make additional changes as follow on PRs:

https://lists.apache.org/thread/hgmd58wrv9yoopcrf61m1bg211l65tbt

From what I can see, it looks like @prtkgaur resolved all blocking comments from @wgtmac 's most recent review (e.g. parquet.thrift change). There are a few more style/clarification suggestions from @wgtmac 's I think that we could do as a follow on.

Thus, @wgtmac could you confirm you are happy with merging this PR as is (perhaps by approving the PR) and if so I'll merge this PR in and we can continue to iterate in follow on PRs

wgtmac

I agree with @alamb's suggestion and fine with current PR. Thanks @prtkgaur and @alamb!

prtkgaur · 2026-07-27T23:44:38Z

+readings) while remaining fully lossless. Each value is encoded independently,
+enabling random access to individual vectors and parallel encode/decode.
+
+#### Overview


Got it. Let's address it in a follow up PR

prtkgaur · 2026-07-27T23:45:07Z

+-------------+------+------+-----+---------+----------+----------+-----+----------+
+```
+
+The compression pipeline below describes *one* way to produce a conforming


Ack. Makes sense. We can do this in a follow up PR

prtkgaur · 2026-07-29T15:45:04Z

+
+This encoding is adapted from the paper
+["ALP: Adaptive Lossless floating-Point Compression"](https://dl.acm.org/doi/10.1145/3626717)
+by Afroozeh, Kuffo, and Boncz (SIGMOD 2024).


prtkgaur · 2026-07-29T15:46:05Z

+    +----------------------------------------------------------+
+    |  3. FRAME OF REFERENCE (FOR)                             |
+    |     min_val = min(encoded[:])                            |
+    |     delta[i] = encoded[i] - min_val                      |


Thanks. Updated.

prtkgaur · 2026-07-29T16:08:13Z

+| FLOAT  | 2^22 + 2^23 = 12,582,912         | `(int32_t)((value + magic) - magic)` | `(int32_t)((value - magic) + magic)` |
+| DOUBLE | 2^51 + 2^52 = 6,755,399,441,055,744 | `(int64_t)((value + magic) - magic)` | `(int64_t)((value - magic) + magic)` |
+
+The sign branching is necessary because the technique relies on `value ± magic`


I over indexed on Ryan's example and updated the wording, thanks Andrew for the correction.
Given the examples that Andrew has brought up i'll keep the wording but instead of "encoder must do it", I can change it to "one of the ways of doing it is..."

prtkgaur · 2026-07-29T16:54:58Z

+ratio and exception count, never correctness or what a reader decodes. The
+`fast_round` technique below is one recommended implementation.
+
+##### Fast Rounding (informative)


@alamb do you recommend I simplify the language here or remove part of the section?

I'm going for simplification for now

alamb · 2026-07-29T20:50:22Z

I just had a brief sync with @prtkgaur - I think he plans to simplify a few things and then we'll merge this one in and iterate in follow on PRs

alamb

I took a look at the most recent edits and I think they look good. I will merge this in and we can continue iterating as follow on PRs

alamb · 2026-07-30T06:40:58Z

+example, places `8,388,608` in a binade whose ULP is far below 1.0 and no rounding
+occurs.
+
+Each form has a limited valid domain — roughly (-2^23, 2^23) for floats and


I like this wording

# Which issue does this PR close?  - Part of #10541 - Closes #8748 . - Related spec PR: apache/parquet-format#557 # Rationale for this change check issue  # What changes are included in this PR?  # Are these changes tested?  # Are there any user-facing changes?  --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

# Which issue does this PR close?  - Part of apache#10541 - Closes apache#8748 . - Related spec PR: apache/parquet-format#557 # Rationale for this change check issue  # What changes are included in this PR?  # Are these changes tested?  # Are there any user-facing changes?  --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

alamb changed the title ~~Add ALP (Adaptive Lossless floating-Point) encoding specification~~ GH-533: Add ALP (Adaptive Lossless floating-Point) encoding specification Mar 11, 2026

This was referenced Mar 11, 2026

GH-533: Adaptive Lossless Floating-Point (ALP) Encoding #548

Closed

[Parquet] ALP encoder/decoder support apache/arrow-rs#9372

Merged

emkornfield reviewed Mar 24, 2026

View reviewed changes

Comment thread Encodings.md Resolved

emkornfield reviewed Mar 24, 2026

View reviewed changes

alamb reviewed Apr 29, 2026

View reviewed changes

prtkgaur pushed a commit to prtkgaur/parquet-format that referenced this pull request Apr 29, 2026

Address review feedback on ALP encoding specification

c8ce8a7

Incorporate review comments from emkornfield and alamb on PR apache#557:

alamb mentioned this pull request Apr 30, 2026

[Proposal] Add ALP encoding support in parquet file format #533

Closed

alamb mentioned this pull request May 1, 2026

Write a Parquet Blog on new encoding Adaptive Lossless Floating Point (ALP) apache/parquet-site#175

Closed

alamb reviewed May 1, 2026

View reviewed changes

alamb mentioned this pull request May 3, 2026

[Parquet] Prototype ALP encoding apache/arrow-rs#8748

Closed

pitrou reviewed May 5, 2026

View reviewed changes

alamb mentioned this pull request May 9, 2026

Align structure names with spec sdf-jkl/arrow-rs#5

Merged

prtkgaur mentioned this pull request May 13, 2026

GH-48701: [C++][Parquet] Add ALPpd encoding apache/arrow#48345

Merged

sfc-gh-pgaur added 2 commits May 14, 2026 00:47

Address review feedback on ALP encoding specification

69aaf62

Incorporate review comments from emkornfield and alamb on PR apache#557:

wgtmac requested changes Jul 22, 2026

View reviewed changes

wgtmac reviewed Jul 22, 2026

View reviewed changes

sfc-gh-pgaur added 4 commits July 23, 2026 16:23

alamb requested a review from wgtmac July 29, 2026 06:46

wgtmac approved these changes Jul 29, 2026

View reviewed changes

sfc-gh-pgaur added 2 commits July 29, 2026 16:15

Mark ALP fast_round rounding as informative

1293332

Address review: raw-bit exceptions, fast_round reference wording

4752c51

prtkgaur commented Jul 29, 2026

View reviewed changes

Address review: trim informative Fast Rounding section

2447379

prtkgaur commented Jul 29, 2026

View reviewed changes

alamb approved these changes Jul 30, 2026

View reviewed changes

alamb merged commit 2076361 into apache:master Jul 30, 2026
4 checks passed

This was referenced Aug 1, 2026

Add ALP Encoding blog apache/parquet-site#195

Merged

Invitation to provide feedback on New Parquet Encoding: Adaptive Lossless Floating Point Compression pola-rs/polars#26279

Closed

Implement ALP encoding apache/arrow-rs#10541

Open

This was referenced Aug 13, 2026

GH-602: Move ALP encoding specification to AlpEncoding.md #608

Merged

Add ALP encoding to the feature matrix apache/parquet-site#199

Closed

Add ALP encoding to the feature matrix alamb/parquet-site#3

Closed

alamb mentioned this pull request Sep 8, 2026

Add parquet-format 2.14.0 features to the support features page apache/parquet-site#202

Merged

alamb mentioned this pull request Sep 16, 2026

Update parquet-format submodule to 2.14.0 apache/parquet-site#206

Merged

gunnarmorling mentioned this pull request Oct 15, 2026

Support ALP (Adaptive Lossless floating-Point) encoding for reading and writing hardwood-hq/hardwood#581

Closed


		##### Header (7 bytes)

		All multi-byte values are little-endian.

	All multi-byte values are little-endian.
	All multi-byte values are stored in little-endian order.

	\| 2 \| log_vector_size \| 1 byte \| uint8 \| log2(vector\_size). Must be in \[3, 15\]. Default: 10 (vector size 1024) \|
	\| 2 \| log_vector_size \| 1 byte \| uint8 \| log2(vector\_size). Must be in \[3, 15\]. Recommended default: 10 (vector size 1024) \|

	The first offset equals `num_vectors * 4` (pointing just past the offset array).
	The first offset always equals `num_vectors * 4` (pointing just past the offset array).

	\| PackedValues \| ceil(N * bit\_width / 8) \| Bit-packed delta values \|
	\| PackedValues \| ceil(`vector_size` * bit\_width / 8) \| Bit-packed delta values \|

	\| ExceptionValues \| num\_exceptions * sizeof(T) \| Original float/double values \|
	\| ExceptionValues \| num\_exceptions * sizeof(encoded type) (float=4 and double=8) \| Original float/double values \|

	\| DOUBLE \| 2^51 + 2^52 = 6,755,399,441,055,744 \| `(long)((value + magic) - magic)` \|
	\| DOUBLE \| 2^51 + 2^52 = 6,755,399,441,055,744 \| `(int64_t)((value + magic) - magic)` \|

	\| FLOAT \| 2^22 + 2^23 = 12,582,912 \| `(int)((value + magic) - magic)` \|
	\| FLOAT \| 2^22 + 2^23 = 12,582,912 \| `(int32_t)((value + magic) - magic)` \|

	\| encoded = round( value * 10^e * 10^(-f) ) \|
	\| encoded = fast_round( value * 10^e * 10^(-f) ) \|

	### Supported Encodings

	For details on current implementation status, see the [Implementation Status](https://parquet.apache.org/docs/file-format/implementationstatus/#encodings) page.

	\| Encoding type \| Encoding enum \| Supported Types \|
	\| ------------------------------------------------ \| --------------------------------------------------------- \| ------------------------------------------------- \|
	\| [Plain](#PLAIN) \| PLAIN = 0 \| All Physical Types \|
	\| [Dictionary Encoding](#DICTIONARY) \| PLAIN_DICTIONARY = 2 (Deprecated) <br> RLE_DICTIONARY = 8 \| All Physical Types \|
	\| [Run Length Encoding / Bit-Packing Hybrid](#RLE) \| RLE = 3 \| BOOLEAN, Dictionary Indices \|
	\| [Delta Encoding](#DELTAENC) \| DELTA_BINARY_PACKED = 5 \| INT32, INT64 \|
	\| [Delta-length byte array](#DELTALENGTH) \| DELTA_LENGTH_BYTE_ARRAY = 6 \| BYTE_ARRAY \|
	\| [Delta Strings](#DELTASTRING) \| DELTA_BYTE_ARRAY = 7 \| BYTE_ARRAY, FIXED_LEN_BYTE_ARRAY \|
	\| [Byte Stream Split](#BYTESTREAMSPLIT) \| BYTE_STREAM_SPLIT = 9 \| INT32, INT64, FLOAT, DOUBLE, FIXED_LEN_BYTE_ARRAY \|


		##### Fast Rounding

		The rounding function uses a "magic number" technique for branchless rounding:

-The rounding function uses a "magic number" technique for branchless rounding:
+The `fast_round` function uses a "magic number" technique for branchless rounding.
+`fast_round(value)` is defined as follows:

	##### Frame of Reference and Bit-Packing
	##### Example: Frame of Reference and Bit-Packing


		##### Frame of Reference and Bit-Packing

		After decimal encoding and exception substitution:

	After decimal encoding and exception substitution:
	Given the following data after decimal encoding and exception substitution:

	enabling random access to individual vectors and parallel encode/decode.
	enabling random access to individual values and parallel encode/decode.

		floating-point rounding across languages. Implementations must ensure that the
		encoder and decoder use identical power-of-10 values for a given exponent.

		Byte: 0 1 2 3 4 5 6
		+----------------+---------------+--------------+----+----+----+----+

	\| Select (exponent, factor) pair for this vector \|
	\| Select (exponent, factor) pair for this array \|

	\| 3 \| num_elements \| 4 bytes \| int32 \| Total number of floating-point values in the page \|
	\| 3 \| num_elements \| 4 bytes \| int32 \| Total number of non-null floating-point values in the page \|

Conversation

prtkgaur commented Mar 11, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Do these changes have PoC implementations?

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

prtkgaur commented Mar 11, 2026 •

edited by alamb

Loading