Activity

1. pitrou commented on Jul 21, 2026

   

   pitrou

   on Jul 21, 2026

   Member

   More actions

   • All possible ALP bit widths sizes (1 -> 15 == 65k)

   Do you mean log2 vector size?

   👍React with 👍1Reacted by Andrew Lamb
2. alamb commented on Aug 3, 2026

   

   alamb

   on Aug 3, 2026

   ContributorAuthor

   More actions

   I spent some time designing what I think would be an example file that covers all the relevant corner cases.

   1. A single parquet file with multiple columns (to keep the size down)
   2. Columns with the same values encoded with multiple encodings (so downstream decoders can compare with known results, following the model of byte_stream_split_extended.gzip.parquet)

   Columns

   Column	Encoding	Rationale / coverage
   float_plain, double_plain	PLAIN + zstd	In-file reference: readers can bit-compare the ALP columns against these
   float_alp_1024, double_alp_1024	ALP, 1024-value vectors	The default vector size
   float_alp_4096, double_alp_4096	ALP, 4096-value vectors	Readers must honor log_vector_size from the page header rather than assume 1024
   float_alp_32, double_alp_32	ALP, 32-value vectors	many vectors per page, stress per-vector metadata loop

   Data distribution (9000 rows)

   Base distribution: random values in [-10.00, 10.00] with exactly 2 decimal digits (e.g. 9.43) — losslessly encodable by ALP, so base values produce zero exceptions.

   Total row count is deliberately not a multiple of any vector size to ensure coverage of incomplete vectors

   Rows	Contents	Rationale / coverage
   0–1023	base	Happy path: full vector, small FOR bit width, no exceptions
   1024–2047	base, plus: NaN at 1024 and 2047 (first/last element of the vector), NaN at 1500, +inf at 2000, −inf at 2001, −0.0 at 2002, subnormal at 2003 (5e-324 double / 1e-45 float)	NaN / Inf and sign/precision edge values via the exception mechanism; exceptions at exact vector boundaries; statistics conventions (NaN excluded from min/max, ±inf included, nan_count, −0.0/+0.0 normalization)
   2048–3071	base, plus 3.141592653589793 at 2500	Exactly one exception: the full-mantissa value cannot round-trip as a decimal, and nothing else in the vector is exceptional
   3072–4095	base, plus 44974934523.343 at 3100 and -1243432432.3432 at 3711	Large-magnitude values
   4096–5119	every 2nd value full-mantissa random (e.g. from random() * 20 - 10 unrounded), rest base	Many (but not all) exceptions
   5120–6143	all values full-mantissa random	All exceptions (the "random float data" case)
   6144–7167	base but with 4 decimal digits (e.g. 3.1416)	different exponent/factor than the other vectors
   7168–8191	constant (all 7.77)	bit_width = 0 vectors
   8192–8999	base, with nulls at every 100th row	Trailing partial vector + null handling

   Documentation

   Following the model of Additional types and the variant README, the PR should add a data/README.md section with: the schema, the row-range table above, and verification instructions.
3. alamb commented on Aug 4, 2026

   

   alamb

   on Aug 4, 2026

   ContributorAuthor

   More actions

   Here is my proposal:

   • Add single ALP test file (216KB, covers corner cases) #119
4. alamb

   closed this as completedin #119on Aug 21, 2026