You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 0751940
Browse filesBrowse the repository at this point in the historyBrowse files
Refine FoRScheme to per-chunk references when fastlanes.for.v2 is allowed (#10136)
Stacked on #10135.
## Summary
`FoRScheme` compresses with one reference per 1024-element chunk when
the compressor's allowed serialized IDs include `fastlanes.for.v2`,
following `DecimalScheme`. Default writes are unchanged: the core
edition allows only `fastlanes.for`, so `refine` keeps v1.
## `FoRScheme`
- `FoRScheme` gains v1 and v2 modes with `v1()` and `v2()` constructors;
`Default` is v1. The session registers `FOR_V1`.
- `refine` returns v2 when `fastlanes.for.v2` is allowed, and v1
otherwise.
- `produced_encodings`: v1 declares `fastlanes.for`; v2 declares
`fastlanes.for` and `fastlanes.for.v2`.
- **Compress**: v2 always encodes with `FoR::encode_chunked`. Per-chunk
references never pack wider than one reference, and the chunked encoder
is faster than the single-reference one. The encoded child is bit-packed
as before, and the references are compressed as child 0 with
`compress_child`. When every chunk has the same minimum, the references
compress to a constant and the array serializes as `fastlanes.for`.
- **Estimate**: both modes use v1's estimate, unchanged. That is
conservative for v2: it can understate v2's ratio, and it skips arrays
where only per-chunk references would help, such as those whose minimum
is zero or whose single-reference width is no narrower than plain
BitPacking.
## CUDA
The CUDA preset excludes `fastlanes.for.v2`, as it does multi-part
decimals. The GPU FoR decoder returns an error for per-chunk references.
## Follow-ups
- **Estimate per-chunk references.** The existing samples can't estimate
them: the sampler stitches 64-value runs from across the array, so no
sampled chunk resembles a real one. Options are chunk-aligned sampling,
or a read-only pass computing each chunk's min and max, which is much
cheaper than `encode_chunked` because it writes nothing. The ratio
should then count the references child. Done when
`estimate_skips_arrays_only_chunk_references_narrow` expects v2, the
golden snapshots don't regress, and the change in compression time is
measured.
---------
Signed-off-by: Matt Katz <[email protected]>
0 commit comments