Skip to content

Commit c7934d3

Browse files
larsk-dbvkorukanti
authored andcommitted
Update Spec to use the portable Deletion Vector serialization format
## Description The RoaringBitmap community has recently standardised on a ["portable" 64-bit serialization format](https://github.com/RoaringBitmap/RoaringFormatSpec#extention-for-64-bit-implementations). We should switch the Deletion Vector feature to use this format, as it will make adoption by 3rd parties easier, enabling them to use an off-the-shelf 64-bit implementation from the RoaringBitmap library of their choice, as long as it supports this format (which CRoaring already does and the next release of Java RoaringBitmap will do as well). This PR makes the appropriate changes to the DV format description in the appendix of the Delta spec. N/A ## Does this PR introduce _any_ user-facing changes? No. Closes #1494 Signed-off-by: Venki Korukanti <[email protected]> GitOrigin-RevId: 1503f42337a553879ad527bee8b78bf2d034bfd2
1 parent a1fbbc6 commit c7934d3

1 file changed

Lines changed: 30 additions & 10 deletions

File tree

‎PROTOCOL.md‎

Lines changed: 30 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -782,28 +782,48 @@ Deletion Vectors are basically sets of row indexes, that is 64-bit integers that
782782
- Bitmap-compressed, when the number of values in the block is large and scattered.
783783
- Run-length encoded, when the number of values in the block is large, but clustered.
784784

785-
The serialization format is standardized, and both [Java](https://github.com/lemire/RoaringBitmap/) and [C/C++](https://github.com/RoaringBitmap/CRoaring) implementations are available (among others).
785+
The serialization format is [standardized](https://github.com/RoaringBitmap/RoaringFormatSpec), and both [Java](https://github.com/lemire/RoaringBitmap/) and [C/C++](https://github.com/RoaringBitmap/CRoaring) implementations are available (among others).
786786

787-
Since RoaringBitmap only covers 32-bit integers, we extend the format in a simple manner by keeping an array of 32-bit RoaringBitmaps and using the upper 32-bits to index into the array.
787+
The above description only applies to 32-bit bitmaps, but Deletion Vectors use 64-bit integers. In order to extend coverage from 32 to 64 bits, RoaringBitmaps defines a "portable" serialization format in the [RoaringBitmaps Specification](https://github.com/RoaringBitmap/RoaringFormatSpec#extention-for-64-bit-implementations). This format essentially splits the space into an outer part with the most significant 32-bit "keys" indexing the least significant 32-bit RoaringBitmaps in ascending sequence. The spec calls these least signficant 32-bit RoaringBitmaps "buckets".
788788

789-
The serialization format for such a `RoaringBitmapArray` is as follows (all numerical values are written in little endian byte order):
789+
Bytes | Name | Description
790+
-|-|-
791+
0 – 7 | numBuckets | The number of distinct 32-bit buckets in this bitmap.
792+
`repeat for each bucket b` | | For each bucket in ascending order of keys.
793+
`<start of b>` – `<start of b> + 3` | key | The most significant 32-bit of all the values in this bucket.
794+
`<start of b> + 4` – `<end of b>` | bucketData | A serialized 32-bit RoaringBitmap with all the least signficant 32-bit entries in this bucket.
795+
796+
The 32-bit serialization format then consists of a header that describes all the (least signficant) 16-bit containers, their types (s. above), and their their key (most significant 16-bits).
797+
This is followed by the data for each individual container in a container-specific format.
798+
799+
Reference Implementations of the Roaring format:
800+
801+
- [32-bit Java RoaringBitmap](https://github.com/RoaringBitmap/RoaringBitmap/blob/c7993318d7224cd3cc0244dcc99c8bbc5ddb0c87/RoaringBitmap/src/main/java/org/roaringbitmap/RoaringArray.java#L905-L949)
802+
- [64-bit Java RoaringNavigableBitmap](https://github.com/RoaringBitmap/RoaringBitmap/blob/c7993318d7224cd3cc0244dcc99c8bbc5ddb0c87/RoaringBitmap/src/main/java/org/roaringbitmap/longlong/Roaring64NavigableMap.java#L1253-L1260)
803+
804+
Delta uses the format described above as a black box, but with two additions:
805+
806+
1. We prepend a "magic number", which can be used to make sure we are reading the correct format and also retains the ability to evolve the format in the future.
807+
2. We require that every "key" (s. above) in the bitmap has a 0 as its most significant bit. This ensures that in Java, where values are read signed, we never read negative keys.
808+
809+
The concrete serialization format is as follows (all numerical values are written in little endian byte order):
790810

791811
Bytes | Name | Description
792812
-|-|-
793-
0 — 3 | magicNumber | 1681511376; Indicates that the following bytes are serialised in this exact format. Future alternative—but related—formats must have a different magic number, for example by incrementing this one.
794-
4 — 7 | numBitmaps | The number of 32-bit bitmaps in the array
795-
`repeat for i in 0 to length` | | For each 32-bit RoaringBitmap
796-
`<start of i>` — `<start of i> + 3` | bitmapDataSize | Number of bytes of this bitmap’s serialized representation.
797-
`<start of i> + 4` — `<start of i> + 4 + bitmapDataSize` | bitmapData | Serialized bytes in the RoaringBitmap standard serialization format.
813+
0 — 3 | magicNumber | 1681511377; Indicates that the following bytes are serialized in this exact format. Future alternative—but related—formats must have a different magic number, for example by incrementing this one.
814+
4 — end | bitmap | A serialized 64-bit bitmap in the portable standard format as defined in the [RoaringBitmaps Specification](https://github.com/RoaringBitmap/RoaringFormatSpec#extention-for-64-bit-implementations). This can be treated as a black box by any Delta implementation that has a native, standard-compliant RoaringBitmap library available to pass these bytes to.
815+
816+
### Deletion Vector File Storage Format
798817

799-
The format for storing DVs in file storage is one (or more) of these `RoaringBitmapArray`s per file, together with a checksum for each DV:
818+
Deletion Vectors can be stored in files in cloud storage or inline in the Delta log.
819+
The format for storing DVs in file storage is one (or more) of DV, using the 64-bit RoaringBitmaps described in the previous section, per file, together with a checksum for each DV:
800820

801821
Bytes | Name | Description
802822
-|-|-
803823
0 — 1 | version | The format version of this file: `1` for the format described here.
804824
`repeat for each DV i` | | For each DV
805825
`<start of i>` — `<start of i> + 3` | dataSize | Size of this DV’s data (without the checksum)
806-
`<start of i> + 4` — `<start of i> + 4 + dataSize - 1` | bitmapData | One `RoaringBitmapArray` serialised as described above.
826+
`<start of i> + 4` — `<start of i> + 4 + dataSize - 1` | bitmapData | One 64-bit RoaringBitmap serialised as described above.
807827
`<start of i> + 4 + dataSize` — `<start of i> + 4 + dataSize + 3` | checksum | CRC-32 checksum of `bitmapData`
808828

809829
## Per-file Statistics

0 commit comments

Comments
 (0)