You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c7934d3
Browse filesBrowse the repository at this point in the historyBrowse files
Update Spec to use the portable Deletion Vector serialization format
## Description
The RoaringBitmap community has recently standardised on a ["portable" 64-bit serialization format](https://github.com/RoaringBitmap/RoaringFormatSpec#extention-for-64-bit-implementations).
We should switch the Deletion Vector feature to use this format, as it will make adoption by 3rd parties easier, enabling them to use an off-the-shelf 64-bit implementation from the RoaringBitmap library of their choice, as long as it supports this format (which CRoaring already does and the next release of Java RoaringBitmap will do as well).
This PR makes the appropriate changes to the DV format description in the appendix of the Delta spec.
N/A
## Does this PR introduce _any_ user-facing changes?
No.
Closes#1494
Signed-off-by: Venki Korukanti <[email protected]>
GitOrigin-RevId: 1503f42337a553879ad527bee8b78bf2d034bfd2
Copy file name to clipboardExpand all lines: PROTOCOL.md
+30-10Lines changed: 30 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -782,28 +782,48 @@ Deletion Vectors are basically sets of row indexes, that is 64-bit integers that
782
782
- Bitmap-compressed, when the number of values in the block is large and scattered.
783
783
- Run-length encoded, when the number of values in the block is large, but clustered.
784
784
785
-
The serialization format is standardized, and both [Java](https://github.com/lemire/RoaringBitmap/) and [C/C++](https://github.com/RoaringBitmap/CRoaring) implementations are available (among others).
785
+
The serialization format is [standardized](https://github.com/RoaringBitmap/RoaringFormatSpec), and both [Java](https://github.com/lemire/RoaringBitmap/) and [C/C++](https://github.com/RoaringBitmap/CRoaring) implementations are available (among others).
786
786
787
-
Since RoaringBitmap only covers 32-bit integers, we extend the format in a simple manner by keeping an array of 32-bit RoaringBitmaps and using the upper 32-bits to index into the array.
787
+
The above description only applies to 32-bit bitmaps, but Deletion Vectors use 64-bit integers. In order to extend coverage from 32 to 64 bits, RoaringBitmaps defines a "portable" serialization format in the [RoaringBitmaps Specification](https://github.com/RoaringBitmap/RoaringFormatSpec#extention-for-64-bit-implementations). This format essentially splits the space into an outer part with the most significant 32-bit "keys" indexing the least significant 32-bit RoaringBitmaps in ascending sequence. The spec calls these least signficant 32-bit RoaringBitmaps "buckets".
788
788
789
-
The serialization format for such a `RoaringBitmapArray` is as follows (all numerical values are written in little endian byte order):
789
+
Bytes | Name | Description
790
+
-|-|-
791
+
0 – 7 | numBuckets | The number of distinct 32-bit buckets in this bitmap.
792
+
`repeat for each bucket b` | | For each bucket in ascending order of keys.
793
+
`<start of b>` – `<start of b> + 3` | key | The most significant 32-bit of all the values in this bucket.
794
+
`<start of b> + 4` – `<end of b>` | bucketData | A serialized 32-bit RoaringBitmap with all the least signficant 32-bit entries in this bucket.
795
+
796
+
The 32-bit serialization format then consists of a header that describes all the (least signficant) 16-bit containers, their types (s. above), and their their key (most significant 16-bits).
797
+
This is followed by the data for each individual container in a container-specific format.
Delta uses the format described above as a black box, but with two additions:
805
+
806
+
1. We prepend a "magic number", which can be used to make sure we are reading the correct format and also retains the ability to evolve the format in the future.
807
+
2. We require that every "key" (s. above) in the bitmap has a 0 as its most significant bit. This ensures that in Java, where values are read signed, we never read negative keys.
808
+
809
+
The concrete serialization format is as follows (all numerical values are written in little endian byte order):
790
810
791
811
Bytes | Name | Description
792
812
-|-|-
793
-
0 — 3 | magicNumber | 1681511376; Indicates that the following bytes are serialised in this exact format. Future alternative—but related—formats must have a different magic number, for example by incrementing this one.
794
-
4 — 7 | numBitmaps | The number of 32-bit bitmaps in the array
795
-
`repeat for i in 0 to length` | | For each 32-bit RoaringBitmap
796
-
`<start of i>` — `<start of i> + 3` | bitmapDataSize | Number of bytes of this bitmap’s serialized representation.
797
-
`<start of i> + 4` — `<start of i> + 4 + bitmapDataSize` | bitmapData | Serialized bytes in the RoaringBitmap standard serialization format.
813
+
0 — 3 | magicNumber | 1681511377; Indicates that the following bytes are serialized in this exact format. Future alternative—but related—formats must have a different magic number, for example by incrementing this one.
814
+
4 — end | bitmap | A serialized 64-bit bitmap in the portable standard format as defined in the [RoaringBitmaps Specification](https://github.com/RoaringBitmap/RoaringFormatSpec#extention-for-64-bit-implementations). This can be treated as a black box by any Delta implementation that has a native, standard-compliant RoaringBitmap library available to pass these bytes to.
815
+
816
+
### Deletion Vector File Storage Format
798
817
799
-
The format for storing DVs in file storage is one (or more) of these `RoaringBitmapArray`s per file, together with a checksum for each DV:
818
+
Deletion Vectors can be stored in files in cloud storage or inline in the Delta log.
819
+
The format for storing DVs in file storage is one (or more) of DV, using the 64-bit RoaringBitmaps described in the previous section, per file, together with a checksum for each DV:
800
820
801
821
Bytes | Name | Description
802
822
-|-|-
803
823
0 — 1 | version | The format version of this file: `1` for the format described here.
804
824
`repeat for each DV i` | | For each DV
805
825
`<start of i>` — `<start of i> + 3` | dataSize | Size of this DV’s data (without the checksum)
806
-
`<start of i> + 4` — `<start of i> + 4 + dataSize - 1` | bitmapData | One `RoaringBitmapArray` serialised as described above.
826
+
`<start of i> + 4` — `<start of i> + 4 + dataSize - 1` | bitmapData | One 64-bit RoaringBitmap serialised as described above.
807
827
`<start of i> + 4 + dataSize` — `<start of i> + 4 + dataSize + 3` | checksum | CRC-32 checksum of `bitmapData`
0 commit comments