|
1 | 1 | # IPC Format |
2 | 2 |
|
| 3 | +The IPC format streams Vortex arrays between processes as a sequence of self-delimiting messages. |
| 4 | +Arrays keep their encodings on the wire, so compressed data crosses the process boundary without |
| 5 | +being decompressed. The format is implemented by the `vortex-ipc` crate, re-exported as |
| 6 | +[`vortex::ipc`](https://docs.rs/vortex/latest/vortex/ipc/index.html). |
| 7 | + |
3 | 8 | :::{warning} |
4 | | -This page is under construction. |
| 9 | +The IPC format is unstable. It is not covered by the [versioning](versioning.md) guarantees of the |
| 10 | +[file format](file-format.md), and it has a single message version, `V0`, which readers require |
| 11 | +exactly. If you want to persist data, please consider writing a Vortex file instead. |
5 | 12 | ::: |
6 | 13 |
|
7 | | -Planned content: |
| 14 | +## Streams |
| 15 | + |
| 16 | +An IPC stream carries a single logical array as a sequence of chunks: |
| 17 | + |
| 18 | +``` |
| 19 | +[DType message] [Array message] [Array message] ... |
| 20 | +``` |
| 21 | + |
| 22 | +1. The first message is a `DTypeMessage` carrying the stream's data type. |
| 23 | +2. Every following message is an `ArrayMessage` holding one chunk, whose data type must equal the |
| 24 | + stream's. |
| 25 | +3. The stream ends at end-of-input after a complete message. There is no end-of-stream marker, so |
| 26 | + a stream with no chunks is a lone `DTypeMessage`. |
| 27 | + |
| 28 | +## Message framing |
| 29 | + |
| 30 | +Every message has the same framing: |
| 31 | + |
| 32 | +``` |
| 33 | +<4 bytes> u32 little-endian header length, H |
| 34 | +<H bytes> Message FlatBuffer |
| 35 | +<body_size bytes> message body |
| 36 | +``` |
| 37 | + |
| 38 | +The `Message` FlatBuffer is the header, which contains the following fields: |
| 39 | + |
| 40 | +- **`version`** (`MessageVersion`, default `V0`): the message format version. Readers reject any |
| 41 | + value other than `V0`. |
| 42 | +- **`header`** (`MessageHeader` union): the message type, which also determines how to interpret |
| 43 | + the body. |
| 44 | +- **`body_size`** (`uint64`): the exact length of the body, so a reader always knows where the next |
| 45 | + message starts. |
| 46 | + |
| 47 | +Messages are written back-to-back, with no padding between or within them. This means that |
| 48 | +header or body bytes are not guaranteed to land on an aligned offset. Thus, after receiving a |
| 49 | +message, a reader copies each FlatBuffer and data buffer into appropriately aligned memory. |
| 50 | + |
| 51 | +## Message types |
| 52 | + |
| 53 | +### DTypeMessage |
| 54 | + |
| 55 | +The body is a [`DType`](dtype-format.md) FlatBuffer (`root_type DType` in `dtype.fbs`). The |
| 56 | +`DTypeMessage` table itself has no fields. |
| 57 | + |
| 58 | +### ArrayMessage |
| 59 | + |
| 60 | +The body is a serialized array. The header carries the information needed to decode it: |
| 61 | + |
| 62 | +- **`row_count`** (`uint32`) — the length of the root array. |
| 63 | +- **`encodings`** (`[string]`) — the encoding IDs the array references, such as `vortex.primitive`. |
| 64 | + |
| 65 | +The body has the same shape as any other serialized array: |
| 66 | + |
| 67 | +``` |
| 68 | +[buffer 0] [buffer 1] ... [buffer N-1] [Array FlatBuffer] [u32 little-endian FlatBuffer length] |
| 69 | +``` |
| 70 | + |
| 71 | +A reader decodes the body from the end: |
| 72 | + |
| 73 | +1. Read the trailing `u32` to find the length of the `Array` FlatBuffer that precedes it. The |
| 74 | + bytes before the FlatBuffer hold the data buffers. |
| 75 | +2. Locate each data buffer from the `Array.buffers` table, in order. Buffer `i` starts `padding` |
| 76 | + bytes after the end of buffer `i - 1`, where the first buffer's predecessor ends at offset 0, |
| 77 | + and spans `length` bytes. Align it to 2<sup>`alignment_exponent`</sup> bytes. |
| 78 | +3. Decode the `ArrayNode` tree from `Array.root`. Each node's `encoding` is an index into the |
| 79 | + message's `encodings` list, and its `buffers` are indices into the `Array.buffers` table. |
| 80 | +4. Decode the root node with the stream's data type and `row_count`. Each encoding derives the data |
| 81 | + types and lengths of its children from its own metadata. Any `stats` present on a node are |
| 82 | + attached to the decoded array. |
| 83 | + |
| 84 | +Each message is self-contained: its `encodings` list and buffer table apply only to that message. |
8 | 85 |
|
9 | | -- Overview of the Vortex IPC wire format |
10 | | -- Message framing and serialization |
11 | | -- Compressed array transfer |
12 | | -- Use by the Scan API for remote interchange |
| 86 | +The IPC writer never pads buffers, so `padding` is always 0. Readers should still honor it, |
| 87 | +because [Vortex files](file-format.md) use the same array layout, which do pad. The writer always sets |
| 88 | +`compression` to `None`. `LZ4` is reserved in the schema but not implemented; current readers |
| 89 | +don't check the field. |
| 90 | + |
| 91 | +The `ArrayNode` tree and its statistics are described in |
| 92 | +[Serialization](../developer-guide/internals/serialization.md). |
| 93 | + |
| 94 | +### BufferMessage |
| 95 | + |
| 96 | +The body is a single raw byte buffer. `alignment_exponent` is the alignment the receiver must give |
| 97 | +it, as a power of two. Readers reject exponents above 16 (64 KiB), since the value is untrusted. |
| 98 | + |
| 99 | +The stream writers never emit a `BufferMessage`, and the stream readers reject one. It is available |
| 100 | +only through the message-level API below. |
| 101 | + |
| 102 | +## Rust API |
| 103 | + |
| 104 | +The crate has three layers. Most callers only need the array-stream layer. |
| 105 | + |
| 106 | +| Layer | Write | Read | |
| 107 | +|---------------|----------------------------------------------------------------|---------------------------------------------------------------------------------------------| |
| 108 | +| Array streams | `stream::ArrayStreamIPC`, `iterator::ArrayIteratorIPC` | `stream::AsyncIPCReader`, `iterator::SyncIPCReader` | |
| 109 | +| Messages | `messages::AsyncMessageWriter`, `messages::SyncMessageWriter` | `messages::AsyncMessageReader`, `messages::SyncMessageReader`, `messages::BufMessageReader` | |
| 110 | +| Framing | `messages::MessageEncoder` | `messages::MessageDecoder` | |
| 111 | + |
| 112 | +`ArrayStreamIPC` and `ArrayIteratorIPC` are implemented for every `ArrayStream` and |
| 113 | +`ArrayIterator`. They write the `DTypeMessage`, then one `ArrayMessage` per item. The iterator and |
| 114 | +stream from `ArrayRef::to_array_iterator` and `ArrayRef::to_array_stream` yield one item per chunk |
| 115 | +of a chunked array. |
| 116 | + |
| 117 | +`AsyncIPCReader` and `SyncIPCReader` read such a stream back as an `ArrayStream` or `ArrayIterator`. |
| 118 | +They fail if the first message is not a `DTypeMessage`, if a later message is not an |
| 119 | +`ArrayMessage`, or if a decoded array's data type differs from the stream's. |
| 120 | + |
| 121 | +`MessageDecoder` holds no IO. It reports the total number of bytes it needs to make progress, so |
| 122 | +callers can drive it over any transport. |
| 123 | + |
| 124 | +```rust |
| 125 | +use std::io::Cursor; |
| 126 | + |
| 127 | +use vortex::VortexSessionDefault; |
| 128 | +use vortex::array::IntoArray; |
| 129 | +use vortex::array::iter::ArrayIteratorExt; |
| 130 | +use vortex::buffer::buffer; |
| 131 | +use vortex::error::VortexResult; |
| 132 | +use vortex::ipc::iterator::ArrayIteratorIPC; |
| 133 | +use vortex::ipc::iterator::SyncIPCReader; |
| 134 | +use vortex::session::VortexSession; |
| 135 | + |
| 136 | +fn round_trip() -> VortexResult<()> { |
| 137 | + let session = VortexSession::default(); |
| 138 | + let array = buffer![1i32, 2, 3].into_array(); |
| 139 | + |
| 140 | + let bytes = array.to_array_iterator().write_ipc(Vec::new(), &session)?; |
| 141 | + |
| 142 | + let reader = SyncIPCReader::try_new(Cursor::new(bytes), &session)?; |
| 143 | + assert_eq!(reader.read_all()?.len(), 3); |
| 144 | + Ok(()) |
| 145 | +} |
| 146 | +``` |
| 147 | + |
| 148 | +Decoding resolves encoding IDs through the session, so the reader's session must register every |
| 149 | +encoding the writer used. |
| 150 | + |
| 151 | +## Limitations |
| 152 | + |
| 153 | +- **No zero-copy reads.** Buffers are unpadded on the wire, so the reader copies them into aligned |
| 154 | + memory. |
| 155 | +- **No state shared across messages.** Each message repeats its own encoding list, and data such |
| 156 | + as a dictionary shared between chunks is sent again with every chunk. |
| 157 | +- **Row count limit.** An `ArrayMessage` holds at most `u32::MAX` rows. Larger arrays must be split |
| 158 | + into chunks. |
| 159 | +- **No end-of-stream marker.** A stream cut exactly at a message boundary is indistinguishable from |
| 160 | + a complete one. Transports that can drop data must detect truncation themselves. |
| 161 | + |
| 162 | +## FlatBuffer definition |
| 163 | + |
| 164 | +:::{literalinclude} ../../vortex-ipc/flatbuffers/vortex-serde/message.fbs |
| 165 | +::: |
0 commit comments