Skip to content

Commit 38fa7e3

Browse files
mprammerclaude
andauthored
docs: improve IPC documentation (#10036)
Replaces the IPC format stub with a spec of the wire format as `vortex-ipc` implements it, and fixes two pages that misdescribed IPC. Co-authored-by: Claude <[email protected]> Signed-off-by: Martin Prammer <[email protected]>
1 parent 0ff150e commit 38fa7e3

3 files changed

Lines changed: 167 additions & 35 deletions

File tree

‎docs/concepts/scanning.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -55,11 +55,10 @@ Splits can also be serialized for distributed execution across remote workers.
5555
### Remote Sources
5656

5757
A source may front remote storage rather than local files. In this case, the split's execution
58-
issues a remote call and receives the result over the network. The
59-
[Vortex IPC format](../specs/ipc-format.md) can be used as the wire protocol for these calls, allowing
60-
compressed arrays to be transferred without decompression. This gives remote sources the same
61-
zero-decompression benefits as local scans -- the data stays in its compressed encoding end-to-end,
62-
from remote storage through the network and into the query engine.
58+
issues a remote call and receives the result over the network. Vortex does not ship such a source,
59+
but the [Vortex IPC format](../specs/ipc-format.md) is suited to be its wire protocol: arrays cross
60+
the network in their compressed encodings, so the data is not decompressed between remote storage
61+
and the query engine.
6362

6463
## Filter Pushdown
6564

‎docs/developer-guide/internals/serialization.md‎

Lines changed: 4 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -35,30 +35,10 @@ the current implementation.
3535

3636
## IPC Format
3737

38-
The IPC format wraps serialized arrays in a message-oriented protocol for streaming between
39-
processes. Each message consists of:
40-
41-
```
42-
[u32 flatbuffer length] [flatbuffer Message] [body bytes]
43-
```
44-
45-
Three message types are defined:
46-
47-
- **ArrayMessage** -- a serialized array with its row count and an encoding context that maps
48-
dictionary indices to encoding IDs.
49-
- **BufferMessage** -- a raw buffer with an alignment exponent, used for transferring individual
50-
segments.
51-
- **DTypeMessage** -- a serialized dtype, used to communicate the schema before data transfer
52-
begins.
53-
54-
The IPC format is used both for inter-process communication and as the wire protocol for remote
55-
source execution in the [Scan API](/concepts/scanning).
56-
57-
:::{note}
58-
The IPC format is unstable and subject to change. It does not yet support shared arrays (e.g.
59-
a dictionary shared across multiple chunked arrays), which limits its efficiency for certain
60-
workloads. This is an area of active development.
61-
:::
38+
The IPC format streams serialized arrays between processes: a dtype message, then one array
39+
message per chunk, each carrying a serialized array as its body. IPC writes buffers without
40+
padding, so unlike file reads, IPC reads copy buffers into aligned memory. The
41+
[IPC format specification](../../specs/ipc-format.md) defines the framing and message types.
6242

6343
## Segment Storage
6444

‎docs/specs/ipc-format.md‎

Lines changed: 159 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,165 @@
11
# IPC Format
22

3+
The IPC format streams Vortex arrays between processes as a sequence of self-delimiting messages.
4+
Arrays keep their encodings on the wire, so compressed data crosses the process boundary without
5+
being decompressed. The format is implemented by the `vortex-ipc` crate, re-exported as
6+
[`vortex::ipc`](https://docs.rs/vortex/latest/vortex/ipc/index.html).
7+
38
:::{warning}
4-
This page is under construction.
9+
The IPC format is unstable. It is not covered by the [versioning](versioning.md) guarantees of the
10+
[file format](file-format.md), and it has a single message version, `V0`, which readers require
11+
exactly. If you want to persist data, please consider writing a Vortex file instead.
512
:::
613

7-
Planned content:
14+
## Streams
15+
16+
An IPC stream carries a single logical array as a sequence of chunks:
17+
18+
```
19+
[DType message] [Array message] [Array message] ...
20+
```
21+
22+
1. The first message is a `DTypeMessage` carrying the stream's data type.
23+
2. Every following message is an `ArrayMessage` holding one chunk, whose data type must equal the
24+
stream's.
25+
3. The stream ends at end-of-input after a complete message. There is no end-of-stream marker, so
26+
a stream with no chunks is a lone `DTypeMessage`.
27+
28+
## Message framing
29+
30+
Every message has the same framing:
31+
32+
```
33+
<4 bytes> u32 little-endian header length, H
34+
<H bytes> Message FlatBuffer
35+
<body_size bytes> message body
36+
```
37+
38+
The `Message` FlatBuffer is the header, which contains the following fields:
39+
40+
- **`version`** (`MessageVersion`, default `V0`): the message format version. Readers reject any
41+
value other than `V0`.
42+
- **`header`** (`MessageHeader` union): the message type, which also determines how to interpret
43+
the body.
44+
- **`body_size`** (`uint64`): the exact length of the body, so a reader always knows where the next
45+
message starts.
46+
47+
Messages are written back-to-back, with no padding between or within them. This means that
48+
header or body bytes are not guaranteed to land on an aligned offset. Thus, after receiving a
49+
message, a reader copies each FlatBuffer and data buffer into appropriately aligned memory.
50+
51+
## Message types
52+
53+
### DTypeMessage
54+
55+
The body is a [`DType`](dtype-format.md) FlatBuffer (`root_type DType` in `dtype.fbs`). The
56+
`DTypeMessage` table itself has no fields.
57+
58+
### ArrayMessage
59+
60+
The body is a serialized array. The header carries the information needed to decode it:
61+
62+
- **`row_count`** (`uint32`) — the length of the root array.
63+
- **`encodings`** (`[string]`) — the encoding IDs the array references, such as `vortex.primitive`.
64+
65+
The body has the same shape as any other serialized array:
66+
67+
```
68+
[buffer 0] [buffer 1] ... [buffer N-1] [Array FlatBuffer] [u32 little-endian FlatBuffer length]
69+
```
70+
71+
A reader decodes the body from the end:
72+
73+
1. Read the trailing `u32` to find the length of the `Array` FlatBuffer that precedes it. The
74+
bytes before the FlatBuffer hold the data buffers.
75+
2. Locate each data buffer from the `Array.buffers` table, in order. Buffer `i` starts `padding`
76+
bytes after the end of buffer `i - 1`, where the first buffer's predecessor ends at offset 0,
77+
and spans `length` bytes. Align it to 2<sup>`alignment_exponent`</sup> bytes.
78+
3. Decode the `ArrayNode` tree from `Array.root`. Each node's `encoding` is an index into the
79+
message's `encodings` list, and its `buffers` are indices into the `Array.buffers` table.
80+
4. Decode the root node with the stream's data type and `row_count`. Each encoding derives the data
81+
types and lengths of its children from its own metadata. Any `stats` present on a node are
82+
attached to the decoded array.
83+
84+
Each message is self-contained: its `encodings` list and buffer table apply only to that message.
885

9-
- Overview of the Vortex IPC wire format
10-
- Message framing and serialization
11-
- Compressed array transfer
12-
- Use by the Scan API for remote interchange
86+
The IPC writer never pads buffers, so `padding` is always 0. Readers should still honor it,
87+
because [Vortex files](file-format.md) use the same array layout, which do pad. The writer always sets
88+
`compression` to `None`. `LZ4` is reserved in the schema but not implemented; current readers
89+
don't check the field.
90+
91+
The `ArrayNode` tree and its statistics are described in
92+
[Serialization](../developer-guide/internals/serialization.md).
93+
94+
### BufferMessage
95+
96+
The body is a single raw byte buffer. `alignment_exponent` is the alignment the receiver must give
97+
it, as a power of two. Readers reject exponents above 16 (64 KiB), since the value is untrusted.
98+
99+
The stream writers never emit a `BufferMessage`, and the stream readers reject one. It is available
100+
only through the message-level API below.
101+
102+
## Rust API
103+
104+
The crate has three layers. Most callers only need the array-stream layer.
105+
106+
| Layer | Write | Read |
107+
|---------------|----------------------------------------------------------------|---------------------------------------------------------------------------------------------|
108+
| Array streams | `stream::ArrayStreamIPC`, `iterator::ArrayIteratorIPC` | `stream::AsyncIPCReader`, `iterator::SyncIPCReader` |
109+
| Messages | `messages::AsyncMessageWriter`, `messages::SyncMessageWriter` | `messages::AsyncMessageReader`, `messages::SyncMessageReader`, `messages::BufMessageReader` |
110+
| Framing | `messages::MessageEncoder` | `messages::MessageDecoder` |
111+
112+
`ArrayStreamIPC` and `ArrayIteratorIPC` are implemented for every `ArrayStream` and
113+
`ArrayIterator`. They write the `DTypeMessage`, then one `ArrayMessage` per item. The iterator and
114+
stream from `ArrayRef::to_array_iterator` and `ArrayRef::to_array_stream` yield one item per chunk
115+
of a chunked array.
116+
117+
`AsyncIPCReader` and `SyncIPCReader` read such a stream back as an `ArrayStream` or `ArrayIterator`.
118+
They fail if the first message is not a `DTypeMessage`, if a later message is not an
119+
`ArrayMessage`, or if a decoded array's data type differs from the stream's.
120+
121+
`MessageDecoder` holds no IO. It reports the total number of bytes it needs to make progress, so
122+
callers can drive it over any transport.
123+
124+
```rust
125+
use std::io::Cursor;
126+
127+
use vortex::VortexSessionDefault;
128+
use vortex::array::IntoArray;
129+
use vortex::array::iter::ArrayIteratorExt;
130+
use vortex::buffer::buffer;
131+
use vortex::error::VortexResult;
132+
use vortex::ipc::iterator::ArrayIteratorIPC;
133+
use vortex::ipc::iterator::SyncIPCReader;
134+
use vortex::session::VortexSession;
135+
136+
fn round_trip() -> VortexResult<()> {
137+
let session = VortexSession::default();
138+
let array = buffer![1i32, 2, 3].into_array();
139+
140+
let bytes = array.to_array_iterator().write_ipc(Vec::new(), &session)?;
141+
142+
let reader = SyncIPCReader::try_new(Cursor::new(bytes), &session)?;
143+
assert_eq!(reader.read_all()?.len(), 3);
144+
Ok(())
145+
}
146+
```
147+
148+
Decoding resolves encoding IDs through the session, so the reader's session must register every
149+
encoding the writer used.
150+
151+
## Limitations
152+
153+
- **No zero-copy reads.** Buffers are unpadded on the wire, so the reader copies them into aligned
154+
memory.
155+
- **No state shared across messages.** Each message repeats its own encoding list, and data such
156+
as a dictionary shared between chunks is sent again with every chunk.
157+
- **Row count limit.** An `ArrayMessage` holds at most `u32::MAX` rows. Larger arrays must be split
158+
into chunks.
159+
- **No end-of-stream marker.** A stream cut exactly at a message boundary is indistinguishable from
160+
a complete one. Transports that can drop data must detect truncation themselves.
161+
162+
## FlatBuffer definition
163+
164+
:::{literalinclude} ../../vortex-ipc/flatbuffers/vortex-serde/message.fbs
165+
:::

0 commit comments

Comments
 (0)