Skip to content

Commit df3d421

Browse files
connortsui20connortsui20
committed
some adjustments
Signed-off-by: Connor Tsui <[email protected]>
1 parent 6c3756f commit df3d421

33 files changed

Lines changed: 96 additions & 341 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -666,7 +666,7 @@ jobs:
666666
matrix:
667667
include:
668668
- { shard: 1, name: "Core foundation", packages: "vortex-buffer vortex-error" }
669-
- { shard: 2, name: "Arrays", packages: "vortex-array vortex-dtype", features: "--features _test-harness" }
669+
- { shard: 2, name: "Arrays", packages: "vortex-array", features: "--features _test-harness" }
670670
- { shard: 3, name: "Main library", packages: "vortex" }
671671
- { shard: 4, name: "Encodings 1", packages: "vortex-alp vortex-bytebool vortex-datetime-parts" }
672672
- { shard: 5, name: "Encodings 2", packages: "vortex-decimal-byte-parts vortex-fastlanes vortex-fsst", features: "--features _test-harness" }

‎CLAUDE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616

1717
* `vortex-buffer` defines zero-copy aligned `Buffer<T>` and `BufferMut<T>` that are guaranteed
1818
to be aligned to `T` (or whatever requested runtime alignment).
19-
* `vortex-dtype` contains the basic `DType` logical type enum that is the basis of the Vortex
19+
* `vortex-array/src/dtype` contains the basic `DType` logical type enum that is the basis of the Vortex
2020
type system
2121
* `vortex-array` contains the basic `Array` trait, as well as several encodings which impl
2222
that trait for each encoding. It includes all of most of the Apache Arrow encodings.

‎docs/developer-guide/internals/architecture.md‎

Lines changed: 38 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -24,60 +24,60 @@ query engine integrations build on the file reading and scan APIs exposed throug
2424
The core crates provide the foundation for the Vortex type system, array representation, file
2525
format, and I/O.
2626

27-
| Crate | Role |
28-
|----------------------|--------------------------------------------------------------------------------|
29-
| `vortex-error` | `VortexError` and `VortexResult` types, `vortex_err!` / `vortex_bail!` macros |
30-
| `vortex-buffer` | Zero-copy aligned `Buffer<T>` with guaranteed alignment |
31-
| `vortex-dtype` | `DType` enum: Null, Bool, Primitive, UTF8, Binary, Struct, List, Extension |
32-
| `vortex-scalar` | Single-value representations of each dtype |
33-
| `vortex-mask` | Bitmask operations for validity and selection |
34-
| `vortex-session` | Session object holding registries for encodings, layouts, and extension types |
35-
| `vortex-array` | `Array` trait, canonical encodings, vtable system, statistics |
36-
| `vortex-io` | Async I/O abstraction (local filesystem, object store, HTTP) |
37-
| `vortex-layout` | Layout traits and built-in layouts (Flat, Struct, Chunked) |
38-
| `vortex-ipc` | IPC format for inter-process communication |
39-
| `vortex-file` | `.vortex` file reading and writing |
40-
| `vortex-scan` | Table scan with filter and projection pushdown |
41-
| `vortex-expr` | Expression representation and optimization |
42-
| `vortex-flatbuffers` | FlatBuffer schema definitions |
27+
| Crate | Role |
28+
| ------------------------- | ----------------------------------------------------------------------------- |
29+
| `vortex-error` | `VortexError` and `VortexResult` types, `vortex_err!` / `vortex_bail!` macros |
30+
| `vortex-buffer` | Zero-copy aligned `Buffer<T>` with guaranteed alignment |
31+
| `vortex-array/src/dtype` | `DType` enum: Null, Bool, Primitive, UTF8, Binary, Struct, List, Extension |
32+
| `vortex-array/src/scalar` | Single-value representations of each dtype |
33+
| `vortex-mask` | Bitmask operations for validity and selection |
34+
| `vortex-session` | Session object holding registries for encodings, layouts, and extension types |
35+
| `vortex-array` | `Array` trait, canonical encodings, vtable system, statistics |
36+
| `vortex-io` | Async I/O abstraction (local filesystem, object store, HTTP) |
37+
| `vortex-layout` | Layout traits and built-in layouts (Flat, Struct, Chunked) |
38+
| `vortex-ipc` | IPC format for inter-process communication |
39+
| `vortex-file` | `.vortex` file reading and writing |
40+
| `vortex-scan` | Table scan with filter and projection pushdown |
41+
| `vortex-expr` | Expression representation and optimization |
42+
| `vortex-flatbuffers` | FlatBuffer schema definitions |
4343

4444
## Encodings
4545

4646
Encodings live in separate crates under `/encodings/`. Each encoding implements the array vtable
4747
and registers itself with the session. The standard encodings are bundled into the `vortex` crate.
4848

49-
| Crate | Technique |
50-
|------------------------------|--------------------------------------------------------|
51-
| `vortex-alp` | Adaptive Lossless floating-Point compression |
52-
| `vortex-fastlanes` | FastLanes bit-packing, delta, and frame-of-reference |
53-
| `vortex-fsst` | Fast Static Symbol Table compression for strings |
54-
| `vortex-runend` | Run-end encoding for repetitive data |
55-
| `vortex-sparse` | Sparse array encoding |
56-
| `vortex-zigzag` | ZigZag encoding for signed integers |
57-
| `vortex-roaring` | Roaring bitmap encoding |
58-
| `vortex-dict` | Dictionary encoding |
59-
| `vortex-bytebool` | Byte-per-boolean encoding |
60-
| `vortex-datetime-parts` | DateTime field decomposition |
61-
| `vortex-decimal-byte-parts` | Decimal byte decomposition |
62-
| `vortex-sequence` | Arithmetic sequence encoding |
49+
| Crate | Technique |
50+
| --------------------------- | ---------------------------------------------------- |
51+
| `vortex-alp` | Adaptive Lossless floating-Point compression |
52+
| `vortex-fastlanes` | FastLanes bit-packing, delta, and frame-of-reference |
53+
| `vortex-fsst` | Fast Static Symbol Table compression for strings |
54+
| `vortex-runend` | Run-end encoding for repetitive data |
55+
| `vortex-sparse` | Sparse array encoding |
56+
| `vortex-zigzag` | ZigZag encoding for signed integers |
57+
| `vortex-roaring` | Roaring bitmap encoding |
58+
| `vortex-dict` | Dictionary encoding |
59+
| `vortex-bytebool` | Byte-per-boolean encoding |
60+
| `vortex-datetime-parts` | DateTime field decomposition |
61+
| `vortex-decimal-byte-parts` | Decimal byte decomposition |
62+
| `vortex-sequence` | Arithmetic sequence encoding |
6363

6464
## Language Bindings
6565

6666
Language bindings expose Vortex to non-Rust environments.
6767

68-
| Directory | Role |
69-
|--------------------|----------------------------------------|
70-
| `vortex-python/` | Python bindings via PyO3 and Maturin |
71-
| `java/vortex-jni/` | Java JNI bindings |
72-
| `vortex-ffi/` | C FFI bindings (generates `vortex.h`) |
73-
| `vortex-cxx/` | C++ wrapper around the C FFI |
68+
| Directory | Role |
69+
| ------------------ | ------------------------------------- |
70+
| `vortex-python/` | Python bindings via PyO3 and Maturin |
71+
| `java/vortex-jni/` | Java JNI bindings |
72+
| `vortex-ffi/` | C FFI bindings (generates `vortex.h`) |
73+
| `vortex-cxx/` | C++ wrapper around the C FFI |
7474

7575
## Integrations
7676

7777
Query engine integrations allow Vortex files to be queried through existing analytics engines.
7878

7979
| Crate / Directory | Engine | Notes |
80-
|----------------------|------------|----------------------------------------------|
80+
| -------------------- | ---------- | -------------------------------------------- |
8181
| `vortex-datafusion/` | DataFusion | `TableProvider` and `FileFormat` integration |
8282
| `vortex-duckdb/` | DuckDB | Table function integration |
8383
| `java/vortex-spark/` | Spark | DataSource V2 connector via JNI |
@@ -86,7 +86,7 @@ Query engine integrations allow Vortex files to be queried through existing anal
8686
## Other Crates
8787

8888
| Crate | Role |
89-
|----------------|--------------------------------------------------------|
89+
| -------------- | ------------------------------------------------------ |
9090
| `vortex-cuda` | GPU-accelerated decompression and compute (Linux only) |
9191
| `vortex-tui` | Terminal UI for inspecting Vortex files |
9292
| `vortex-bench` | Benchmark harness and data generators |

‎docs/developer-guide/internals/session.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@ Each Vortex crate defines a session variable that holds a registry for its exten
2727

2828
| Session Variable | Crate | Registry Contents |
2929
|-------------------|------------------|----------------------------------------------|
30-
| `DTypeSession` | `vortex-dtype` | Extension dtype vtables (Date, Time, ...) |
30+
| `DTypeSession` | `vortex-array` | Extension dtype vtables (Date, Time, ...) |
3131
| `ArraySession` | `vortex-array` | Array encoding vtables (ALP, FSST, ...) |
3232
| `ExprSession` | `vortex-array` | Scalar expression vtables |
3333
| `LayoutSession` | `vortex-layout` | Layout encoding vtables (Flat, Chunked, ...) |

0 commit comments

Comments
 (0)