Skip to content

Commit 38fcf6b

Browse files
clauderobert3005
authored andcommitted
Add docs on how to use Spark vortex datasource
Fix minor usage annoyances w.r.t. managed tables and direct file queries
1 parent d2b2378 commit 38fcf6b

9 files changed

Lines changed: 737 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
[![CodSpeed Badge](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/vortex-data/vortex)
77
[![Crates.io](https://img.shields.io/crates/v/vortex.svg)](https://crates.io/crates/vortex)
88
[![PyPI - Version](https://img.shields.io/pypi/v/vortex-data)](https://pypi.org/project/vortex-data/)
9-
[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark)
9+
[![Maven - Version](https://img.shields.io/maven-central/v/dev.vortex/vortex-spark_2.13)](https://central.sonatype.com/artifact/dev.vortex/vortex-spark_2.13)
1010
[![codecov](https://codecov.io/github/vortex-data/vortex/graph/badge.svg)](https://codecov.io/github/vortex-data/vortex)
1111
[![Cite](https://img.shields.io/badge/cite-CITATION.cff-blue)](CITATION.cff)
1212

‎docs/user-guide/spark.md‎

Lines changed: 113 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,34 +1,71 @@
11
# Spark
22

33
Vortex provides a Spark DataSource V2 connector for reading and writing Vortex files. The
4-
connector is published to Maven Central as `dev.vortex:vortex-spark`.
4+
connector is published to Maven Central in two flavors:
55

6-
## Installation
6+
- `dev.vortex:vortex-spark_2.13` for Spark 4.x (Scala 2.13)
7+
- `dev.vortex:vortex-spark_2.12` for Spark 3.5.x (Scala 2.12)
78

8-
Add the dependency to your build. The connector is built against Spark 4.x with Scala 2.13.
9+
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.78.0-all.jar`). It is self-contained:
10+
it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
11+
(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
12+
conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
13+
references relocated classes that only ship in the `all` JAR.
14+
15+
## Getting Vortex into Spark
16+
17+
For `spark-shell`, `spark-submit`, or `pyspark`, pass the `all` JAR with `--jars`. Spark
18+
accepts either a local path or a URL, so you can point directly at Maven Central:
19+
20+
```shell
21+
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.78.0/vortex-spark_2.13-0.78.0-all.jar
22+
```
23+
24+
Or equivalently when building a session programmatically, e.g. in PySpark:
25+
26+
```python
27+
spark = (
28+
SparkSession.builder
29+
.config("spark.jars", "/path/to/vortex-spark_2.13-0.78.0-all.jar")
30+
.getOrCreate()
31+
)
32+
```
33+
34+
```{note}
35+
`--packages dev.vortex:vortex-spark_2.13:0.78.0` does not work: `--packages` cannot select
36+
the `all` classifier and resolves the thin JAR, which fails at runtime with
37+
`NoClassDefFoundError: dev/vortex/relocated/...`.
38+
```
39+
40+
Once the JAR is on the classpath, the connector registers itself automatically under the
41+
format name `vortex` — no session configuration is required.
42+
43+
## Installation as a Build Dependency
44+
45+
To depend on the connector from a JVM project, add the `all` classifier to the dependency:
946

1047
````{tab} Gradle (Kotlin)
1148
```kotlin
12-
implementation("dev.vortex:vortex-spark:<version>")
49+
implementation("dev.vortex:vortex-spark_2.13:0.78.0:all")
1350
```
1451
````
1552

1653
````{tab} Maven
1754
```xml
1855
<dependency>
1956
<groupId>dev.vortex</groupId>
20-
<artifactId>vortex-spark</artifactId>
21-
<version>${vortex.version}</version>
57+
<artifactId>vortex-spark_2.13</artifactId>
58+
<version>0.78.0</version>
59+
<classifier>all</classifier>
2260
</dependency>
2361
```
2462
````
2563

26-
The connector ships as a shadow JAR that relocates its Arrow, Guava, and Protobuf dependencies
27-
to avoid classpath conflicts with Spark.
28-
2964
## Reading Vortex Files
3065

31-
Use the `vortex` format to read a single file or a directory of Vortex files:
66+
Paths may be local filesystem paths (`/path/to/data`) or URLs (`file:///path/to/data`,
67+
`s3://bucket/path/to/data`). Use the `vortex` format to read a single file or a directory of
68+
Vortex files:
3269

3370
```java
3471
Dataset<Row> df = spark.read()
@@ -65,6 +102,72 @@ Each Spark partition produces one output file named `part-{partitionId}-{taskId}
65102
The connector supports all standard Spark save modes: `Overwrite`, `Append`, `Ignore`, and
66103
`ErrorIfExists`.
67104

105+
## Spark SQL
106+
107+
The connector can also be used from pure SQL. To query existing Vortex files, register them
108+
as a temporary view:
109+
110+
```sql
111+
CREATE TEMPORARY VIEW people
112+
USING vortex
113+
OPTIONS (path '/path/to/data');
114+
115+
SELECT name, age FROM people WHERE age > 30;
116+
```
117+
118+
Tables can be created with `USING vortex`, then written to and read back with plain SQL.
119+
With a `LOCATION` clause the table is external, backed by the files at that path; without
120+
one the table is managed, and Spark stores its data under the warehouse directory (and
121+
deletes it on `DROP TABLE`):
122+
123+
```sql
124+
CREATE TABLE student (id INT, name STRING, age INT)
125+
USING vortex;
126+
127+
INSERT INTO student VALUES (1, 'Alice', 20), (2, 'Bob', 21);
128+
129+
SELECT * FROM student;
130+
```
131+
132+
`CREATE TABLE ... AS SELECT` works the same way:
133+
134+
```sql
135+
CREATE TABLE adults
136+
USING vortex
137+
AS SELECT * FROM people WHERE age >= 18;
138+
```
139+
140+
```{note}
141+
On Spark 3.5, `CREATE TABLE ... USING vortex` additionally requires replacing the session
142+
catalog, because Spark 3.5's built-in catalog cannot read tables backed by a DataSource
143+
V2-only connector:
144+
145+
spark.sql.catalog.spark_catalog=dev.vortex.spark.VortexSessionCatalog
146+
147+
The extension delegates everything to the built-in session catalog (including the Hive
148+
metastore, if configured) and only changes how `vortex` tables are resolved; tables of other
149+
providers are untouched. It is not needed on Spark 4, though setting it is harmless.
150+
```
151+
152+
## Direct File Queries
153+
154+
Spark's built-in ``SELECT * FROM format.`path` `` syntax only works for built-in file
155+
formats, so the connector ships a path-based catalog that provides the equivalent for
156+
Vortex. Register it in the session configuration under the name `vortex`:
157+
158+
```shell
159+
spark-sql --conf spark.sql.catalog.vortex=dev.vortex.spark.VortexCatalog
160+
```
161+
162+
Then query a Vortex file, or a directory of Vortex files, directly by path — no view or
163+
table required:
164+
165+
```sql
166+
SELECT * FROM vortex.`/path/to/data`;
167+
168+
INSERT INTO vortex.`/path/to/data` VALUES (1, 'Alice', 20);
169+
```
170+
68171
## Supported Types
69172

70173
| Spark Type | Vortex Type |

‎java/README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,9 @@
33
We provide two interfaces for working with Vortex from Java:
44

55
- `vortex-java` - a low-level interface JNI for working with Vortex files and arrays on cloud and local storage
6-
- `vortex-spark` - A Spark connector for working with datasets of Vortex files
6+
- `vortex-spark` - A Spark connector for working with datasets of Vortex files. See
7+
[vortex-spark/README.md](vortex-spark/README.md) for how to load the connector into Spark
8+
and query Vortex files from the DataFrame API or Spark SQL.
79

810
## Publishing
911

‎java/vortex-spark/README.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# vortex-spark
2+
3+
A Spark DataSource V2 connector for reading and writing [Vortex](https://vortex.dev) files.
4+
It registers itself under the format name `vortex` and supports both the DataFrame API and
5+
Spark SQL.
6+
7+
Two flavors are published to Maven Central:
8+
9+
| Artifact | Spark | Scala |
10+
|-------------------------------|-----------|-------|
11+
| `dev.vortex:vortex-spark_2.13` | Spark 4.x | 2.13 |
12+
| `dev.vortex:vortex-spark_2.12` | Spark 3.5.x | 2.12 |
13+
14+
Use the `all` classifier JAR (e.g. `vortex-spark_2.13-0.78.0-all.jar`). It is self-contained:
15+
it bundles the Vortex JNI bindings, native libraries for Linux (x86_64 and aarch64) and macOS
16+
(aarch64), and relocates its Arrow, Guava, and Jackson dependencies to avoid classpath
17+
conflicts with Spark. The thin (unclassified) JAR does not work on its own because it
18+
references relocated classes that only ship in the `all` JAR.
19+
20+
## Getting Vortex into Spark
21+
22+
Pass the `all` JAR to `spark-shell`, `spark-submit`, or `pyspark` with `--jars`. Spark accepts
23+
either a local path or a URL, so you can point directly at Maven Central:
24+
25+
```shell
26+
spark-shell --jars https://repo1.maven.org/maven2/dev/vortex/vortex-spark_2.13/0.78.0/vortex-spark_2.13-0.78.0-all.jar
27+
```
28+
29+
Or configure it on the session builder, e.g. in PySpark:
30+
31+
```python
32+
spark = (
33+
SparkSession.builder
34+
.config("spark.jars", "/path/to/vortex-spark_2.13-0.78.0-all.jar")
35+
.getOrCreate()
36+
)
37+
```
38+
39+
Note that `--packages dev.vortex:vortex-spark_2.13:0.78.0` does not work: `--packages` cannot
40+
select the `all` classifier and resolves the thin JAR, which fails at runtime with
41+
`NoClassDefFoundError: dev/vortex/relocated/...`.
42+
43+
To depend on the connector from a JVM project instead, add the `all` classifier to the
44+
dependency:
45+
46+
```kotlin
47+
implementation("dev.vortex:vortex-spark_2.13:0.78.0:all")
48+
```
49+
50+
## Usage
51+
52+
Paths may be local filesystem paths (`/path/to/data`) or URLs (`file:///path/to/data`,
53+
`s3://bucket/path/to/data`).
54+
55+
### DataFrame API
56+
57+
```java
58+
// Write
59+
df.write()
60+
.format("vortex")
61+
.option("path", "/path/to/output")
62+
.mode(SaveMode.Overwrite)
63+
.save();
64+
65+
// Read a single file or a directory of .vortex files
66+
Dataset<Row> df = spark.read()
67+
.format("vortex")
68+
.option("path", "/path/to/output")
69+
.load();
70+
```
71+
72+
### Spark SQL
73+
74+
```sql
75+
-- Query existing Vortex files through a temporary view
76+
CREATE TEMPORARY VIEW people
77+
USING vortex
78+
OPTIONS (path '/path/to/data');
79+
80+
SELECT name, age FROM people WHERE age > 30;
81+
82+
-- Create a table and write to it. With a LOCATION clause the table is external,
83+
-- backed by the files at that path; without one it is managed by Spark.
84+
CREATE TABLE student (id INT, name STRING, age INT)
85+
USING vortex;
86+
87+
INSERT INTO student VALUES (1, 'Alice', 20), (2, 'Bob', 21);
88+
89+
SELECT * FROM student;
90+
```
91+
92+
On Spark 3.5, `CREATE TABLE ... USING vortex` additionally requires replacing the session
93+
catalog with `spark.sql.catalog.spark_catalog=dev.vortex.spark.VortexSessionCatalog`,
94+
because Spark 3.5's built-in catalog cannot read tables backed by a DataSource V2-only
95+
connector. The extension delegates everything else to the built-in session catalog and
96+
leaves tables of other providers untouched; it is not needed on Spark 4.
97+
98+
### Direct file queries
99+
100+
Spark's built-in ``SELECT * FROM format.`path` `` syntax only works for built-in file
101+
formats, so the connector ships a path-based catalog that provides the equivalent for
102+
Vortex. Register it under the name `vortex` with this session config:
103+
104+
```
105+
spark.sql.catalog.vortex=dev.vortex.spark.VortexCatalog
106+
```
107+
108+
Then query (or insert into) Vortex files directly by path:
109+
110+
```sql
111+
SELECT * FROM vortex.`/path/to/data`;
112+
```
113+
114+
See the [Spark user guide](https://docs.vortex.dev/user-guide/spark.html) for the full
115+
documentation, including supported types, write options, and S3 configuration.

0 commit comments

Comments
 (0)