Skip to content

Commit e4b3421

Browse files
authored
duckdb: add release docs (#9575)
- Add release and upgrade docs for maintainers. - Add object store auth section in user docs. Resolves: #9487 Signed-off-by: Mikhail Kot <[email protected]>
1 parent 62648ef commit e4b3421

2 files changed

Lines changed: 115 additions & 94 deletions

File tree

‎docs/user-guide/duckdb.md‎

Lines changed: 41 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,36 @@
11
# DuckDB
22

33
Vortex [extension](https://duckdb.org/docs/stable/core_extensions/vortex) is
4-
available from DuckDB 1.4.2+ on Linux and macOS (amd64, arm64).
4+
available from DuckDB 1.4.2+ on Linux and macOS (amd64, arm64). Windows support
5+
[is planned](https://github.com/vortex-data/vortex/issues/9569).
56

67
## Setup
78

89
```sql
9-
INSTALL vortex;
10-
LOAD vortex;
10+
INSTALL vortex; LOAD vortex;
1111
```
1212

1313
## Reading files
1414

1515
```sql
1616
SELECT * FROM 'data.vortex';
17-
```
18-
19-
Filters and projections are pushed down into Vortex, so only the columns and rows needed by the
20-
query are read and decompressed.
21-
22-
```sql
23-
SELECT name, age FROM 'data.vortex' WHERE age > 30;
17+
# this syntax supports arguments like hive_partitioning=true
18+
SELECT * FROM read_vortex('data.vortex');
2419
```
2520

2621
## Writing files
2722

28-
Export data to Vortex using the `COPY` statement.
29-
3023
```sql
3124
COPY (SELECT * FROM my_table) TO 'output.vortex';
3225
```
3326

34-
## Python
27+
Make sure to call `LOAD vortex` before writing any files. Duckdb writes CSV
28+
files by default, so if Vortex extension wasn't loaded prior to `COPY`,
29+
`output.vortex` will be a CSV file. If you want to prevent this from happening,
30+
you can use `COPY ... TO 'output.vortex' (FORMAT vortex)` syntax instead which
31+
will fail if Vortex is not loaded.
3532

36-
The DuckDB Python client works the same way:
33+
## Python client
3734

3835
```python
3936
import duckdb
@@ -44,3 +41,33 @@ duckdb.sql("LOAD vortex")
4441
result = duckdb.sql("SELECT * FROM 'data.vortex' WHERE age > 30")
4542
result.show()
4643
```
44+
45+
## Object storage secrets
46+
47+
Vortex does not use DuckDB's secret storage. It uses environment variables via
48+
[`object_store`](https://docs.rs/object_store/latest/object_store) instead.
49+
50+
If you want to read or write from an [S3
51+
bucket](https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html),
52+
set the following variables:
53+
54+
```sh
55+
export AWS_ACCESS_KEY_ID="your_access_key"
56+
export AWS_SECRET_ACCESS_KEY="your_secret_key"
57+
export AWS_REGION="your_bucket_region"
58+
```
59+
60+
For [Google Cloud Storage](https://docs.rs/object_store/latest/object_store/gcp/struct.GoogleCloudStorageBuilder.html),
61+
use a key file variable:
62+
63+
```sh
64+
export GOOGLE_APPLICATION_CREDENTIALS="my service account JSON key file"
65+
```
66+
67+
For [Azure Blob Storage](https://docs.rs/object_store/latest/object_store/azure/struct.MicrosoftAzureBuilder.html),
68+
use
69+
70+
```sh
71+
export AZURE_STORAGE_ACCOUNT="my account"
72+
export AZURE_STORAGE_KEY="my key"
73+
```

‎vortex-duckdb/README.md‎

Lines changed: 74 additions & 80 deletions
Original file line numberDiff line numberDiff line change
@@ -1,80 +1,74 @@
1-
# Vortex DuckDB
2-
3-
Rust bindings for DuckDB. Supports DuckDB precompiled libraries for fast builds and from source builds for debugging.
4-
5-
## Prerequisites
6-
7-
- **Ninja**: `brew install ninja` (macOS) | `apt-get install ninja-build` (Ubuntu)
8-
- **CMake**: `brew install cmake` (macOS) | `apt-get install cmake` (Ubuntu)
9-
- **C++20 compatible compiler**: GCC or Clang
10-
11-
## Build Modes
12-
13-
### Default (Release)
14-
15-
Link against the precompiled DuckDB release build.
16-
17-
```bash
18-
cargo build -p vortex-duckdb
19-
```
20-
21-
### Debug Build
22-
23-
Opt into DuckDB debug build: `VX_DUCKDB_DEBUG=1`.
24-
25-
```bash
26-
VX_DUCKDB_DEBUG=1 cargo build -p vortex-duckdb
27-
```
28-
29-
### AddressSanitizer & ThreadSanitizer
30-
31-
Enable both ASAN & TSAN: `VX_DUCKDB_SAN=1`.
32-
33-
```bash
34-
VX_DUCKDB_DEBUG=1 VX_DUCKDB_SAN=1 cargo build -p vortex-duckdb
35-
```
36-
37-
## Environment Variables
38-
39-
| Variable | Effect |
40-
| ----------------- | ------------------------------- |
41-
| `VX_DUCKDB_DEBUG` | Build from source in debug mode |
42-
| `VX_DUCKDB_ASAN` | Enable AddressSanitizer |
43-
44-
## Running Tests
45-
46-
```bash
47-
# By default, link against the precompiled DuckDB release build.
48-
cargo test -p vortex-duckdb
49-
50-
# Link against the DuckDB debug build from source.
51-
VX_DUCKDB_DEBUG=1 cargo test -p vortex-duckdb
52-
53-
# Link against the DuckDB debug build from source with ASAN & TSAN.
54-
ASAN_OPTIONS=detect_container_overflow=0 VX_DUCKDB_DEBUG=1 VX_DUCKDB_SAN=1 cargo test -p vortex-duckdb
55-
```
56-
57-
## Testing the extension with DuckDB
58-
59-
By default, our tests use a precompiled build which means you don't get an
60-
.extension which you can load in DuckDB. If you want to test a full setup,
61-
62-
1. Clone [duckdb-vortex](https://github.com/vortex-data/duckdb-vortex)
63-
repository.
64-
65-
2. If there is an api difference between duckdb-vortex's duckdb submodule and
66-
vortex's vortex-duckdb/duckdb submodule, checkout duckdb-vortex to previous
67-
commit. For example, if duckdb-vortex's HEAD uses 1.6 API but vortex's HEAD
68-
uses 1.5.2, checkout duckdb-vortex at 8a41ee6ebd9.
69-
70-
3. Update duckdb-vortex's submodules. Replace vortex/ submodule by a softlink to
71-
your local vortex repository.
72-
4. Inside duckdb-vortex, run make -j.
73-
74-
./target/release/duckdb will be a duckdb instance with vortex-duckdb already
75-
loaded.
76-
77-
## Testing a custom DuckDB tag
78-
79-
Change `DUCKDB_VERSION` environment variable value to a preferred hash or commit
80-
(local build), or change build.rs (for testing in CI).
1+
# DuckDB extension for Vortex
2+
3+
User documentation is on the [docs website](https://docs.vortex.dev/user-guide/duckdb).
4+
5+
## Build
6+
7+
Install `ninja`, `cmake`, and a C++20 compiler.
8+
9+
- For debug builds, launch `VX_DUCKDB_DEBUG=1 cargo build -p vortex-duckdb`.
10+
- For ASAN/TSAN builds, launch `VX_DUCKDB_DEBUG=1 VX_DUCKDB_SAN=1 cargo build -p
11+
vortex-duckdb`.
12+
- For builds against a custom DuckDB tag or commit, change `build.rs`'s
13+
`DEFAULT_DUCKDB_VERSION`.
14+
15+
## Test
16+
17+
Our sqllogic tests use a build which links against `libduckdb.so`. This means
18+
you don't get an extension file to load in DuckDB. If you want to test just
19+
sqllogic, run `cargo test -p vortex-sqllogictest --test sqllogictests`. If you
20+
want to test a full setup,
21+
22+
1. Clone [duckdb-vortex](https://github.com/vortex-data/duckdb-vortex).
23+
2. Update duckdb-vortex's submodules. Update vortex reference in duckdb-vortex's
24+
`vortex-extension/Cargo.toml` with this Vortex checkout.
25+
4. Inside duckdb-vortex, run `make reldebug -j`.
26+
27+
`./build/reldebug/duckdb` will be a DuckDB shell with Vortex linked statically.
28+
29+
## Update extension with same DuckDB version
30+
31+
1. Merge the PR with your changes, if any.
32+
2. Make a new Vortex release.
33+
3. Get the commit of the release, i.e.
34+
`afb005379dd9a1b7dc7ae2cb49f4f465d91cc2b3` is the commit for
35+
[0.85](https://github.com/vortex-data/vortex/releases/tag/0.85.0).
36+
4. In duckdb-vortex, update vortex reference in `vortex-extension/Cargo.toml` to
37+
the release commit hash.
38+
5. Open a PR in duckdb-vortex, check the tests and merge it.
39+
6. Ensure the PR has been backported to the current release branch, i.e. for
40+
duckdb 1.5.* the release branch is
41+
[v1.5-variegata](https://github.com/vortex-data/duckdb-vortex/tree/v1.5-variegata).
42+
This should happen automatically.
43+
7. Notify DuckDB maintainers in the Discord `#spiraldb` channel of the new
44+
commit and ask to update the extension version. This will likely take some
45+
days since they need to bump the extension version, trigger a deploy job and
46+
update the artifacts on their website. Provide the commit in duckdb-vortex's
47+
release branch, i.e.
48+
[`5390e62`](https://github.com/vortex-data/duckdb-vortex/commit/5390e6270da964f04afbd6df4a5c9bc83a2e7880).
49+
8. Check `UPDATE EXTENSIONS; INSTALL vortex; LOAD vortex` in your local duckdb
50+
loads the latest version.
51+
52+
## Update duckdb version
53+
54+
- Update the version in `build.rs`'s `DEFAULT_DUCKDB_VERSION`.
55+
- Create a PR with your changes and ensure it builds. Merge the PR. If your PR
56+
is for a pre-release build, CI will build DuckDB from source and upload the
57+
artifacts to R2 so you can reuse them locally.
58+
- Create a new Vortex release.
59+
- In duckdb-vortex, change all occurrences of old version to the new version. See
60+
[1.5.5](https://github.com/vortex-data/duckdb-vortex/pull/105/changes) upgrade
61+
as an example.
62+
- In duckdb-vortex, upgrade submodules to the new version.
63+
- Follow from step (3) in extension update plan.
64+
65+
## Build artifacts storage
66+
67+
CI uses duckdb prebuilt (`.so/.a/.dylib`) artifacts from Github releases.
68+
However, release downloads from Github are very unstable, and we're observed a
69+
~25% build rate failure when doing so. Thus we mirror release archives to R2.
70+
71+
A job is triggered on every PR which checks artifacts are present in R2, and if
72+
they aren't, tries to mirror them from DuckDB Github releases via a deployment
73+
request. If `DEFAULT_DUCKDB_VERSION` is a hash and files weren't present, the
74+
job builds DuckDB from source and uploads the artifacts to R2 as well.

0 commit comments

Comments
 (0)