|
1 | | -# Vortex DuckDB |
2 | | - |
3 | | -Rust bindings for DuckDB. Supports DuckDB precompiled libraries for fast builds and from source builds for debugging. |
4 | | - |
5 | | -## Prerequisites |
6 | | - |
7 | | -- **Ninja**: `brew install ninja` (macOS) | `apt-get install ninja-build` (Ubuntu) |
8 | | -- **CMake**: `brew install cmake` (macOS) | `apt-get install cmake` (Ubuntu) |
9 | | -- **C++20 compatible compiler**: GCC or Clang |
10 | | - |
11 | | -## Build Modes |
12 | | - |
13 | | -### Default (Release) |
14 | | - |
15 | | -Link against the precompiled DuckDB release build. |
16 | | - |
17 | | -```bash |
18 | | -cargo build -p vortex-duckdb |
19 | | -``` |
20 | | - |
21 | | -### Debug Build |
22 | | - |
23 | | -Opt into DuckDB debug build: `VX_DUCKDB_DEBUG=1`. |
24 | | - |
25 | | -```bash |
26 | | -VX_DUCKDB_DEBUG=1 cargo build -p vortex-duckdb |
27 | | -``` |
28 | | - |
29 | | -### AddressSanitizer & ThreadSanitizer |
30 | | - |
31 | | -Enable both ASAN & TSAN: `VX_DUCKDB_SAN=1`. |
32 | | - |
33 | | -```bash |
34 | | -VX_DUCKDB_DEBUG=1 VX_DUCKDB_SAN=1 cargo build -p vortex-duckdb |
35 | | -``` |
36 | | - |
37 | | -## Environment Variables |
38 | | - |
39 | | -| Variable | Effect | |
40 | | -| ----------------- | ------------------------------- | |
41 | | -| `VX_DUCKDB_DEBUG` | Build from source in debug mode | |
42 | | -| `VX_DUCKDB_ASAN` | Enable AddressSanitizer | |
43 | | - |
44 | | -## Running Tests |
45 | | - |
46 | | -```bash |
47 | | -# By default, link against the precompiled DuckDB release build. |
48 | | -cargo test -p vortex-duckdb |
49 | | - |
50 | | -# Link against the DuckDB debug build from source. |
51 | | -VX_DUCKDB_DEBUG=1 cargo test -p vortex-duckdb |
52 | | - |
53 | | -# Link against the DuckDB debug build from source with ASAN & TSAN. |
54 | | -ASAN_OPTIONS=detect_container_overflow=0 VX_DUCKDB_DEBUG=1 VX_DUCKDB_SAN=1 cargo test -p vortex-duckdb |
55 | | -``` |
56 | | - |
57 | | -## Testing the extension with DuckDB |
58 | | - |
59 | | -By default, our tests use a precompiled build which means you don't get an |
60 | | -.extension which you can load in DuckDB. If you want to test a full setup, |
61 | | - |
62 | | -1. Clone [duckdb-vortex](https://github.com/vortex-data/duckdb-vortex) |
63 | | - repository. |
64 | | - |
65 | | -2. If there is an api difference between duckdb-vortex's duckdb submodule and |
66 | | - vortex's vortex-duckdb/duckdb submodule, checkout duckdb-vortex to previous |
67 | | - commit. For example, if duckdb-vortex's HEAD uses 1.6 API but vortex's HEAD |
68 | | - uses 1.5.2, checkout duckdb-vortex at 8a41ee6ebd9. |
69 | | - |
70 | | -3. Update duckdb-vortex's submodules. Replace vortex/ submodule by a softlink to |
71 | | - your local vortex repository. |
72 | | -4. Inside duckdb-vortex, run make -j. |
73 | | - |
74 | | -./target/release/duckdb will be a duckdb instance with vortex-duckdb already |
75 | | -loaded. |
76 | | - |
77 | | -## Testing a custom DuckDB tag |
78 | | - |
79 | | -Change `DUCKDB_VERSION` environment variable value to a preferred hash or commit |
80 | | -(local build), or change build.rs (for testing in CI). |
| 1 | +# DuckDB extension for Vortex |
| 2 | + |
| 3 | +User documentation is on the [docs website](https://docs.vortex.dev/user-guide/duckdb). |
| 4 | + |
| 5 | +## Build |
| 6 | + |
| 7 | +Install `ninja`, `cmake`, and a C++20 compiler. |
| 8 | + |
| 9 | +- For debug builds, launch `VX_DUCKDB_DEBUG=1 cargo build -p vortex-duckdb`. |
| 10 | +- For ASAN/TSAN builds, launch `VX_DUCKDB_DEBUG=1 VX_DUCKDB_SAN=1 cargo build -p |
| 11 | + vortex-duckdb`. |
| 12 | +- For builds against a custom DuckDB tag or commit, change `build.rs`'s |
| 13 | + `DEFAULT_DUCKDB_VERSION`. |
| 14 | + |
| 15 | +## Test |
| 16 | + |
| 17 | +Our sqllogic tests use a build which links against `libduckdb.so`. This means |
| 18 | +you don't get an extension file to load in DuckDB. If you want to test just |
| 19 | +sqllogic, run `cargo test -p vortex-sqllogictest --test sqllogictests`. If you |
| 20 | +want to test a full setup, |
| 21 | + |
| 22 | +1. Clone [duckdb-vortex](https://github.com/vortex-data/duckdb-vortex). |
| 23 | +2. Update duckdb-vortex's submodules. Update vortex reference in duckdb-vortex's |
| 24 | + `vortex-extension/Cargo.toml` with this Vortex checkout. |
| 25 | +4. Inside duckdb-vortex, run `make reldebug -j`. |
| 26 | + |
| 27 | +`./build/reldebug/duckdb` will be a DuckDB shell with Vortex linked statically. |
| 28 | + |
| 29 | +## Update extension with same DuckDB version |
| 30 | + |
| 31 | +1. Merge the PR with your changes, if any. |
| 32 | +2. Make a new Vortex release. |
| 33 | +3. Get the commit of the release, i.e. |
| 34 | + `afb005379dd9a1b7dc7ae2cb49f4f465d91cc2b3` is the commit for |
| 35 | + [0.85](https://github.com/vortex-data/vortex/releases/tag/0.85.0). |
| 36 | +4. In duckdb-vortex, update vortex reference in `vortex-extension/Cargo.toml` to |
| 37 | + the release commit hash. |
| 38 | +5. Open a PR in duckdb-vortex, check the tests and merge it. |
| 39 | +6. Ensure the PR has been backported to the current release branch, i.e. for |
| 40 | + duckdb 1.5.* the release branch is |
| 41 | + [v1.5-variegata](https://github.com/vortex-data/duckdb-vortex/tree/v1.5-variegata). |
| 42 | + This should happen automatically. |
| 43 | +7. Notify DuckDB maintainers in the Discord `#spiraldb` channel of the new |
| 44 | + commit and ask to update the extension version. This will likely take some |
| 45 | + days since they need to bump the extension version, trigger a deploy job and |
| 46 | + update the artifacts on their website. Provide the commit in duckdb-vortex's |
| 47 | + release branch, i.e. |
| 48 | + [`5390e62`](https://github.com/vortex-data/duckdb-vortex/commit/5390e6270da964f04afbd6df4a5c9bc83a2e7880). |
| 49 | +8. Check `UPDATE EXTENSIONS; INSTALL vortex; LOAD vortex` in your local duckdb |
| 50 | + loads the latest version. |
| 51 | + |
| 52 | +## Update duckdb version |
| 53 | + |
| 54 | +- Update the version in `build.rs`'s `DEFAULT_DUCKDB_VERSION`. |
| 55 | +- Create a PR with your changes and ensure it builds. Merge the PR. If your PR |
| 56 | + is for a pre-release build, CI will build DuckDB from source and upload the |
| 57 | + artifacts to R2 so you can reuse them locally. |
| 58 | +- Create a new Vortex release. |
| 59 | +- In duckdb-vortex, change all occurrences of old version to the new version. See |
| 60 | + [1.5.5](https://github.com/vortex-data/duckdb-vortex/pull/105/changes) upgrade |
| 61 | + as an example. |
| 62 | +- In duckdb-vortex, upgrade submodules to the new version. |
| 63 | +- Follow from step (3) in extension update plan. |
| 64 | + |
| 65 | +## Build artifacts storage |
| 66 | + |
| 67 | +CI uses duckdb prebuilt (`.so/.a/.dylib`) artifacts from Github releases. |
| 68 | +However, release downloads from Github are very unstable, and we're observed a |
| 69 | +~25% build rate failure when doing so. Thus we mirror release archives to R2. |
| 70 | + |
| 71 | +A job is triggered on every PR which checks artifacts are present in R2, and if |
| 72 | +they aren't, tries to mirror them from DuckDB Github releases via a deployment |
| 73 | +request. If `DEFAULT_DUCKDB_VERSION` is a hash and files weren't present, the |
| 74 | +job builds DuckDB from source and uploads the artifacts to R2 as well. |
0 commit comments