Repository navigation
Commit c7f59af
committed
[feature](lance) Support time travel, tags, branches and REST Namespace managed versioning
### What problem does this PR solve?
Issue Number: close #66491
Related PR: #65730 (baseline Lance read integration, which already carried `FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation)
Problem Summary:
Lance catalogs could select a dataset version on the FE, but nothing proved it end to end, the error messages leaked SDK internals (`NumberFormatException: For input string`, storage paths and Rust source locations), and tables whose versions are managed by a REST Namespace (`DescribeTable` returns `managed_versioning = true`) were rejected outright with "not supported by the current BE reader".
This PR completes the four scope items of the issue:
1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF '<timestamp>'` are covered by regression suites on a filesystem catalog and a REST catalog, with a committed three-version fixture (`time_travel.lance`) and the existing `multi_frag.lance` (three appends, then a delete) so that reading an older version across a deletion file is exercised on the BE.
The semantics follow the Iceberg and Paimon integrations: the selected version is the latest one whose commit time is not later than the timestamp, parsed in the session time zone with second or millisecond precision and compared with commit times at millisecond precision (the precision a namespace reports them in, so storage and namespace resolution agree); a timestamp earlier than the first version is an error.
2. **REST Namespace managed versioning** – the FE now opens such a table through the SDK's namespace client (`Dataset.open().namespaceClient(..).tableId(..)`), so the latest version comes from `ListTableVersions`, an explicit version from `DescribeTableVersion`, and `FOR TIME AS OF` from the commit times (`timestamp_millis`) the namespace reports. The SDK finalizes a still-staged manifest to its canonical `_versions/` path while opening, so the BE keeps opening the dataset by URI and version number through lance-c exactly as before; no BE or thrift change is needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables.
3. **Selected version fixed for the whole statement** – schema binding, fragment planning, predicate pushdown, metadata COUNT and the BE scan all use the selected version; two references to the same table in one statement can select different versions (self join / union tests).
4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch and version it points at, read from the dataset's `_refs/tags/` for both storage-versioned and managed tables (Lance's namespace tag APIs expose the same refs; the selected version of a managed table is then resolved through the namespace); a tag created on a branch selects that branch's version, not the same number on `main`. As for Iceberg and Paimon, a non-numeric `FOR VERSION AS OF 'v2'` names a tag. `@branch(main)` is `main`; `@tag` / `@branch` take exactly one name and reject extra parameters. Inside a branch, versions and times select among the branch's own versions, which start at the version it was created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`, optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the BE receives the branch directory as the dataset URI and needs no change (a branch is a shallow clone with its own `_versions/`). For managed tables the SDK derives the branch from that directory and sends it in the `branch` field of the version APIs. Previously `@tag` / `@branch` were silently ignored and the query read the latest version of `main`.
5. **Errors** – a missing version (never committed, or removed by `cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')] was not found[ in the namespace]` without the storage path; a missing tag or branch, an out-of-range or non-positive version (a signed number is a version, not a tag name), a timestamp before the first version and an unparsable timestamp have their own messages; a namespace that lists no versions is reported as such; a table that is only declared in the namespace fails with "declared in the namespace but has no data yet"; for a managed table, a namespace `location` that is empty or differs from `table_uri` fails the query before anything is read; every other failure keeps the sanitized provider message.
Design decisions worth calling out:
- For a managed table the namespace is the source of truth. A version whose manifest is still in storage but which the namespace does not record is reported as not found, and the table's latest version is the namespace's latest even when storage already holds a newer manifest. Reading unrecorded manifests would expose staged or rolled-back commits.
- `timestamp_millis` is optional in the Lance Namespace spec. If a namespace lists its versions without it, the only commit times available are the ones in the storage manifests, so `FOR TIME AS OF` uses those, restricted to the versions the namespace lists so that the selected version is always one the namespace records. A manifest records when the writer committed the data, which can be earlier than when the namespace registered the version, so the selection follows the writer's clock in that case.
- The SDK opens and checks out a namespace-managed dataset with its own native namespace client (the JNI takes the native handle of a REST or Directory namespace instead of calling back into the Java object), so those operations are not serialized on the FE's namespace lock; only the FE's own namespace requests are.
- `FOR VERSION AS OF` on a dataset whose directory is missing altogether reports "version N was not found", because the SDK probes the version's manifest path first; the provider's own message is kept in the exception cause, handled the same way as every other Lance error.
- `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its `lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace client, whose `branch` field on the version requests is what managed-table branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled unchanged and every Lance unit test class and regression suite passes on it. The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and never writes a dataset, so the newer FE release does not change what the BE can read.
- An expired version (removed by cleanup) is indistinguishable from one that never existed, on storage and in a namespace, so both get the same message. "in the namespace" is only added when the namespace itself reported the miss.
- Every selector other than an explicit version on `main` is a checkout from one open of the latest `main`, and a managed `FOR TIME AS OF` whose namespace reports commit times is resolved before anything is opened. The namespace version list is fetched completely, once per read, without a page size: Lance's Directory namespace applies a limit without returning a page token, and neither the order nor monotonic commit times can be relied on to stop early.
- Reading a managed table whose manifest is still staged makes the SDK copy it to its canonical path, so read credentials must allow that write; an access-denied failure on a managed table says so.
- The BE reads by URI and version number, which assumes the manifest a namespace records ends up at its canonical path `_versions/<u64::MAX - v>.manifest`. A staged manifest from a normal commit is moved there on read; a namespace that keeps finalized manifests elsewhere cannot be detected by the FE, and the BE then either fails to find the version or reads a different manifest with the same number at the canonical path. The documentation lists this as a limitation.
- Known limitations: two references to the same branch in one statement resolve the branch head independently, and the optimizer's row count is that of the latest `main` (both shared with the Iceberg integration). `vector_search()`, `full_text_search()` and index inspection use the latest `main` and reject a version, tag or branch in their `table` argument. Index paths open by URI and are only reachable for filesystem catalogs, whose Directory namespace never manages versions; an assertion guards that.
- A Lance branch is a shallow clone whose manifests record the parent's location as an absolute URI (`Manifest.base_paths`), so a branch is readable only where it was created. The committed `time_travel.lance` branch was therefore created against the fixture's final location `s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py --create-branch`) and synced back; the fixture's self-check verifies that reference.
Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions 1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3 / 4..6 / 7..9):
```sql
mysql> SET enable_file_scanner_v2 = true;
mysql> SET time_zone = 'UTC';
mysql> CREATE CATALOG lance_demo_fs PROPERTIES (
-> "type" = "lance",
-> "lance.catalog.type" = "filesystem",
-> "warehouse" = "s3://warehouse/lance",
-> "s3.endpoint" = "http://127.0.0.1:19000",
-> "s3.access_key" = "admin",
-> "s3.secret_key" = "password",
-> "s3.region" = "us-east-1",
-> "use_path_style" = "true"
-> );
mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 ORDER BY row_id;
+--------+------+
| row_id | tag |
+--------+------+
| 1 | v1 |
| 2 | v1 |
| 3 | v1 |
+--------+------+
mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:10';
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3
-> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a
-> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON a.row_id = b.row_id
-> ORDER BY a.row_id;
+--------+--------+--------+
| row_id | tag_v1 | tag_v3 |
+--------+--------+--------+
| 1 | v1 | v1 |
| 2 | v1 | v1 |
| 3 | v1 | v1 |
+--------+--------+--------+
mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 99;
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of default.time_travel was not found
mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:07';
ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to load Lance table metadata for default.time_travel: IllegalArgumentException: Lance dataset has no version at or before '2026-09-19 13:06:07'
```
A REST Namespace with managed versioning. `time_travel_managed` records versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage already holds version 3, and `time_travel_managed_partial` records 1 and 3:
```sql
mysql> CREATE CATALOG lance_demo_rest PROPERTIES (
-> "type" = "lance",
-> "lance.catalog.type" = "rest",
-> "lance.rest.uri" = "http://127.0.0.1:19102",
-> "lance.rest.security.type" = "bearer",
-> "lance.rest.bearer-token" = "doris-lance-rest-test-token",
-> "lance.namespace.root_database" = "default",
-> "s3.endpoint" = "http://127.0.0.1:19000",
-> "s3.region" = "us-east-1",
-> "use_path_style" = "true"
-> );
mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2;
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_lagging;
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3;
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of default.time_travel_managed_lagging was not found in the namespace
mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF '2026-09-19 13:06:10';
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 3 | 3 |
+----------+-------------+
mysql> EXPLAIN SELECT row_id FROM lance_demo_rest.`default`.time_travel_managed_lagging;
...
0:VLANCE_SCAN_NODE(44)
table: lance_demo_rest.default.time_travel_managed_lagging
lanceCatalogType=rest
lanceVersion=2
lanceManagedVersioning=true
lanceFragments=2
lanceFragmentGrouping=FRAGMENT
```
Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch `dev` forked from version 2 with one extra row; on `time_travel_managed_partial` the tag `v2` points at the version the namespace no longer records):
```sql
mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY row_id;
+--------+------+
| row_id | tag |
+--------+------+
| 1 | v1 |
| 2 | v1 |
| 3 | v1 |
| 4 | v2 |
| 5 | v2 |
| 6 | v2 |
+--------+------+
mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER BY row_id;
+--------+------+
| row_id | tag |
+--------+------+
| 1 | v1 |
| 2 | v1 |
| 3 | v1 |
| 4 | v2 |
| 5 | v2 |
| 6 | v2 |
| 100 | dev |
+--------+------+
mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2;
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 6 | 6 |
+----------+-------------+
mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@tag(no_such_tag);
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of default.time_travel was not found
mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope);
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of default.time_travel was not found
mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@branch(dev);
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 7 | 100 |
+----------+-------------+
mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@tag(v1);
+----------+-------------+
| count(*) | max(row_id) |
+----------+-------------+
| 3 | 3 |
+----------+-------------+
mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial@tag(v2);
ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of default.time_travel_managed_partial (tag 'v2') was not found in the namespace
mysql> EXPLAIN SELECT row_id FROM lance_demo_fs.`default`.time_travel@branch(dev);
...
0:VLANCE_SCAN_NODE(44)
table: lance_demo_fs.default.time_travel
lanceCatalogType=filesystem
lanceVersion=3
lanceManagedVersioning=false
lanceBranch=dev
lanceFragments=3
lanceFragmentGrouping=FRAGMENT
```
Test environment changes: the Lance REST fixture (`lance_rest_server.py`) gains `ListTableVersions` / `DescribeTableVersion` for tables listed in `LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed three-version dataset with tags and a branch, carried over as-is by `lance_build_preinstalled_catalog.py` because the suites hard-code its commit times (`lance_build_time_travel.py` regenerates it and prints the times to update; `--create-branch` forks the branch at the uploaded location). The stub additionally serves the `branch` parameter of the version APIs.
### Release note
Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)` and `@branch(name)`, including tables whose versions are managed by a REST Namespace (`managed_versioning = true`). Missing versions, tags and branches and invalid selectors now report clear errors.
### Check List (For Author)
- Test: Regression test / Unit Test
- `external_table_p0/lance/test_lance_time_travel`, `test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture + MinIO)
- `LanceManagedVersioningTest` (real SDK dataset behind a REST stub: managed / partial / lagging / staged manifest / untimed / growing / server error / location checks / declared-only / cleaned-up version / tags / branches / paging / empty version list), plus `test_lance_vector_search` and `test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest`, `LanceTableAccessCacheTest`, `LanceRestCatalogTest`, `LanceScanNodeTest`, `LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest`
- Behavior changed: Yes. Tables with `managed_versioning = true` were rejected and are now readable; error messages for invalid or missing versions changed to the user-facing form above; `@tag` / `@branch` on a Lance table now select the tag or branch (including a tag that points into a branch) instead of silently reading the latest version of `main`; FE dependency `lance-core` 11.0.0 → 12.0.0.
- Does this need documentation: Yes (doris-website PR to follow: feature table, REST catalog caution, new "Time Travel" section with the managed-versioning limitations)1 parent cbc1933 commit c7f59af
45 files changed
Lines changed: 2825 additions & 76 deletions
File tree
- docker/thirdparties/docker-compose/iceberg
- scripts
- preinstalled_data/lance/time_travel.lance
- _refs
- branches
- tags
- _transactions
- _versions
- data
- tree/dev
- _transactions
- _versions
- data
- fe
- fe-core/src
- main/java/org/apache/doris
- datasource/lance
- metadata
- source
- tablefunction
- test/java/org/apache/doris
- datasource/lance
- metadata
- tablefunction
- regression-test
- data/external_table_p0/lance
- suites/external_table_p0/lance
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
118 | 118 | | |
119 | 119 | | |
120 | 120 | | |
121 | | - | |
| 121 | + | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
125 | 138 | | |
126 | 139 | | |
127 | 140 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
32 | 35 | | |
33 | 36 | | |
34 | 37 | | |
| |||
88 | 91 | | |
89 | 92 | | |
90 | 93 | | |
| 94 | + | |
91 | 95 | | |
92 | 96 | | |
93 | 97 | | |
| |||
103 | 107 | | |
104 | 108 | | |
105 | 109 | | |
| 110 | + | |
106 | 111 | | |
107 | 112 | | |
108 | 113 | | |
| |||
931 | 936 | | |
932 | 937 | | |
933 | 938 | | |
934 | | - | |
| 939 | + | |
935 | 940 | | |
| 941 | + | |
| 942 | + | |
936 | 943 | | |
937 | 944 | | |
938 | 945 | | |
| |||
1726 | 1733 | | |
1727 | 1734 | | |
1728 | 1735 | | |
| 1736 | + | |
1729 | 1737 | | |
1730 | 1738 | | |
1731 | 1739 | | |
| |||
1793 | 1801 | | |
1794 | 1802 | | |
1795 | 1803 | | |
| 1804 | + | |
| 1805 | + | |
| 1806 | + | |
| 1807 | + | |
1796 | 1808 | | |
1797 | 1809 | | |
1798 | 1810 | | |
1799 | 1811 | | |
1800 | | - | |
| 1812 | + | |
1801 | 1813 | | |
1802 | 1814 | | |
1803 | 1815 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
22 | | - | |
| 22 | + | |
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
26 | | - | |
| 26 | + | |
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| |||
0 commit comments