Skip to content

Commit c7f59af

Browse files
committed
[feature](lance) Support time travel, tags, branches and REST Namespace managed versioning
### What problem does this PR solve? Issue Number: close #66491 Related PR: #65730 (baseline Lance read integration, which already carried `FOR VERSION AS OF` / `FOR TIME AS OF` on the FE without tests or documentation) Problem Summary: Lance catalogs could select a dataset version on the FE, but nothing proved it end to end, the error messages leaked SDK internals (`NumberFormatException: For input string`, storage paths and Rust source locations), and tables whose versions are managed by a REST Namespace (`DescribeTable` returns `managed_versioning = true`) were rejected outright with "not supported by the current BE reader". This PR completes the four scope items of the issue: 1. **Time travel** – `FOR VERSION AS OF <n>` and `FOR TIME AS OF '<timestamp>'` are covered by regression suites on a filesystem catalog and a REST catalog, with a committed three-version fixture (`time_travel.lance`) and the existing `multi_frag.lance` (three appends, then a delete) so that reading an older version across a deletion file is exercised on the BE. The semantics follow the Iceberg and Paimon integrations: the selected version is the latest one whose commit time is not later than the timestamp, parsed in the session time zone with second or millisecond precision and compared with commit times at millisecond precision (the precision a namespace reports them in, so storage and namespace resolution agree); a timestamp earlier than the first version is an error. 2. **REST Namespace managed versioning** – the FE now opens such a table through the SDK's namespace client (`Dataset.open().namespaceClient(..).tableId(..)`), so the latest version comes from `ListTableVersions`, an explicit version from `DescribeTableVersion`, and `FOR TIME AS OF` from the commit times (`timestamp_millis`) the namespace reports. The SDK finalizes a still-staged manifest to its canonical `_versions/` path while opening, so the BE keeps opening the dataset by URI and version number through lance-c exactly as before; no BE or thrift change is needed. `EXPLAIN` shows `lanceManagedVersioning=true` for these tables. 3. **Selected version fixed for the whole statement** – schema binding, fragment planning, predicate pushdown, metadata COUNT and the BE scan all use the selected version; two references to the same table in one statement can select different versions (self join / union tests). 4. **Tags and branches** – `tbl@tag(name)` resolves the tag to the branch and version it points at, read from the dataset's `_refs/tags/` for both storage-versioned and managed tables (Lance's namespace tag APIs expose the same refs; the selected version of a managed table is then resolved through the namespace); a tag created on a branch selects that branch's version, not the same number on `main`. As for Iceberg and Paimon, a non-numeric `FOR VERSION AS OF 'v2'` names a tag. `@branch(main)` is `main`; `@tag` / `@branch` take exactly one name and reject extra parameters. Inside a branch, versions and times select among the branch's own versions, which start at the version it was created from. `tbl@branch(name)` opens the branch under `<table>/tree/<name>/`, optionally with `FOR VERSION AS OF` / `FOR TIME AS OF` inside the branch; the BE receives the branch directory as the dataset URI and needs no change (a branch is a shallow clone with its own `_versions/`). For managed tables the SDK derives the branch from that directory and sends it in the `branch` field of the version APIs. Previously `@tag` / `@branch` were silently ignored and the query read the latest version of `main`. 5. **Errors** – a missing version (never committed, or removed by `cleanup_old_versions`) is reported as `Lance version N of db.tbl[ (tag 'x')] was not found[ in the namespace]` without the storage path; a missing tag or branch, an out-of-range or non-positive version (a signed number is a version, not a tag name), a timestamp before the first version and an unparsable timestamp have their own messages; a namespace that lists no versions is reported as such; a table that is only declared in the namespace fails with "declared in the namespace but has no data yet"; for a managed table, a namespace `location` that is empty or differs from `table_uri` fails the query before anything is read; every other failure keeps the sanitized provider message. Design decisions worth calling out: - For a managed table the namespace is the source of truth. A version whose manifest is still in storage but which the namespace does not record is reported as not found, and the table's latest version is the namespace's latest even when storage already holds a newer manifest. Reading unrecorded manifests would expose staged or rolled-back commits. - `timestamp_millis` is optional in the Lance Namespace spec. If a namespace lists its versions without it, the only commit times available are the ones in the storage manifests, so `FOR TIME AS OF` uses those, restricted to the versions the namespace lists so that the selected version is always one the namespace records. A manifest records when the writer committed the data, which can be earlier than when the namespace registered the version, so the selection follows the writer's clock in that case. - The SDK opens and checks out a namespace-managed dataset with its own native namespace client (the JNI takes the native handle of a REST or Directory namespace instead of calling back into the Java object), so those operations are not serialized on the FE's namespace lock; only the FE's own namespace requests are. - `FOR VERSION AS OF` on a dataset whose directory is missing altogether reports "version N was not found", because the SDK probes the version's manifest path first; the provider's own message is kept in the exception cause, handled the same way as every other Lance error. - `org.lance:lance-core` is bumped from 11.0.0 to 12.0.0 (its `lance-namespace` client from 0.7.7 to 0.11.1). The bump is for the namespace client, whose `branch` field on the version requests is what managed-table branches need; `Dataset.checkout(Ref)` already existed. The FE code compiled unchanged and every Lance unit test class and regression suite passes on it. The BE stays on lance-c 0.1.9 (Lance 11.0.0): the FE only reads metadata and never writes a dataset, so the newer FE release does not change what the BE can read. - An expired version (removed by cleanup) is indistinguishable from one that never existed, on storage and in a namespace, so both get the same message. "in the namespace" is only added when the namespace itself reported the miss. - Every selector other than an explicit version on `main` is a checkout from one open of the latest `main`, and a managed `FOR TIME AS OF` whose namespace reports commit times is resolved before anything is opened. The namespace version list is fetched completely, once per read, without a page size: Lance's Directory namespace applies a limit without returning a page token, and neither the order nor monotonic commit times can be relied on to stop early. - Reading a managed table whose manifest is still staged makes the SDK copy it to its canonical path, so read credentials must allow that write; an access-denied failure on a managed table says so. - The BE reads by URI and version number, which assumes the manifest a namespace records ends up at its canonical path `_versions/<u64::MAX - v>.manifest`. A staged manifest from a normal commit is moved there on read; a namespace that keeps finalized manifests elsewhere cannot be detected by the FE, and the BE then either fails to find the version or reads a different manifest with the same number at the canonical path. The documentation lists this as a limitation. - Known limitations: two references to the same branch in one statement resolve the branch head independently, and the optimizer's row count is that of the latest `main` (both shared with the Iceberg integration). `vector_search()`, `full_text_search()` and index inspection use the latest `main` and reject a version, tag or branch in their `table` argument. Index paths open by URI and are only reachable for filesystem catalogs, whose Directory namespace never manages versions; an assertion guards that. - A Lance branch is a shallow clone whose manifests record the parent's location as an absolute URI (`Manifest.base_paths`), so a branch is readable only where it was created. The committed `time_travel.lance` branch was therefore created against the fixture's final location `s3://warehouse/lance/time_travel.lance` (`lance_build_time_travel.py --create-branch`) and synced back; the fixture's self-check verifies that reference. Examples (mysql client against a 4.1 FE; `time_travel.lance` has versions 1..3 committed at 13:06:07.597 / 13:06:09.113 / 13:06:10.621 UTC with rows 1..3 / 4..6 / 7..9): ```sql mysql> SET enable_file_scanner_v2 = true; mysql> SET time_zone = 'UTC'; mysql> CREATE CATALOG lance_demo_fs PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "filesystem", -> "warehouse" = "s3://warehouse/lance", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.access_key" = "admin", -> "s3.secret_key" = "password", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT * FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT a.row_id, a.tag AS tag_v1, b.tag AS tag_v3 -> FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 1 a -> JOIN lance_demo_fs.`default`.time_travel FOR VERSION AS OF 3 b ON a.row_id = b.row_id -> ORDER BY a.row_id; +--------+--------+--------+ | row_id | tag_v1 | tag_v3 | +--------+--------+--------+ | 1 | v1 | v1 | | 2 | v1 | v1 | | 3 | v1 | v1 | +--------+--------+--------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR VERSION AS OF 99; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 99 of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel FOR TIME AS OF '2026-09-19 13:06:07'; ERROR 1105 (HY000): errCode = 2, detailMessage = Failed to load Lance table metadata for default.time_travel: IllegalArgumentException: Lance dataset has no version at or before '2026-09-19 13:06:07' ``` A REST Namespace with managed versioning. `time_travel_managed` records versions 1..3, `time_travel_managed_lagging` records only 1..2 while storage already holds version 3, and `time_travel_managed_partial` records 1 and 3: ```sql mysql> CREATE CATALOG lance_demo_rest PROPERTIES ( -> "type" = "lance", -> "lance.catalog.type" = "rest", -> "lance.rest.uri" = "http://127.0.0.1:19102", -> "lance.rest.security.type" = "bearer", -> "lance.rest.bearer-token" = "doris-lance-rest-test-token", -> "lance.namespace.root_database" = "default", -> "s3.endpoint" = "http://127.0.0.1:19000", -> "s3.region" = "us-east-1", -> "use_path_style" = "true" -> ); mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_lagging; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_lagging FOR VERSION AS OF 3; ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 3 of default.time_travel_managed_lagging was not found in the namespace mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed_partial FOR TIME AS OF '2026-09-19 13:06:10'; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> EXPLAIN SELECT row_id FROM lance_demo_rest.`default`.time_travel_managed_lagging; ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_rest.default.time_travel_managed_lagging lanceCatalogType=rest lanceVersion=2 lanceManagedVersioning=true lanceFragments=2 lanceFragmentGrouping=FRAGMENT ``` Tags and branches (`time_travel.lance` carries tags `v1`..`v3` and a branch `dev` forked from version 2 with one extra row; on `time_travel_managed_partial` the tag `v2` points at the version the namespace no longer records): ```sql mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@tag(v2) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | +--------+------+ mysql> SELECT * FROM lance_demo_fs.`default`.time_travel@branch(dev) ORDER BY row_id; +--------+------+ | row_id | tag | +--------+------+ | 1 | v1 | | 2 | v1 | | 3 | v1 | | 4 | v2 | | 5 | v2 | | 6 | v2 | | 100 | dev | +--------+------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_fs.`default`.time_travel@branch(dev) FOR VERSION AS OF 2; +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 6 | 6 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@tag(no_such_tag); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance tag 'no_such_tag' of default.time_travel was not found mysql> SELECT count(*) FROM lance_demo_fs.`default`.time_travel@branch(nope); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance branch 'nope' of default.time_travel was not found mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@branch(dev); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 7 | 100 | +----------+-------------+ mysql> SELECT count(*), max(row_id) FROM lance_demo_rest.`default`.time_travel_managed@tag(v1); +----------+-------------+ | count(*) | max(row_id) | +----------+-------------+ | 3 | 3 | +----------+-------------+ mysql> SELECT count(*) FROM lance_demo_rest.`default`.time_travel_managed_partial@tag(v2); ERROR 1105 (HY000): errCode = 2, detailMessage = Lance version 2 of default.time_travel_managed_partial (tag 'v2') was not found in the namespace mysql> EXPLAIN SELECT row_id FROM lance_demo_fs.`default`.time_travel@branch(dev); ... 0:VLANCE_SCAN_NODE(44) table: lance_demo_fs.default.time_travel lanceCatalogType=filesystem lanceVersion=3 lanceManagedVersioning=false lanceBranch=dev lanceFragments=3 lanceFragmentGrouping=FRAGMENT ``` Test environment changes: the Lance REST fixture (`lance_rest_server.py`) gains `ListTableVersions` / `DescribeTableVersion` for tables listed in `LANCE_REST_MANAGED_TABLES_JSON`; `time_travel.lance` is a committed three-version dataset with tags and a branch, carried over as-is by `lance_build_preinstalled_catalog.py` because the suites hard-code its commit times (`lance_build_time_travel.py` regenerates it and prints the times to update; `--create-branch` forks the branch at the uploaded location). The stub additionally serves the `branch` parameter of the version APIs. ### Release note Lance catalogs support `FOR VERSION AS OF`, `FOR TIME AS OF`, `@tag(name)` and `@branch(name)`, including tables whose versions are managed by a REST Namespace (`managed_versioning = true`). Missing versions, tags and branches and invalid selectors now report clear errors. ### Check List (For Author) - Test: Regression test / Unit Test - `external_table_p0/lance/test_lance_time_travel`, `test_lance_rest_time_travel`, `test_lance_rest_catalog` (Lance REST fixture + MinIO) - `LanceManagedVersioningTest` (real SDK dataset behind a REST stub: managed / partial / lagging / staged manifest / untimed / growing / server error / location checks / declared-only / cleaned-up version / tags / branches / paging / empty version list), plus `test_lance_vector_search` and `test_lance_full_text_search` re-run on lance-core 12.0.0, `LanceSnapshotTest`, `LanceTableAccessCacheTest`, `LanceRestCatalogTest`, `LanceScanNodeTest`, `LanceMetadataLoaderTest`, `LanceCatalogLifecycleTest` - Behavior changed: Yes. Tables with `managed_versioning = true` were rejected and are now readable; error messages for invalid or missing versions changed to the user-facing form above; `@tag` / `@branch` on a Lance table now select the tag or branch (including a tag that points into a branch) instead of silently reading the latest version of `main`; FE dependency `lance-core` 11.0.0 → 12.0.0. - Does this need documentation: Yes (doris-website PR to follow: feature table, REST catalog caution, new "Time Travel" section with the managed-versioning limitations)
1 parent cbc1933 commit c7f59af

45 files changed

Lines changed: 2825 additions & 76 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -118,10 +118,23 @@ services:
118118
- ./scripts/lance_rest_server.py:/opt/lance-rest/server.py:ro
119119
environment:
120120
LANCE_REST_BEARER_TOKEN: doris-lance-rest-test-token
121-
LANCE_REST_TABLES_JSON: '{"all_types":"s3://warehouse/lance/all_types.lance","all_types_unprefixed":"s3://warehouse/lance/all_types.lance"}'
121+
LANCE_REST_TABLES_JSON: '{"all_types":"s3://warehouse/lance/all_types.lance","all_types_unprefixed":"s3://warehouse/lance/all_types.lance","time_travel":"s3://warehouse/lance/time_travel.lance"}'
122122
# all_types_unprefixed serves the same dataset but vends its credentials under the
123123
# unprefixed object-store spelling, which is what real namespace servers emit.
124-
LANCE_REST_UNPREFIXED_TABLES_JSON: '["all_types_unprefixed"]'
124+
LANCE_REST_UNPREFIXED_TABLES_JSON: '["all_types_unprefixed","time_travel_managed_unprefixed"]'
125+
# All managed tables are the same three-version dataset; they differ in what the namespace
126+
# records. A reader that resolves versions through the namespace cannot see a version
127+
# missing there even though its manifest is still in storage, and its latest version is
128+
# the namespace's latest, not storage's:
129+
# time_travel_managed every version, with commit times, and branch dev
130+
# (versions 2 and 3 under tree/dev/)
131+
# time_travel_managed_partial versions 1 and 3
132+
# time_travel_managed_lagging versions 1 and 2, storage already has 3
133+
# time_travel_managed_untimed versions 1 and 3, no commit times reported
134+
# time_travel_managed_unprefixed every version, credentials vended unprefixed
135+
# Versions and commit times (epoch millis, UTC) match the committed time_travel.lance,
136+
# see lance_build_time_travel.py.
137+
LANCE_REST_MANAGED_TABLES_JSON: '{"time_travel_managed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}],"branches":{"dev":{"versions":[2,3]}}},"time_travel_managed_partial":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":3,"timestamp_millis":1789823170621}]},"time_travel_managed_lagging":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113}]},"time_travel_managed_untimed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[1,3]},"time_travel_managed_unprefixed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}]}}'
125138
LANCE_S3_ACCESS_KEY: admin
126139
LANCE_S3_SECRET_KEY: password
127140
LANCE_S3_REGION: us-east-1

‎docker/thirdparties/docker-compose/iceberg/scripts/lance_build_preinstalled_catalog.py‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,9 @@
2929
- __manifest Directory Namespace V2 manifest table (with its scalar indexes).
3030
- all_types.lance The pre-existing compatibility-mode root table, re-registered as-is.
3131
- nested_null.lance Nullable Null leaves inside lists, structs, and maps.
32+
- time_travel.lance Three uncompacted versions for FOR VERSION / TIME AS OF; carried
33+
over as-is because the suites hard-code its commit times
34+
(see lance_build_time_travel.py).
3235
- The `doris` namespace with two full-text-search fixtures, one indexed vector table per cell of the
3336
algorithm x element type x metric matrix (hash-prefixed directories), listed in
3437
VECTOR_TABLES below; BREADTH_TABLE, one table carrying the remaining cells at plan
@@ -88,6 +91,7 @@
8891
import pyarrow.ipc as ipc
8992
from lance_build_multivector import build as build_multivector, check as check_multivector
9093
from lance_build_nested_null import build as build_nested_null, check as check_nested_null
94+
from lance_build_time_travel import check as check_time_travel
9195
from lance_namespace_urllib3_client.models import (
9296
CreateNamespaceRequest,
9397
CreateTableRequest,
@@ -103,6 +107,7 @@
103107
NAMESPACE = "doris"
104108
ALL_TYPES_DIR = "all_types.lance"
105109
NESTED_NULL_DIR = "nested_null.lance"
110+
TIME_TRAVEL_DIR = "time_travel.lance"
106111
MANIFEST_DIR = "__manifest"
107112

108113
# 4-bit PQ keeps codebook training comfortable on 1024 rows. This only serves fixture
@@ -931,8 +936,10 @@ def build_multi_frag(root: Path) -> None:
931936
lance.dataset(location).delete(f"row_id in ({deleted})")
932937

933938

934-
def build(root: Path, all_types_source: Path) -> None:
939+
def build(root: Path, all_types_source: Path, time_travel_source: Path) -> None:
935940
shutil.copytree(all_types_source, root / ALL_TYPES_DIR)
941+
# Not rebuilt: its commit times are hard-coded in the time-travel suites.
942+
shutil.copytree(time_travel_source, root / TIME_TRAVEL_DIR)
936943
build_multi_frag(root)
937944
# Recreate this fixture in staging because promotion replaces the entire catalog tree.
938945
build_nested_null(root / NESTED_NULL_DIR)
@@ -1726,6 +1733,7 @@ def check_catalog(root: Path) -> None:
17261733
check_nested_dataset(nested.location)
17271734
check_multi_frag(root)
17281735
check_nested_null(root / NESTED_NULL_DIR)
1736+
check_time_travel(root / TIME_TRAVEL_DIR)
17291737
check_multivector(root / "multivector.lance")
17301738

17311739
full_fts = namespace.describe_table(DescribeTableRequest(id=[NAMESPACE, FTS_TABLE]))
@@ -1793,11 +1801,15 @@ def main() -> int:
17931801
if not all_types_source.is_dir():
17941802
print(f"missing all_types source: {all_types_source}", file=sys.stderr)
17951803
return 1
1804+
time_travel_source = output / TIME_TRAVEL_DIR
1805+
if not time_travel_source.is_dir():
1806+
print(f"missing time_travel source: {time_travel_source}", file=sys.stderr)
1807+
return 1
17961808

17971809
with tempfile.TemporaryDirectory(prefix="lance_fixture_") as staging_name:
17981810
staging = Path(staging_name) / "lance"
17991811
staging.mkdir()
1800-
build(staging, all_types_source)
1812+
build(staging, all_types_source, time_travel_source)
18011813
check_catalog(staging)
18021814
backup = output.with_name(output.name + ".old")
18031815
if backup.exists():
Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
"""Generate the Lance time-travel regression fixture.
19+
20+
time_travel.lance is a root table of the preinstalled Directory catalog next to
21+
all_types.lance. Every other fixture is compacted to a single version, so this is the one
22+
dataset whose history survives: three commits, none of them cleaned up.
23+
24+
version 1 create row_id 1..3 tag column "v1" Lance tag v1
25+
version 2 append row_id 4..6 tag column "v2" Lance tag v2
26+
version 3 append row_id 7..9 tag column "v3" Lance tag v3
27+
28+
Each version also carries a Lance tag of the same name (stored under _refs/tags/), which the
29+
suites select with tbl@tag(v2). A branch "dev" (stored under tree/dev/, metadata in
30+
_refs/branches/) is forked from version 2 and carries one extra commit:
31+
32+
dev version 3 append row_id 100 tag column "dev"
33+
34+
so the branch's version 3 has rows 1..6 and 100, while main's version 3 has rows 1..9; the
35+
suites select it with tbl@branch(dev). The main chain is not touched by the branch. A tag "rel"
36+
points at the branch's version 3, so tbl@tag(rel) must read the branch, not main's version 3.
37+
38+
A Lance branch is a shallow clone: its manifests record the parent's location as an absolute
39+
URI (Manifest.base_paths), so a branch only reads where it was created. The committed
40+
branch was therefore created against the fixture's final location, s3://warehouse/lance/
41+
time_travel.lance, and synced back into this directory; build() writes the main chain and
42+
the tags locally, and --create-branch <uri> creates the branch at the uploaded location:
43+
44+
python3 lance_build_time_travel.py --create-branch s3://warehouse/lance/time_travel.lance \
45+
--storage-option endpoint=http://127.0.0.1:19000 --storage-option access_key_id=admin \
46+
--storage-option secret_access_key=password --storage-option region=us-east-1 \
47+
--storage-option allow_http=true
48+
# then sync tree/ and _refs/branches/ from that location into preinstalled_data/lance/time_travel.lance/
49+
50+
check() verifies the branch files and that they point at that URI without opening the branch,
51+
which is not possible offline.
52+
53+
The commits are spaced apart so that FOR TIME AS OF can land between two of them. Lance
54+
stores the commit time of each version in its manifest, and that time is whatever wall
55+
clock this script ran at. Doris has no SQL to read those times back, so the regression
56+
suites hard-code them (test_lance_time_travel and test_lance_rest_time_travel, together
57+
with their .out files). Regenerating this dataset therefore means updating those suites
58+
from the times this script prints; that is why lance_build_preinstalled_catalog.py carries
59+
the committed directory over as-is, like all_types.lance, instead of rebuilding it, and only
60+
runs check() on it.
61+
62+
The same directory serves three catalogs in the regression suites: the filesystem catalog,
63+
the REST catalog with storage-native versions, and the REST catalog with namespace-managed
64+
versions, where lance_rest_server.py answers the version endpoints from a static list
65+
matching the versions written here.
66+
67+
Run it with the writer pinned in lance_fixture_requirements.txt; the main script's
68+
check_pinned_writer() enforces the pin for the whole catalog, this script alone does not.
69+
70+
Usage:
71+
python3 lance_build_time_travel.py preinstalled_data/lance/time_travel.lance
72+
python3 lance_build_time_travel.py --check preinstalled_data/lance/time_travel.lance
73+
"""
74+
import argparse
75+
import shutil
76+
import time
77+
from pathlib import Path
78+
79+
import lance
80+
import pyarrow as pa
81+
82+
COMMITS = (("create", 1, 3, "v1"), ("append", 4, 6, "v2"), ("append", 7, 9, "v3"))
83+
BRANCH = "dev"
84+
BRANCH_ROW_ID = 100
85+
BRANCH_TAG = "rel"
86+
BRANCH_PARENT_URI = "s3://warehouse/lance/time_travel.lance"
87+
COMMIT_GAP_SECONDS = 1.5
88+
89+
90+
def rows_of(low: int, high: int, tag: str) -> pa.Table:
91+
return pa.table({
92+
"row_id": pa.array(range(low, high + 1), pa.int32()),
93+
"tag": pa.array([tag] * (high - low + 1), pa.string()),
94+
})
95+
96+
97+
def build(output: Path) -> None:
98+
if output.exists():
99+
shutil.rmtree(output)
100+
for index, (mode, low, high, tag) in enumerate(COMMITS):
101+
if index > 0:
102+
time.sleep(COMMIT_GAP_SECONDS)
103+
# Match all_types.lance (data storage version 2.2) so every committed Lance data file
104+
# shares one on-disk format with the rest of the fixture.
105+
lance.write_dataset(rows_of(low, high, tag), str(output), mode=mode,
106+
data_storage_version="2.2")
107+
dataset = lance.dataset(str(output))
108+
for version, (_, _, _, tag) in zip((1, 2, 3), COMMITS):
109+
dataset.tags.create(tag, version)
110+
print(f"main chain and tags written; create the branch with --create-branch {BRANCH_PARENT_URI}")
111+
112+
113+
def create_branch(uri: str, storage_options: dict) -> None:
114+
"""Forks the branch at the dataset's final location and appends its extra row there."""
115+
dataset = lance.dataset(uri, storage_options=storage_options)
116+
assert [v["version"] for v in dataset.versions()] == [1, 2, 3], "upload the main chain first"
117+
dataset.create_branch(BRANCH, 2)
118+
lance.write_dataset(rows_of(BRANCH_ROW_ID, BRANCH_ROW_ID, BRANCH), f"{uri}/tree/{BRANCH}",
119+
mode="append", data_storage_version="2.2", storage_options=storage_options)
120+
branch = lance.dataset(f"{uri}/tree/{BRANCH}", storage_options=storage_options)
121+
assert branch.version == 3 and sorted(branch.to_table()["row_id"].to_pylist()) == [1, 2, 3, 4, 5, 6, BRANCH_ROW_ID]
122+
dataset.tags.create(BRANCH_TAG, (BRANCH, 3))
123+
print(f"branch {BRANCH} and tag {BRANCH_TAG} created at {uri}; sync tree/, _refs/branches/ and _refs/tags/ back")
124+
125+
126+
def check(output: Path) -> None:
127+
dataset = lance.dataset(str(output))
128+
versions = dataset.versions()
129+
assert [v["version"] for v in versions] == [1, 2, 3], (
130+
f"time-travel fixture must keep exactly versions 1..3: {versions}")
131+
timestamps = [v["timestamp"] for v in versions]
132+
assert timestamps == sorted(timestamps) and len(set(timestamps)) == 3, (
133+
f"time-travel fixture commit times must be distinct and increasing: {timestamps}")
134+
assert all((b - a).total_seconds() >= 1 for a, b in zip(timestamps, timestamps[1:])), (
135+
f"time-travel fixture commits must be at least one second apart: {timestamps}")
136+
for version, (_, _, high, tag) in zip((1, 2, 3), COMMITS):
137+
table = dataset.checkout_version(version).to_table().sort_by("row_id")
138+
assert table["row_id"].to_pylist() == list(range(1, high + 1)), (
139+
f"version {version} rows differ from expected: {table}")
140+
assert table["tag"].to_pylist()[-1] == tag, f"version {version} tag differs: {table}"
141+
tags = {name: (ref["branch"], ref["version"]) for name, ref in dataset.tags.list().items()}
142+
assert tags == {"v1": (None, 1), "v2": (None, 2), "v3": (None, 3), BRANCH_TAG: (BRANCH, 3)}, (
143+
f"time-travel fixture tags differ: {tags}")
144+
assert list(dataset.branches.list()) == [BRANCH], f"time-travel fixture branches differ: {dataset.branches.list()}"
145+
branch_manifests = sorted((output / "tree" / BRANCH / "_versions").glob("*.manifest"))
146+
assert len(branch_manifests) == 2, f"branch {BRANCH} must carry versions 2 and 3: {branch_manifests}"
147+
for manifest in branch_manifests:
148+
assert BRANCH_PARENT_URI.encode() in manifest.read_bytes(), (
149+
f"{manifest} must reference the parent at {BRANCH_PARENT_URI}; a branch created elsewhere is unreadable there")
150+
for version in versions:
151+
print(f"time_travel.lance version {version['version']} committed at "
152+
f"{version['timestamp'].isoformat()}")
153+
154+
155+
def main() -> int:
156+
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
157+
parser.add_argument("output", type=Path, help="path of time_travel.lance")
158+
parser.add_argument("--check", action="store_true", help="verify the existing fixture")
159+
parser.add_argument("--create-branch", metavar="URI",
160+
help="create the dev branch at the uploaded dataset URI instead of building")
161+
parser.add_argument("--storage-option", action="append", default=[], metavar="KEY=VALUE",
162+
help="Lance storage option for --create-branch (repeatable)")
163+
args = parser.parse_args()
164+
if args.create_branch:
165+
create_branch(args.create_branch, dict(item.split("=", 1) for item in args.storage_option))
166+
elif args.check:
167+
check(args.output)
168+
else:
169+
build(args.output)
170+
return 0
171+
172+
173+
if __name__ == "__main__":
174+
raise SystemExit(main())

‎docker/thirdparties/docker-compose/iceberg/scripts/lance_fixture_requirements.txt‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,11 +19,11 @@
1919
#
2020
# The fixture has several readers and they do not all run the same Lance version:
2121
#
22-
# lance-c v0.1.6 (lance-rs 7.0.0-beta.7) BE, both from source and from the prebuilt
22+
# lance-c v0.1.9 (lance-rs 11.0.0, patched) BE, both from source and from the prebuilt
2323
# doris-thirdparty package - thirdparty/vars.sh
2424
# lance-java 4.0.0 Spark, via lance-spark-bundle 0.4.0; it
2525
# registers runtime tables into the same __manifest
26-
# lance-java 9.1.0-beta.3 Doris FE Directory Namespace client - fe/pom.xml
26+
# lance-java 12.0.0 Doris FE Directory Namespace client - fe/pom.xml
2727
#
2828
# The writer is pinned to the Lance generation the BE actually reads. It used to be pinned
2929
# to the oldest reader instead, on the theory that an older writer is readable by every

0 commit comments

Comments
 (0)