Skip to content

Commit 157fc36

Browse files
committed
[fix](lance) Address review comments on Lance time travel
- Managed tables: pin every latest read (plain queries, search TVFs, metadata reads, branch heads) to the newest version the namespace records, and reject an empty history instead of letting the SDK fall back to the newest manifest in storage. - Managed tables: if the SDK opens another location than the cached table access, drop that access and read once more, so the FE and the BE use the same location. Hand the SDK the vended options in the namespace's own spelling so its fresh values replace them key for key. - FOR TIME AS OF: resolve from manifest commit times for all tables, as Lance's asof does, choosing only among namespace-recorded versions for managed tables; fail when the neighbouring version was removed by cleanup instead of returning an older snapshot; probe a still-staged recorded version only next to the selection. - Reject a tag name in FOR VERSION AS OF with every explicit @Branch, @Branch(main) included. - Read only the requested tag's file for @tag, deriving its branch from the checked-out URI. - Do not report a branch the namespace records as missing when its head manifest fails to open; use a dedicated exception for "no version at or before"; ignore null versions; compare table_uri and location without a trailing slash. - Point the Linux x86_64 JNI pins at lance-jni-12.0.0-glibc2.17-r1. - lance_build_time_travel.py: output is optional with --create-branch.
1 parent c7f59af commit 157fc36

16 files changed

Lines changed: 631 additions & 152 deletions

File tree

‎docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,8 @@ services:
133133
# time_travel_managed_untimed versions 1 and 3, no commit times reported
134134
# time_travel_managed_unprefixed every version, credentials vended unprefixed
135135
# Versions and commit times (epoch millis, UTC) match the committed time_travel.lance,
136-
# see lance_build_time_travel.py.
136+
# see lance_build_time_travel.py. Doris resolves FOR TIME AS OF from the manifests' commit
137+
# times, so the reported ones are not what the time-travel results depend on.
137138
LANCE_REST_MANAGED_TABLES_JSON: '{"time_travel_managed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}],"branches":{"dev":{"versions":[2,3]}}},"time_travel_managed_partial":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":3,"timestamp_millis":1789823170621}]},"time_travel_managed_lagging":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113}]},"time_travel_managed_untimed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[1,3]},"time_travel_managed_unprefixed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}]}}'
138139
LANCE_S3_ACCESS_KEY: admin
139140
LANCE_S3_SECRET_KEY: password

‎docker/thirdparties/docker-compose/iceberg/scripts/lance_build_time_travel.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -154,13 +154,16 @@ def check(output: Path) -> None:
154154

155155
def main() -> int:
156156
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
157-
parser.add_argument("output", type=Path, help="path of time_travel.lance")
157+
parser.add_argument("output", type=Path, nargs="?",
158+
help="path of time_travel.lance (not used with --create-branch)")
158159
parser.add_argument("--check", action="store_true", help="verify the existing fixture")
159160
parser.add_argument("--create-branch", metavar="URI",
160161
help="create the dev branch at the uploaded dataset URI instead of building")
161162
parser.add_argument("--storage-option", action="append", default=[], metavar="KEY=VALUE",
162163
help="Lance storage option for --create-branch (repeatable)")
163164
args = parser.parse_args()
165+
if not args.create_branch and args.output is None:
166+
parser.error("the output path is required unless --create-branch is given")
164167
if args.create_branch:
165168
create_branch(args.create_branch, dict(item.split("=", 1) for item in args.storage_option))
166169
elif args.check:

‎docker/thirdparties/docker-compose/iceberg/scripts/lance_rest_server.py‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,8 @@
3737
2, 3],
3838
"branches": {"dev": {"versions": [2, 3]}}}}
3939
40-
A version given as an object also reports its commit time, which is what Doris resolves
41-
FOR TIME AS OF against for a managed table; a bare integer reports none. Tags are not served:
40+
A version given as an object also reports its commit time, as a real namespace does; a bare
41+
integer reports none. Doris resolves FOR TIME AS OF from the manifests' commit times either way. Tags are not served:
4242
they live in the dataset's _refs/tags/ for managed tables too. "branches" lists the versions
4343
recorded on each branch (manifests under <table>/tree/<branch>/_versions/), answering the
4444
version endpoints when a request carries a branch.
@@ -112,8 +112,7 @@ def _load_managed_tables() -> dict[tuple[str, ...], dict]:
112112
113113
Each entry maps to {"versions": {version: commit_millis_or_None}, "branches": {name: [versions]}}.
114114
A version may be given as a bare integer, in which case no commit time is reported, or as
115-
{"version": n, "timestamp_millis": ms}, which is what a real namespace returns and what
116-
FOR TIME AS OF resolves against.
115+
{"version": n, "timestamp_millis": ms}, which is what a real namespace returns.
117116
"""
118117
raw = os.environ.get("LANCE_REST_MANAGED_TABLES_JSON", "{}")
119118
managed = json.loads(raw)

‎docker/thirdparties/lance-jni-helpers.sh‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,11 @@
1717
# under the License.
1818

1919
# Keep these pins in sync with lance.version in fe/pom.xml.
20-
LANCE_JNI_VERSION="11.0.0"
21-
LANCE_JNI_ASSET="liblance_jni-11.0.0-linux-x86_64-glibc2.17-r1.so.gz"
22-
LANCE_JNI_URL="https://github.com/apache/doris-thirdparty/releases/download/lance-jni-11.0.0-glibc2.17-r1/${LANCE_JNI_ASSET}"
23-
LANCE_JNI_ARCHIVE_SHA256="f9dc713269632e26c06ea2ea20637ca16ec9d76772dc20db0bead11e5cd34a6b"
24-
LANCE_JNI_LIBRARY_SHA256="b6540edd3bdcd76b04f96a9e78fe804a2f1bffbd75eb35abe84de9091ba432ac"
20+
LANCE_JNI_VERSION="12.0.0"
21+
LANCE_JNI_ASSET="liblance_jni-12.0.0-linux-x86_64-glibc2.17-r1.so.gz"
22+
LANCE_JNI_URL="https://github.com/apache/doris-thirdparty/releases/download/lance-jni-12.0.0-glibc2.17-r1/${LANCE_JNI_ASSET}"
23+
LANCE_JNI_ARCHIVE_SHA256="678d9454ad1a24ff07a504d8c318479f6b01d095f04945248dc170eecb303801"
24+
LANCE_JNI_LIBRARY_SHA256="72f26651b5500bf1483da1cd35130887274a42c2e8505fcfdcd846a521fc1549"
2525

2626
lance_jni_download() (
2727
set -eo pipefail

‎docs/lance-jni-packaging.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,17 @@
11
# Lance JNI packaging for Linux x86_64
22

33
`bash build.sh --fe` automatically replaces the Linux x86_64 JNI entry in
4-
`lance-core-11.0.0.jar` with the glibc 2.17 rebuild. No local Lance checkout,
4+
`lance-core-12.0.0.jar` with the glibc 2.17 rebuild. No local Lance checkout,
55
Rust toolchain, or `LANCE_JNI_SO` variable is required on the Doris build host.
66

77
The artifact is pinned in `docker/thirdparties/lance-jni-helpers.sh` and published
8-
in [apache/doris-thirdparty](https://github.com/apache/doris-thirdparty/releases/tag/lance-jni-11.0.0-glibc2.17-r1).
8+
in [apache/doris-thirdparty](https://github.com/apache/doris-thirdparty/releases/tag/lance-jni-12.0.0-glibc2.17-r1).
99
Both the compressed archive and the extracted library have fixed SHA256 values.
1010

1111
The archive is cached under:
1212

1313
```text
14-
thirdparty/installed/lance-jni/liblance_jni-11.0.0-linux-x86_64-glibc2.17-r1.so.gz
14+
thirdparty/installed/lance-jni/liblance_jni-12.0.0-linux-x86_64-glibc2.17-r1.so.gz
1515
```
1616

1717
With a custom `DORIS_THIRDPARTY`, the cache is under that directory's

0 commit comments

Comments
 (0)