Skip to content

Commit c57da5d

Browse files
committed
[fix](lance) Read managed tables by URI and version, as the BE does
The FE opened a table with managed_versioning through the SDK's namespace client while the BE opened it by URI and version through lance-c, and the previous rounds kept adding code to align the two (store identity, credentials, the SDK's second describe, the manifest it opened). The premise of namespace mode, that a reader must finalize a staged manifest, does not hold for Lance's namespaces: they finalize inside CreateTableVersion and list the canonical manifests. - The FE reads a managed version by the location and options of the read's own DescribeTable and the version number, and hands the same access to the BE. The namespace still decides which versions exist: Doris lists or describes them itself, and a managed table's access is not cached. LanceSdkNamespace and its tests are removed. - The manifest the namespace records for the selected version must be its canonical path on the chain, or a staged manifest beside it (LanceManifestPaths); otherwise the read fails and asks to retry. The canonical name no longer depends on the FE's locale. - A version recorded only at a staged manifest whose canonical manifest does not exist cannot be read; nothing is written on read. - On a managed chain, a version number the namespace no longer records between recorded versions cuts the FOR TIME AS OF history like a removed version. A branch's FOR TIME AS OF only checks the version it selects, as main does, and @Branch no longer needs a recorded main version. - Version requests run outside the catalog's namespace lock. A missing table from a branch's version request, after the table was described, is reported as the missing branch (Lance's Directory namespace reports it that way). A time before every candidate on main is a user-facing error. - The REST fixture can record a version at a staged or arbitrary manifest path; three managed tables cover an unfinalized head, a finalized staged record and a misrecorded manifest.
1 parent 8e0eda2 commit c57da5d

21 files changed

Lines changed: 1025 additions & 2088 deletions

‎docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -130,12 +130,18 @@ services:
130130
# (versions 2 and 3 under tree/dev/)
131131
# time_travel_managed_partial versions 1 and 3
132132
# time_travel_managed_lagging versions 1 and 2, storage already has 3
133-
# time_travel_managed_untimed versions 1 and 3, no commit times reported
133+
# time_travel_managed_untimed every version, no commit times reported
134134
# time_travel_managed_unprefixed every version, credentials vended unprefixed
135+
# time_travel_managed_pending every version, and version 4 (main and dev) recorded at a
136+
# staged manifest whose commit never finalized
137+
# time_travel_managed_staged every version, version 3 recorded at its staged manifest
138+
# although its canonical one exists
139+
# time_travel_managed_misrecorded every version, version 3 recorded at a manifest Doris
140+
# does not read
135141
# Versions and commit times (epoch millis, UTC) match the committed time_travel.lance,
136142
# see lance_build_time_travel.py. Doris resolves FOR TIME AS OF from the manifests' commit
137143
# times, so the reported ones are not what the time-travel results depend on.
138-
LANCE_REST_MANAGED_TABLES_JSON: '{"time_travel_managed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}],"branches":{"dev":{"versions":[2,3]}}},"time_travel_managed_partial":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":3,"timestamp_millis":1789823170621}]},"time_travel_managed_lagging":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113}]},"time_travel_managed_untimed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[1,3]},"time_travel_managed_unprefixed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}]}}'
144+
LANCE_REST_MANAGED_TABLES_JSON: '{"time_travel_managed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}],"branches":{"dev":{"versions":[2,3]}}},"time_travel_managed_partial":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":3,"timestamp_millis":1789823170621}]},"time_travel_managed_lagging":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113}]},"time_travel_managed_untimed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[1,2,3]},"time_travel_managed_unprefixed":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621}]},"time_travel_managed_pending":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621},{"version":4,"staged":true}],"branches":{"dev":{"versions":[2,3,{"version":4,"staged":true}]}}},"time_travel_managed_staged":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621,"staged":true}]},"time_travel_managed_misrecorded":{"uri":"s3://warehouse/lance/time_travel.lance","versions":[{"version":1,"timestamp_millis":1789823167597},{"version":2,"timestamp_millis":1789823169113},{"version":3,"timestamp_millis":1789823170621,"manifest_path":"lance/time_travel.lance/_versions/custom.manifest"}]}}'
139145
LANCE_S3_ACCESS_KEY: admin
140146
LANCE_S3_SECRET_KEY: password
141147
LANCE_S3_REGION: us-east-1

‎docker/thirdparties/docker-compose/iceberg/scripts/lance_rest_server.py‎

Lines changed: 61 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -41,11 +41,14 @@
4141
integer reports none. Doris resolves FOR TIME AS OF from the manifests' commit times either way. Tags are not served:
4242
they live in the dataset's _refs/tags/ for managed tables too. "branches" lists the versions
4343
recorded on each branch (manifests under <table>/tree/<branch>/_versions/), answering the
44-
version endpoints when a request carries a branch.
45-
46-
Manifests are expected at their canonical V2 path,
47-
``<table>/_versions/<u64::MAX - version>.manifest``, which is where pylance
48-
writes them.
44+
version endpoints when a request carries a branch; its versions take the same forms.
45+
46+
A version is recorded at its canonical V2 path,
47+
``<chain>/_versions/<u64::MAX - version>.manifest``, which is where pylance
48+
writes it. An object may record it elsewhere: ``"staged": true`` records the
49+
staged manifest beside the canonical path (``<canonical>-<id>``), as a Lance
50+
namespace does for a commit it has not finalized, and ``"manifest_path"``
51+
records the given path as is.
4952
"""
5053

5154
import json
@@ -106,13 +109,45 @@ def _load_unprefixed_tables() -> set[tuple[str, ...]]:
106109

107110
U64_MAX = 2**64 - 1
108111

112+
# The id a staged manifest carries after the canonical name; any id Lance generates looks like this.
113+
STAGED_ID = "3c9d0e1f-2a4b-4c6d-8e0f-1a2b3c4d5e6f"
114+
115+
116+
def _load_versions(entries) -> dict[int, dict]:
117+
"""Parses recorded versions into {version: {"timestamp_millis", "staged", "manifest_path"}}.
118+
119+
A version may be given as a bare integer, in which case no commit time is reported, or as
120+
{"version": n, "timestamp_millis": ms}, which is what a real namespace returns, optionally
121+
with "staged" or "manifest_path" to record the version elsewhere than its canonical path.
122+
"""
123+
if not isinstance(entries, list):
124+
raise ValueError("Managed Lance versions must be a list")
125+
recorded: dict[int, dict] = {}
126+
for entry in entries:
127+
spec = entry if isinstance(entry, dict) else {"version": entry}
128+
version = spec.get("version")
129+
timestamp = spec.get("timestamp_millis")
130+
staged = spec.get("staged", False)
131+
manifest_path = spec.get("manifest_path")
132+
if not isinstance(version, int) or isinstance(version, bool) or version <= 0:
133+
raise ValueError("Managed Lance versions must be positive integers")
134+
if timestamp is not None and not isinstance(timestamp, int):
135+
raise ValueError("Managed Lance timestamp_millis must be an integer")
136+
if not isinstance(staged, bool) or (manifest_path is not None and not isinstance(manifest_path, str)):
137+
raise ValueError("Managed Lance 'staged' must be a boolean and 'manifest_path' a string")
138+
recorded[version] = {
139+
"timestamp_millis": timestamp,
140+
"staged": staged,
141+
"manifest_path": manifest_path,
142+
}
143+
return dict(sorted(recorded.items()))
144+
109145

110146
def _load_managed_tables() -> dict[tuple[str, ...], dict]:
111147
"""Tables whose versions this namespace manages.
112148
113-
Each entry maps to {"versions": {version: commit_millis_or_None}, "branches": {name: [versions]}}.
114-
A version may be given as a bare integer, in which case no commit time is reported, or as
115-
{"version": n, "timestamp_millis": ms}, which is what a real namespace returns.
149+
Each entry maps to {"versions": {version: record}, "branches": {name: {version: record}}},
150+
with records as _load_versions parses them.
116151
"""
117152
raw = os.environ.get("LANCE_REST_MANAGED_TABLES_JSON", "{}")
118153
managed = json.loads(raw)
@@ -124,30 +159,16 @@ def _load_managed_tables() -> dict[tuple[str, ...], dict]:
124159
if not parts or not isinstance(spec, dict):
125160
raise ValueError("A managed Lance table needs an identifier and a spec object")
126161
uri = spec.get("uri")
127-
versions = spec.get("versions")
128-
if not isinstance(uri, str) or not isinstance(versions, list):
162+
if not isinstance(uri, str) or "versions" not in spec:
129163
raise ValueError("A managed Lance table spec needs 'uri' and 'versions'")
130-
recorded: dict[int, int | None] = {}
131-
for entry in versions:
132-
if isinstance(entry, dict):
133-
version, timestamp = entry.get("version"), entry.get("timestamp_millis")
134-
else:
135-
version, timestamp = entry, None
136-
if not isinstance(version, int) or version <= 0:
137-
raise ValueError("Managed Lance versions must be positive integers")
138-
if timestamp is not None and not isinstance(timestamp, int):
139-
raise ValueError("Managed Lance timestamp_millis must be an integer")
140-
recorded[version] = timestamp
141164
branches = spec.get("branches", {})
142165
if not isinstance(branches, dict) or any(
143-
not isinstance(name, str) or not isinstance(b, dict) or not isinstance(b.get("versions"), list)
144-
or any(not isinstance(v, int) or v <= 0 for v in b["versions"])
145-
for name, b in branches.items()):
146-
raise ValueError("Managed Lance branches must map branch names to {'versions': [ints]}")
166+
not isinstance(name, str) or not isinstance(b, dict) for name, b in branches.items()):
167+
raise ValueError("Managed Lance branches must map branch names to {'versions': [...]}")
147168
TABLES[parts] = uri
148169
result[parts] = {
149-
"versions": dict(sorted(recorded.items())),
150-
"branches": {name: sorted(b["versions"]) for name, b in branches.items()},
170+
"versions": _load_versions(spec["versions"]),
171+
"branches": {name: _load_versions(b.get("versions")) for name, b in branches.items()},
151172
}
152173
return result
153174

@@ -166,28 +187,26 @@ def _object_store_path(table_uri: str) -> str:
166187

167188

168189
def _table_version(identifier: tuple[str, ...], version: int, branch: str | None = None) -> dict:
169-
root = _object_store_path(TABLES[identifier])
190+
# A branch is its own manifest chain under <table>/tree/<branch>/.
191+
chain = _object_store_path(TABLES[identifier])
170192
if branch is not None:
171-
# A branch is its own manifest chain under <table>/tree/<branch>/; the fixture reports
172-
# no commit times for branch versions.
173-
return {
174-
"version": version,
175-
"manifest_path": f"{root}/tree/{branch}/_versions/{U64_MAX - version}.manifest",
176-
}
177-
result = {
178-
"version": version,
179-
"manifest_path": f"{root}/_versions/{U64_MAX - version}.manifest",
180-
}
181-
timestamp = MANAGED_TABLES[identifier]["versions"][version]
182-
if timestamp is not None:
183-
result["timestamp_millis"] = timestamp
193+
chain = f"{chain}/tree/{branch}"
194+
record = _branch_versions(MANAGED_TABLES[identifier], branch)[version]
195+
manifest_path = record["manifest_path"]
196+
if manifest_path is None:
197+
manifest_path = f"{chain}/_versions/{U64_MAX - version:020d}.manifest"
198+
if record["staged"]:
199+
manifest_path = f"{manifest_path}-{STAGED_ID}"
200+
result = {"version": version, "manifest_path": manifest_path}
201+
if record["timestamp_millis"] is not None:
202+
result["timestamp_millis"] = record["timestamp_millis"]
184203
return result
185204

186205

187206
def _branch_versions(managed: dict, branch: str | None):
188207
"""Versions recorded on the main chain (branch None) or on a branch; None if unknown."""
189208
if branch is None:
190-
return list(managed["versions"])
209+
return managed["versions"]
191210
return managed["branches"].get(branch)
192211

193212

0 commit comments

Comments
 (0)