fix(commit): verify finalized external manifests#7578
Conversation
9d0fd93 to
6971630
Compare
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
6971630 to
442310c
Compare
📝 WalkthroughWalkthroughChangesFinalized manifest verification
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant ExternalManifestStore
participant ExternalManifestCommitHandler
participant ObjectStore
ExternalManifestStore->>ExternalManifestCommitHandler: return finalized ManifestLocation
ExternalManifestCommitHandler->>ObjectStore: head manifest path
ObjectStore-->>ExternalManifestCommitHandler: return size and e_tag
ExternalManifestCommitHandler-->>ExternalManifestStore: return verified or fallback location
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
🧹 Nitpick comments (2)
rust/lance-table/src/io/commit/external_manifest.rs (2)
404-409: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winLog the stale-finalized-path fallback.
The
NotFoundbranch silently falls back todefault_resolve_versionwith no runtime log, even though this indicates the external store's finalized-path record is stale/wrong — an unexpected condition. The rest of this file (e.g. theput_if_existsfailure path) already useswarn!for comparable best-effort/unexpected situations, so a similar log here would help operators detect a systemically stale external store instead of it silently degrading every lookup.As per coding guidelines, "Choose log levels by audience: use
debug!for routine/high-frequency operations,info!for infrequent operator-visible state changes, andwarn!for unexpected conditions" and "Log warnings for silent no-op skipped operations, but omit warnings before errors because the error message is sufficient."📝 Suggested log addition
Err(ObjectStoreError::NotFound { .. }) => { // The external store may hold a stale finalized V2 path while // the object store still has the manifest at the V1 location. + warn!( + "finalized manifest path {} for version {} not found in object store; falling back to default resolution", + location.path, location.version + ); default_resolve_version(base_path, location.version, object_store).await }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-table/src/io/commit/external_manifest.rs` around lines 404 - 409, Add a warn!-level log in the NotFound branch before calling default_resolve_version, indicating that the external store’s finalized path is stale and the V1/default resolution fallback is being used. Keep the existing fallback and other error handling unchanged, and include relevant path/version context in the warning.Source: Coding guidelines
507-525: 🚀 Performance & Scalability | 🔵 TrivialVerified finalized-path flow looks correct.
Matches the tested behavior (happy path fills in missing size/e_tag, V1 fallback on
NotFound). Note this adds a mandatory HEAD round-trip to every "resolve latest" call that lands on a finalized manifest (confirmed by thes3_test.rsread_iops bump from 1 to 2) — an accepted correctness-over-latency trade-off, but worth keeping in mind for high-frequency open/checkout paths under load.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance-table/src/io/commit/external_manifest.rs` around lines 507 - 525, Preserve the finalized-manifest handling in verify_finalized_manifest_location, including the HEAD-based metadata resolution and V1 NotFound fallback. Do not remove or bypass this verification when resolving the latest manifest, and retain the existing behavior that fills missing size and e_tag values.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@rust/lance-table/src/io/commit/external_manifest.rs`:
- Around line 404-409: Add a warn!-level log in the NotFound branch before
calling default_resolve_version, indicating that the external store’s finalized
path is stale and the V1/default resolution fallback is being used. Keep the
existing fallback and other error handling unchanged, and include relevant
path/version context in the warning.
- Around line 507-525: Preserve the finalized-manifest handling in
verify_finalized_manifest_location, including the HEAD-based metadata resolution
and V1 NotFound fallback. Do not remove or bypass this verification when
resolving the latest manifest, and retain the existing behavior that fills
missing size and e_tag values.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: QUIET
Plan: Pro Plus
Run ID: df7d26c6-c493-40ef-9763-876c00376580
📒 Files selected for processing (3)
rust/lance-table/src/io/commit/external_manifest.rsrust/lance/src/io/commit/external_manifest.rsrust/lance/src/io/commit/s3_test.rs
(cherry picked from commit 741548c)
Bug Fix
What is the bug?
ExternalManifestCommitHandler trusted finalized
.manifestpaths returned by the external manifest store without checking that the object existed.What issues or incorrect behavior does the bug cause?
A stale finalized V2 path could bypass object-store verification and skip the default V2-to-V1 manifest fallback.
How does this PR fix the problem?
Finalized external manifest locations are now HEAD-checked before use. If the finalized path is missing, resolution falls back through the default version resolver, preserving the V2-first then V1 fallback behavior. The same verification path is used for latest and explicit version resolution.
Validation
cargo fmt --allgit diff --check -- rust/lance-table/src/io/commit/external_manifest.rs rust/lance/src/io/commit/external_manifest.rsPROTOC=/private/tmp/protoc-35.1-osx-aarch_64/bin/protoc PROTOC_INCLUDE=/private/tmp/protoc-35.1-osx-aarch_64/include cargo test -p lance finalized_external_manifest_location --libPROTOC=/private/tmp/protoc-35.1-osx-aarch_64/bin/protoc PROTOC_INCLUDE=/private/tmp/protoc-35.1-osx-aarch_64/include cargo clippy --all --tests --benches -- -D warnings