feat(io): add shared-memory:// scheme for cross-component in-memory state#6753
Conversation
…tate The existing memory:// provider returns a fresh InMemory backend on every new_store call, which makes it impossible for independent components in the same process to coordinate through a shared in-memory object store — each component writes to its own isolated bytes even when the URL matches. Add a sibling shared-memory:// scheme that wraps MemoryStoreProvider and swaps the inner backend for a process-global Arc<InMemory> keyed by URL authority. Two callers resolving shared-memory://bucket/... see the same bytes regardless of which ObjectStoreRegistry instance they used; distinct authorities (shared-memory://bucket-a vs shared-memory://bucket-b) stay isolated. memory:// is unchanged so existing tests that rely on per-call isolation are unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
@claude review once |
| let mut providers: HashMap<String, Arc<dyn ObjectStoreProvider>> = HashMap::new(); | ||
|
|
||
| providers.insert("memory".into(), Arc::new(memory::MemoryStoreProvider)); | ||
| providers.insert( | ||
| "shared-memory".into(), |
There was a problem hiding this comment.
🔴 The new shared-memory:// scheme isn't registered in commit_handler_from_url (rust/lance-table/src/io/commit.rs:747), so any Lance dataset opened/created at a shared-memory:// URL falls through to the default arm and gets UnsafeCommitHandler — manifests are written to the final path with no conflict detection. This defeats the PR's stated 'manifest-store + WAL-writer agreement on the same backing bytes' use case: two concurrent writers against shared-memory://bucket-a/ds will silently clobber each other instead of returning CommitConflict, whereas the equivalent memory:// URL correctly conflict-detects. Fix is a one-line change: add "shared-memory" to the existing "memory" arm at line 747.
Extended reasoning...
The bug
commit_handler_from_url in rust/lance-table/src/io/commit.rs picks a commit handler by URL scheme (lines 745–810):
match url.scheme() {
"file" | "file-object-store" => Ok(local_handler),
"s3" | "gs" | "az" | "abfss" | "memory" | "oss" | "cos" => {
Ok(Arc::new(ConditionalPutCommitHandler))
}
...
_ => Ok(Arc::new(UnsafeCommitHandler)),
}memory is in the conditional-put list, but the new shared-memory scheme added by this PR is not. It therefore hits the wildcard arm at line 809 and gets UnsafeCommitHandler — which writes the manifest directly to the final path without any compare-and-swap, so concurrent writers cannot detect a conflict.
Why existing code doesn't prevent it
The PR only touches rust/lance-io (the object-store provider layer). Nothing in lance-table was updated. A grep for shared-memory/shared_memory in the repo shows the string appears only in the three files this PR touches; there is no integration in the commit-handler selection. The PR also adds no Dataset-level tests for the new scheme — its tests exercise ObjectStore::from_uri_and_params directly — so the gap is invisible to the test suite.
Impact
The PR description specifically advertises this use case: 'manifest-store + WAL-writer agreement on the same backing bytes'. That coordination requires CommitConflict semantics on the manifest path, which is exactly what UnsafeCommitHandler removes. Result: a user following the PR's stated use case gets silent data loss on concurrent commits, while the analogous memory:// URL behaves correctly.
The underlying backend is the same (shared-memory wraps the same object_store::memory::InMemory, just with a process-global Arc), and InMemory supports conditional puts — that's precisely why memory is in the ConditionalPutCommitHandler arm in the first place. So the existing handler is the correct choice for shared-memory too.
Step-by-step proof
- Caller invokes
Dataset::write(data, "shared-memory://bucket-a/ds", ...)without supplying an explicitcommit_handler. - The write path (e.g.
rust/lance/src/dataset/write.rs) callscommit_handler_from_url("shared-memory://bucket-a/ds", ...). url.scheme()returns"shared-memory".- Line 746 (
file | file-object-store) does not match. - Line 747 (
s3 | gs | az | abfss | memory | oss | cos) does not match —shared-memoryis not listed. - None of the
s3+ddbarms match. - Control falls through to line 809:
_ => Ok(Arc::new(UnsafeCommitHandler)). UnsafeCommitHandler::commitwrites the manifest directly to its final path with no conditional-put guard.- A second concurrent writer to the same dataset does the same — its manifest write succeeds and overwrites the first writer's manifest. No
CommitConflictis ever raised.
Repeating step 1 with memory://bucket-a/ds instead routes through ConditionalPutCommitHandler at line 748 and correctly returns CommitConflict on the racing writer.
Fix
Add "shared-memory" to the existing match arm:
"s3" | "gs" | "az" | "abfss" | "memory" | "shared-memory" | "oss" | "cos" => {
Ok(Arc::new(ConditionalPutCommitHandler))
}A Dataset-level concurrent-write test against a shared-memory:// URL would also make this gap visible going forward.
There was a problem hiding this comment.
Updated to register this correctly. It's a little curious that the fencing tests succeeded without this, I would expect the ConditionalPutCommitHandler to be critical here.
The new shared-memory:// scheme fell through to UnsafeCommitHandler, which writes manifests with no compare-and-swap guard — concurrent writers would silently clobber each other on the manifest path, defeating the scheme's manifest-coordination use case. The underlying backend is the same InMemory store that backs memory://, which already supports conditional puts, so the same handler applies. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
westonpace
left a comment
There was a problem hiding this comment.
Should we scope this to cfg test? If not, should we log a warning when it is not cfg test explaining that these stores live forever?
The shared-memory:// scheme exists solely so test harnesses can coordinate state across registries — it has no production use case and a process-global mutable backend pool is a sharp edge to ship in release builds. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
## Summary The `shared-memory://` object store provider (added in #6753) was gated behind `#[cfg(test)]`, so it only compiled for this project's own test builds. Downstream crates and integration harnesses depending on `lance-io` could never resolve the scheme — which defeats the purpose of having a shareable, cross-component in-memory store. This removes the `#[cfg(test)]` gates from the four production sites required for the provider to function: - **`providers.rs`** — module declaration + registration in `ObjectStoreRegistry::default()` - **`object_store.rs`** — `is_cloud()` classification (folded into the existing local/`memory` check) - **`commit.rs`** — commit-handler routing to `ConditionalPutCommitHandler` (folded into the existing cloud-scheme arm) The provider stays strictly **opt-in**: callers must explicitly use the `shared-memory://` scheme, so existing `memory://` per-call isolation is unchanged. The in-module `#[cfg(test)] mod tests` remains test-only. ## Testing - `cargo check -p lance-io -p lance-table` — clean - `cargo clippy -p lance-io -p lance-table --tests -- -D warnings` — clean - `cargo test -p lance-io shared_memory` — 5/5 pass - `cargo test -p lance-table test_commit_handler_from_url_memory_schemes` — pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
…ce-format#6805) ## Summary The `shared-memory://` object store provider (added in lance-format#6753) was gated behind `#[cfg(test)]`, so it only compiled for this project's own test builds. Downstream crates and integration harnesses depending on `lance-io` could never resolve the scheme — which defeats the purpose of having a shareable, cross-component in-memory store. This removes the `#[cfg(test)]` gates from the four production sites required for the provider to function: - **`providers.rs`** — module declaration + registration in `ObjectStoreRegistry::default()` - **`object_store.rs`** — `is_cloud()` classification (folded into the existing local/`memory` check) - **`commit.rs`** — commit-handler routing to `ConditionalPutCommitHandler` (folded into the existing cloud-scheme arm) The provider stays strictly **opt-in**: callers must explicitly use the `shared-memory://` scheme, so existing `memory://` per-call isolation is unchanged. The in-module `#[cfg(test)] mod tests` remains test-only. ## Testing - `cargo check -p lance-io -p lance-table` — clean - `cargo clippy -p lance-io -p lance-table --tests -- -D warnings` — clean - `cargo test -p lance-io shared_memory` — 5/5 pass - `cargo test -p lance-table test_commit_handler_from_url_memory_schemes` — pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Summary
The existing
memory://object store provider returns a freshInMemorybackend on everynew_storecall, which makes it impossible for independent components in the same process to coordinate through a shared in-memory store — each writes to its own isolated bytes even when the URL matches.This PR adds a sibling
shared-memory://scheme that wrapsMemoryStoreProviderand swaps the inner backend for a process-globalArc<InMemory>keyed by URL authority.shared-memory://bucket-a/...URLs resolve to the same underlyingInMemorybackend (the URL path is the object key within that backend), so a write toshared-memory://bucket-a/xfrom one component is visible to another component readingshared-memory://bucket-a/x— acrossObjectStoreRegistryinstances, threads, and unrelated callers in the same process.shared-memory://bucket-a/...andshared-memory://bucket-b/...resolve to separate backends and remain isolated.memory://is unchanged, so existing tests that rely on per-call isolation are unaffected.The cache lives in a process-global
LazyLock<Mutex<HashMap<String, Arc<InMemory>>>>rather than on the provider instance becauseObjectStore::from_uriconstructs a freshObjectStoreRegistryper call — a per-provider cache wouldn't actually share state across components.Why a new scheme rather than caching
memory://"memory://"(no path) is reused as a generic "give me a temp store" URL across ~70+ test sites in the codebase (e.g.Dataset::write(data, "memory://", ...)). Withcargo testrunning in parallel, adding caching tomemory://would cause cross-test contamination at the dataset root. Opt-in via a new scheme avoids any migration churn.Use case
Tests and harnesses that need multiple actors to coordinate through a common in-memory object store — e.g. a writer and an independent reader, multi-pod fence simulations, manifest-store + WAL-writer agreement on the same backing bytes.
Test plan
cargo test -p lance-io --lib shared_memory— 5 new tests pass (cross-registry sharing, per-authority isolation, path extraction, prefix uniqueness, end-to-end viafrom_uri_and_params)cargo test -p lance-io --lib memory— existingmemory://tests still passcargo clippy -p lance-io --tests -- -D warningsclean🤖 Generated with Claude Code