Skip to content

feat(io): add shared-memory:// scheme for cross-component in-memory state#6753

Merged
westonpace merged 3 commits into
lance-format:mainfrom
hamersaw:ci/persistent-memory-objectstore
May 14, 2026
Merged

feat(io): add shared-memory:// scheme for cross-component in-memory state#6753
westonpace merged 3 commits into
lance-format:mainfrom
hamersaw:ci/persistent-memory-objectstore

Conversation

@hamersaw

@hamersaw hamersaw commented May 13, 2026

Copy link
Copy Markdown
Contributor

Summary

The existing memory:// object store provider returns a fresh InMemory backend on every new_store call, which makes it impossible for independent components in the same process to coordinate through a shared in-memory store — each writes to its own isolated bytes even when the URL matches.

This PR adds a sibling shared-memory:// scheme that wraps MemoryStoreProvider and swaps the inner backend for a process-global Arc<InMemory> keyed by URL authority.

  • All shared-memory://bucket-a/... URLs resolve to the same underlying InMemory backend (the URL path is the object key within that backend), so a write to shared-memory://bucket-a/x from one component is visible to another component reading shared-memory://bucket-a/x — across ObjectStoreRegistry instances, threads, and unrelated callers in the same process.
  • shared-memory://bucket-a/... and shared-memory://bucket-b/... resolve to separate backends and remain isolated.
  • memory:// is unchanged, so existing tests that rely on per-call isolation are unaffected.

The cache lives in a process-global LazyLock<Mutex<HashMap<String, Arc<InMemory>>>> rather than on the provider instance because ObjectStore::from_uri constructs a fresh ObjectStoreRegistry per call — a per-provider cache wouldn't actually share state across components.

Why a new scheme rather than caching memory://

"memory://" (no path) is reused as a generic "give me a temp store" URL across ~70+ test sites in the codebase (e.g. Dataset::write(data, "memory://", ...)). With cargo test running in parallel, adding caching to memory:// would cause cross-test contamination at the dataset root. Opt-in via a new scheme avoids any migration churn.

Use case

Tests and harnesses that need multiple actors to coordinate through a common in-memory object store — e.g. a writer and an independent reader, multi-pod fence simulations, manifest-store + WAL-writer agreement on the same backing bytes.

Test plan

  • cargo test -p lance-io --lib shared_memory — 5 new tests pass (cross-registry sharing, per-authority isolation, path extraction, prefix uniqueness, end-to-end via from_uri_and_params)
  • cargo test -p lance-io --lib memory — existing memory:// tests still pass
  • cargo clippy -p lance-io --tests -- -D warnings clean

🤖 Generated with Claude Code

…tate

The existing memory:// provider returns a fresh InMemory backend on every
new_store call, which makes it impossible for independent components in
the same process to coordinate through a shared in-memory object store —
each component writes to its own isolated bytes even when the URL matches.

Add a sibling shared-memory:// scheme that wraps MemoryStoreProvider and
swaps the inner backend for a process-global Arc<InMemory> keyed by URL
authority. Two callers resolving shared-memory://bucket/... see the same
bytes regardless of which ObjectStoreRegistry instance they used; distinct
authorities (shared-memory://bucket-a vs shared-memory://bucket-b) stay
isolated. memory:// is unchanged so existing tests that rely on per-call
isolation are unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the enhancement New feature or request label May 13, 2026
@codecov

codecov Bot commented May 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.96907% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-io/src/object_store.rs 85.71% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@westonpace

Copy link
Copy Markdown
Member

@claude review once

Comment on lines 293 to +297
let mut providers: HashMap<String, Arc<dyn ObjectStoreProvider>> = HashMap::new();

providers.insert("memory".into(), Arc::new(memory::MemoryStoreProvider));
providers.insert(
"shared-memory".into(),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 The new shared-memory:// scheme isn't registered in commit_handler_from_url (rust/lance-table/src/io/commit.rs:747), so any Lance dataset opened/created at a shared-memory:// URL falls through to the default arm and gets UnsafeCommitHandler — manifests are written to the final path with no conflict detection. This defeats the PR's stated 'manifest-store + WAL-writer agreement on the same backing bytes' use case: two concurrent writers against shared-memory://bucket-a/ds will silently clobber each other instead of returning CommitConflict, whereas the equivalent memory:// URL correctly conflict-detects. Fix is a one-line change: add "shared-memory" to the existing "memory" arm at line 747.

Extended reasoning...

The bug

commit_handler_from_url in rust/lance-table/src/io/commit.rs picks a commit handler by URL scheme (lines 745–810):

match url.scheme() {
    "file" | "file-object-store" => Ok(local_handler),
    "s3" | "gs" | "az" | "abfss" | "memory" | "oss" | "cos" => {
        Ok(Arc::new(ConditionalPutCommitHandler))
    }
    ...
    _ => Ok(Arc::new(UnsafeCommitHandler)),
}

memory is in the conditional-put list, but the new shared-memory scheme added by this PR is not. It therefore hits the wildcard arm at line 809 and gets UnsafeCommitHandler — which writes the manifest directly to the final path without any compare-and-swap, so concurrent writers cannot detect a conflict.

Why existing code doesn't prevent it

The PR only touches rust/lance-io (the object-store provider layer). Nothing in lance-table was updated. A grep for shared-memory/shared_memory in the repo shows the string appears only in the three files this PR touches; there is no integration in the commit-handler selection. The PR also adds no Dataset-level tests for the new scheme — its tests exercise ObjectStore::from_uri_and_params directly — so the gap is invisible to the test suite.

Impact

The PR description specifically advertises this use case: 'manifest-store + WAL-writer agreement on the same backing bytes'. That coordination requires CommitConflict semantics on the manifest path, which is exactly what UnsafeCommitHandler removes. Result: a user following the PR's stated use case gets silent data loss on concurrent commits, while the analogous memory:// URL behaves correctly.

The underlying backend is the same (shared-memory wraps the same object_store::memory::InMemory, just with a process-global Arc), and InMemory supports conditional puts — that's precisely why memory is in the ConditionalPutCommitHandler arm in the first place. So the existing handler is the correct choice for shared-memory too.

Step-by-step proof

  1. Caller invokes Dataset::write(data, "shared-memory://bucket-a/ds", ...) without supplying an explicit commit_handler.
  2. The write path (e.g. rust/lance/src/dataset/write.rs) calls commit_handler_from_url("shared-memory://bucket-a/ds", ...).
  3. url.scheme() returns "shared-memory".
  4. Line 746 (file | file-object-store) does not match.
  5. Line 747 (s3 | gs | az | abfss | memory | oss | cos) does not match — shared-memory is not listed.
  6. None of the s3+ddb arms match.
  7. Control falls through to line 809: _ => Ok(Arc::new(UnsafeCommitHandler)).
  8. UnsafeCommitHandler::commit writes the manifest directly to its final path with no conditional-put guard.
  9. A second concurrent writer to the same dataset does the same — its manifest write succeeds and overwrites the first writer's manifest. No CommitConflict is ever raised.

Repeating step 1 with memory://bucket-a/ds instead routes through ConditionalPutCommitHandler at line 748 and correctly returns CommitConflict on the racing writer.

Fix

Add "shared-memory" to the existing match arm:

"s3" | "gs" | "az" | "abfss" | "memory" | "shared-memory" | "oss" | "cos" => {
    Ok(Arc::new(ConditionalPutCommitHandler))
}

A Dataset-level concurrent-write test against a shared-memory:// URL would also make this gap visible going forward.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated to register this correctly. It's a little curious that the fencing tests succeeded without this, I would expect the ConditionalPutCommitHandler to be critical here.

The new shared-memory:// scheme fell through to UnsafeCommitHandler,
which writes manifests with no compare-and-swap guard — concurrent
writers would silently clobber each other on the manifest path,
defeating the scheme's manifest-coordination use case. The underlying
backend is the same InMemory store that backs memory://, which already
supports conditional puts, so the same handler applies.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we scope this to cfg test? If not, should we log a warning when it is not cfg test explaining that these stores live forever?

The shared-memory:// scheme exists solely so test harnesses can coordinate
state across registries — it has no production use case and a process-global
mutable backend pool is a sharp edge to ship in release builds.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@westonpace
westonpace merged commit 0a6acba into lance-format:main May 14, 2026
28 checks passed
jackye1995 pushed a commit that referenced this pull request May 16, 2026
## Summary

The `shared-memory://` object store provider (added in #6753) was gated
behind `#[cfg(test)]`, so it only compiled for this project's own test
builds. Downstream crates and integration harnesses depending on
`lance-io` could never resolve the scheme — which defeats the purpose of
having a shareable, cross-component in-memory store.

This removes the `#[cfg(test)]` gates from the four production sites
required for the provider to function:

- **`providers.rs`** — module declaration + registration in
`ObjectStoreRegistry::default()`
- **`object_store.rs`** — `is_cloud()` classification (folded into the
existing local/`memory` check)
- **`commit.rs`** — commit-handler routing to
`ConditionalPutCommitHandler` (folded into the existing cloud-scheme
arm)

The provider stays strictly **opt-in**: callers must explicitly use the
`shared-memory://` scheme, so existing `memory://` per-call isolation is
unchanged. The in-module `#[cfg(test)] mod tests` remains test-only.

## Testing

- `cargo check -p lance-io -p lance-table` — clean
- `cargo clippy -p lance-io -p lance-table --tests -- -D warnings` —
clean
- `cargo test -p lance-io shared_memory` — 5/5 pass
- `cargo test -p lance-table
test_commit_handler_from_url_memory_schemes` — pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
geruh pushed a commit to geruh/lance that referenced this pull request May 16, 2026
…ce-format#6805)

## Summary

The `shared-memory://` object store provider (added in lance-format#6753) was gated
behind `#[cfg(test)]`, so it only compiled for this project's own test
builds. Downstream crates and integration harnesses depending on
`lance-io` could never resolve the scheme — which defeats the purpose of
having a shareable, cross-component in-memory store.

This removes the `#[cfg(test)]` gates from the four production sites
required for the provider to function:

- **`providers.rs`** — module declaration + registration in
`ObjectStoreRegistry::default()`
- **`object_store.rs`** — `is_cloud()` classification (folded into the
existing local/`memory` check)
- **`commit.rs`** — commit-handler routing to
`ConditionalPutCommitHandler` (folded into the existing cloud-scheme
arm)

The provider stays strictly **opt-in**: callers must explicitly use the
`shared-memory://` scheme, so existing `memory://` per-call isolation is
unchanged. The in-module `#[cfg(test)] mod tests` remains test-only.

## Testing

- `cargo check -p lance-io -p lance-table` — clean
- `cargo clippy -p lance-io -p lance-table --tests -- -D warnings` —
clean
- `cargo test -p lance-io shared_memory` — 5/5 pass
- `cargo test -p lance-table
test_commit_handler_from_url_memory_schemes` — pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants