feat: allow using object_store as opendal's backend by flaneur2020 · Pull Request #6283 · apache/opendal

flaneur2020 · 2025-06-11T13:29:25Z

Which issue does this PR close?

Closes #6171.

Rationale for this change

with allowing object_store as opendal's backend, we can leverage opendal's advanced operation like parallel fetching, chunking, caching for users object_store.

What changes are included in this PR?

New Backend Implementation: Added complete object_store backend in src/backend/ with full CRUD operations.

Are there any user-facing changes?

New APIs: ObjectStoreBackend and ObjectStoreBuilder for creating object_store-based OpenDAL operators
New Feature: services-object-store feature flag to enable the backend

let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
let operator = Operator::new(ObjectStore::default().store(store))?.finish();

Xuanwo · 2025-06-11T16:23:29Z

Thanks a lot for building this, but I think it's better to be part of opendal's object_store integration instead of in opendal core.

flaneur2020 · 2025-06-12T04:57:53Z

Thanks a lot for building this, but I think it's better to be part of opendal's object_store integration instead of in opendal core.

thank you for the feedback, let me move it there

Xuanwo · 2025-07-04T10:28:07Z

Give Github Copilot a chance.

Copilot

Pull Request Overview

This PR integrates object_store as a custom backend for opendal, enabling users to leverage object_store’s advanced features (parallel fetching, chunking, caching) through the opendal interface.

Introduce a new backend module with ObjectStoreBuilder and ObjectStoreBackend.
Implement reader, writer, lister, deleter adapters and unified error mapping.
Update core/Cargo.toml to add the services-object-store feature and optional object_store dependency.

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
integrations/object_store/src/lib.rs	Expose `backend` module and re-export builder & backend
integrations/object_store/src/backend/writer.rs	Add `ObjectStoreWriter` and `parse_write_args`
integrations/object_store/src/backend/reader.rs	Add `ObjectStoreReader` and `parse_read_args`
integrations/object_store/src/backend/lister.rs	Add `ObjectStoreLister`
integrations/object_store/src/backend/deleter.rs	Add `ObjectStoreDeleter`
integrations/object_store/src/backend/error.rs	Map `object_store::Error` variants to `opendal::Error`
integrations/object_store/src/backend/mod.rs	Wire up `Builder` and `Access` implementation
core/Cargo.toml	Register `services-object-store` and optional `object_store` dependency

Comments suppressed due to low confidence (2)

integrations/object_store/src/backend/writer.rs:77

There are no unit tests covering parse_write_args. Consider adding tests to verify mapping of OpWrite options (content_type, content_disposition, cache_control, user_metadata) into PutOptions.

pub(crate) fn parse_write_args(args: &OpWrite) -> Result<PutOptions> {

integrations/object_store/src/backend/reader.rs:70

The parse_read_args function isn't covered by tests. Adding unit tests for version, conditional headers, and range options would help ensure correct behavior across scenarios.

fn parse_read_args(args: &OpRead) -> Result<object_store::GetOptions> {

integrations/object_store/src/backend/lister.rs

integrations/object_store/src/backend/deleter.rs

…d-objstore

Xuanwo

Thank you, I believe are are very close now!

integrations/object_store/src/service/deleter.rs

Xuanwo · 2025-08-05T09:36:47Z

integrations/object_store/src/service/mod.rs

+                stat_with_if_match: true,
+                stat_with_if_unmodified_since: true,
+                read: true,
+                write: true,


object_store should support many memtadata, but it's fine to leave them in next PR.

integrations/object_store/src/service/mod.rs

Xuanwo · 2025-08-05T09:38:06Z

integrations/object_store/src/service/mod.rs

+        let writer = ObjectStoreWriter::new(self.store.clone(), path, args);
+        Ok((
+            RpWrite::default(),
+            MultipartWriter::new(self.info(), writer, 10),


We could make concurrent configurable; that might be a nice improvement for the next PR.

Xuanwo · 2025-08-05T09:38:55Z

integrations/object_store/src/service/mod.rs

+}
+
+#[cfg(test)]
+mod tests {


I wish we could integrate with opendal's test suites, but so far I haven't figured out how to make it work. It might be worth opening an issue.

integrations/object_store/src/service/reader.rs

Xuanwo · 2025-08-05T09:44:24Z

integrations/object_store/src/service/writer.rs

+
+        // Generate a proper ETag for the part based on its content
+        // This follows the pattern used by many object stores where ETags are MD5 hashes
+        let etag = calculate_part_etag(&bytes, part_number);


We don't need to do this, and it might be incorrect. Let's just return an empty string since it not used anyways.

fixed in dd71f57

integrations/object_store/src/service/writer.rs

integrations/object_store/Cargo.toml

…er readability

…th consistent naming

flaneur2020 · 2025-08-16T04:59:41Z

@Xuanwo sorry for the late response. i updated the PR with following the feedbacks. please take another look when you have time at your convenience qwq

there're still some follow ups that might be in future prs:

make it integrated with opendal's existing test suites
enhance the metadata handling
make concurrent configurable in writer

Xuanwo

Thank you for working on this!

flaneur2020 added 9 commits June 7, 2025 13:45

add object-store as service backend

e424a77

add object store as features

64c24ad

add scaffold for impl Access

52f6876

add parse_error

b5aa86c

add read()

b54326f

add reader

abd5586

factor out reader.rs

51c9c8f

add writer

9717838

Merge branch 'apache:main' into add-objstore

babd9be

flaneur2020 added 7 commits June 15, 2025 20:28

move the code from core/ to integration/object_store

b0cd0e4

move backend.rs to mod.rs

1af671d

add deleter.rs

0a485f0

add deleter

6c57e37

add lister.rs

8c73e9e

add test case

040adad

add test cases

0c166af

Xuanwo requested a review from Copilot July 4, 2025 10:27

Copilot AI reviewed Jul 4, 2025

View reviewed changes

integrations/object_store/src/backend/lister.rs Outdated Show resolved Hide resolved

integrations/object_store/src/backend/deleter.rs Outdated Show resolved Hide resolved

flaneur2020 added 5 commits July 20, 2025 22:09

fix apache header

3945395

Merge remote-tracking branch 'origin/main' into add-objstore

663ae77

Merge branch 'add-objstore' of github.com:flaneur2020/opendal into ad…

4b2c58d

…d-objstore

cargo fmt

0430963

fix clippy

7495e59

flaneur2020 force-pushed the add-objstore branch from 5d8c912 to 7495e59 Compare July 20, 2025 14:21

flaneur2020 marked this pull request as ready for review July 20, 2025 14:36

flaneur2020 requested a review from Xuanwo as a code owner July 20, 2025 14:36

dosubot bot added the size:XL This PR changes 500-999 lines, ignoring generated files. label Jul 20, 2025

flaneur2020 added 8 commits August 4, 2025 20:33

revert cargo.toml

05a30cd

fix set_err to track the cause

4b21fa8

fix build

d559661

fix oio::Read

cb1c3bb

extract a format_metadata

8e45f6d

implement BatchDelete

a022cd7

do not allow delete with version

a539490

refactor the multipart upload

a6076c7

flaneur2020 force-pushed the add-objstore branch from 5a2755f to a6076c7 Compare August 4, 2025 14:34

flaneur2020 requested a review from Xuanwo August 4, 2025 14:36

Xuanwo reviewed Aug 5, 2025

View reviewed changes

clean up unused code

dd71f57

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XXL This PR changes 1000+ lines, ignoring generated files. labels Aug 6, 2025

refactor: import object_store::path::Path as ObjectStorePath for bett…

f6b905f

…er readability

dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Aug 16, 2025

refactor: move OpXxx to GetOptions conversion functions to core.rs wi…

89086e0

…th consistent naming

flaneur2020 force-pushed the add-objstore branch from 594eaa6 to 89086e0 Compare August 16, 2025 04:20

handle op_stat by get_opts

9299efc

flaneur2020 requested a review from Xuanwo August 16, 2025 04:57

Merge branch 'main' into add-objstore

1796b52

flaneur2020 added 2 commits August 16, 2025 12:59

explicitly impl Sync to replace Mutex

ed9655c

cargo fmt

bdcac6d

Xuanwo approved these changes Aug 26, 2025

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Aug 26, 2025

Xuanwo merged commit f00fd1c into apache:main Aug 26, 2025
36 checks passed

mbutrovich mentioned this pull request Oct 13, 2025

chore(deps): bump opendal from 0.54.0 to 0.54.1 in /native apache/datafusion-comet#2559

Merged

flaneur2020 mentioned this pull request Nov 23, 2025

Make a PoC about using OpenDAL to replace object_store slatedb/slatedb#1043

Open

Comments

Conversation

flaneur2020 commented Jun 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

Uh oh!

Xuanwo commented Jun 11, 2025

Uh oh!

flaneur2020 commented Jun 12, 2025

Uh oh!

Xuanwo commented Jul 4, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Xuanwo Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Xuanwo Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Xuanwo Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Xuanwo Aug 5, 2025

Choose a reason for hiding this comment

Uh oh!

flaneur2020 Aug 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

flaneur2020 commented Aug 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

flaneur2020 commented Jun 11, 2025 •

edited

Loading

flaneur2020 commented Aug 16, 2025 •

edited

Loading