Skip to content

Comments

refactor: Introduce IpfsCore for improved service structure#5950

Merged
Xuanwo merged 2 commits intoapache:mainfrom
miroim:split-ipfs-core
Apr 5, 2025
Merged

refactor: Introduce IpfsCore for improved service structure#5950
Xuanwo merged 2 commits intoapache:mainfrom
miroim:split-ipfs-core

Conversation

@miroim
Copy link
Member

@miroim miroim commented Apr 3, 2025

Which issue does this PR close?

Part of #5702
Part of #5677

Rationale for this change

What changes are included in this PR?

  • This pull request refactors the IpfsBackend by introducing a new IpfsCore struct to encapsulate the core functionality and state. The changes aim to improve code organization and maintainability.
  • Use the http_client from AccessorInfo, instead of directly accessing the backend, to migrate the service to a context-based http client.

Are there any user-facing changes?

@miroim miroim requested a review from Xuanwo as a code owner April 3, 2025 02:30
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. core services/ipfs labels Apr 3, 2025
@miroim
Copy link
Member Author

miroim commented Apr 3, 2025

Behavior tests
cargo test behavior --features tests,services-ipfs -- --show-output
    Finished `test` profile [unoptimized + debuginfo] target(s) in 3.08s
     Running unittests src\lib.rs (target\debug\deps\opendal-90b9ea2d0dddfb7c.exe)

running 0 tests

successes:

successes:

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 134 filtered out; finished in 0.00s

     Running tests\behavior\main.rs (target\debug\deps\behavior-c0c901956f19c793.exe)

running 23 tests
test behavior::test_read_only_read_with_range              ... ok
test behavior::test_read_only_read_full_with_special_chars ... ok
test behavior::test_read_only_read_with_dir_path           ... ok
test behavior::test_read_only_read_with_if_match           ... ok
test behavior::test_read_only_read_with_if_none_match      ... ok
test behavior::test_reader_only_read_with_if_match         ... ok
test behavior::test_reader_only_read_with_if_none_match    ... ok
test behavior::test_read_only_read_full                    ... ok
test behavior::test_read_only_read_not_exist               ... ok
test behavior::test_read_only_stat_not_cleaned_path        ... ok
test behavior::test_read_only_stat_file_and_dir            ... ok
test behavior::test_read_only_stat_with_if_match           ... ok
test behavior::test_read_only_stat_with_if_none_match      ... ok
test behavior::test_read_only_stat_root                    ... ok
test behavior::test_read_only_stat_not_exist               ... ok
test behavior::test_read_only_stat_special_chars           ... ok
test behavior::test_blocking_read_only_stat_file_and_dir   ... ok
test behavior::test_blocking_read_only_stat_not_exist      ... ok
test behavior::test_blocking_read_only_stat_special_chars  ... ok
test behavior::test_blocking_read_only_read_with_range     ... ok
test behavior::test_blocking_read_only_read_not_exist      ... ok
test behavior::test_blocking_read_only_read_full           ... ok
test behavior::test_list_only                              ... ok

test result: ok. 23 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.47s

Copy link
Member

@erickguan erickguan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for getting to IPFS!

Can we use core directly in DirStream?

impl oio::PageList for DirStream {
async fn next_page(&self, ctx: &mut oio::PageContext) -> Result<()> {
let resp = self.backend.ipfs_list(&self.path).await?;
let resp = self.backend.core.ipfs_list(&self.path).await?;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we reference self.core directly without going from backend?

Because you have IpfsCore, we can have:

pub struct DirStream {
    core: Arc<IpfsCore>,
    path: String,
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your review!

DirStream needs some backend functionality

let meta = self
.backend
.stat(&name, OpStat::new())
.await?
.into_metadata();

or we can have:

pub struct DirStream {
    backend: Arc<IpfsBackend>,
    core: Arc<IpfsCore>,
    path: String,
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right - we are depending on IpfsBackend::stat. Could we take it a step further and build a function like pub async fn IpfsCore::ipfs_stat(name: &str) -> Result<Metadata>? This would allow you to retrieve metadata for each name directly via the core for DirStream.

As aside, while this change improves the structure, it doesn't resolve the N+1 issue... That is not related to the issue. So we can ignore it for now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we take it a step further and build a function like pub async fn IpfsCore::ipfs_stat(name: &str) -> Result?

Yes, I'll raise another PR

@erickguan
Copy link
Member

Good plan. Thanks for sharing some love for IPFS!

Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @miroim for this! And thank you @erickguan for the review.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Apr 5, 2025
@Xuanwo Xuanwo merged commit 7636db9 into apache:main Apr 5, 2025
241 checks passed
@miroim miroim deleted the split-ipfs-core branch April 5, 2025 08:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core lgtm This PR has been approved by a maintainer services/ipfs size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants