fix(services/webdav): Add possibility to answer without response if file isn't exist by AJIOB · Pull Request #4170 · apache/opendal

AJIOB · 2024-02-07T16:29:21Z

While testing sccache with JFrog Artifactory as a WebDAV storage, I found the problem: sccache cannot store marker file while the curl can do it.

After deeper investigation & debugging I found that Artifactory answers to the PROPFIND request with this XML when file is not presented on the remote server:

<?xml version="1.0" encoding="UTF-8"?>
<D:multistatus xmlns:D="DAV:" xmlns:ns0="DAV:" />

And on the next step Multistatus cannot be deserialized => sccache was panicked with this logs:

[2024-02-07T14:24:29Z DEBUG opendal::services::webdav::backend] backend stat response bytes (string): "<?xml version=\"1.0\" encoding=\"utf-8\" ?>\n<D:multistatus xmlns:D=\"DAV:\" xmlns:ns0=\"DAV:\"></D:multistatus>\n"
[2024-02-07T14:24:29Z ERROR opendal::services] service=webdav operation=stat path=.sccache_check -> Unexpected (permanent) at stat, context: { service: webdav, path: .sccache_check } => deserialize xml, source: missing field `response`
[2024-02-07T14:24:30Z ERROR sccache::server] storage check failed for: cache storage failed to read: Unexpected (permanent) at stat => deserialize xml

    Context:
       service: webdav
       path: .sccache_check

    Source:
       missing field `response`

    Backtrace:
       0: opendal::types::error::Error::new
                 at /home/alex/Projects/waste/opendal/core/src/types/error.rs:338:24
       1: opendal::raw::serde_util::new_xml_deserialize_error
                 at /home/alex/Projects/waste/opendal/core/src/raw/serde_util.rs:33:5
       2: core::ops::function::FnOnce::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
       3: core::result::Result<T,E>::map_err
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/result.rs:829:27
       4: <opendal::services::webdav::backend::WebdavBackend as opendal::raw::accessor::Accessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/services/webdav/backend.rs:315:13
       5: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
       6: <F as futures_core::future::TryFuture>::try_poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.30/src/future.rs:82:9
       7: <futures_util::future::try_future::into_future::IntoFuture<Fut> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/try_future/into_future.rs:34:9
       8: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/future/future/map.rs:55:37
       9: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
      10: <futures_util::future::try_future::MapErr<Fut,F> as core::future::future::Future>::poll
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.30/src/lib.rs:91:13
      11: <opendal::layers::error_context::ErrorContextAccessor<A> as opendal::raw::layer::LayeredAccessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/error_context.rs:173:14
      12: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      13: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      14: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      15: <alloc::sync::Arc<T> as opendal::raw::accessor::Accessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/accessor.rs:408:40
      16: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      17: opendal::layers::complete::CompleteAccessor<A>::complete_stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/complete.rs:226:37
      18: <opendal::layers::complete::CompleteAccessor<A> as opendal::raw::layer::LayeredAccessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/complete.rs:566:40
      19: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      20: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      21: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      22: <opendal::layers::logging::LoggingAccessor<A> as opendal::raw::layer::LayeredAccessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/layers/logging.rs:464:14
      23: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      24: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      25: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      26: opendal::raw::layer::LayeredAccessor::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:167:39
      27: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      28: opendal::raw::layer::<impl opendal::raw::accessor::Accessor for L>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/layer.rs:246:39
      29: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      30: <alloc::sync::Arc<T> as opendal::raw::accessor::Accessor>::stat::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/raw/accessor.rs:408:40
      31: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      32: opendal::types::operator::operator::Operator::read_with::{{closure}}::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/types/operator/operator.rs:536:26
      33: opendal::types::operator::operator::Operator::read::{{closure}}
                 at /home/alex/Projects/waste/opendal/core/src/types/operator/operator.rs:421:30
      34: <opendal::types::operator::operator::Operator as sccache::cache::cache::Storage>::check::{{closure}}
                 at /home/alex/Projects/waste/sccache/src/cache/cache.rs:487:31
      35: <core::pin::Pin<P> as core::future::future::Future>::poll
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/future/future.rs:125:9
      36: sccache::server::start_server::{{closure}}
                 at /home/alex/Projects/waste/sccache/src/server.rs:446:31
      37: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
      38: tokio::runtime::coop::with_budget
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
      39: tokio::runtime::coop::budget
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
      40: tokio::runtime::park::CachedParkThread::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
      41: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
      42: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
      43: tokio::runtime::context::runtime::enter_runtime
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
      44: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
      45: tokio::runtime::runtime::Runtime::block_on
                 at /home/alex/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:45
      46: sccache::server::start_server
                 at /home/alex/Projects/waste/sccache/src/server.rs:445:22
      47: sccache::commands::run_command
                 at /home/alex/Projects/waste/sccache/src/commands.rs:663:13
      48: sccache::main
                 at /home/alex/Projects/waste/sccache/src/lib.rs:88:30
      49: sccache::main
                 at /home/alex/Projects/waste/sccache/src/main.rs:18:5
      50: core::ops::function::FnOnce::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
      51: std::sys_common::backtrace::__rust_begin_short_backtrace
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/sys_common/backtrace.rs:154:18
      52: std::rt::lang_start::{{closure}}
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:167:18
      53: core::ops::function::impls::<impl core::ops::function::FnOnce<A> for &F>::call_once
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:284:13
      54: std::panicking::try::do_call
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      55: std::panicking::try
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      56: std::panic::catch_unwind
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      57: std::rt::lang_start_internal::{{closure}}
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:48
      58: std::panicking::try::do_call
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:552:40
      59: std::panicking::try
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:516:19
      60: std::panic::catch_unwind
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panic.rs:142:14
      61: std::rt::lang_start_internal
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:148:20
      62: std::rt::lang_start
                 at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/rt.rs:166:17
      63: main
      64: __libc_start_call_main
                 at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
      65: __libc_start_main_impl
                 at ./csu/../csu/libc-start.c:392:3
      66: _start

With my patch all works on the JFrog Artifactory WebDAV and on this WebDAV servers.
Without them, only the second server really works.

P.S. I've rebuilt the latest SCCache 0.7.7 with the latest OpenDAL 0.45.0 for collecting this backtrace & investigation.
P.P.S. After patches both SCCache cache hits & writes are works nice, tested locally.

Xuanwo

Perfect, thanks a lot!

Xuanwo · 2024-02-07T16:38:30Z

Does jfrog-artifactory-oss also provides webdav features? Maybe we can setup a new service to test webdav.

AJIOB · 2024-02-07T16:42:06Z

@Xuanwo, there is no info about this here.

As I can see, they wrote if feature is available in the JFrog Artifactory Pro and higher, as it was written for S3 backend here.

Xuanwo · 2024-02-07T16:45:01Z

@Xuanwo, there is no info about this here.

As I can see, they wrote if feature is available in the JFrog Artifactory Pro and higher, as it was written for S3 backend here.

Thanks for the info! I'll give it a shot. By adding a test for WebDAV, we can ensure that OpenDAL doesn't accidentally break it.

Add possibility to answer without response if file isn't exist

df799c0

AJIOB requested a review from Xuanwo as a code owner February 7, 2024 16:29

github-actions bot requested review from G-XD and Young-Flash February 7, 2024 16:29

Xuanwo changed the title ~~Add possibility to answer without response if file isn't exist~~ fix(services/webdav): Add possibility to answer without response if file isn't exist Feb 7, 2024

github-actions bot added the releases-note/fix The PR fixes a bug or has a title that begins with "fix" label Feb 7, 2024

Xuanwo approved these changes Feb 7, 2024

View reviewed changes

Xuanwo merged commit e628198 into apache:main Feb 7, 2024

Xuanwo mentioned this pull request Feb 7, 2024

Add new service test for webdav: JFrog Artifactory as a WebDAV storage #4171

Closed

AJIOB deleted the AJIOB/webdav-artifactory-sccache-workaround branch February 8, 2024 13:10

Xuanwo mentioned this pull request Feb 22, 2024

Bump to opendal version 0.45.1 #4246

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

fix(services/webdav): Add possibility to answer without response if file isn't exist#4170

fix(services/webdav): Add possibility to answer without response if file isn't exist#4170
Xuanwo merged 1 commit intoapache:mainfrom
AJIOB:AJIOB/webdav-artifactory-sccache-workaround

AJIOB commented Feb 7, 2024

Uh oh!

Xuanwo left a comment

Uh oh!

Xuanwo commented Feb 7, 2024

Uh oh!

AJIOB commented Feb 7, 2024

Uh oh!

Xuanwo commented Feb 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Comments

Conversation

AJIOB commented Feb 7, 2024

Uh oh!

Xuanwo left a comment

Choose a reason for hiding this comment

Uh oh!

Xuanwo commented Feb 7, 2024

Uh oh!

AJIOB commented Feb 7, 2024

Uh oh!

Xuanwo commented Feb 7, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants