Port object_store integration tests, use github actions#2148
Port object_store integration tests, use github actions#2148alamb merged 15 commits intoapache:masterfrom
object_store integration tests, use github actions#2148Conversation
object_store_integration testsobject_store integration tests to arrow-rs / github actions:
object_store integration tests to arrow-rs / github actions:object_store integration tests to arrow-rs / github actions
| - object_store/** | ||
| - .github/** | ||
|
|
||
| jobs: |
There was a problem hiding this comment.
This is more or less a direct port of https://github.com/influxdata/object_store_rs/blob/fe716c3/.circleci/config.yml
| let (is_emulator, storage_account_client) = if use_emulator { | ||
| check_if_emulator_works()?; | ||
| (true, StorageAccountClient::new_emulator_default()) | ||
| // Allow overriding defaults. Values taken from |
There was a problem hiding this comment.
This is required to specify a URL different than 127.0.0.1 -- the github actions "services" run with some actual dns name rather than on the local loopback (e.g. you need to specify http://auzure:10000 rather than http://127.0.0.1:10000)
Codecov Report
@@ Coverage Diff @@
## master #2148 +/- ##
==========================================
- Coverage 82.82% 82.82% -0.01%
==========================================
Files 237 237
Lines 61425 61428 +3
==========================================
+ Hits 50874 50875 +1
- Misses 10551 10553 +2
Help us with your feedback. Take ten seconds to tell us how you rate us. |
|
|
||
| impl GoogleCloudConfig { | ||
| fn build(self) -> Result<GoogleCloudStorage> { | ||
| // ignore HTTPS errors in tests so we can use fake-gcs server |
There was a problem hiding this comment.
Note that this is only done in the tests, whereas the original approach allowed it in production as well)
|
|
||
| We don't have a good story yet for testing the GCP integration locally. You will need to create a GCS bucket, a | ||
| service account that has access to it, and use this to run the tests. | ||
| To test the GCS integration, we use [Fake GCS Server](https://github.com/fsouza/fake-gcs-server) |
There was a problem hiding this comment.
Added documentation on how to run these integration tests
|
Here is an example of the new test running: https://github.com/apache/arrow-rs/runs/7483480192?check_suite_focus=true |
object_store integration tests to arrow-rs / github actionsobject_store integration tests, use github actions
Co-authored-by: Raphael Taylor-Davies <[email protected]>
…tore_integration_test
| let store = memory::InMemory::new(); | ||
| let stream = list_store(&store, "path").await.unwrap(); | ||
| assert_eq!(stream.count().await, 0); | ||
| let mut stream = list_store(&store, "path").await.unwrap(); |
There was a problem hiding this comment.
I am not sure why, but locally the stream.count() construction started erroring like this for me:
cd /Users/alamb/Software/arrow-rs2 && RUST_LOG=trace RUST_BACKTRACE=1 CARGO_TARGET_DIR=/Users/alamb/Software/df-target2 cargo test -p object_store --features=aws,azure,azure_test,gcp
Compiling object_store v0.3.0 (/Users/alamb/Software/arrow-rs2/object_store)
error[E0599]: `Pin<Box<dyn futures::Stream<Item = std::result::Result<ObjectMeta, Error>> + std::marker::Send>>` is not an iterator
--> object_store/src/lib.rs:699:27
|
699 | assert_eq!(stream.count().await, 0);
| ^^^^^ `Pin<Box<dyn futures::Stream<Item = std::result::Result<ObjectMeta, Error>> + std::marker::Send>>` is not an iterator
|
::: /Users/alamb/.rustup/toolchains/stable-x86_64-apple-darwin/lib/rustlib/src/rust/library/core/src/pin.rs:408:1
|
408 | pub struct Pin<P> {
| ----------------- doesn't satisfy `_: Iterator`
|
::: /Users/alamb/.cargo/registry/src/github.com-1ecc6299db9ec823/futures-core-0.3.17/src/stream.rs:27:1
|
27 | pub trait Stream {
| ---------------- doesn't satisfy `_: Iterator`
|
= note: the following trait bounds were not satisfied:
`Pin<Box<dyn futures::Stream<Item = std::result::Result<ObjectMeta, Error>> + std::marker::Send>>: Iterator`
which is required by `&mut Pin<Box<dyn futures::Stream<Item = std::result::Result<ObjectMeta, Error>> + std::marker::Send>>: Iterator`
`dyn futures::Stream<Item = std::result::Result<ObjectMeta, Error>> + std::marker::Send: Iterator`
which is required by `&mut dyn futures::Stream<Item = std::result::Result<ObjectMeta, Error>> + std::marker::Send: Iterator`
There was a problem hiding this comment.
That sounds like the StreamExt is not in scope for some reason
There was a problem hiding this comment.
Yeah, I tried a few ways to get that to work, but rustc gave me the same error and told me that StreamExt was unused ...
|
🤔 now the test https://github.com/apache/arrow-rs/runs/7499943986?check_suite_focus=true is failing with: I guess I will continue on the path laid out in #2149 |
|
However I don't see any reason to block merging this PR due to an unrelated issue, so merging |
|
Benchmark runs are scheduled for baseline = bcaa0b6 and contender = e68852c. e68852c is a master commit associated with this PR. Results will be available as each benchmark for each run completes. |
* Add github test skeleton * Cleanups and fmt * Run on changes to object_store * Update name * Broken yaml? * Remove uneeded lint job * Run only object store tests * Add local gcp test instructions * Allow custom http client for gcs * remove unused error * Also run clippy * Update object_store/src/gcp.rs Co-authored-by: Raphael Taylor-Davies <[email protected]> * rename more * Fixup test Co-authored-by: Raphael Taylor-Davies <[email protected]>
Which issue does this PR close?
re #2030
Rationale for this change
#2081 brought in the
object_store_rscode, but did not include the test scripts for running object store tests against s3/azure/gcp simulatorsWhat changes are included in this PR?
Adds github action based tests for running object store integration tests
Adds scripts + small object store configuration changes required to run them
It is based on my PR influxdata/object_store_rs#51 which I used to prototype the changes
I ported this using these commands:
Are there any user-facing changes?
Not really