Add integration test for Shard Snapshot API with S3 storage#7
Merged
kemkemG0 merged 1 commit intofeature/make-s3-available-for-snapshotsfrom May 10, 2024
Merged
Conversation
kemkemG0
added a commit
that referenced
this pull request
May 11, 2024
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation * [refactor] use snapshots_config instead of s3_config * update config * add AWS official`aws-sdk-s3` * implement store_file() WITHOUT error handling * implement list_snapshots * implement delete_snapshot * run `cargo +nightly fmt` * delete println * implement get_stored_file * Add error handlings * Refactor AWS S3 configuration and error handling * fix bugs * create an empty test file * fix `alias_test.rs` for StorageConfig type * tempolary delete some test and try s3 test * Update integration-tests.yml to use snap instead of apt-get for installing yq * Update integration-tests.yml to use sudo when installing yq * add sudo * make (full/non-full) snapshots downloadable * debug * small fix * Add S3 endpoint URL configuration option * fix * fix * debug * fix endpoint * update to http://127.0.0.1:9000/ * update * fix * fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3 * put original tests back * refactor * small fix (delete println & echo) * use object_store and refactor * create snapshot_storage_ops and implement * Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size * cargo +nightly fmt --all * make it more abstract * Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig * small update * small fix * Update dependencies in Cargo.lock * Update minio image to satantime/minio-server * Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs * Fix issue with downloaded file size not matching expected size in download_snapshot function * add flush * Use Streaming instead of donloading once * apply `cargo +nightly fmt --all` * Fix issue with opening file in SnapshotStream::LocalFS variant * Fix error handling in SnapshotStream::LocalFS variant * Add integration test for Shard Snapshot API with S3 storage (#7)
kemkemG0
added a commit
that referenced
this pull request
Jun 3, 2024
* Add SnapshotsStorageConfig enum(Local or S3) and deserialize implementation * [refactor] use snapshots_config instead of s3_config * update config * add AWS official`aws-sdk-s3` * implement store_file() WITHOUT error handling * implement list_snapshots * implement delete_snapshot * run `cargo +nightly fmt` * delete println * implement get_stored_file * Add error handlings * Refactor AWS S3 configuration and error handling * fix bugs * create an empty test file * fix `alias_test.rs` for StorageConfig type * tempolary delete some test and try s3 test * Update integration-tests.yml to use snap instead of apt-get for installing yq * Update integration-tests.yml to use sudo when installing yq * add sudo * make (full/non-full) snapshots downloadable * debug * small fix * Add S3 endpoint URL configuration option * fix * fix * debug * fix endpoint * update to http://127.0.0.1:9000/ * update * fix * fix `#[get("/collections/{collection}/shards/{shard}/snapshots/{snapshot}")]` for s3 * put original tests back * refactor * small fix (delete println & echo) * use object_store and refactor * create snapshot_storage_ops and implement * Refactor get_appropriate_chunk_size function to adjust chunk size based on service limits and file size * cargo +nightly fmt --all * make it more abstract * Refactor SnapshotsStorageConfig deserialization in SnapShotsConfig * small update * small fix * Update dependencies in Cargo.lock * Update minio image to satantime/minio-server * Refactor snapshot storage paths in snapshots_manager.rs and snapshot_storage_ops.rs * Fix issue with downloaded file size not matching expected size in download_snapshot function * add flush * Use Streaming instead of donloading once * apply `cargo +nightly fmt --all` * Fix issue with opening file in SnapshotStream::LocalFS variant * Fix error handling in SnapshotStream::LocalFS variant * Add integration test for Shard Snapshot API with S3 storage (#7)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
All Submissions:
devbranch. Did you create your branch fromdev?New Feature Submissions:
cargo +nightly fmt --allcommand prior to submission?cargo clippy --all --all-featurescommand?Changes to Core Features: