Support S3-compatible provider as storage backend#317
Conversation
|
This is awesome! |
How about
I'd define
I'd stick with official AWS SDK |
Do you mean something like this? storage_type = "local" # default value if omitted
[server]
hostname = "https://example-host.com"
port = 8080
data_dir = "/data/podsync/"
# ...storage_type = "s3"
[server]
hostname = "https://example-bucket-name.s3-website-us-west-2.amazonaws.com"
[s3]
endpoint = "https://s3.us-west-2.amazonaws.com"
region = "us-west-2"
bucket = "example-bucket-name"
# ... |
|
May be something like [storage]
type="local|s3"
[storage.s3]
...
[storage.local]
...WDYT? |
|
What would be under |
|
I think [storage]
# type="local" by default
[storage.local]
data_dir = "/data/podsync/"and remove |
|
Sounds good to me.
…Sent from my iPhone
On Jun 11, 2022, at 4:38 PM, Contextualist ***@***.***> wrote:
I think storage.local.data_dir makes more sense in the current model anyway. Maybe we could deprecate server.data_dir: emit a warning to the user and encourage the following settings
[storage]
# type="local" by default
[storage.local]
data_dir = "/data/podsync/"
and remove server.data_dir in the future. These changes will be transparent to the Docker users.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
This also deprecate `server.data_dir` in favor of `storage.local.data_dir`
|
Everything is ready except for documentation updates (README.md, config.toml.example, cloud_formation.yml). Since the |
|
Next release will include these changes, so it’s fine to include documentation updates too. |
|
Ok, done! |
|
Thanks! |
Fix two regressions introduced by PR #317
|
Next release will include these changes, so it’s fine to include documentation updates too.
…Sent from my iPhone
On Jun 12, 2022, at 11:37 AM, Contextualist ***@***.***> wrote:
Everything is ready except for documentation updates (README.md, config.toml.example, cloud_formation.yml). Since the storage section is a major change to the config, should we postpone the documentation change until the next release?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.
|
2 participants
Currently, this is a working implementation, but still work and discussion are needed. This resolves #159 and achieve part of the goals in #180.
To try this out, set a config as the following:
And set the access key and secret in environment variables
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY(credential should also works per aws-sdk-go's documentation, but I haven't tried it out).Optional: use CLI arg
--headlessto quit after first round of updates.To be discussed:
ShouldRenamed to--update-oncebe a config entry instead? Should we rename it?--headlessWhere should theIn the news3config entry be located?storageconfig sections3config entries?fs.Storageinterface includehttp.FileSystem?Dependency aws/aws-sdk-go is quite heavy (compiled binary size is increased from 4MB to 22MB); should we use a lighter implementation for the S3 API (e.g. rhnvrm/simples3)?No changeTo do:
fs.S3Any other discussions are welcome. I hope this PR can contribute some ideas on the S3 backend. If you already have something better, feel free to close this in favor of another PR!