Skip to content

allow the aws credentials profile to be specified - #6

Merged
pwinckles merged 3 commits into
masterfrom
issue-5
Jul 15, 2021
Merged

allow the aws credentials profile to be specified#6
pwinckles merged 3 commits into
masterfrom
issue-5

Conversation

@pwinckles

@pwinckles pwinckles commented Jul 15, 2021

Copy link
Copy Markdown
Owner

Adds a new --profile option at the root level that allows an aws credentials profile to be specified. This option can also be specified in the config file.

Example usage:

rocfl -p my-profile -b my-bucket -R us-east-2 ls

Resolves #5

@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 02:05 Inactive
@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 02:05 Inactive
@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 02:05 Inactive
@ives1227

Copy link
Copy Markdown

@pwinckles Thank you for such a quick turnaround! Happy to test and I did. Unfortunately, I'm receiving errors. If I do the following aws cli command, I get a list of items in the s3 bucket:
aws s3 ls s3://mybucket --profile abc

If I do this in rocfl,
rocfl -p abc -b mybucket -R us-east-1 ls

then the following error occurs:

[ERROR] Failed to list storage root extensions: Request ID: None Body: <?xml version="1.0" encoding="UTF-8"?>
    <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>XXXX (omitted)</RequestId><HostId>ALONGDIGESTTHATIREMOVED</HostId></Error>
[ERROR] Failed to load OCFL layout: unknown variant `NNNN-flat-omit-prefix-storage-layout`, expected one of `0002-flat-direct-storage-layout`, `0004-hashed-n-tuple-storage-layout`, `0003-hash-and-id-n-tuple-storage-layout`, `0006-flat-omit-prefix-storage-layout` at line 2 column 54

@pwinckles

Copy link
Copy Markdown
Owner Author

@ives1227 Is that all of the output? I would have expected it to have scanned the repo for objects even if it couldn't load your storage layout.

The errors it reported are two different problems:

  1. It attempted to do a list objects request on a prefix and got an AccesDenied response. Does the user have permission to list objects?
  2. The ocfl_layout.json file in the root of the repository references the pre-adoption version of your storage layout. You can fix this problem by making the following changes:
    1. In ocfl_layout.json change the extension to be 0006-flat-omit-prefix-storage-layout
    2. You should also have an object at extensions/NNNN-flat-omit-prefix-storage-layout/config.json, yes? This object should be moved to extensions/0006-flat-omit-prefix-storage-layout/config.json
    3. Inside the config.json file that you just moved, the extensionName should be changed to 0006-flat-omit-prefix-storage-layout

That said, if this is a repository that you are also interacting with using ocfl-java, then you probably don't want to make these changes until after ocfl-java has been updated to support the adopted version of the extension, either officially or by updating your local implementation of the extension.

Out of curiosity, do you ever use the config file to avoid having to specify so many options?

@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 11:48 Inactive
@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 11:48 Inactive
@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 11:48 Inactive
@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 11:59 Inactive
@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 11:59 Inactive
@pwinckles
pwinckles temporarily deployed to ci July 15, 2021 11:59 Inactive
@pwinckles

Copy link
Copy Markdown
Owner Author

@ives1227 Is your repository located in the bucket root or is it using a prefix? I almost always put repositories under a prefix, and I just noticed that there is a bug when you do not use a prefix, which could explain why you didn't see any objects listed earlier.

The most recent commit to this PR should address the problem.

@ives1227

Copy link
Copy Markdown

@pwinckles our bucket is at root. Our ops team sets up the buckets and credentials and this is the way they choose to do it.

To answer your question from #6 (comment):

  1. Yes, I have permission to list objects with or without a prefix.
  2. These configs are actually in place.

I'm still receiving the same errors with the latest release. My suspicion is that we set up our repos differently. I have a work around for what I need this for but it would be nice to have it eventually. If I have time in the future, I'll see if I can help a bit more.

@pwinckles
pwinckles merged commit 360236d into master Jul 15, 2021
@pwinckles
pwinckles deleted the issue-5 branch October 25, 2021 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Profile option for S3 connection

2 participants