Skip to content

feat: Allow providing a service account key directly for GCS#3489

Merged
tustvold merged 3 commits intoapache:masterfrom
GlareDB:gcp-sa-key-no-file
Jan 9, 2023
Merged

feat: Allow providing a service account key directly for GCS#3489
tustvold merged 3 commits intoapache:masterfrom
GlareDB:gcp-sa-key-no-file

Conversation

@scsmithr
Copy link
Contributor

@scsmithr scsmithr commented Jan 7, 2023

Which issue does this PR close?

Closes apache/arrow-rs-object-store#201

Rationale for this change

Use case:

We're storing service accounts keys external to where the object store client is
being created. We do not want to have to write the key to a file before creating
the object store client. This change allows for providing the key directly.

What changes are included in this PR?

Adds an appropriate method to the GCS object store builder for supplying the service account key directly. Only one of service account path or service account key may be provided, otherwise build will return an appropriate error.

Are there any user-facing changes?

An additional method on GCS object store builder.

There are currently no breaking changes, however I believe the ServiceAccount variant for the GoogleConfigKey should be renamed to ServiceAccountPath to better represent what that option is for. I held off on making that change because I saw that the changelog was already generated for 0.5.3 which includes the new GoogleConfigKey stuff, making that a breaking change. If that's an acceptable breaking change, I'm down to go ahead and do that in this PR as well.

Use case:

We're storing service accounts keys external to where the object store client is
being created. We do not want to have to write the key to a file before creating
the object store client. This change allows for providing the key directly.
@tustvold
Copy link
Contributor

tustvold commented Jan 8, 2023

Thank you, makes sense to me.

I believe the ServiceAccount variant for the GoogleConfigKey should be renamed to ServiceAccountPath to better represent what that option is for.

Yeah, probably not worth introducing a breaking change over, but perhaps we could deserialize "service_account_path" and "google_service_account_path" to allow for users to opt-in to using a more explicit config key. What do you think?

"google_service_account_path" and "service_account_path" can now be used.
@scsmithr
Copy link
Contributor Author

scsmithr commented Jan 8, 2023

perhaps we could deserialize "service_account_path" and "google_service_account_path" to allow for users to opt-in to using a more explicit config key. What do you think?

Seems reasonable to me. Latest commit has those included.

@tustvold tustvold merged commit eae993f into apache:master Jan 9, 2023
@ursabot
Copy link

ursabot commented Jan 9, 2023

Benchmark runs are scheduled for baseline = c746658 and contender = eae993f. eae993f is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-rs-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

alamb pushed a commit to alamb/arrow-rs that referenced this pull request Mar 20, 2025
…3489)

* feat: Allow providing a service account key directly for GCP

Use case:

We're storing service accounts keys external to where the object store client is
being created. We do not want to have to write the key to a file before creating
the object store client. This change allows for providing the key directly.

* Add additional aliases for specifying service account path

"google_service_account_path" and "service_account_path" can now be used.

* Add test asserting aliases set appropriate config option
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow providing service account key directly when building GCP object store client

3 participants