Support GCP Auth by edwardfeng-db · Pull Request #444 · databricks/databricks-sdk-py

edwardfeng-db · 2023-11-14T20:50:11Z

Changes

Support auth in GCP
- With Google Credentials
- With Google Service Account

Tests

make test run locally
make fmt applied
relevant integration tests applied

codecov-commenter · 2023-11-14T20:52:55Z

Codecov Report

Attention: 32 lines in your changes are missing coverage. Please review.

Comparison is base (2b952c9) 51.67% compared to head (38cf1b7) 51.62%.
Report is 1 commits behind head on main.

Files	Patch %	Lines
databricks/sdk/core.py	23.80%	32 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #444      +/-   ##
==========================================
- Coverage   51.67%   51.62%   -0.06%     
==========================================
  Files          38       38              
  Lines       22277    22318      +41     
==========================================
+ Hits        11512    11521       +9     
- Misses      10765    10797      +32

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mgyucht

I thought I left the review but haven't apparently. Some minor notes, but this is very close.

databricks/sdk/__init__.py

databricks/sdk/core.py

setup.py

mgyucht · 2023-11-16T14:22:55Z

databricks/sdk/core.py

    return refreshed_headers


+@credentials_provider('google-credentials', ['host', 'google_credentials'])


I guess this is the same as with the Go library. But in theory, google_credentials does not even need to be specified, because there is also a default directory that the google-auth library looks in, if I'm not mistaken. We might be able to remove this and allow users to auto-login with their google credentials set up via the default app credentials pathway.

Yeah I thought so as well at the beginning but after checking more I think we do need to point it to the service account json file, which is different from the default auth json file as the service account one contains more info about the keys and secrets for signing the jwt etc. So it seems like we need this to make it work

mgyucht

LGTM provided nightly tests pass!

Major changes: * GCP Auth is now supported in the Python SDK. To use Google credentials-based authentication, specify your Default Application Credentials in the `GOOGLE_CREDENTIALS` environment variable or corresponding `google_credentials` parameter in `Config` or the client constructors. You may provide either the path to the file containing your credentials or the credentials themselves serialized as JSON. To use Google impersonation, specify the service principal to impersonate in the `DATABRICKS_GOOGLE_SERVICE_ACCOUNT` environment variable or the corresponding `google_service_account` parameter in `Config` or the client constructors. See [#444](#444) for the changes. Bug fixes: * Fix flask app example ([#445](#445)). * Fix deserialization of repeated enums ([#450](#450), [#452](#452)). * Capture stdout and stderr separately when calling Azure CLI ([#460](#460)). Other changes: * Change the name of retries logger to `databricks.sdk.retries` ([#453](#453)). API Changes: * Added `pipeline_id` field for `databricks.sdk.service.catalog.TableInfo`. * Added `enable_predictive_optimization` field for `databricks.sdk.service.catalog.UpdateCatalog` and `databricks.sdk.service.catalog.UpdateSchema`. * Removed `databricks.sdk.service.catalog.UpdatePredictiveOptimization` and `databricks.sdk.service.catalog.UpdatePredictiveOptimizationResponse` dataclasses. * Removed `enable_optimization()` method for [w.metastores](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/metastores.html) workspace-level service. * Added `description` field for `databricks.sdk.service.jobs.CreateJob` and `databricks.sdk.service.jobs.JobSettings`. * Added `list_network_connectivity_configurations()` and `list_private_endpoint_rules()` methods for [a.network_connectivity](https://databricks-sdk-py.readthedocs.io/en/latest/account/network_connectivity.html) account-level service. * Added `databricks.sdk.service.settings.ListNccAzurePrivateEndpointRulesResponse`, `databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsRequest`, `databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsResponse`, and `databricks.sdk.service.settings.ListPrivateEndpointRulesRequest` dataclasses. Internal changes: * Make ucws tests skipped when DATABRICKS_ACCOUNT_ID is present ([#448](#448)). OpenAPI SHA: 22f09783eb8a84d52026f856be3b2068f9498db3, Date: 2023-11-23 Dependency updates: * Bump API spec for Python SDK ([#454](#454)).

pietern · 2023-11-29T10:04:50Z

README.md

+                    google_service_account=input('Google Service Account: '))
+
+```
+


@edwardfeng-db Could you take a second pass at this doc? The field names mentioned are the Go SDK field names, not the Python SDK field names (PascalCase vs snake_case).

Yeah I see, sorry about this, let me update that. Some copy pasta issues

Major changes: * GCP Auth is now supported in the Python SDK. To use Google credentials-based authentication, specify your Default Application Credentials in the `GOOGLE_CREDENTIALS` environment variable or corresponding `google_credentials` parameter in `Config` or the client constructors. You may provide either the path to the file containing your credentials or the credentials themselves serialized as JSON. To use Google impersonation, specify the service principal to impersonate in the `DATABRICKS_GOOGLE_SERVICE_ACCOUNT` environment variable or the corresponding `google_service_account` parameter in `Config` or the client constructors. See [#444](#444) for the changes. Bug fixes: * Fix flask app example ([#445](#445)). * Fix deserialization of repeated enums ([#450](#450), [#452](#452)). * Capture stdout and stderr separately when calling Azure CLI ([#460](#460)). Other changes: * Change the name of retries logger to `databricks.sdk.retries` ([#453](#453)). API Changes: * Added `pipeline_id` field for `databricks.sdk.service.catalog.TableInfo`. * Added `enable_predictive_optimization` field for `databricks.sdk.service.catalog.UpdateCatalog` and `databricks.sdk.service.catalog.UpdateSchema`. * Removed `databricks.sdk.service.catalog.UpdatePredictiveOptimization` and `databricks.sdk.service.catalog.UpdatePredictiveOptimizationResponse` dataclasses. * Removed `enable_optimization()` method for [w.metastores](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/metastores.html) workspace-level service. * Added `description` field for `databricks.sdk.service.jobs.CreateJob` and `databricks.sdk.service.jobs.JobSettings`. * Added `list_network_connectivity_configurations()` and `list_private_endpoint_rules()` methods for [a.network_connectivity](https://databricks-sdk-py.readthedocs.io/en/latest/account/network_connectivity.html) account-level service. * Added `databricks.sdk.service.settings.ListNccAzurePrivateEndpointRulesResponse`, `databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsRequest`, `databricks.sdk.service.settings.ListNetworkConnectivityConfigurationsResponse`, and `databricks.sdk.service.settings.ListPrivateEndpointRulesRequest` dataclasses. Internal changes: * Make ucws tests skipped when DATABRICKS_ACCOUNT_ID is present ([#448](#448)). OpenAPI SHA: 22f09783eb8a84d52026f856be3b2068f9498db3, Date: 2023-11-23 Dependency updates: * Introduced "google-auth" dependency to support Google authentication.

edwardfeng-db force-pushed the edwardfeng-db/gcp-auth branch from 2473e67 to 112b27d Compare November 14, 2023 20:51

edwardfeng-db requested a review from mgyucht November 16, 2023 10:57

edwardfeng-db force-pushed the edwardfeng-db/gcp-auth branch from 80e42ba to d2e05b0 Compare November 20, 2023 20:08

mgyucht reviewed Nov 21, 2023

View reviewed changes

update

e404051

edwardfeng-db force-pushed the edwardfeng-db/gcp-auth branch from 58c0159 to e404051 Compare November 21, 2023 14:52

update

70c20ef

mgyucht approved these changes Nov 21, 2023

View reviewed changes

update

38cf1b7

edwardfeng-db added this pull request to the merge queue Nov 22, 2023

Merged via the queue into main with commit d8da7a8 Nov 22, 2023

edwardfeng-db deleted the edwardfeng-db/gcp-auth branch November 22, 2023 12:39

edwardfeng-db mentioned this pull request Nov 22, 2023

[NOT-MERGED] Support GCP auth #434

Closed

3 tasks

mgyucht mentioned this pull request Nov 29, 2023

Release v0.14.0 #463

Merged

pietern reviewed Nov 29, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support GCP Auth#444

Support GCP Auth#444
edwardfeng-db merged 3 commits intomainfrom
edwardfeng-db/gcp-auth

edwardfeng-db commented Nov 14, 2023

Uh oh!

codecov-commenter commented Nov 14, 2023 •

edited

Loading

Uh oh!

mgyucht left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgyucht Nov 16, 2023

Uh oh!

edwardfeng-db Nov 21, 2023

Uh oh!

mgyucht left a comment

Uh oh!

pietern Nov 29, 2023

Uh oh!

edwardfeng-db Nov 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		return refreshed_headers


		@credentials_provider('google-credentials', ['host', 'google_credentials'])

		google_service_account=input('Google Service Account: '))

		```

Conversation

edwardfeng-db commented Nov 14, 2023

Changes

Tests

Uh oh!

codecov-commenter commented Nov 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

mgyucht left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgyucht Nov 16, 2023

Choose a reason for hiding this comment

Uh oh!

edwardfeng-db Nov 21, 2023

Choose a reason for hiding this comment

Uh oh!

mgyucht left a comment

Choose a reason for hiding this comment

Uh oh!

pietern Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

edwardfeng-db Nov 29, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov-commenter commented Nov 14, 2023 •

edited

Loading