Rename of RefreshableCredentials is causing an issue #348
Description
Describe the bug
I'm running dbt debug in a databricks job cluster.
The dbt cli using the databricks adapter is having trouble connecting to databricks. This is the error I'm seeing.
It cannot find the RefreshableCredentials class since it was recently renamed to SessionCredentials.
Steps To Reproduce
The easiest way to reproduce this is by running dbt debug in a databricks job while using the
Expected behavior
It should successfully retrieve the credentials and connect to the cluster.
Screenshots and log output
+ dbt debug
22:32:36 Running with dbt=1.5.0
22:32:36 dbt version: 1.5.0
22:32:36 python version: 3.9.5
22:32:36 python path: /local_disk0/.ephemeral_nfs/cluster_libraries/python/bin/python
22:32:36 os info: Linux-5.15.0-1035-aws-x86_64-with-glibc2.31
22:32:36 Using profiles.yml file at /tmp/tmp-dbt-run-96368-profiles/profiles.yml
22:32:36 Using dbt_project.yml file at /tmp/tmp-dbt-run-96368/dbt_project.yml
22:32:36 Configuration:
22:32:38 Encountered an error:
cannot import name 'RefreshableCredentials' from 'databricks.sdk.oauth' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/databricks/sdk/oauth.py)
22:32:38 Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/requires.py", line 86, in wrapper
result, success = func(*args, **kwargs)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/requires.py", line 71, in wrapper
return func(*args, **kwargs)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/cli/main.py", line 417, in debug
results = task.run()
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/task/debug.py", line 110, in run
self.test_configuration()
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/task/debug.py", line 294, in test_configuration
profile_status = self._load_profile()
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/task/debug.py", line 254, in _load_profile
profile: Profile = Profile.render(
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 436, in render
return cls.from_raw_profiles(
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 401, in from_raw_profiles
return cls.from_raw_profile_info(
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 355, in from_raw_profile_info
credentials: Credentials = cls._credentials_from_profile(
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/config/profile.py", line 165, in _credentials_from_profile
cls = load_plugin(typename)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/factory.py", line 202, in load_plugin
return FACTORY.load_plugin(name)
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/factory.py", line 57, in load_plugin
mod: Any = import_module("." + name, "dbt.adapters")
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 855, in exec_module
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/databricks/__init__.py", line 1, in <module>
from dbt.adapters.databricks.connections import DatabricksConnectionManager # noqa
File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/dbt/adapters/databricks/connections.py", line 54, in <module>
from databricks.sdk.oauth import OAuthClient, RefreshableCredentials
ImportError: cannot import name 'RefreshableCredentials' from 'databricks.sdk.oauth' (/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/databricks/sdk/oauth.py)
System information
dbt version
22:32:36 dbt version: 1.5.0
The operating system you're using:
os info: Linux-5.15.0-1035-aws-x86_64-with-glibc2.31
The output of python --version:
22:32:36 python version: 3.9.5
Additional context
I believe this is caused due to the latest databricks-sdk-py release with the change that was pushed here:
databricks/databricks-sdk-py#116
The best fix would be to just bump the version of the databricks-sdk to 0.1.7 in the dbt-databricks repository.
I'm happy to raise that PR.