-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Enable dbt Cloud provider to interact with single tenant instances #22938
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
|
|
Keeping my eyeballs on this PR |
2771d62 to
80b19c2
Compare
|
Hello~ Following up to see if this needs anything else for review? 🥲 |
|
@epapineau can you post a screenshot of this working on single tenant based on your code changes? |
|
@sungchun12 Added to the PR. Please let me know if you'd like any others :) |
|
@epapineau How would users typically connect to these single-tenant instances? Does one API key have access to multiple, single-tenant instances or should one API only have access to an individual single-tenant instance? The reason I ask is, if it's the latter, it might be better to have users provide a URL or tenant name in the Airflow connection rather than having to pass it in every operator/sensor instance. |
|
@josh-fell one API key for one single tenant instance. A large customer may have multiple single tenant instances. Considering that, do you think it would be best to have the single tenant option on the connection, but leave it override-able at the operator/sensor, like with |
|
@josh-fell Added tenant as an option to the Airflow connection. Let me know what you think. |
So it sounds like users would need to configure an Airflow connection per API key they use? Or is it possible that a single API key could access a single tenant instance + a multi-tenant instance therefore the need for an override when using a single Airflow connection? |
|
@josh-fell It is not possible that a single API key could access a single tenant instance + a multi-tenant instance. Each instance will require its own API key |
So if users will always need to use a separate connection for each API key they want to use because said key can't cross-pollinate with other instances, it seems like the move is to have tenant optionally configurable in the connection rather than overriding at the task level. If it's not populated in the connection, then it's assumed to be a multi-tenant instance which always has a "cloud" tenant name. Am I in the ballpark with that? Or what is a good use case to have the override at the task level? |
|
@josh-fell Yep, totally follow. I removed the arg from the Operator. Do you have an opinion whether the tenant should be stored in the connection as the schema or as an extra? |
Schema is great I think. |
|
@josh-fell okay changes pushed 🎉 |
josh-fell
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you remove the tenant parameter from the hook and sensor as well as update the unit test?
| def test_init_hook_single_tenant(self): | ||
| hook = DbtCloudHook(tenant="single.tenant") | ||
| assert hook.dbt_cloud_conn_id == "dbt_cloud_default" | ||
| assert hook.base_url == "https://single.tenant.getdbt.com/api/v2/accounts/" | ||
| assert hook.auth_type == TokenAuth | ||
| assert hook.method == "POST" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you update this test accordingly too?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Definitely. I tried my hand at parameterizing this for an additional test connection. Please lmk what you think.
josh-fell
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we're almost there! Just need to address the unit test failures.
|
@josh-fell I was able to resolve the unit test failures locally by updating the value of |
I don't think that will work for CI unfortunately. The test will be looking for a connection that's accessible by the CI environment. I can see a couple options to help resolve the failure:
@cached_property
def base_url(self) -> str:
tenant = self.connection.schema if self.connection.schema else "cloud"
return f"https://{tenant}.getdbt.com/api/v2/accounts/"I don't have a strong opinion either way. |
|
@josh-fell cool I went with option one & it's passing now 🥳 |
|
Looks fantastic!! Would you mind rebasing to the latest main when you get a chance please? We can go ahead and merge afterwards. |
1827f65 to
203fe71
Compare
closes: #22730
Enables dbt Cloud provider to make API calls to single tenant dbt Cloud instances using the tenant arg on
DbtCloudRunJobOperatorScreenshots
dbt Cloud Single Tentant Instance Job + DAG
Line 31 of dbt_cloud_provider_example.py ==
default_args={"dbt_cloud_conn_id": "dbt_cloud", "tenant": "staging.singletenant"},dbt Cloud Multi Tentant Instance Job + DAG
Line 31 of dbt_cloud_provider_example.py ==


default_args={"dbt_cloud_conn_id": "dbt_cloud"},^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.