Replace API server’s direct Connection access workaround in BaseHook#54083
Replace API server’s direct Connection access workaround in BaseHook#54083amoghrajesh merged 20 commits intoapache:mainfrom
Conversation
amoghrajesh
changed the title
|
Created the PR, let's see what tests start to fail. |
|
Core failures are handled by #54084 |
amoghrajesh
requested review from
eladkal,
hussein-awala and
jedcunningham
as code owners
|
Alright, fighting with the CI for compat + provider tests! |
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/hooks/test_kubernetes.py
Outdated
Show resolved
Hide resolved
providers/google/tests/unit/google/cloud/operators/test_dataflow.py
Outdated
Show resolved
Hide resolved
|
Alright, kube tests should be fixed by: #54261 Last set of compat failing! |
|
@ashb I made updates on this one, could you take another look when possible? |
|
Failing K8s tests to be fixed by #54385 |
github-project-automation
bot
moved this from In Progress
to Done
in AIP-72 - Task Execution Interface and SDK
|
@seanghaeli the eventual goal is to move towards using the task sdk for these sort of things, so I'd suggest that the task sdk is installed |
Ah you are right :) |
|
But also @amoghrajesh -> we should handle the case where system tests are run on Airflow 2 -> I guess we cannot install task.sdk on 2 and we will never aim to do it ? |
|
@potiuk @o-nikolas I do not think that should be an issue at all. For system tests running on Airflow 2, we use the Most hook imports for amazon systests are from base_aws, and see the import for BaseHook there: https://github.com/apache/airflow/blob/main/providers/amazon/src/airflow/providers/amazon/aws/hooks/base_aws.py#L63 (its from a compat layer) |
3 participants
closes: #52520
During the migration of BaseHook to task sdk, I introduced a temporary workaround in the API server context:
https://github.com/apache/airflow/pull/51873/files#diff-acfa34467241916c1189174e98f87a4094e8cbf58d391fa084e6afb2594decbcR61-R71 to help solve some compat issues which arrived along the way.
When in the API server context, the system uses the Connection model (via custom backend > env var > DB fallback) directly.
This issue was on hold because it was dependent on coupling between Git bundle and API server. For view_url use case, a slim git bundle is instantiated to form the view_url leading to it using BaseHook to get a connection and BaseHook uses Connection from SDK, it goes back to API server causing a self loop scenario.
That has been fixed via: #52876 so attempting the removal again
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.