Added asynchronous aget_hook method to BaseHook#68506
Conversation
8112a3f to
eb3edec
Compare
jscheffl
left a comment
There was a problem hiding this comment.
It is a (very) core change but looks reasonable to be added as async alternative to existing sync API. I also think there is alternative debatable.
I am approving assuning you can master the glitch in static checks, then LGTM in my view.
…async tasks so we don't block the event look when resolving a hook based on conn_id
eb3edec to
a147398
Compare
Adding asynchronous Hook (async aget_connection already exists) like async Task State Management (AIP-103) and async XCom operations is the next logical step since the introduction of the async PtyhonOperator with AIP-98. |
2241de9 to
d73a407
Compare
…an async version of mask_secret
d73a407 to
b006462
Compare
|
Not for this PR specifically, but I’m wondering if we can unify the implementation of sync and async function? Similar to how |
That would be indeed a good idea TP, which would automatically make the code more DRY in general. |
75ce83b to
6a52a6f
Compare
Summary
Adds
BaseHook.aget_hook()— the missing async counterpart ofBaseHook.get_hook()— andamask_secret()— the async counterpart ofmask_secret()— for use in async tasks so we don't block the event loop.Root cause
Async tasks that called
BaseHook.get_hook()from insideasync def/aexecute()wereinadvertently using the sync connection-fetch path:
get_hook()callsget_connection()→Connection.get()→_get_connection(), whichultimately calls
comms.send()— the blocking supervisor round-trip — on the event loopthread. Blocking the event loop thread while other coroutines are concurrently in-flight via
comms.asend()causes a deadlock or severe stall.Even after switching to
aget_hook(), a second deadlock source remained:_async_get_connection()called the sync
mask_secret()which internally usescomms.send(), triggeringDeadlockImminentError.Changes
1.
BaseHook.aget_hook()(task-sdk/src/airflow/sdk/bases/hook.py)Provides a proper async alternative so user code never needs to call the blocking sync path
from inside a coroutine:
aget_hook()delegates to the existingaget_connection(), which already routes throughcomms.asend()correctly.2.
amask_secret()(task-sdk/src/airflow/sdk/log.py)Adds async version of
mask_secret()that usescomms.asend()instead ofcomms.send():3.
_amask_connection_secrets()/_async_get_connection()(task-sdk/src/airflow/sdk/execution_time/context.py)_amask_connection_secrets()— async helper that callsamask_secret()_async_get_connection()to useawait _amask_connection_secrets(conn)instead of the sync_mask_connection_secrets(conn)This ensures the entire async connection-fetch path is fully non-blocking.
Was generative AI tooling used to co-author this PR?
Claude 4.6
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.