Fix deferrable sensors not respecting soft_fail on timeout#61132
Merged
Conversation
When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
nathadfield
force-pushed
the
fix-deferrable-sensor-soft-fail-timeout
branch
from
0ece79a to
ef1aa5e
Compare
Contributor
Author
|
Hey folks! If any of you get the time to review this PR soon I'd be most appreciative. Many thanks! |
kaxil
approved these changes
Feb 4, 2026
github-actions Bot
pushed a commit
that referenced
this pull request
Feb 4, 2026
#61132) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: #61130 (cherry picked from commit cec8ba6) Co-authored-by: Nathan Hadfield <[email protected]> Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Contributor
Backport successfully created: v3-1-test
|
github-actions Bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Feb 4, 2026
apache#61132) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 (cherry picked from commit cec8ba6) Co-authored-by: Nathan Hadfield <[email protected]> Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Alok-kumar-priyadarshi
pushed a commit
to Alok-kumar-priyadarshi/airflow
that referenced
this pull request
Feb 5, 2026
) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
jhgoebbert
pushed a commit
to jhgoebbert/airflow_Owen-CH-Leung
that referenced
this pull request
Feb 8, 2026
) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Ratasa143
pushed a commit
to Ratasa143/airflow
that referenced
this pull request
Feb 15, 2026
) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
potiuk
pushed a commit
that referenced
this pull request
Feb 18, 2026
#61132) (#61421) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: #61130 (cherry picked from commit cec8ba6) Co-authored-by: Nathan Hadfield <[email protected]> Co-authored-by: Claude Sonnet 4.5 <[email protected]>
choo121600
pushed a commit
to choo121600/airflow
that referenced
this pull request
Feb 22, 2026
) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
Subham-KRLX
pushed a commit
to Subham-KRLX/airflow
that referenced
this pull request
Mar 4, 2026
) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
vatsrahul1001
pushed a commit
that referenced
this pull request
Mar 4, 2026
#61132) (#61421) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: #61130 (cherry picked from commit cec8ba6) Co-authored-by: Nathan Hadfield <[email protected]> Co-authored-by: Claude Sonnet 4.5 <[email protected]>
Ankurdeewan
pushed a commit
to Ankurdeewan/airflow
that referenced
this pull request
Mar 15, 2026
) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
radhwene
pushed a commit
to radhwene/airflow
that referenced
this pull request
Mar 21, 2026
) When a deferrable sensor with soft_fail=True times out, the task fails with AirflowSensorTimeout instead of being marked as SKIPPED. This is a regression from Airflow 2.x behavior. The issue was in resume_execution() where TaskDeferralTimeout was converted to AirflowSensorTimeout before checking soft_fail. This fix uses nested exception handling to check soft_fail and never_fail before the conversion, ensuring timeouts are properly skipped. closes: apache#61130 Co-Authored-By: Claude Sonnet 4.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #61130
Description
Fixes a regression in Airflow 3.x where deferrable sensors with
soft_fail=Truefail withAirflowSensorTimeoutinstead of being marked as SKIPPED when they timeout. This breaks compatibility with Airflow 2.x behavior and user expectations.Root Cause
In
BaseSensorOperator.resume_execution(), the exception handling order was incorrect:TaskDeferralTimeoutwas caught and immediately converted toAirflowSensorTimeoutsoft_fail/never_failchecks only caughtAirflowExceptionandTaskDeferralErrorSolution
TaskDeferralTimeout→AirflowSensorTimeoutwhile ensuring both exceptions are caught by the outer handlersoft_failandnever_failflags before raisingAirflowSensorTimeoutexecute()method for consistencyChanges
task-sdk/src/airflow/sdk/bases/sensor.py:resume_execution()to use nested exception handlingsoft_failandnever_failchecks with explicit error messagestask-sdk/tests/task_sdk/bases/test_sensor.py:soft_fail=True/Falsenever_fail=Truesoft_fail=True/Falsenever_fail=TrueTriggerFailureReasonenum valuesWas generative AI tooling used to co-author this PR?
Generated-by: Claude Code following the guidelines