Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.0
What happened and how to reproduce it?
There is a race condition that can cause deferred tasks to fail. Below I will outline a timeline of events (all from May 2) as an example:
15:30:16.795 worker TI started (try_number=1)
15:30:19.712 worker Pausing task as DEFERRED (defer() called, worker exits cleanly)
15:30:21.445 triggerer Trigger 47218 starting
15:30:21.484 triggerer Trigger fired → success
15:30:21.485 triggerer Trigger completed
15:30:22.787 scheduler Trigger event processed → TI rescheduled
15:30:22.789 scheduler Celery success event arrives (from defer-exit) *after* trigger event
15:30:22.820 scheduler TaskInstance finished (state=scheduled, executor_state=success)
15:30:22.822 scheduler ERROR: state mismatch (scheduled vs success)
15:30:23.386 scheduler Task marked as FAILED
This is the error that appears in the audit logs:
Executor CeleryExecutor(parallelism=88) reported that the task instance <TaskInstance: transform_daily_account_balances_chime_finance.sensor_task_ftr_transaction_signal scheduled__2026-05-02T15:30:00+00:00 [scheduled]> finished with state success, but the task instance's state attribute is scheduled. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally
The race condition cannot be reliably reproduced. However, it is related to this known issue where the race condition happens between the pre-deferral task being marked as success and the post-deferral task being in the queued state. The only difference in this case is that the post-deferral task is in the scheduled state, not queued state. This PR was created to fix the known issue. However, it does not account for the post-deferral task being in the scheduled state.
What you think should happen instead?
There should be a fix that checks whether the task is scheduled after deferral (similar to what is done for queued tasks) in order to prevent the state mismatch failure.
Operating System
No response
Deployment
Astronomer
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
Under which category would you file this issue?
Airflow Core
Apache Airflow version
3.2.0
What happened and how to reproduce it?
There is a race condition that can cause deferred tasks to fail. Below I will outline a timeline of events (all from May 2) as an example:
This is the error that appears in the audit logs:
The race condition cannot be reliably reproduced. However, it is related to this known issue where the race condition happens between the pre-deferral task being marked as success and the post-deferral task being in the queued state. The only difference in this case is that the post-deferral task is in the scheduled state, not queued state. This PR was created to fix the known issue. However, it does not account for the post-deferral task being in the scheduled state.
What you think should happen instead?
There should be a fix that checks whether the task is scheduled after deferral (similar to what is done for queued tasks) in order to prevent the state mismatch failure.
Operating System
No response
Deployment
Astronomer
Apache Airflow Provider(s)
No response
Versions of Apache Airflow Providers
No response
Official Helm Chart version
Not Applicable
Kubernetes Version
No response
Helm Chart configuration
No response
Docker Image customizations
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct