Skip to content

Race condition between scheduler processing events and trigger completion #66374

Description

@karenbraganz

Under which category would you file this issue?

Airflow Core

Apache Airflow version

3.2.0

What happened and how to reproduce it?

There is a race condition that can cause deferred tasks to fail. Below I will outline a timeline of events (all from May 2) as an example:

15:30:16.795  worker     TI started (try_number=1)
15:30:19.712  worker     Pausing task as DEFERRED (defer() called, worker exits cleanly)
15:30:21.445  triggerer  Trigger 47218 starting
15:30:21.484  triggerer  Trigger fired → success
15:30:21.485  triggerer  Trigger completed
15:30:22.787  scheduler  Trigger event processed → TI rescheduled
15:30:22.789  scheduler  Celery success event arrives (from defer-exit) *after* trigger event
15:30:22.820  scheduler  TaskInstance finished (state=scheduled, executor_state=success)
15:30:22.822  scheduler  ERROR: state mismatch (scheduled vs success)
15:30:23.386  scheduler  Task marked as FAILED 

This is the error that appears in the audit logs:

Executor CeleryExecutor(parallelism=88) reported that the task instance <TaskInstance: transform_daily_account_balances_chime_finance.sensor_task_ftr_transaction_signal scheduled__2026-05-02T15:30:00+00:00 [scheduled]> finished with state success, but the task instance's state attribute is scheduled. Learn more: https://airflow.apache.org/docs/apache-airflow/stable/troubleshooting.html#task-state-changed-externally

The race condition cannot be reliably reproduced. However, it is related to this known issue where the race condition happens between the pre-deferral task being marked as success and the post-deferral task being in the queued state. The only difference in this case is that the post-deferral task is in the scheduled state, not queued state. This PR was created to fix the known issue. However, it does not account for the post-deferral task being in the scheduled state.

What you think should happen instead?

There should be a fix that checks whether the task is scheduled after deferral (similar to what is done for queued tasks) in order to prevent the state mismatch failure.

Operating System

No response

Deployment

Astronomer

Apache Airflow Provider(s)

No response

Versions of Apache Airflow Providers

No response

Official Helm Chart version

Not Applicable

Kubernetes Version

No response

Helm Chart configuration

No response

Docker Image customizations

No response

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:Schedulerincluding HA (high availability) schedulerarea:Triggererkind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions