Backfill job marked as completed immediately, though backfill dag runs still are executing

Apache Airflow version

3.1.6

If "Other Airflow 3 version" selected, which one?

3.1.6

What happened?

I have been running backfill commands multiple times for a specific DAG recently. Initially, it worked as expected; however, at some point, the behavior changed. Now, whenever I execute the backfill command, the process transitions to a "completed" status almost instantaneously.

Interestingly, while the backfill job itself marks itself as finished immediately, the actual DAG runs triggered by that backfill appear to be running and processing normally in the background.

What you think should happen instead?

No response

How to reproduce

The schedule information for this DAG is as follows:

schedule=CronTriggerTimetable('0 6 * * *', timezone='Asia/Seoul') # +09:00

The backfill job was executed using the following CLI command:

airflow backfill create \
    --dag-id etl_vitess_load_v2 \
    --from-date 2023-11-20 \
    --to-date 2024-06-10 \
    --max-active-runs 1 \
    --run-backwards

Operating System

Ubuntu 24.04.3 LTS

Versions of Apache Airflow Providers

From airflow info

apache-airflow-providers-amazon           | 9.19.0
apache-airflow-providers-celery           | 3.15.0
apache-airflow-providers-cncf-kubernetes  | 10.12.0
apache-airflow-providers-common-compat    | 1.11.0
apache-airflow-providers-common-io        | 1.7.0
apache-airflow-providers-common-messaging | 2.0.1
apache-airflow-providers-common-sql       | 1.30.2
apache-airflow-providers-edge3            | 2.0.1
apache-airflow-providers-fab              | 3.1.1
apache-airflow-providers-http             | 5.6.2
apache-airflow-providers-mysql            | 6.4.0
apache-airflow-providers-postgres         | 6.5.1
apache-airflow-providers-redis            | 4.4.1
apache-airflow-providers-smtp             | 2.4.1
apache-airflow-providers-standard         | 1.10.2

Deployment

Official Apache Airflow Helm Chart

Deployment details

normally deployed into k8s (1.23) using official helm chart.

meta-database postgresql 16.x also deployed in the same k8s cluster with ceph pv

Anything else?

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

I would like to work on this could you please assign it to me?

In the current implementation, it appears that the Backfill record and its associated DagRuns may be committed in separate database transactions or phases. Specifically, the Backfill record is persisted first to ensure the request is registered.

However, the logic used by the scheduler to monitor and mark a backfill as completed seems to rely on the absence of "unfinished" dag runs associated with that backfill.

Since the query for unfinished DagRuns returns zero at that precise moment, the system might incorrectly mark the backfill as Completed before any actual work has been queued.

if some synchronization guards already in place, let me know.

Reproducible on Airflow 3.1.7 as well

Reproducible on Airflow 3.1.7 as well

I wasn't able to reproduce the issue in 3.0.6. Seems to be working properly

@roykoand

@roykoand how are you reproducing?

I was able to reproduce the issue. WIll submit a potential fix soon.

There is a race condition in creating the backfill entry, and creating the dag runs for the backfill. Scheduler assumes a backfill is complete before the dags are created.

This unit test replicates the problem

def test_mark_backfills_complete_race_condition(dag_maker, session):
    """
    Simulate the window where a Backfill row exists but DagRuns
    have not been created yet. _mark_backfills_complete() should
    NOT mark it as completed.
    """
    with dag_maker(serialized=True, dag_id="test_race", schedule="@daily"):
        BashOperator(task_id="hi", bash_command="echo hi")

    # Phase 1 only: insert a Backfill row with no DagRuns
    b = Backfill(
        dag_id="test_race",
        from_date=pendulum.parse("2021-01-01"),
        to_date=pendulum.parse("2021-01-03"),
        max_active_runs=10,
        dag_run_conf={},
        reprocess_behavior=ReprocessBehavior.NONE,
    )
    session.add(b)
    session.commit()

    runner = SchedulerJobRunner(
        job=Job(job_type=SchedulerJobRunner.job_type),
        executors=[MockExecutor(do_update=False)],
    )
    runner._mark_backfills_complete()

    session.refresh(b)
    # BUG: with current code, b.completed_at is NOT None here
    assert b.completed_at is None, "Backfill was marked complete before DagRuns were created"

To fix the issue: Backfill gets created first and then the associated dag runs. If the scheduler checks for backfill completion before the dag runs are created the scheduler marks the backfill as completed We can simply check if there is at least one dag run linked to the backfill before we check all the linked dags are completed. I am going to do some local testing and submit a PR i couple of hours

Confirmed that version 3.2 is also affected.

Fix backfill marked complete before DagRuns are created (apache#61375)

@shivaam

@shivaam We are unassigning you from this issue as part of our updated assignment policy.

This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.

We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.

Fix backfill marked complete before DagRuns are created (apache#61375)

Navigation Menu

Repository navigation

Backfill job marked as completed immediately, though backfill dag runs still are executing #61375

Apache Airflow version

If "Other Airflow 3 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Backfill job marked as completed immediately, though backfill dag runs still are executing #61375

Description

Apache Airflow version

If "Other Airflow 3 version" selected, which one?

What happened?

What you think should happen instead?

How to reproduce

Operating System

Versions of Apache Airflow Providers

Deployment

Deployment details

Anything else?

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on Feb 3, 2026

SakshamSinghal20 commented on Feb 3, 2026

assurah commented on Feb 4, 2026

roykoand commented on Feb 18, 2026

shivaam commented on Feb 27, 2026

shivaam commented on Feb 27, 2026

shivaam commented on Feb 27, 2026

shivaam commented on Feb 27, 2026

shivaam commented on Feb 27, 2026

shivaam commented on Feb 27, 2026

potiuk commented on Mar 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions