Skip to content

Sub-hour scheduling without catchup result in unaligned data intervals #32346

Description

@AlexisBRENON

Apache Airflow version

2.6.2

What happened

I setup a DAG to run every 15 minutes, with a start date aligned on hour start, to have it scheduled with data interval like:

  • hh:00 - hh:15
  • hh:15 - hh:30
  • hh:30 - hh:45
  • hh:45 - hh+1:00

But if I disable catchup, the resulting intervals are not hour aligned, whatever the start date, but aligned with the time when I unpaused the DAG.

What you think should happen instead

I expect the DAG schedules to still be hour aligned, but just skip any "missed" schedule.

How to reproduce

Create two similar DAGs (with just the catchup argument changing):

from airflow.models.dag import DAG
from airflow.operators.empty import EmptyOperator
import datetime

with DAG(dag_id="with_catch",
         start_date=datetime.datetime(2023, 7, 4, 12, tzinfo=datetime.timezone.utc),
        schedule=datetime.timedelta(minutes=30),
    catchup=True) as dag1:
        EmptyOperator(task_id="task")


with DAG(dag_id="no_catch",
         start_date=datetime.datetime(2023, 7, 4, 12, tzinfo=datetime.timezone.utc),
        schedule=datetime.timedelta(minutes=30),
    catchup=False) as dag1:
        EmptyOperator(task_id="task")

Operating System

Arch Linux

Versions of Apache Airflow Providers

No response

Deployment

Other Docker-based deployment

Deployment details

docker run --rm -it -v /tmp/dags:/opt/airflow/dags -p 8080:8080 --entrypoint airflow apache/airflow:2.6.2 standalone

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions