Skip to content

Airflow scheduler does not trigger when schedule_interval is @weekly OR (montlhy) "0 3 1 * *". #10824

@ana-carolina-januario

Description

@ana-carolina-januario

Apache Airflow version:1.10.5

Kubernetes version (if you are using kubernetes) (use kubectl version): NA

Environment: CentOS Linux release 7.7.1908 (Core), without Airflow environment defined variables.

  • Cloud provider or hardware configuration: Azure, B4ms standard (4 cpu, 16GiB memory, 30GiB ssd disk)
  • OS (e.g. from /etc/os-release): CentOS Linux release 7.7.1908 (Core)
  • Kernel (e.g. uname -a): Linux prd-azure-ddp-airflow 3.10.0-1062.1.2.el7.x86_64 Improving the search functionality in the graph view #1 SMP Mon Sep 30 14:19:46 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
  • Install tools:
    $ pip list
    Package Version

alembic 1.2.1
apache-airflow 1.10.5
apispec 3.0.0
attrs 19.3.0
azure-common 1.1.25
azure-cosmos 3.1.2
azure-datalake-store 0.0.48
azure-mgmt-containerinstance 1.5.0
azure-mgmt-hdinsight 1.4.0
azure-mgmt-resource 9.0.0
azure-nspkg 3.0.2
azure-storage 0.36.0
azure-storage-blob 2.1.0
azure-storage-common 2.1.0
Babel 2.7.0
boto 2.49.0
boto3 1.12.31
botocore 1.15.31
cached-property 1.5.1
certifi 2019.9.11
cffi 1.13.0
chardet 3.0.4
Click 7.0
colorama 0.4.1
colorlog 4.0.2
configparser 3.5.3
croniter 0.3.30
cryptography 2.8
defusedxml 0.6.0
dill 0.2.9
docutils 0.15.2
dumb-init 1.2.2
elasticsearch 5.5.3
elasticsearch-dsl 5.4.0
Flask 1.1.1
Flask-Admin 1.5.3
Flask-AppBuilder 1.13.1
Flask-Babel 0.12.2
Flask-Caching 1.3.3
Flask-JWT-Extended 3.24.0
Flask-Login 0.4.1
Flask-OpenID 1.2.5
Flask-SQLAlchemy 2.4.1
flask-swagger 0.2.13
Flask-WTF 0.14.2
funcsigs 1.0.0
future 0.16.0
gunicorn 19.9.0
idna 2.8
importlib-metadata 0.23
iso8601 0.1.12
itsdangerous 1.1.0
JayDeBeApi 1.1.1
Jinja2 2.10.3
jmespath 0.9.5
JPype1 0.6.3
json-merge-patch 0.2
jsonschema 3.1.1
lazy-object-proxy 1.4.2
lockfile 0.12.2
Mako 1.1.0
Markdown 2.6.11
MarkupSafe 1.1.1
marshmallow 2.19.5
marshmallow-enum 1.5.1
marshmallow-sqlalchemy 0.19.0
more-itertools 7.2.0
mysql-connector 2.2.9
mysqlclient 1.3.14
ndg-httpsclient 0.5.1
numpy 1.17.3
ordereddict 1.1
pandas 0.25.2
pendulum 1.4.4
pip 20.2
prison 0.1.0
psutil 5.6.3
psycopg2 2.8.4
pyasn1 0.4.7
pycparser 2.19
Pygments 2.4.2
PyJWT 1.7.1
pyOpenSSL 19.0.0
pyrsistent 0.15.4
python-daemon 2.1.2
python-dateutil 2.8.0
python-editor 1.0.4
python3-openid 3.1.0
pytz 2019.3
pytzdata 2019.3
PyYAML 5.1.2
requests 2.22.0
s3transfer 0.3.3
setproctitle 1.1.10
setuptools 41.4.0
six 1.12.0
SQLAlchemy 1.3.10
tabulate 0.8.5
tenacity 4.12.0
termcolor 1.1.0
text-unidecode 1.2
thrift 0.11.0
tzlocal 1.5.1
unicodecsv 0.14.1
urllib3 1.25.6
Werkzeug 0.16.0
wheel 0.33.6
WTForms 2.2.1
zipp 0.6.0
zope.deprecation 4.4.0

  • Others:

What happened: I have a dag scheduled to be triggered weekly using '@Weekly' in dag creation but the dag was never triggered according to this schedule.
Before the weekly schedule, I've also tried to triggered this same DAG monthly with "0 3 1 * *" which didn't work as well.

What you expected to happen: I've expected the referred DAG to be triggered.

How to reproduce it:
This can be reproduced by simply create a couple of DAGs scheduled to be triggered with the referred schedule intervals(weekly and monthly).

The relevant code would be:

default_args = {
'owner': 'daai',
'depends_on_past': False,
'start_date': airflow.utils.dates.days_ago(1),
'retries': 1,
'retry_delay': timedelta(minutes=2),
'provide_context': True}

dag = DAG(dag_name, concurrency=6, schedule_interval="@Weekly", default_args=default_args, max_active_runs=1, dagrun_timeout=timedelta(hours=3))

Anything else we need to know:
I don't thing the content of the DAGs impact this issue but let me know if you think they impact.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:Schedulerincluding HA (high availability) schedulerkind:bugThis is a clearly a bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions