Apache Airflow version
2.6.1
What happened
A number of Airflow CLI commands will fail with an error like the one below if a DAG uses a custom Timetable:
Traceback (most recent call last):
File "/opt/conda/envs/production-airflow-2.6.1/bin/airflow", line 11, in <module>
sys.exit(main())
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/__main__.py", line 48, in main
args.func(args)
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/cli/cli_config.py", line 51, in command
return func(*args, **kwargs)
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/utils/cli.py", line 112, in wrapper
return f(*args, **kwargs)
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/cli/commands/dag_command.py", line 139, in dag_backfill
_run_dag_backfill(dags, args)
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/cli/commands/dag_command.py", line 79, in _run_dag_backfill
ti.dry_run()
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1730, in dry_run
self.render_templates()
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 2172, in render_templates
context = self.get_template_context()
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/models/taskinstance.py", line 1931, in get_template_context
data_interval = dag.get_run_data_interval(dag_run)
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/models/dag.py", line 861, in get_run_data_interval
return self.infer_automated_data_interval(run.execution_date)
File "/opt/conda/envs/production-airflow-2.6.1/lib/python3.9/site-packages/airflow/models/dag.py", line 882, in infer_automated_data_interval
raise ValueError(f"Not a valid timetable: {self.timetable!r}")
ValueError: Not a valid timetable: <workday.AfterWorkdayTimetable object at 0x7f50f9cd1700>
Some commands I have found that demonstrate this issue (though there may be more that I haven't found):
airflow dags backfill <dag_id> -s <date> -e <date> --dry-run (works fine without --dry-run)airflow dags test <dag_id> [date]airflow tasks render <dag_id> <task_id> <date> (works fine if you pass a valid run_id instead of an execution_date)
What you think should happen instead
Custom Timetables should work in all contexts that built-in timetables do.
How to reproduce
Set up Airflow in a such a way that example plugins can be used (see comments below). Add this DAG script:
#!/usr/bin/env python
from datetime import datetime
from airflow.decorators import dag, task
# copy airflow/example_dags/plugins/workday.py into your $PLUGINS_FOLDER
# OR update airflow.cfg to set core.plugins_folder = /path/to/airflow/example_dags/plugins
from workday import AfterWorkdayTimetable
@dag(
start_date=datetime(2023, 5, 13),
timetable=AfterWorkdayTimetable(),
catchup=True,
)
def test_timetable():
@task()
def t():
print('hello')
t()
test_timetable()
To verify that the timetable is working correctly, you should be able to unpause the DAG and see tasks complete. Then run one of the CLI commands that I mentioned above to see the error.
Operating System
CentOS Stream 8
Versions of Apache Airflow Providers
N/A
Deployment
Other
Deployment details
Self-hosted deployment/standalone mode
Postgres DB backend
Anything else
Possibly related:
Are you willing to submit PR?
Code of Conduct
Apache Airflow version
2.6.1
What happened
A number of Airflow CLI commands will fail with an error like the one below if a DAG uses a custom Timetable:
Some commands I have found that demonstrate this issue (though there may be more that I haven't found):
airflow dags backfill <dag_id> -s <date> -e <date> --dry-run(works fine without--dry-run)airflow dags test <dag_id> [date]airflow tasks render <dag_id> <task_id> <date>(works fine if you pass a validrun_idinstead of anexecution_date)What you think should happen instead
Custom Timetables should work in all contexts that built-in timetables do.
How to reproduce
Set up Airflow in a such a way that example plugins can be used (see comments below). Add this DAG script:
To verify that the timetable is working correctly, you should be able to unpause the DAG and see tasks complete. Then run one of the CLI commands that I mentioned above to see the error.
Operating System
CentOS Stream 8
Versions of Apache Airflow Providers
N/A
Deployment
Other
Deployment details
Self-hosted deployment/
standalonemodePostgres DB backend
Anything else
Possibly related:
Are you willing to submit PR?
Code of Conduct