Description
For the description below, assume that the DAG:
- is scheduled to run once a day
- has been paused more than one day
- has
max_active_runs=1
- was not running when it was paused (e.g. all tasks are success or failed)
When triggering a paused DAG, it would be nice if the manually triggered run starts before any scheduled run.
If the DAG has parameters, there is a "Unpause DAG when triggered" toggle, but this doesn't help.
This is the behavior currently seen, running with the official Helm chart, using the Celery scheduler:
- With "Unpause DAG when triggered" = true: regularly scheduled run starts running, the manually triggered run can run only after the other one completes (naturally, or because marked success / failure)
- With "Unpause DAG when triggered" = false: only the manually triggered run is scheduled - until the DAG is unpaused, at which time the regularly scheduled run starts running, and the manually triggered run can run only after the other one completes (naturally, or because marked success / failure)
Use case/motivation
We have an Integration environment, but most of the DAGs are usually paused on Integration. Some of the DAGs process a lot of data, and take a long time to run. These DAGs are run on Integration only occasionally, for example when testing changes. These DAGs often have parameters to reduce the amount of data processed (e.g. only process the list of ids (1,2,3), or only process the source data within a given date/time range).
It's an annoyance for us, and counterintuitive, that when we trigger a paused DAG, the manually triggered run, with the parameters we have specified, is not the one that starts running - it's the regularly scheduled DAG run that goes first (obviously, without the parameters that have been specified). We have to then mark the regularly scheduled run as fail / success, before our parameterized manual run starts.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
Description
For the description below, assume that the DAG:
max_active_runs=1When triggering a paused DAG, it would be nice if the manually triggered run starts before any scheduled run.
If the DAG has parameters, there is a "Unpause DAG when triggered" toggle, but this doesn't help.
This is the behavior currently seen, running with the official Helm chart, using the Celery scheduler:
Use case/motivation
We have an Integration environment, but most of the DAGs are usually paused on Integration. Some of the DAGs process a lot of data, and take a long time to run. These DAGs are run on Integration only occasionally, for example when testing changes. These DAGs often have parameters to reduce the amount of data processed (e.g. only process the list of ids (1,2,3), or only process the source data within a given date/time range).
It's an annoyance for us, and counterintuitive, that when we trigger a paused DAG, the manually triggered run, with the parameters we have specified, is not the one that starts running - it's the regularly scheduled DAG run that goes first (obviously, without the parameters that have been specified). We have to then mark the regularly scheduled run as fail / success, before our parameterized manual run starts.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct