Add an option to load the dags from db for command tasks run#32038
Conversation
Signed-off-by: Hussein Awala <[email protected]>
vandonr-amz
left a comment
vandonr-amz
left a comment
There was a problem hiding this comment.
don't you think making reading from the DB the default and having a flag to force a full re-parse would make more sense ?
Why make the slow command the default when the fast one is going to be what the user wants 99% of the time ?
Also, it'd probably more likely occur to me to look in the doc for an option to force a refresh than for an option that does not refresh.
|
Agree with @vandonr-amz on making the default option as optimized version and then letting customers decide if they want to refresh the DAGs. |
It will be a breaking change for those that uses |
|
@hussein-awala , even though you linked the issue you are working on in this PR, it's necessary to provide a good commit message for the PR |
@ephraimbuddy I agree, but I think we should we can add a new Airflow configuration to specify the default value for this parameter. wdyt? DEFAULT_DAGS_SOURCE_FOR_TASKS_RUN = "dags_folder"
# or
DEFAULT_DAGS_SOURCE_FOR_TASKS_RUN = "metadata_db"By implementing this approach, the worker will be capable of executing the task without requiring the parsing of the DAG file. Consequently, it will be possible to run the task without including them in the worker container. (testing is necessary to ensure its effectiveness) |
👍 |
…32038) Signed-off-by: Hussein Awala <[email protected]>
|
Reading from SerializedModel was originally added as part of AIP-45 in https://github.com/apache/airflow/pull/21877/files#diff-ad618185a072910e49c11770954af009d1cc070b120a4fde5f2fc095a588271bR360-R363 . It was changed in #26750 . Not reading from database was intentional as @ephraimbuddy mentioned. I am not sure if I am misunderstanding something where serialized models where changed after 2.4 to allow this. cc: @pingzh |
7 participants
closes: #32020
This PR adds a new param
--read-from-dbtotasks runcommand, when it is provided, the CLI will try to load the dag from the Metadata instead of parsing the dags files.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.