Description
expanding tasks in batches to allow mapped tasks spawn more than 1024 processes.
Use case/motivation
Maximum length of a list is limited to 1024 by max_map_length (AIRFLOW__CORE__MAX_MAP_LENGTH).
during scheduling of the new tasks, an UPDATE query is ran that tries to set all the new tasks at once. Increasing max_map_length more than 4K makes airflow scheduler completely unresponsive.
Also, Postgres throws stack depth limit exceeded error which can be fixed by updating to a newer version and setting max_stack_depth higher. But it doesn't really matter because airflow scheduler freezes up.
As a workaround, I split the dag runs into subdag runs which works but it would be much nicer if we didn't have to worry about exceeding max_map_length.
Related issues
It was discussed here:
Increasing 'max_map_length' leads to SQL 'max_stack_depth' error with 5000 dags to be spawned #28478
Are you willing to submit a PR?
Code of Conduct
Description
expanding tasks in batches to allow mapped tasks spawn more than 1024 processes.
Use case/motivation
Maximum length of a list is limited to 1024 by
max_map_length (AIRFLOW__CORE__MAX_MAP_LENGTH).during scheduling of the new tasks, an UPDATE query is ran that tries to set all the new tasks at once. Increasing
max_map_lengthmore than 4K makes airflow scheduler completely unresponsive.Also, Postgres throws
stack depth limit exceedederror which can be fixed by updating to a newer version and settingmax_stack_depthhigher. But it doesn't really matter because airflow scheduler freezes up.As a workaround, I split the dag runs into subdag runs which works but it would be much nicer if we didn't have to worry about exceeding
max_map_length.Related issues
It was discussed here:
Increasing 'max_map_length' leads to SQL 'max_stack_depth' error with 5000 dags to be spawned #28478
Are you willing to submit a PR?
Code of Conduct