Rename DAG concurrency settings for easier understanding by kaxil · Pull Request #16267 · apache/airflow

kaxil · 2021-06-04T12:32:33Z

dag_concurrency -> max_active_tasks_per_dag

ToDo:

Add an example scenario where these settings will be useful:
max_active_tasks_per_dag : "To stop a new dag with an early start date from stealing all the executor slots in a cluster?"
Rename DAG.concurrency
Add note in UPDATING.md
Rename BaseOperator.task_concurrency settings too. (Will do this as a separate PR as the change is becoming quite BIG)

Some of Airflow's concurrency settings have been a source of confusion for a lot of users (including me), for example:

This PR is an attempt to make the settings easier to understand

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

ryanahamilton

I appreciate how I very clearly know what this means by just reading the name.

ephraimbuddy

This makes a lot of sense now! Thanks Kaxil!

Just some REST API test fix. LGTM

github-actions · 2021-06-16T14:06:03Z

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

``dag_conccurency`` -> ``max_active_tasks_per_dag``

potiuk · 2022-08-03T10:55:28Z

Very clear name max_active_tasks_per_dag became obscured max_active_tis_per_dag somehow. How did it happen? This "tis" is not clear at all disappointed

Your praised very simple max_active_tasks_per_dag was simply wrong name. There is a difference between task and task instance. The tis is often used all over the code for task_instance and this is right. max_activet_tasks_instances_per_dag would be simply too long.

It's all perfectly logical and accurate. As opposed to previous name which was wrong and illogical.

jbmeerkat · 2022-08-03T12:04:50Z

Your praised very simple max_active_tasks_per_dag was simply wrong name. There is a difference between task and task instance. The tis is often used all over the code for task_instance and this is right. max_activet_tasks_instances_per_dag would be simply too long.

It's all perfectly logical and accurate. As opposed to previous name which was wrong and illogical.

Jarec thank you very much for clarification. Now I see why that changed!

To be more clear I'm looking not from an Airflow developer perspective but from an Airflow user perspective, so I don't know how it's named in the code. I just updated my Airflow instance and started to see a deprecation warnings about this "tis" 😄

I'm not trying to dispute that name, just telling about my experience.

potiuk · 2022-08-03T12:40:41Z

I'm not trying to dispute that name, just telling about my experience.

And I am just answering your question - very precisely and accurately - "How did it happen?".

jbmeerkat · 2022-08-03T12:53:17Z

I'm not trying to dispute that name, just telling about my experience.

And I am just answering your question - very precisely and accurately - "How did it happen?".

Thank you for that!