Note: we want to refactor runtime code in #2926 before implementing this.

With the least-build-containers container placement strategy implemented in #2577, we also want a way to limit the max number of build containers on the workers.

The proposal in #2577 wants to add --max-tasks-per-worker to ATC:

if max-tasks-per-worker is set:

• If the current worker has less than max-tasks-per-worker tasks running, then dispatch to it the new task (and directly or indirectly update the priority queue).
• If on the other hand the current worker has >= than max-tasks-per-worker tasks running, then since the list is a priority queue, all other workers will be in a similar situation, no need to keep traversing the list. Do not dispatch the runnable task and wait for the next event to wake up the scheduler.

If max-tasks-per-worker is not set:

• Take the current worker from the priority queue and dispatch the runnable task to it. This is still better than random placement because it dispatches to the matching worker will less running tasks.

This option requires least-build-containers to be set, otherwise it will error.

Additionally,

Add option max-tasks-for-worker to concourse worker and modify the scheduling algorithm on the ATC to allow to take care of differences between workers, while max-tasks-per-worker on the ATC would still be a maximum cap.

We may want to change the flag names to say build-containers instead of tasks, since the least-build-containers strategy includes get, task and put containers.