The change apache#33364 removed accidentally --builder command that has been
used in CI to build image cache.

This PR restores it and also moves it to the "cache" option group where
it belongs.

The change #33364 removed accidentally --builder command that has been
used in CI to build image cache.

This PR restores it and also moves it to the "cache" option group where
it belongs.

The canary builds continued failing after fixing aibotocore
in #33364 when aiobotocore dependency limit dependency limit. This was
because the main build is now using "empty" eager upgrade requirements
which override the embedded aiobotocore and continue trigger
backtracking.

This is not easily visible on CI because the backtracking
error is under a folded CI image build group in CI.

This PR fixes both:

* only overrides eager requirements if they are not empty
* make sure that eager upgrade output is in the unfolded output
  of CI job

The canary builds continued failing after fixing aibotocore
in #33364 when aiobotocore dependency limit dependency limit. This was
because the main build is now using "empty" eager upgrade requirements
which override the embedded aiobotocore and continue trigger
backtracking.

This is not easily visible on CI because the backtracking
error is under a folded CI image build group in CI.

This PR fixes both:

* only overrides eager requirements if they are not empty
* make sure that eager upgrade output is in the unfolded output
  of CI job

Even the latest pip can enter into a long loop of backtracking
when trying to find latest "good" set of dependencies with
eager upgrade. This happened on August 10th 2023 with aiobotocore
causing backtracking.

This PR adds a complete set of tools and instructions that can
help in such cases and figure out which newly released dependency
causes backtracking.

The toolset consists of:

* adding timeout on the image build, so that it can fail before
  the job timeout and provide useful instructions what to do

* adding `ci find-backtracking-candidates` that allows to identify
  the packages released after the last successful constraint update
  that could be the reason for backtracking

* running the `find-backtracking-candidates` command in the CI
  when timeout occurs - this will help to see the candidates as
  early as possible - at the first build that will fail with
  timeout. This should help with narrowing down the root cause
  much faster

* adding detailed explanation why we have the problem and how to
  deal with it step-by-step, including example based on the
  August 2023 backtracking issue with aiobotocore

* finally removing `--empty-image` switch and pushing empty images
  in CI. This was an attempt to speed up waiting for image in case
  the image failed, but what it did, it has hidden the failures
  of the images when they failed. It does not really add value
  any more, since "image waiting" is now always done using small
  public runners, waiting till timeout for those is not a big issue.

(cherry picked from commit 1cf960d)

The change #33364 removed accidentally --builder command that has been
used in CI to build image cache.

This PR restores it and also moves it to the "cache" option group where
it belongs.

(cherry picked from commit 2ab0d2d)

The canary builds continued failing after fixing aibotocore
in #33364 when aiobotocore dependency limit dependency limit. This was
because the main build is now using "empty" eager upgrade requirements
which override the embedded aiobotocore and continue trigger
backtracking.

This is not easily visible on CI because the backtracking
error is under a folded CI image build group in CI.

This PR fixes both:

* only overrides eager requirements if they are not empty
* make sure that eager upgrade output is in the unfolded output
  of CI job

(cherry picked from commit 3857d33)

Even the latest pip can enter into a long loop of backtracking
when trying to find latest "good" set of dependencies with
eager upgrade. This happened on August 10th 2023 with aiobotocore
causing backtracking.

This PR adds a complete set of tools and instructions that can
help in such cases and figure out which newly released dependency
causes backtracking.

The toolset consists of:

* adding timeout on the image build, so that it can fail before
  the job timeout and provide useful instructions what to do

* adding `ci find-backtracking-candidates` that allows to identify
  the packages released after the last successful constraint update
  that could be the reason for backtracking

* running the `find-backtracking-candidates` command in the CI
  when timeout occurs - this will help to see the candidates as
  early as possible - at the first build that will fail with
  timeout. This should help with narrowing down the root cause
  much faster

* adding detailed explanation why we have the problem and how to
  deal with it step-by-step, including example based on the
  August 2023 backtracking issue with aiobotocore

* finally removing `--empty-image` switch and pushing empty images
  in CI. This was an attempt to speed up waiting for image in case
  the image failed, but what it did, it has hidden the failures
  of the images when they failed. It does not really add value
  any more, since "image waiting" is now always done using small
  public runners, waiting till timeout for those is not a big issue.

The change apache#33364 removed accidentally --builder command that has been
used in CI to build image cache.

This PR restores it and also moves it to the "cache" option group where
it belongs.

The canary builds continued failing after fixing aibotocore
in apache#33364 when aiobotocore dependency limit dependency limit. This was
because the main build is now using "empty" eager upgrade requirements
which override the embedded aiobotocore and continue trigger
backtracking.

This is not easily visible on CI because the backtracking
error is under a folded CI image build group in CI.

This PR fixes both:

* only overrides eager requirements if they are not empty
* make sure that eager upgrade output is in the unfolded output
  of CI job