Unnecessary downloading of dependencies when they are constrained #9232
Description
When we try to install 1.10.14rc1 release of Airflow with fully consistent set of requirements, At least google-cloud-bigquery and oauthlib libraries are needlessly downloaded (all versions) despite being properly constrained in https://github.com/apache/airflow/blob/constraints-1.10.14/constraints-3.6.txt
If you look to the file you will see:
google-cloud-bigquery==2.5.0oauthlib==3.1.0
Running it with old resolver or pip 20.2.4 works perfectly fine:
pyenv virtualenv 3.6.12 test-new-resolver4
pyenv activate test-new-resolver4
pip --upgrade pip=20.2.4
pip install "apache-airflow[devel_ci]==1.10.14rc1" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt"
When I run pip check I got:
[jarek:~/code/airflow] [test-new-resolver4] add-per-provider-javascript(+0/-0,+12/-2)+ 1m8s ± pip check
No broken requirements found.
Then it fails with 20.3:
pip install --upgrade pip
pip --version
pip 20.3.1 from /home/jarek/.pyenv/versions/3.6.12/envs/test-new-resolver4/lib/python3.6/site-packages/pip (python 3.6)
pip install "apache-airflow[devel_ci]==1.10.14rc1" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt"
What we get (despite having both google-cloud-bigquery==2.5.0:
Collecting google-cloud-bigquery[bqstorage,pandas]<3.0.0dev,>=1.11.1
Using cached google_cloud_bigquery-2.4.0-py2.py3-none-any.whl (210 kB)
Using cached google_cloud_bigquery-2.3.1-py2.py3-none-any.whl (208 kB)
Using cached google_cloud_bigquery-2.2.0-py2.py3-none-any.whl (200 kB)
Using cached google_cloud_bigquery-2.1.0-py2.py3-none-any.whl (190 kB)
Using cached google_cloud_bigquery-2.0.0-py2.py3-none-any.whl (193 kB)
Using cached google_cloud_bigquery-1.28.0-py2.py3-none-any.whl (187 kB)
Using cached google_cloud_bigquery-1.27.2-py2.py3-none-any.whl (172 kB)
Using cached google_cloud_bigquery-1.26.1-py2.py3-none-any.whl (170 kB)
Using cached google_cloud_bigquery-1.26.0-py2.py3-none-any.whl (170 kB)
Using cached google_cloud_bigquery-1.25.0-py2.py3-none-any.whl (169 kB)
Using cached google_cloud_bigquery-1.24.0-py2.py3-none-any.whl (165 kB)
Using cached google_cloud_bigquery-1.23.1-py2.py3-none-any.whl (162 kB)
Using cached google_cloud_bigquery-1.23.0-py2.py3-none-any.whl (162 kB)
Using cached google_cloud_bigquery-1.22.0-py2.py3-none-any.whl (161 kB)
Using cached google_cloud_bigquery-1.21.0-py2.py3-none-any.whl (158 kB)
Using cached google_cloud_bigquery-1.20.0-py2.py3-none-any.whl (154 kB)
Using cached google_cloud_bigquery-1.19.1-py2.py3-none-any.whl (149 kB)
Using cached google_cloud_bigquery-1.19.0-py2.py3-none-any.whl (149 kB)
Using cached google_cloud_bigquery-1.18.1-py2.py3-none-any.whl (146 kB)
Using cached google_cloud_bigquery-1.18.0-py2.py3-none-any.whl (146 kB)
Using cached google_cloud_bigquery-1.17.1-py2.py3-none-any.whl (142 kB)
Using cached google_cloud_bigquery-1.17.0-py2.py3-none-any.whl (140 kB)
Using cached google_cloud_bigquery-1.16.1-py2.py3-none-any.whl (140 kB)
Using cached google_cloud_bigquery-1.16.0-py2.py3-none-any.whl (139 kB)
Using cached google_cloud_bigquery-1.15.1-py2.py3-none-any.whl (137 kB)
Using cached google_cloud_bigquery-1.15.0-py2.py3-none-any.whl (133 kB)
Using cached google_cloud_bigquery-1.14.1-py2.py3-none-any.whl (137 kB)
Using cached google_cloud_bigquery-1.14.0-py2.py3-none-any.whl (132 kB)
Using cached google_cloud_bigquery-1.13.1-py2.py3-none-any.whl (137 kB)
Using cached google_cloud_bigquery-1.13.0-py2.py3-none-any.whl (132 kB)
Using cached google_cloud_bigquery-1.12.2-py2.py3-none-any.whl (137 kB)
Using cached google_cloud_bigquery-1.12.1-py2.py3-none-any.whl (130 kB)
Using cached google_cloud_bigquery-1.12.0-py2.py3-none-any.whl (130 kB)
Using cached google_cloud_bigquery-1.11.4-py2.py3-none-any.whl (134 kB)
Using cached google_cloud_bigquery-1.11.3-py2.py3-none-any.whl (127 kB)
Using cached google_cloud_bigquery-1.11.2-py2.py3-none-any.whl (127 kB)
Using cached google_cloud_bigquery-1.11.1-py2.py3-none-any.whl (127 kB)
And then despite Having oauthlib==3.1.0 in the constrains, we get:
Collecting oauthlib[signedtoken]>=1.0.0
Using cached oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
Using cached oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
Using cached oauthlib-3.0.0-py2.py3-none-any.whl (142 kB)
Using cached oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
Using cached oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
Using cached oauthlib-2.0.6.tar.gz (127 kB)
Using cached oauthlib-2.0.5.tar.gz (129 kB)
Using cached oauthlib-2.0.4.tar.gz (127 kB)
Using cached oauthlib-2.0.3.tar.gz (127 kB)
Using cached oauthlib-2.0.2.tar.gz (125 kB)
Using cached oauthlib-2.0.1.tar.gz (122 kB)
Using cached oauthlib-2.0.0.tar.gz (122 kB)
Using cached oauthlib-1.1.2.tar.gz (111 kB)
Using cached oauthlib-1.1.1.tar.gz (108 kB)
Downloading oauthlib-1.0.3.tar.gz (109 kB)
|████████████████████████████████| 109 kB 2.1 MB/s
Downloading oauthlib-1.0.2.tar.gz (108 kB)
|████████████████████████████████| 108 kB 4.7 MB/s
Downloading oauthlib-1.0.1.tar.gz (108 kB)
|████████████████████████████████| 108 kB 4.7 MB/s
Downloading oauthlib-1.0.0.tar.gz (108 kB)
|████████████████████████████████| 108 kB 4.1 MB/s
Using cached oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[devel_ci]==1.10.14rc1) has different version in metadata: '1.1.0'
And installation crashes here. This crash was originally raised in #9203 and it is likely workarounded (not solved yet - this is only possible because we asked oauthlib maintainers to yank the release and they kindly agreed) in #9226 .
I would love to know when we can expect fix to that.