nspkg dependency issues during pip install with python 3 #3447
Description
We have been experiencing very strange behaviour which I believe is related to #3412
some context.
We have a local pypi proxy in nexus, the weird behaviour seems to be limited to when we point to this private pypi.
Behaviour with Python 3.5
Nexus pypi
<redacted>@<redacted> ✗ pip install azure-storage
Looking in indexes: https://nexus.<redacted>/repository/pypi-all/simple
Collecting azure-storage
Using cached https://nexus.<redacted>/repository/pypi-all/packages/76/26/482c033f8f3a48d16cf75aad91c3e1256856719f4117fabb4696d33aa5da/azure_storage-0.36.0-py2.py3-none-any.whl
Collecting cryptography (from azure-storage)
Using cached https://nexus.<redacted>/repository/pypi-all/packages/59/32/92cade62c645756a83598edf56289e9b19aae5370642a7ce690cd06bc72f/cryptography-2.3.1-cp34-abi3-manylinux1_x86_64.whl
Collecting requests (from azure-storage)
Using cached https://nexus.<redacted>/repository/pypi-all/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl
Collecting azure-nspkg>=2.0.0 (from azure-storage)
Using cached https://nexus.<redacted>/repository/pypi-all/packages/1e/a4/84eb1251c2f3d2392bae60c7999f070ce1c68412880dfc80e644a854354e/azure_nspkg-3.0.0-py2.py3-none-any.whl
azure-nspkg requires Python '<3' but the running Python is 3.5.6
for some reason an attempt to pull azure-nspkg==3.0.0 happens even though we're running python 3. The install fails.
Public pypi
<redacted>@<redacted> ✓ pip install azure-storage
Collecting azure-storage
Using cached https://files.pythonhosted.org/packages/76/26/482c033f8f3a48d16cf75aad91c3e1256856719f4117fabb4696d33aa5da/azure_storage-0.36.0-py2.py3-none-any.whl
Collecting requests (from azure-storage)
Using cached https://files.pythonhosted.org/packages/65/47/7e02164a2a3db50ed6d8a6ab1d6d60b69c4c3fdf57a284257925dfc12bda/requests-2.19.1-py2.py3-none-any.whl
Collecting azure-nspkg>=2.0.0 (from azure-storage)
Using cached https://files.pythonhosted.org/packages/cd/a0/76ca6659ae9afd7567fdbb5b9c85e9c9a0b48238cfcacd92525484408f66/azure_nspkg-2.0.0-py2.py3-none-any.whl
Collecting azure-common>=1.1.5 (from azure-storage)
Using cached https://files.pythonhosted.org/packages/ac/d3/055ce7ad06459a415ff9ca210e04c6cbb51bd6564815b7c8ac34bf5a1c39/azure_common-1.1.16-py2.py3-none-any.whl
Collecting python-dateutil (from azure-storage)
Using cached https://files.pythonhosted.org/packages/cf/f5/af2b09c957ace60dcfac112b669c45c8c97e32f94aa8b56da4c6d1682825/python_dateutil-2.7.3-py2.py3-none-any.whl
Collecting cryptography (from azure-storage)
Using cached https://files.pythonhosted.org/packages/59/32/92cade62c645756a83598edf56289e9b19aae5370642a7ce690cd06bc72f/cryptography-2.3.1-cp34-abi3-manylinux1_x86_64.whl
Collecting urllib3<1.24,>=1.21.1 (from requests->azure-storage)
Using cached https://files.pythonhosted.org/packages/bd/c9/6fdd990019071a4a32a5e7cb78a1d92c53851ef4f56f62a3486e6a7d8ffb/urllib3-1.23-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->azure-storage)
Using cached https://files.pythonhosted.org/packages/df/f7/04fee6ac349e915b82171f8e23cee63644d83663b34c539f7a09aed18f9e/certifi-2018.8.24-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->azure-storage)
Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.8,>=2.5 (from requests->azure-storage)
Using cached https://files.pythonhosted.org/packages/4b/2a/0276479a4b3caeb8a8c1af2f8e4355746a97fab05a372e4a2c6a6b876165/idna-2.7-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil->azure-storage)
Using cached https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Collecting asn1crypto>=0.21.0 (from cryptography->azure-storage)
Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.7 (from cryptography->azure-storage)
Using cached https://files.pythonhosted.org/packages/59/cc/0e1635b4951021ef35f5c92b32c865ae605fac2a19d724fb6ff99d745c81/cffi-1.11.5-cp35-cp35m-manylinux1_x86_64.whl
Collecting pycparser (from cffi!=1.11.3,>=1.7->cryptography->azure-storage)
Installing collected packages: urllib3, certifi, chardet, idna, requests, azure-nspkg, azure-common, six, python-dateutil, asn1crypto, pycparser, cffi, cryptography, azure-storage
Successfully installed asn1crypto-0.24.0 azure-common-1.1.16 azure-nspkg-2.0.0 azure-storage-0.36.0 certifi-2018.8.24 cffi-1.11.5 chardet-3.0.4 cryptography-2.3.1 idna-2.7 pycparser-2.19 python-dateutil-2.7.3 re
quests-2.19.1 six-1.11.0 urllib3-1.23
azure_nspkg==2.0.0 is collected. The install works.
In #3412 the *_nspkg modules are added as requires_extra with a python<3 tag so I'm confused as to why they are being fetched at all during a pip install on python 3.
Current fix
Currently I have explicitly installed all required *_nspkg modules pinned at version 2.0.0 in the my respective setup.py files as a work around.
Help!
- I have tried removing
azure_nspkg==3.0.0from nexus but it keeps coming back - I've tried invalidating the nexus cache
- I've tried reindexing the nexus pypi repo
This was all working until 2 days ago which coincides with the changes introduced during #3412.