[WinError 5] Access is denied during cleanup/rmtree #7280
Description
Environment
- pip version: 19.2.3+
- Python version: python 3.7
- OS: Windows 10
- AntiVirus: CylancePROTECT
Description
pip install or download fails on Windows during cleanup because (real-time) virus protection is still scanning files that pip is trying to remove.
using Process Monitor (procmon), I was able to confirm that CylanceSvc.exe (virus scanner) was reading the files while python was trying to the same files.
I tried both increasing the retry wait (in misc.py) and ignoring the rmtree error (in temp_dir.py). both approaches worked.
This issue indicates that other anti-virus scanners cause similar problems:
aws/aws-cli#2654
Expected behavior
The package should install.
How to Reproduce
pip install or download a large package. I was usually able to reproduce when installing or downloading awscli. If I have difficulty reproducing, I can lower the priority of the Cylance.exe to Background (4), and the symptoms are easier to reproduce consistently.
using install:
pip install awscli
using download:
pip download awscli
pip download is better because it provides a stacktrace.
Output
using pip install, the error message is quite terse.
>pip install awscli
Collecting awscli
Using cached https://files.pythonhosted.org/packages/f2/f5/d682aa32100edc6908a784630aa2903ef7d2735130e6df98a05af6c33096/awscli-1.16.271-py2.py3-none-any.whl
ERROR: Could not install packages due to an EnvironmentError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-5e4a1ye3\\awscli-1.16.271-py2.py3-none-any.whl'
Consider using the `--user` option or check the permissions.
. . .
using pip download, the stacktrace is more helpful.
>pip download awscli
Collecting awscli
Using cached https://files.pythonhosted.org/packages/f2/f5/d682aa32100edc6908a784630aa2903ef7d2735130e6df98a05af6c33096/awscli-1.16.271-py2.py3-none-any.whl
Saved c:\users\rich9002\tmp\awscli-1.16.271-py2.py3-none-any.whl
ERROR: Exception:
Traceback (most recent call last):
File "c:\sw\python37\lib\shutil.py", line 398, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-4gdrskd3\\awscli-1.16.271-py2.py3-none-any.whl'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\sw\python37\lib\site-packages\pip\_internal\cli\base_command.py", line 188, in main
status = self.run(options, args)
File "c:\sw\python37\lib\site-packages\pip\_internal\commands\download.py", line 156, in run
resolver.resolve(requirement_set)
File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 196, in resolve
self._resolve_one(requirement_set, req)
File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 359, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "c:\sw\python37\lib\site-packages\pip\_internal\legacy_resolve.py", line 307, in _get_abstract_dist_for
self.require_hashes
File "c:\sw\python37\lib\site-packages\pip\_internal\operations\prepare.py", line 199, in prepare_linked_requirement
progress_bar=self.progress_bar
File "c:\sw\python37\lib\site-packages\pip\_internal\download.py", line 1064, in unpack_url
progress_bar=progress_bar
File "c:\sw\python37\lib\site-packages\pip\_internal\download.py", line 935, in unpack_http_url
os.unlink(from_path)
File "c:\sw\python37\lib\site-packages\pip\_internal\utils\temp_dir.py", line 60, in __exit__
self.cleanup()
File "c:\sw\python37\lib\site-packages\pip\_internal\utils\temp_dir.py", line 83, in cleanup
rmtree(self.path)
File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 49, in wrapped_f
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 212, in call
raise attempt.get()
File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 247, in get
six.reraise(self.value[0], self.value[1], self.value[2])
File "c:\sw\python37\lib\site-packages\pip\_vendor\six.py", line 693, in reraise
raise value
File "c:\sw\python37\lib\site-packages\pip\_vendor\retrying.py", line 200, in call
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "c:\sw\python37\lib\site-packages\pip\_internal\utils\misc.py", line 166, in rmtree
onerror=rmtree_errorhandler)
File "c:\sw\python37\lib\shutil.py", line 516, in rmtree
return _rmtree_unsafe(path, onerror)
File "c:\sw\python37\lib\shutil.py", line 400, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "c:\sw\python37\lib\site-packages\pip\_internal\utils\misc.py", line 176, in rmtree_errorhandler
os.chmod(path, stat.S_IWRITE)
PermissionError: [WinError 5] Access is denied: 'c:\\tmp\\pip\\pip-unpack-4gdrskd3\\awscli-1.16.271-py2.py3-none-any.whl'