retry delete directory while shutil.rmtree raises "OSError: [Errno 39] Directory not empty" exception

ISSUE TYPE

Bug Report

COMPONENT NAME

file module

ANSIBLE VERSION

currently i am using ansible 2.2 ,python 2.7, but i think it exists in the latest version, because file module fails when shutil.rmtree throws an exception

ansible/lib/ansible/modules/files/file.py

Lines 267 to 282 in 571f878

    
           if state == 'absent': 
        
               if state_change: 
        
                   if not module.check_mode: 
        
                       if prev_state == 'directory': 
        
                           try: 
        
                               shutil.rmtree(b_path, ignore_errors=False) 
        
                           except Exception as e: 
        
                               module.fail_json(msg="rmtree failed: %s" % to_native(e)) 
        
                       else: 
        
                           try: 
        
                               os.unlink(b_path) 
        
                           except Exception as e: 
        
                               module.fail_json(path=path, msg="unlinking failed: %s " % to_native(e)) 
        
                   module.exit_json(path=path, changed=True, diff=diff) 
        
               else: 
        
                   module.exit_json(path=path, changed=False)

OS / ENVIRONMENT

N/A

SUMMARY

when i use file module ( state=absent )to delete a non-empty directory, ansible throws :

failed:

[20.20.20.22] (item=/usr/libexec/kubernetes) => {"failed": true, "item": "/usr/libexec/kubernetes", "msg": >"rmtree failed: [Errno 39] Directory not empty: '/usr/libexec/kubernetes/kubelet-plugins/volume'"}

related reports:
https://groups.google.com/forum/#!topic/ansible-project/XrKPpnOxD0I

STEPS TO REPRODUCE

this is randomly reproducible, our codes works well for over a year, now we meet one failure.

it seems to be a bug in shutil.rmtree, easybuild workaround this bug by retry invoking shutil.rmtree
https://github.com/easybuilders/easybuild-framework/blob/master/easybuild/tools/filetools.py#L1185

hashdist also workarounds this bug by retry invoking shutil.rmtree
https://github.com/hashdist/hashdist/blob/master/hashdist/core/fileutils.py#L69-L89

Files identified in the description:

lib/ansible/modules/files/file.py

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

I could not reproduce this OSError.
It works well as follows.

$ mkdir -p /tmp/xxx/yyy/zzz
$ touch /tmp/xxx/aaa.txt
$ ansible localhost -m file -a 'path=/tmp/xxx state=absent'
$ find /tmp/xxx
find: /tmp/xxx: No such file or directory

i think this bug can be reproduced with these steps:

shutil.rmtree found A is a non-empty directory
shutil.rmtree try to remove all contents recursively
shutil.rmtree has finished remote all contents recursively
another process produces some file / directory in A
shutil try to remove A itself and failed

Can confirm this happens if the directory is active and updated by multiple processes, likely leading to a race condition where files are written in after rmtree stats the files to delete but actually deletes them.

Yup, I don't think you can really fix this, unless you rename the directory prior to recursively removing.
And even then, if a process has the directory entry open, changes can still occur in the directory you're trying to recursively remove.

If you would retry removing files/directories, you might be fighting indefinitely with other processes.
Maybe a clarification in the file module documentation is in order ?

This seems very counter intuitive, maybe I'm interpreting this answer wrong or shutil.rmtree has changed since 2014?

ansible/lib/ansible/modules/files/file.py

Line 421 in 1891827

shutil.rmtree(b_path, ignore_errors=False)

Seems like the SO answer is saying this should be:

shutil.rmtree(b_path, ignore_errors=True)

@techfishio Which would end up not removing the whole tree. Which is not what the user intended. If the user asks operation A to be performed, and operation A did not succeed, we do not want to silently ignore this and continue. So ignore_error=True is pretending everything worked fine.

  I had encounted this problem recently  when i create a lmdb-object to open a source lmdb databases  under a directory (first created by my program) each times and then merge these databases to one target database in merge_db() method. After merge ,i delete source lmdb databases by calling 'shutil.rmtree()'  ,but the lasted direcoty could not be rightly deleted, but the database files under the directory had been sucessfully deleted, along with the errors:"OSError: [Errno 39] Directory not empty" exception .
  But ,the code  works well when i call del() to manually destory lmdb object , otherwise it also works when I start other thread to excute merge_db() method without calling del().

Files identified in the description:

lib/ansible/modules/file.py

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

I do not consider this a bug but a race condition, if your left and right hand are doing opposite actions, you expect to get this kind of error. Using retry/until are a good way to deal with this at the task level and should be the playbook author's call and not built into the module as such event should really require a decision from a human closer to the context in which it is run.

So I'm closing this issue.

Navigation Menu

Repository navigation

retry delete directory while shutil.rmtree raises "OSError: [Errno 39] Directory not empty" exception #34335

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if state == 'absent':
	if state_change:
	if not module.check_mode:
	if prev_state == 'directory':
	try:
	shutil.rmtree(b_path, ignore_errors=False)
	except Exception as e:
	module.fail_json(msg="rmtree failed: %s" % to_native(e))
	else:
	try:
	os.unlink(b_path)
	except Exception as e:
	module.fail_json(path=path, msg="unlinking failed: %s " % to_native(e))
	module.exit_json(path=path, changed=True, diff=diff)
	else:
	module.exit_json(path=path, changed=False)

retry delete directory while shutil.rmtree raises "OSError: [Errno 39] Directory not empty" exception #34335

Description

ISSUE TYPE

COMPONENT NAME

ANSIBLE VERSION

OS / ENVIRONMENT

SUMMARY

STEPS TO REPRODUCE

Activity

ansibot commented on Jan 2, 2018

yuji-k64613 commented on Feb 4, 2018

bruceauyeung commented on Feb 5, 2018

orenwolf commented on Jun 19, 2018

dagwieers commented on Jul 4, 2018

DownRangeDevOps commented on Sep 8, 2018

dagwieers commented on Sep 9, 2018

litaozijin commented on Jul 28, 2019

ansibot commented on May 16, 2020

bcoca commented on Apr 23, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions