Atomic function to rename a file

@pitrou

BPO	8828
Nosy	@pitrou, @vstinner, @giampaolo, @tarekziade, @merwok, @bitdancer, @Trundle
Files	atomic_move_file-windows.py osreplace.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-01-30.21:13:07.802>
created_at = <Date 2010-05-26.23:41:40.453>
labels = ['type-feature', 'library', 'OS-windows']
title = 'Atomic function to rename a file'
updated_at = <Date 2012-01-30.21:13:07.801>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2012-01-30.21:13:07.801>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2012-01-30.21:13:07.802>
closer = 'pitrou'
components = ['Library (Lib)', 'Windows']
creation = <Date 2010-05-26.23:41:40.453>
creator = 'vstinner'
dependencies = []
files = ['17603', '24328']
hgrepos = []
issue_num = 8828
keywords = ['patch']
message_count = 47.0
messages = ['106587', '106599', '106600', '106601', '106602', '106603', '106604', '106605', '106608', '106609', '107070', '107257', '107264', '107269', '107433', '107434', '107449', '143995', '144011', '146226', '146273', '146274', '146280', '146282', '146307', '150149', '150166', '150168', '150170', '150175', '150176', '150177', '150179', '150180', '150181', '150182', '150183', '150186', '150188', '150192', '150193', '150194', '150195', '150196', '152019', '152348', '152349']
nosy_count = 11.0
nosy_names = ['pitrou', 'vstinner', 'draghuram', 'techtonik', 'giampaolo.rodola', 'tarek', 'eric.araujo', 'r.david.murray', 'Trundle', 'neologix', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue8828'
versions = ['Python 3.3']

os.rename() is atomic on Linux, but on Windows it raises an error if the destination does already exist.

Not atomic pseudo-code for Windows:
if exists(b):
unlink(b)
rename(a, b)

Windows offers different functions depending on the version:

MoveFileTransacted(): atomic! version >= (Windows Vista, Windows Server 2008)
ReplaceFile(): version >= Windows 2000
MoveFileEx() with MOVEFILE_REPLACE_EXISTING and MOVEFILE_WRITE_THROUGH flags: not atomic (eg. "If the file is to be moved to a different volume, the function simulates the move by using the CopyFile and DeleteFile functions."), version >= Windows 2000

I don't think that it's possible to write an atomic rename (file) function for any OS, so it's only a "best effort" atomic function. The documentation will give a list of OS on which the operation *is* atomic (eg. Linux).

Note: os.rename() uses MoveFileW() on Windows.

A first implementation can be:

if os.name in ('nt', 'ce'):
   def atomic_rename(a, b):
      if os.path.exists(b):
         unlink(b)
      rename(a, b)
else:
   atomic_rename = os.rename

This implementation is atomic on POSIX, and not atomic on Windows. Tell me if I am wrong.

It can be improved later by adding the support of better Windows functions.

See issue bpo-8604: proposal of a new "with atomic_write(filename) as fp: ..." context manager.

Begin by removing the dest file is maybe not the safer approach :-) Here is a new try: begin by renaming the dest file to a new file.

------

# use maybe a PRNG instead of a dummy counter or tempfile
def _create_old_filename(filename):
   old = filename + '.old'
   index = 2
   while os.path.exists(old):
      old = filename + '-%s.old' % index
      index += 1
   return old

if os.name in ('nt', 'ce'):
   def atomic_rename(src, dst):
      if os.path.exists(dst):
         old = _create_old_filename(dst)
         rename(dst, old)
         rename(src, dst)
         unlink(old)
      else:
         rename(src, dst)
else:
   atomic_rename = os.rename

What can we do if "rename(src, dst)" fails?

This implementation is atomic on POSIX, ...

Wrong :-)

"On how rename is broken in Mac OS X"
http://www.weirdnet.nl/apple/rename.html

"Update January 8, 2010: ... the original bug (5398777) was resolved in Snow Leopard but the issue I reported was not fixed. ..."

We have to think about network file systems like NFS. Gnome (nautilus) had a bug on rename because NFS emitted a delete notification on a rename:
http://linux-nfs.org/pipermail/nfsv4/2009-March/010134.html
https://bugzilla.gnome.org/show_bug.cgi?id=575684

It looks like rename is atomic, it's just a bug about notification. But other virtual file systems may not implement atomic rename (eg. is rename atomic with sshfs?). Should Python detect the file system type to choose the algorithm? I would like to say no, because I consider that as a file system (or kernel) bug, not a Python bug.

--

Should we also implement a atomic version of shutil.move()? Support rename if the source and the destination are on different file systems. Or is shutil.move() already atomic?

Note: this issue is only about renaming a *file*. Atomic rename of a directory is much more complex :-)

It seems you are proposing to call "atomic" something which isn't atomic:

   def atomic_rename(src, dst):
      if os.path.exists(dst):
         old = _create_old_filename(dst)
         rename(dst, old)
         rename(src, dst)
         unlink(old)

@pitrou

@pitrou: Yes, as I wrote: it's not possible to write an atomic function for all OS. The documentation must give a list of the OS on which the function is atomic. Would you prefer to not define the function instead of writing a pseudo-atomic function?

--

Java bug opened in 1996, closed with "Will Not Fix", last comment ("i am facing similar issue...") in 2008:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4017593

"How to do atomic writes in a file" in a MSDN blog:
http://blogs.msdn.com/b/adioltean/archive/2005/12/28/507866.aspx

Extract: "Sometimes shell operations like Delete, Rename can fail for various reasons. For example, it might just happen that an antivirus or content indexing application randomly scans the whole file system once in a while. So, potentially, the file Foo.Tmp.txt will be opened for a short period which will cause ... failed delete. And, not only that, but also Rename can fail if the old file already exists, and someone has an open handle on it."

To avoid that "antivirus or content indexing application" open the file, we may need to use a lock on the files.

Would you prefer to not define the function instead of writing a pseudo-
atomic function?

Your current implementation is useless, since it doesn't achieve anything new.
Besides, if the function isn't atomic, don't name it atomic_XXX.

Antoine Pitrou <[email protected]> added the comment:
> Would you prefer to not define the function instead of writing a pseudo-
> atomic function?

Your current implementation is useless, since it doesn't achieve anything
new. Besides, if the function isn't atomic, don't name it atomic_XXX.

Someone may reimplement it with unlink+rename which is worse :-)

But ok, you prefer to not define the function if no real atomic implementation
can be written.

Victor: you could always name it best_effort_at_atomic_rename :)

Atomic file renames are reimplemented by:

Mercurial
Trac
Dulwich (aka HgGit)

and essential for many other projects out there, so this should be in standard library. Atomic renames depend on filesystem more that on the OS.

Implementation of "atomic" rename() for Windows in Mercurial:
http://selenic.com/repo/hg/file/tip/mercurial/windows.py
(search for "def rename")

I'm good with None/True, but that would imply that for posix rename
we'll need to implement the overwrite=False option...which would be a
nice thing (the shell mv command has -i for that).

My point was rather to forbid False as a value (on all OSes) :)

I think a warning would be good, because a unix programmer will assume
rename will work the same on windows as it does on posix, and vice
versa for a windows programmer.

It is already documented. I don't think we want to add a warning for
every documented peculiar piece of behaviour.

How about overwrite=[None, True] with None meaning "OS default"?

+1.

One of the Python advantages is providing predictable cross-platform
behavior. If we can't introduce nice API without BC break, it is not
a reason to introduce ulgy API.

We cannot make rename() overwrite existing files by default (on Windows).
It's out of question, too much code might rely on this, and this may very well introduce security flaws.
If you're concerned with the semantics difference between platforms, well, there's not much we can do about it now. Java behaves in the same way, for example.

I'm good with None/True, but that would imply that for posix rename
we'll need to implement the overwrite=False option...which would be a
nice thing (the shell mv command has -i for that).

Why?
The problem is that it's simply impossible to implement reliably.
I didn't check mv source code, but it must be using something like:

if '-i' and os.path.exists(target_path):
    continue
os.rename(src_path, target_path).

But there's a TOCTTOU race.

Ah, you are right about the race of course. So yes, I agree with your proposal. It's a weird API, but probably the best we can do.

2011/12/23 Charles-François Natali <[email protected]>

> One of the Python advantages is providing predictable cross-platform
> behavior. If we can't introduce nice API without BC break, it is not
> a reason to introduce ulgy API.

We cannot make rename() overwrite existing files by default (on Windows).
It's out of question, too much code might rely on this, and this may very
well introduce security flaws.
If you're concerned with the semantics difference between platforms, well,
there's not much we can do about it now. Java behaves in the same way, for
example.

I propose quite the opposite. rename() should not overwrite existing files
by default.

I propose quite the opposite. rename() should not overwrite existing
files by default.

That's not what I understood from:

os.rename(overwrite=True) to produce consistent cross-platform
behavior.

The above argument on backward incompatible change applies in exactly the same way (just exchange Windows for POSIX).
As explained above, it can not be done reliably on POSIX (TOCTTOU race).

2011/12/23 Charles-François Natali <[email protected]>

Charles-François Natali <[email protected]> added the comment:

> I propose quite the opposite. rename() should not overwrite existing
> files by default.

That's not what I understood from:
> os.rename(overwrite=True) to produce consistent cross-platform
> behavior.

The above argument on backward incompatible change applies in exactly
the same way (just exchange Windows for POSIX).

As explained above, it can not be done reliably on POSIX (TOCTTOU race).

os.rename(overwrite=False) by default will do less harm than the opposite,
so I'd say it is a way to go even if it can not be reliably done on POSIX.
But I believe correct file system that supports transactions will make it
possible to do reliably on POSIX too.

Then I believe that having a small chance of overwriting file just created
at exactly the same moment on a POSIX is a small price to pay for function
that does safety check before rename on a platform that doesn't have such
functionality at all. If you need this functionality - you will still do
'if exists ... rename' and the TOCTTOU race will affect you. So, the only
exit is to place a little sign "in some extreme cases Linux suffers from
TOCTTOU problem when renaming files without overwriting, so keep that in
mind".

BUT let me remind you that this bug is about "Atomic rename" which should
rollback in case of errors.

os.rename(overwrite=False) by default will do less harm than the opposite,

Disagreed.

Then I believe that having a small chance of overwriting file just created
at exactly the same moment on a POSIX is a small price to pay for function
that does safety check before rename on a platform that doesn't have such
functionality at all.

Disagreed.
If you need the functionality, it's *one* additional line of code. Not
every trivial function deserves to be in the stdlib.

On Fri, Dec 23, 2011 at 10:35 PM, Antoine Pitrou <[email protected]>wrote:

> os.rename(overwrite=False) by default will do less harm than the
opposite,

Disagreed.

Fine. No arguments == no consensus.

> Then I believe that having a small chance of overwriting file just
created
> at exactly the same moment on a POSIX is a small price to pay for
function
> that does safety check before rename on a platform that doesn't have such
> functionality at all.

Disagreed.
If you need the functionality, it's *one* additional line of code. Not
every trivial function deserves to be in the stdlib.

As a Windows programmer I am quite surprised to read this thread with
information that on Linux os.rename() overwrites files without questions,
so as I Windows programmer I want os.rename() to stop that. I always guard
my code against accidental rewrite by catching the exception. EAFP and all
that stuff. But there is no way I can ask forgiveness when files are
already overwritten.

So maybe my warning idea isn't such a bad idea :)

As a unix programmer, I was very surprised to read in this thread that Windows doesn't overwrite the file on rename. As a unix programmer, I don't check for errors on a rename, because I expect it to just work. I'd like the windows rename call to stop throwing errors if the file exists, it breaks my programs if they run on windows.

(Actually, very few of my programs ever get run on Windows, but you get the idea.)

Thus, the only possible course is to maintain backward compatibility, and allow the programmers who care to specify the desired behavior. Since one of the important aspects of 'rename' in unix programmers' minds is that it is atomic, a race condition is not acceptable, therefore overwrite=False is not an acceptable option for os.rename.

As a Windows programmer I am quite surprised to read this thread with
information that on Linux os.rename() overwrites files without questions,
so as I Windows programmer I want os.rename() to stop that.

As a Windows programmer, you are not really qualified to criticize the
behaviour of Unix systems.
As a Linux programmer, I don't want to see the default behaviour
changed.

No arguments == no consensus.

We don't strive to achieve consensus. Agreement among the majority of
core developers is enough, and you aren't part of the population anyway.

Here is a patch adding os.replace().

New changeset 80ddbd822227 by Antoine Pitrou in branch 'default':
Issue bpo-8828: Add new function os.replace(), for cross-platform renaming with overwriting.
http://hg.python.org/cpython/rev/80ddbd822227

os.replace() committed in 3.3!

Navigation Menu

Repository navigation

Atomic function to rename a file #53074

22 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Atomic function to rename a file #53074

Description

Activity

vstinner commented on May 26, 2010

vstinner commented on May 27, 2010

vstinner commented on May 27, 2010

vstinner commented on May 27, 2010

vstinner commented on May 27, 2010

vstinner commented on May 27, 2010

pitrou commented on May 27, 2010

vstinner commented on May 27, 2010

pitrou commented on May 27, 2010

vstinner commented on May 27, 2010

bitdancer commented on Jun 4, 2010

techtonik commented on Jun 7, 2010

vstinner commented on Jun 7, 2010

22 remaining items

pitrou commented on Dec 23, 2011

neologix commented on Dec 23, 2011

bitdancer commented on Dec 23, 2011

techtonik commented on Dec 23, 2011

neologix commented on Dec 23, 2011

techtonik commented on Dec 23, 2011

pitrou commented on Dec 23, 2011

techtonik commented on Dec 23, 2011

bitdancer commented on Dec 23, 2011

pitrou commented on Dec 23, 2011

pitrou commented on Jan 26, 2012

python-dev commented on Jan 30, 2012

pitrou commented on Jan 30, 2012

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions