Skip to content

BUG: Fixed file handle leak in array_tofile.#17598

Merged
mattip merged 6 commits intonumpy:masterfrom
simon-graham:array_tofile_cleanup
Nov 9, 2020
Merged

BUG: Fixed file handle leak in array_tofile.#17598
mattip merged 6 commits intonumpy:masterfrom
simon-graham:array_tofile_cleanup

Conversation

@simon-graham
Copy link
Copy Markdown
Contributor

Closes #17589

If the PyArray_ToFile call inside array_tofile fails for any reason, we leak a dup-ed file handle. This causes problems especially on Windows OS, where any subsequent attempt to delete the file will fail because NumPy is still holding an open handle.

This update ensures that we always close the handle.

The added test case forces PyArray_ToFile to fail, which triggers the original problem.

@charris charris added the 09 - Backport-Candidate PRs tagged should be backported label Oct 21, 2020
@simon-graham simon-graham force-pushed the array_tofile_cleanup branch 2 times, most recently from b0c1eea to 6e301f6 Compare October 21, 2020 03:56
@eric-wieser
Copy link
Copy Markdown
Member

Thanks for the patch. I think the error handling is still generally wrong in this function. I think we need to aim for the C equivalent of:

file = ...
try:
    fd = npy_PyFile_Dup2(file)
    try:
        PyArray_ToFile(fd)
    finally:
        npy_PyFile_DupClose2(fd)
finally:
    if own:
        npy_PyFile_CloseFile(file)

This would almost certainly be much easier to get right if the contents of the outer try were extracted to a helper function. A finally clause is a PyErr_Fetch followed by an unconditional call to npy_PyErr_ChainExceptions.

Comment on lines 5051 to 5061
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you repeat this test with x.tofile(self.filename), to check it closes both the python handle and the fd?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This second check is now added.

If we pass a str to tofile() rather than an open file object, it will open a Python handle. Make sure that it gets closed correctly.
@simon-graham
Copy link
Copy Markdown
Contributor Author

Hi @eric-wieser, is there anything else to address for this PR? I believe I've implemented all of your review feedback. Thanks.

Copy link
Copy Markdown
Member

@eric-wieser eric-wieser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for forgetting about this - looks good, thanks!

@mattip mattip merged commit a747890 into numpy:master Nov 9, 2020
@mattip
Copy link
Copy Markdown
Member

mattip commented Nov 9, 2020

Thanks @simon-graham

@charris charris removed the 09 - Backport-Candidate PRs tagged should be backported label Nov 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

array_tofile does not close file when PyArray_ToFile fails

4 participants