Skip to content

Make overwrite=True work with Path arguments - #17552

Merged
saimn merged 12 commits into
astropy:mainfrom
cmarmo:overwrite-path
Jan 16, 2025
Merged

saimn merged 12 commits into
astropy:mainfrom
cmarmo:overwrite-path

Conversation

@cmarmo

@cmarmo cmarmo commented Dec 17, 2024

Copy link
Copy Markdown
Member

Description

This pull request takes overwrite=True into account in writing fits tables.

Fixes #15350

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

Comment thread astropy/io/fits/tests/test_connect.py Resolved
Comment thread astropy/io/fits/tests/test_connect.py Outdated
Comment on lines +72 to +75
try:
t1.write(Path(filename), format="fits", overwrite=True)
except OSError as error:
pytest.fail(f"overwrite true raises {error}")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

filename is already a path using tmp_path. No need to explicitly catch the exception because any exception or unhandled warning is going to fail the test anyway.

Suggested change
try:
t1.write(Path(filename), format="fits", overwrite=True)
except OSError as error:
pytest.fail(f"overwrite true raises {error}")
t1.write(filename, format="fits", overwrite=True)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment as above ... for some reason just using filename does not make the test fail on main...

Comment thread docs/changes/io.fits/17552.bugfix.rst Resolved
@pllim pllim added this to the v7.0.1 milestone Dec 17, 2024
@cmarmo
cmarmo marked this pull request as draft December 18, 2024 08:00
@cmarmo

cmarmo commented Dec 18, 2024 •

Copy link
Copy Markdown
Member Author

I have investigated a bit more the issue on the test (see this comment) and decided to remove the check on the type of output.
I'm still puzzled about how to write an effective test because the issue also depends on the path of the output itself not only on the type of the variable.
I don't particularly like the idea of checking against multiple types... this is more a mypy job ...
Is that solution acceptable? (apart from the test, which I'd be happy to change but I don't know how for now)

Comment thread astropy/io/fits/connect.py Resolved
@cmarmo

cmarmo commented Dec 18, 2024

Copy link
Copy Markdown
Member Author

apart from the test, which I'd be happy to change but I don't know how for now

I have added a context manager to move to the temporary directory and come back after the test.... @pllim is that ok with you?

@cmarmo
cmarmo marked this pull request as ready for review December 18, 2024 20:43
Comment thread astropy/io/fits/tests/test_connect.py Outdated
)

def test_path(self, tmp_path):
with working_directory(tmp_path):

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like you, I am now very perplexed on why this bug only shows up when the filename has no path attached to it. 🤯

Perhaps we should wait for @saimn to review. He might have extra understanding in regards to io.fits. 🙏

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

input : Table
The table to write out.
output : str
output : str or os.PathLike[str]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what the [str] suffix means in this case ?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

os.PathLike comes in two forms: os.PathLike[bytes] and os.PathLike[str]

@neutrinoceros neutrinoceros Jan 10, 2025 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps a simpler way to go would be

Suggested change
output : str or os.PathLike[str]
output : str or pathlib.Path

(it's more specific than it needs to, but should cover 100% of what real users need or want)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not an expert at all about paths.... I just did a git grep in astropy and there are no os.Pathlike, while 6 occurrences of os.Pathlike[str] ... this is why I put os.Pathlike[str] since ced653d for consistency

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I probably authored most of these annotations, so it is fine as is, AFAIC, I'm just offering an alternative in case @saimn has a different preference

Comment thread astropy/io/fits/tests/test_connect.py Resolved
Comment thread astropy/io/fits/tests/test_connect.py Resolved
Comment thread docs/changes/io.fits/17552.bugfix.rst Resolved
Comment thread astropy/io/fits/tests/test_connect.py Resolved
@neutrinoceros

Copy link
Copy Markdown
Contributor

Okay I've played a little with this locally and I think I got a complete solution to the issue with your test

diff --git a/astropy/io/fits/tests/test_connect.py b/astropy/io/fits/tests/test_connect.py
index 969eee2691..f2eb77a33d 100644
--- a/astropy/io/fits/tests/test_connect.py
+++ b/astropy/io/fits/tests/test_connect.py
@@ -1,6 +1,4 @@
-import contextlib
 import gc
-import os
 import warnings
 from pathlib import Path
 
@@ -60,21 +58,6 @@ def equal_data(a, b):
     return all(np.all(a[name] == b[name]) for name in a.dtype.names)
 
 
-@contextlib.contextmanager
-def working_directory(path):
-    """
-    A context manager changing the working directory to the given
-    path. The directory is changed back on exit.
-    """
-
-    cwd = os.getcwd()
-    os.chdir(path)
-    try:
-        yield
-    finally:
-        os.chdir(cwd)
-
-
 class TestSingleTable:
     def setup_class(self):
         self.data = np.array(
@@ -83,11 +66,10 @@ class TestSingleTable:
         )
 
     def test_path(self, tmp_path):
-        with working_directory(tmp_path):
-            filename = "temp.fits"
-            t1 = Table(self.data)
-            t1.write(filename, overwrite=True)
-            t1.write(Path(filename), format="fits", overwrite=True)
+        filename = tmp_path / "temp.fits"
+        t1 = Table(self.data)
+        t1.write(filename, overwrite=True)
+        t1.write(Path(filename), format="fits", overwrite=True)
 
     def test_simple(self, tmp_path):
         filename = tmp_path / "test_simple.fts"
diff --git a/astropy/io/registry/core.py b/astropy/io/registry/core.py
index b7325c4beb..3c0e1fad25 100644
--- a/astropy/io/registry/core.py
+++ b/astropy/io/registry/core.py
@@ -15,7 +15,7 @@ def _expand_user_in_args(args):
     # Conservatively attempt to apply `os.path.expanduser` to the first
     # argument, which can be either a path or the contents of a table.
     if len(args) and isinstance(args[0], PATH_TYPES):
-        ex_user = os.path.expanduser(args[0])
+        ex_user = type(args[0])(os.path.expanduser(args[0]))
         if ex_user != args[0] and os.path.exists(os.path.dirname(ex_user)):
             args = (ex_user,) + args[1:]
     return args
diff --git a/astropy/io/votable/connect.py b/astropy/io/votable/connect.py
index 8c58a9b105..68dcb63366 100644
--- a/astropy/io/votable/connect.py
+++ b/astropy/io/votable/connect.py
@@ -163,7 +163,7 @@ def write_table_votable(
         )
 
     # Check if output file already exists
-    if isinstance(output, str) and os.path.exists(output):
+    if isinstance(output, (str, os.PathLike)) and os.path.exists(output):
         if overwrite:
             os.remove(output)
         else:
diff --git a/astropy/io/votable/util.py b/astropy/io/votable/util.py
index 970a9cbd03..c8537da4f3 100644
--- a/astropy/io/votable/util.py
+++ b/astropy/io/votable/util.py
@@ -43,7 +43,7 @@ def convert_to_writable_filelike(fd, compressed=False):
     -------
     fd : :term:`file-like (writeable)`
     """
-    if isinstance(fd, str):
+    if isinstance(fd, (str, os.PathLike)):
         fd = os.path.expanduser(fd)
         if fd.endswith(".gz") or compressed:
             with gzip.GzipFile(filename=fd, mode="wb") as real_fd:

(I checked that all tests pass locally)

@pllim

pllim commented Dec 19, 2024

Copy link
Copy Markdown
Member

Re: patch -- Does this mean this PR would also add Path support to votable unified I/O if accepted?

@neutrinoceros

Copy link
Copy Markdown
Contributor

I guess so.

@saimn

saimn commented Dec 19, 2024

Copy link
Copy Markdown
Contributor

We should keep this PR about the original bug.

@neutrinoceros

neutrinoceros commented Dec 20, 2024 •

Copy link
Copy Markdown
Contributor

I'm not sure these are actually separable unfortunately. My patch appears to be needed to allow this PR's test to be rewritten in an acceptable form, but my patch also doesn't work on its own: it requires this PR's change in astropy/io/fits/connect.py.

@cmarmo
cmarmo requested a review from nstarman as a code owner December 20, 2024 17:09
@cmarmo

cmarmo commented Dec 20, 2024

Copy link
Copy Markdown
Member Author

I also believe that this PR should keep focus on a specific bug.
Also, I am quite uncomfortable in keeping credit for someone else's patches.
So, please, have a look at this last version.

  • the output variable is a string modified in io/registry/core.py also in the case of no directory.
  • connect.py is unmodified
  • tests have been added either changing directory or not using contextlib.chdir()

Supporting Path is delayed to another PR, which perhaps @neutrinoceros would like to submit soon , as the code has already been written.
@pllim , @saimn, what do you think?

Comment thread astropy/io/fits/tests/test_connect.py Resolved
Comment thread docs/changes/io.fits/17552.bugfix.rst Resolved
Comment thread astropy/io/registry/core.py Resolved
@neutrinoceros

Copy link
Copy Markdown
Contributor

Supporting Path is delayed to another PR, which perhaps @neutrinoceros would like to submit soon , as the code has already been written.

happy to submit my patch as a follow up PR if we can finish this one without it !

@cmarmo

cmarmo commented Jan 9, 2025

Copy link
Copy Markdown
Member Author

This time I did not forget to pull... :)
Also I'm delegating the check on the type to os.path.exists : I'm not sure what explicit type checking adds to the code.
Please, let me know if this solution is acceptable.

@saimn saimn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @cmarmo !

@saimn
saimn merged commit ea49b85 into astropy:main Jan 16, 2025
meeseeksmachine pushed a commit to meeseeksmachine/astropy that referenced this pull request Jan 16, 2025
pllim added a commit that referenced this pull request Jan 16, 2025
…552-on-v7.0.x

Backport PR #17552 on branch v7.0.x (Make `overwrite=True` work with `Path` arguments)
@cmarmo
cmarmo deleted the overwrite-path branch January 16, 2025 20:20
neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Jan 17, 2025
neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Jan 17, 2025
neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Jan 17, 2025
neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Jan 20, 2025
neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Feb 6, 2025
neutrinoceros added a commit to neutrinoceros/astropy that referenced this pull request Jul 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

overwrite=True ignored in FITS writing for Path filename

4 participants