ResourceWarning when reading ascii tables

Using an example ecsv file from the astropy documentation:

# %ECSV 0.9
# ---
# datatype:
# - {name: a, unit: m / s, datatype: int64, format: '%03d'}
# - {name: b, unit: km, datatype: int64, description: This is column b}
a b
001 2
004 3

(although this works for anything I tried, including rdb files) when I try to read it with a python with warnings enabled (e.g. python -Wd) I get a resource warning:

>>> Table.read("test.ecsv")
/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/io/ascii/connect.py:37: ResourceWarning: unclosed file <_io.TextIOWrapper name='test.ecsv' encoding='UTF-8'>
  return read(filename, format=format, **kwargs)
<Table length=2>
  a     b  
m / s   km 
int64 int64
----- -----
  001     2
  004     3

Specifying the format on the read does not trigger a warning:

>>> Table.read("test.ecsv", format="ascii.ecsv")
<Table length=2>
  a     b  
m / s   km 
int64 int64
----- -----
  001     2
  004     3

and neither does using astropy.io.ascii.read directly. So far "ascii.rdb" also behaves like this but "ascii.latex" does not.

This is with astropy 3.1.2 from conda.

Fast reader does not seem to be the issue:

>>> t = Table.read("test.ecsv", fast_reader=False)
/Users/timj/work/lsstsw3/miniconda/envs/lsst-scipipe/lib/python3.7/site-packages/astropy/io/ascii/connect.py:37: ResourceWarning: unclosed file <_io.TextIOWrapper name='test.ecsv' encoding='UTF-8'>
  return read(filename, format=format, **kwargs)

@taldcroft

My hunch is that something done during the guessing game wasn't closed properly. And since disabling fast reader did not get rid of the warning, it is probably some Python reader in the "things to guess with" list. I tried code diving but got lost in 🐰 🕳 . @taldcroft , any ideas?

Also, it seems like this warning only appears when used with python -Wd or explicitly caught in IPython session. Otherwise, you won't see it. I also wonder why our CI didn't catch this, as we have --open-files check.

Yes, maybe I buried the -Wd in my initial report. I always run with warnings maximally enabled so that I can stay ahead of problems. I'm pretty sure that ResourceWarning is telling you that python eventually works it out and closes the file for you. open-files only looks for files that have been left open which haven't been automatically closed by Python. It's warning us to ensure that we can work out we have a potential problem.

Here is what I've figured out. The issue is primarily related to get_readable_fileobj (in astropy/utils/data.py), which appears to not be closing files in its context manager when the input is itself a file handle. Here is the upshot of I/O that happens during the Table.read() call:

ctx = get_readable_fileobj('junk.ecsv', encoding='binary')
fileobj = ctx.__enter__()
with get_readable_fileobj(fileobj, encoding='UTF-8') as fileobj2:
    out = fileobj2.read()
print(fileobj2.closed)  # prints False, I would have expected True
ctx.__exit__(*sys.exc_info())
print(fileobj2.closed)  # prints True at this point

I confirmed with print statements in the actual code that the io.ascii file object used to read the file is not closed after the context manager returns. The __exit__ for the get_readable_fileobj context manager is looping over some lists of file descriptors that are to be closed, but that list is empty in this case.

So one thing relates to the very end the get_readable_fileobj which has a list of file descriptors to close. In the code above one does not actually get a ResourceWarning and the list of FDs to close is [<_io.FileIO name='junk.ecsv' mode='rb' closefd=True>]. But in the actual Table.read('junk.ecsv') call that final list is [<_io.FileIO [closed]>] . So maybe that fileobj is getting closed somewhere else and mucking things up.

@taldcroft

I came into get_readable_fileobj as well, and I think that the TextIOWrapper created in the needs_textio_wrapper block must be closed:

--- a/astropy/utils/data.py
+++ b/astropy/utils/data.py
@@ -295,6 +295,7 @@ def get_readable_fileobj(name_or_obj, encoding=None, cache=False,
 
         fileobj = io.BufferedReader(fileobj)
         fileobj = io.TextIOWrapper(fileobj, encoding=encoding)
+        close_fds.append(fileobj)
 
         # Ensure that file is at the start - io.FileIO will for
         # example not always be at the start:

With this I no more have the warning, and tests pass (I didn't run the full test suite, but io seems ok).
I won't have much more time to investigate, if you want to try with this @taldcroft .

And I just see your PR after pushing the comment button 😁

And I just see your PR after pushing the comment button

Two of you coming to the same conclusion means something :)
Could you then review/approve the PR?

Just a note to say that this is still a problem for us and we worry that it's showing a real problem. We are reading hundreds of these files and the warnings are overwhelming so we've finally cracked and now mask them all (lsst/meas_algorithms#192).

@timj

@timj - I'll try looking into this again. And likewise the astropy.utils maintainers since the root cause seems to be get_readable_fileobj. I remember my head hurting the last time looking at that code.

Or not, just looking at #8675 again as well. Ugh.

These messages still appear when testing 4.1:

[  359s] io/ascii/tests/test_connect.py::test_read_csv
[  359s]   /usr/lib/python3.8/site-packages/astropy/io/ascii/connect.py:18: ResourceWarning: unclosed file <_io.TextIOWrapper name='/usr/lib/python3.8/site-packages/astropy/io/ascii/tests/data/simple_csv.csv' encoding='UTF-8'>
[  359s]     return read(filename, **kwargs)
[  359s] 
[  359s] io/ascii/tests/test_connect.py::test_auto_identify_ecsv
[  359s]   /usr/lib/python3.8/site-packages/astropy/io/ascii/connect.py:18: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-abuild/pytest-0/popen-gw0/test_auto_identify_ecsv0/tmpFile.ecsv' encoding='UTF-8'>
[  359s]     return read(filename, **kwargs)
[  359s] 
[  359s] io/ascii/tests/test_ecsv.py::test_round_trip_masked_table_default
[  359s]   /usr/lib/python3.8/site-packages/astropy/io/ascii/connect.py:18: ResourceWarning: unclosed file <_io.TextIOWrapper name='/tmp/pytest-of-abuild/pytest-0/popen-gw0/test_round_trip_masked_table_d0/test.ecsv' encoding='UTF-8'>
[  359s]     return read(filename, **kwargs)

And more of the same type for many different file type tests.

Navigation Menu

Repository navigation

ResourceWarning when reading ascii tables #8673

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ResourceWarning when reading ascii tables #8673

Description

Activity

timj commented on May 7, 2019

timj commented on May 7, 2019

pllim commented on May 7, 2019

pllim commented on May 7, 2019

timj commented on May 7, 2019

taldcroft commented on May 7, 2019

saimn commented on May 7, 2019

saimn commented on May 7, 2019

bsipocz commented on May 7, 2019

timj commented on Apr 9, 2020

taldcroft commented on Apr 9, 2020

taldcroft commented on Apr 9, 2020

bnavigator commented on Oct 22, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions