Repository navigation
Closed
Labels
Affects-devPRs and issues that do not impact an existing Astropy releasePRs and issues that do not impact an existing Astropy releaseBugPackage-expertio.ascii
Milestone
Description
When testing with Python 3.3 / Mac there I'm seeing consistent test failures ending with OSError: [Errno 24] Too many open files. This seems to have been introduced by #2969. I wasn't able to reproduce in limited testing outside of the test suite (i.e trying to read the same files in the same way). This doesn't happen in linux (travis) testing.
@AmrAS1
@astrofrog - have you been seeing this? Don't you normally develop/test on mac / python 3? I tried doing everything possible to make sure I'm building in a clean way.
========================================== test session starts ==========================================
platform darwin -- Python 3.3.5 -- pytest-2.5.1
Running tests with Astropy version 1.0.dev10059.
Running tests in ./astropy/io/ascii /Users/aldcroft/git/astropy/docs/io/ascii.
Platform: Darwin-13.4.0-x86_64-i386-64bit
Executable: /Users/aldcroft/anaconda/envs/py33-new/bin/python
Full Python Version:
3.3.5 |Continuum Analytics, Inc.| (default, Sep 2 2014, 13:57:31)
[GCC 4.2.1 (Apple Inc. build 5577)]
encodings: sys: utf-8, locale: UTF-8, filesystem: utf-8
byteorder: little
float info: dig: 15, mant_dig: 15
Numpy: 1.9.0
Scipy: not available
Matplotlib: not available
h5py: not available
Running doctests in .rst files is not supported on Python 3.x
collected 266 items
astropy/io/ascii/daophot.py .
astropy/io/ascii/ipac.py .
astropy/io/ascii/tests/test_c_reader.py ...........................x................x.......x.x...........
astropy/io/ascii/tests/test_cds_header_from_readme.py ...
astropy/io/ascii/tests/test_compressed.py .....
astropy/io/ascii/tests/test_connect.py .............ss........
astropy/io/ascii/tests/test_fixedwidth.py ............................
astropy/io/ascii/tests/test_html.py s.ssssss.ssss..s
astropy/io/ascii/tests/test_ipac_definitions.py ...................
astropy/io/ascii/tests/test_read.py .............FF....F.F.....................................................................
astropy/io/ascii/tests/test_types.py ...
astropy/io/ascii/tests/test_write.py ..........
_________________________________ test_read_all_files_via_table[force] __________________________________
fast_reader = 'force'
@pytest.mark.parametrize('fast_reader', [True, False, 'force'])
def test_read_all_files_via_table(fast_reader):
for testfile in get_testfiles():
if testfile.get('skip'):
print('\n\n******** SKIPPING %s' % testfile['name'])
continue
print('\n\n******** READING %s' % testfile['name'])
for guess in (True, False):
test_opts = testfile['opts'].copy()
if 'guess' not in test_opts:
test_opts['guess'] = guess
if 'Reader' in test_opts:
format = 'ascii.{0}'.format(test_opts['Reader']._format_name)
del test_opts['Reader']
else:
format = 'ascii'
if 'fast_{0}'.format(format) in core.FAST_CLASSES:
test_opts['fast_reader'] = fast_reader
> table = Table.read(testfile['name'], format=format, **test_opts)
test_read.py:143:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'astropy.table.table.Table'>, args = ('t/short.rdb',)
kwargs = {'format': 'ascii.rdb', 'guess': True}
@classmethod
def read(cls, *args, **kwargs):
"""
Read and parse a data table and return as a Table.
This function provides the Table interface to the astropy unified I/O
layer. This allows easily reading a file in many supported data formats
using syntax such as::
>>> from astropy.table import Table
>>> dat = Table.read('table.dat', format='ascii')
>>> events = Table.read('events.fits', format='fits')
The arguments and keywords (other than ``format``) provided to this function are
passed through to the underlying data reader (e.g. `~astropy.io.ascii.read`).
"""
> return io_registry.read(cls, *args, **kwargs)
./astropy/table/table.py:1712:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cls = <class 'astropy.table.table.Table'>, args = ('t/short.rdb',), kwargs = {'guess': True}
format = 'ascii.rdb', ctx = None
reader = functools.partial(<function io_read at 0x103b29c20>, 'ascii.rdb')
> ???
./astropy/io/registry.py:330:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
format = 'rdb', filename = 't/short.rdb', kwargs = {'guess': True}, read = <function read at 0x103ae5cb0>
def io_read(format, filename, **kwargs):
from .ui import read
format = re.sub(r'^ascii\.', '', format)
> return read(filename, format=format, **kwargs)
./astropy/io/ascii/connect.py:36:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
table = 't/short.rdb', guess = True, kwargs = {'fill_values': [('', '0')], 'format': 'rdb'}
new_kwargs = {'Outputter': <class 'astropy.io.ascii.core.TableOutputter'>, 'Reader': <class 'astropy.io.ascii.basic.Rdb'>, 'fill_values': [('', '0')]}
fast_reader_param = True, format = 'rdb', Reader = <class 'astropy.io.ascii.basic.Rdb'>
def read(table, guess=None, **kwargs):
"""Read the input ``table`` and return the table. Most of
the default behavior for various parameters is determined by the Reader
class.
:param table: input table (file name, file-like object, list of strings, or single newline-separated string)
:param guess: try to guess the table format (default=True)
:param format: input table format
:param Inputter: Inputter class
:param Outputter: Outputter class (default=TableOutputter)
:param delimiter: column delimiter string
:param comment: regular expression defining a comment line in table
:param quotechar: one-character string to quote fields containing special characters
:param header_start: line index for the header line not counting comment lines
:param data_start: line index for the start of data not counting comment lines
:param data_end: line index for the end of data (can be negative to count from end)
:param converters: dict of converters
:param data_Splitter: Splitter class to split data columns
:param header_Splitter: Splitter class to split header columns
:param names: list of names corresponding to each data column
:param include_names: list of names to include in output (default=None selects all names)
:param exclude_names: list of names to exlude from output (applied after ``include_names``)
:param fill_values: specification of fill values for bad or missing table values (default=('', '0'))
:param fill_include_names: list of names to include in fill_values (default=None selects all names)
:param fill_exclude_names: list of names to exlude from fill_values (applied after ``fill_include_names``)
:param fast_reader: whether to use the C engine, can also be a dict with options which default to False (default=True)
:param Reader: Reader class (DEPRECATED) (default=``ascii.Basic``)
"""
if 'fill_values' not in kwargs:
kwargs['fill_values'] = [('', '0')]
# If an Outputter is supplied in kwargs that will take precedence.
new_kwargs = {}
new_kwargs['Outputter'] = core.TableOutputter
fast_reader_param = kwargs.get('fast_reader', True)
if 'Outputter' in kwargs: # user specified Outputter, not supported for fast reading
fast_reader_param = False
format = kwargs.get('format')
new_kwargs.update(kwargs)
# Get the Reader class based on possible format and Reader kwarg inputs.
Reader = _get_format_class(format, kwargs.get('Reader'), 'Reader')
if Reader is not None:
new_kwargs['Reader'] = Reader
format = Reader._format_name
# Remove format keyword if there, this is only allowed in read() not get_reader()
if 'format' in new_kwargs:
del new_kwargs['format']
if guess is None:
guess = _GUESS
if guess:
> dat = _guess(table, new_kwargs, format, fast_reader_param)
./astropy/io/ascii/ui.py:141:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
table = 't/short.rdb'
read_kwargs = {'Outputter': <class 'astropy.io.ascii.core.TableOutputter'>, 'Reader': <class 'astropy.io.ascii.basic.Rdb'>, 'fill_values': [('', '0')]}
format = 'rdb', fast_reader = True
def _guess(table, read_kwargs, format, fast_reader):
"""Try to read the table using various sets of keyword args. First try the
original args supplied in the read() call. Then try the standard guess
keyword args. For each key/val pair specified explicitly in the read()
call make sure that if there is a corresponding definition in the guess
then it must have the same val. If not then skip this guess."""
# Keep a trace of all failed guesses kwarg
failed_kwargs = []
fast_kwargs = []
first_kwargs = [read_kwargs.copy()]
if fast_reader and format is not None and 'fast_{0}'.format(format) in \
core.FAST_CLASSES:
# If a fast version of the reader is available, try that before the slow version
fast_kwargs = read_kwargs.copy()
fast_kwargs['Reader'] = core.FAST_CLASSES['fast_{0}'.format(format)]
first_kwargs = [fast_kwargs] + first_kwargs
# First try guessing
for guess_kwargs in first_kwargs + _get_guess_kwargs_list():
guess_kwargs_ok = True # guess_kwargs are consistent with user_kwargs?
for key, val in read_kwargs.items():
# Do guess_kwargs.update(read_kwargs) except that if guess_args has
# a conflicting key/val pair then skip this guess entirely.
if key not in guess_kwargs:
guess_kwargs[key] = val
elif val != guess_kwargs[key] and guess_kwargs != fast_kwargs:
guess_kwargs_ok = False
break
if not guess_kwargs_ok:
# User-supplied kwarg is inconsistent with the guess-supplied kwarg, e.g.
# user supplies delimiter="|" but the guess wants to try delimiter=" ",
# so skip the guess entirely.
continue
try:
# If guessing will try all Readers then use strict req'ts on column names
if 'Reader' not in read_kwargs:
guess_kwargs['strict_names'] = True
reader = get_reader(**guess_kwargs)
> dat = reader.read(table)
./astropy/io/ascii/ui.py:206:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <astropy.io.ascii.fastbasic.FastRdb object at 0x104105490>, table = 't/short.rdb'
def read(self, table):
"""
Read input data (file-like object, filename, list of strings, or
single string) into a Table and return the result.
"""
if self.comment is not None and len(self.comment) != 1:
raise core.ParameterError("The C reader does not support a comment regex")
elif self.data_start is None:
raise core.ParameterError("The C reader does not allow data_start to be None")
elif self.header_start is not None and self.header_start < 0 and \
not isinstance(self, FastCommentedHeader):
raise core.ParameterError("The C reader does not allow header_start to be "
"negative except for commented-header files")
elif self.data_start < 0:
raise core.ParameterError("The C reader does not allow data_start to be negative")
elif len(self.delimiter) != 1:
raise core.ParameterError("The C reader only supports 1-char delimiters")
elif len(self.quotechar) != 1:
raise core.ParameterError("The C reader only supports a length-1 quote character")
elif 'converters' in self.kwargs:
raise core.ParameterError("The C reader does not support passing "
"specialized converters")
elif 'Outputter' in self.kwargs:
raise core.ParameterError("The C reader does not use the Outputter parameter")
elif 'Inputter' in self.kwargs:
raise core.ParameterError("The C reader does not use the Inputter parameter")
elif 'data_Splitter' in self.kwargs or 'header_Splitter' in self.kwargs:
raise core.ParameterError("The C reader does not use a Splitter class")
self.engine = cparser.CParser(table, self.strip_whitespace_lines,
self.strip_whitespace_fields,
delimiter=self.delimiter,
header_start=self.header_start,
comment=self.comment,
quotechar=self.quotechar,
data_start=self.data_start,
fill_extra_cols=self.fill_extra_cols,
> **self.kwargs)
./astropy/io/ascii/fastbasic.py:86:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
astropy/io/ascii/cparser.pyx:223:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
astropy/io/ascii/cparser.pyx:265:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E OSError: [Errno 24] Too many open files
astropy/io/ascii/cparser.pyx:124: OSError
-------------------------------------------- Captured stdout --------------------------------------------
Metadata
Metadata
Assignees
Labels
Affects-devPRs and issues that do not impact an existing Astropy releasePRs and issues that do not impact an existing Astropy releaseBugPackage-expertio.ascii
Activity
astrofrog commentedon Oct 6, 2014
I've seen that too but I thought it was just because I have too much going on at once on my computer ;)
taldcroft commentedon Oct 6, 2014
I actually rebooted my laptop to rule out the "too-much-going-on" idea.
mdmueller commentedon Oct 7, 2014
Unfortunately I don't have regular access to a Mac at the moment, but I'll definitely test this out when I get the chance. Just took a look at the code and found that the memory map isn't being closed before closing the file--maybe this is the issue? I added a line to close the memory map in this branch.
astrofrog commentedon Oct 7, 2014
@AmrAS1 - I'll test it out, thanks!
astrofrog commentedon Oct 7, 2014
It still fails, but interestingly I saw the following warnings in the log:
astrofrog commentedon Oct 7, 2014
@AmrAS1 - it might be possible to reproduce the issue on linux by reducing the default
ulimitfor open files? Maybe there is a difference in that for mac vs linux? On mac the default is 256.astrofrog commentedon Oct 7, 2014
I managed to list all open file handles and the list looks like:
looks like something to do with
tempfile, maybe unrelated to the Cython code?astrofrog commentedon Oct 7, 2014
I think it's related to
NamedTemporaryFile, investigatingastrofrog commentedon Oct 7, 2014
Ok, definitely related to this block:
astrofrog commentedon Oct 7, 2014
The namedtemporaryfile was a red herring, it's still happening inside the fast reader.
astrofrog commentedon Oct 7, 2014
@AmrAS1 - I managed to get errors that might be helpful!
Note the BufferError: cannot close exported pointers exist
astrofrog commentedon Oct 14, 2014
Fixed in #3026