get_readable_fileobj behaves differently for file object input in Python 3

In Python 2, the following code works, but not in Python 3:

In [6]: import gzip

In [7]: f = gzip.open('data.txt.gz')

In [8]: from astropy.table import Table

In [9]: t = Table.read(f, format='ascii')
---------------------------------------------------------------------------
InconsistentTableError                    Traceback (most recent call last)
<ipython-input-9-754ec57c7976> in <module>()
----> 1 t = Table.read(f, format='ascii')

/Users/tom/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9998-py3.4-macosx-10.8-x86_64.egg/astropy/table/table.py in read(cls, *args, **kwargs)
   1774         passed through to the underlying data reader (e.g. `~astropy.io.ascii.read`).
   1775         """
-> 1776         return io_registry.read(cls, *args, **kwargs)
   1777 
   1778     def write(self, *args, **kwargs):

/Users/tom/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9998-py3.4-macosx-10.8-x86_64.egg/astropy/io/registry.py in read(cls, *args, **kwargs)
    328 
    329         reader = get_reader(format, cls)
--> 330         data = reader(*args, **kwargs)
    331 
    332         if not isinstance(data, cls):

/Users/tom/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9998-py3.4-macosx-10.8-x86_64.egg/astropy/io/ascii/connect.py in read_asciitable(filename, **kwargs)
     19 def read_asciitable(filename, **kwargs):
     20     from .ui import read
---> 21     return read(filename, **kwargs)
     22 
     23 io_registry.register_reader('ascii', Table, read_asciitable)

/Users/tom/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9998-py3.4-macosx-10.8-x86_64.egg/astropy/io/ascii/ui.py in read(table, guess, **kwargs)
    139         guess = _GUESS
    140     if guess:
--> 141         dat = _guess(table, new_kwargs, format, fast_reader_param)
    142     else:
    143         reader = get_reader(**new_kwargs)

/Users/tom/Library/Python/3.4/lib/python/site-packages/astropy-1.0.dev9998-py3.4-macosx-10.8-x86_64.egg/astropy/io/ascii/ui.py in _guess(table, read_kwargs, format, fast_reader)
    236             lines.append('Check the table and try with guess=False '
    237                          'and appropriate arguments to read()')
--> 238             raise core.InconsistentTableError('\n'.join(lines))
    239 
    240 

InconsistentTableError: 
ERROR: Unable to guess table format with the guesses listed below:
Reader:Basic fill_values: [('', '0')] strict_names: True
Reader:Rdb fill_values: [('', '0')] strict_names: True
Reader:FastTab fill_values: [('', '0')] strict_names: True
Reader:Tab fill_values: [('', '0')] strict_names: True
Reader:Cds fill_values: [('', '0')] strict_names: True
Reader:Daophot fill_values: [('', '0')] strict_names: True
Reader:SExtractor fill_values: [('', '0')] strict_names: True
Reader:Ipac fill_values: [('', '0')] strict_names: True
Reader:Latex fill_values: [('', '0')] strict_names: True
Reader:AASTex fill_values: [('', '0')] strict_names: True
Reader:HTML fill_values: [('', '0')] strict_names: True
Reader:CommentedHeader delimiter: '|' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:CommentedHeader delimiter: '|' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:CommentedHeader delimiter: ',' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:CommentedHeader delimiter: ',' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:CommentedHeader delimiter: ' ' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:CommentedHeader delimiter: ' ' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:CommentedHeader delimiter: '\\s' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:CommentedHeader delimiter: '\\s' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastBasic delimiter: '|' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastBasic delimiter: '|' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastBasic delimiter: ',' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastBasic delimiter: ',' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastBasic delimiter: ' ' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastBasic delimiter: ' ' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastBasic delimiter: '\\s' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastBasic delimiter: '\\s' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:Basic delimiter: '|' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:Basic delimiter: '|' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:Basic delimiter: ',' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:Basic delimiter: ',' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:Basic delimiter: ' ' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:Basic delimiter: ' ' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:Basic delimiter: '\\s' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:Basic delimiter: '\\s' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastNoHeader delimiter: '|' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastNoHeader delimiter: '|' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastNoHeader delimiter: ',' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastNoHeader delimiter: ',' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastNoHeader delimiter: ' ' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastNoHeader delimiter: ' ' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:FastNoHeader delimiter: '\\s' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:FastNoHeader delimiter: '\\s' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:NoHeader delimiter: '|' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:NoHeader delimiter: '|' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:NoHeader delimiter: ',' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:NoHeader delimiter: ',' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:NoHeader delimiter: ' ' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:NoHeader delimiter: ' ' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:NoHeader delimiter: '\\s' fill_values: [('', '0')] quotechar: '"' strict_names: True
Reader:NoHeader delimiter: '\\s' fill_values: [('', '0')] quotechar: "'" strict_names: True
Reader:Basic fill_values: [('', '0')]
ERROR: Unable to guess table format with the guesses listed above.
Check the table and try with guess=False and appropriate arguments to read()

This is because in Python 3, get_readable_fileobj causes the file object cursor to move to the end:

from astropy.utils.data import get_readable_fileobj

from io import StringIO
s = StringIO(u'a\n1\,2\,3')

for test in range(3):
    with get_readable_fileobj(s) as file_obj:
        print(file_obj.read())

gives:

mac-robitaille2:tmp tom$ python2.7 test7.py 
a
1\,2\,3
a
1\,2\,3
a
1\,2\,3
mac-robitaille2:tmp tom$ python test7.py 
a
1\,2\,3
Traceback (most recent call last):
  File "test7.py", line 8, in <module>
    print(s.read())
ValueError: I/O operation on closed file

A simpler example to reproduce the issue:

In [2]: from io import StringIO

In [3]: from astropy.io.ascii import read

In [4]: s = StringIO(u'a\n1\,2\,3')

In [5]: from astropy.io.ascii import read

In [6]: read(s)
<BOOM>

@taldcroft

cc @taldcroft

What confuses me is that this works in Python 2...

It turns out this is a bug in get_readable_fileobj (I have updated the issue description above)

@mdboom

This could be fixed by moving the fileobj.seek(0) out of the if statement at the end of get_readable_fileobj, but I'm not sure if it's the right solution.

cc @mdboom @eteq @embray

There seems to be a common theme of needing to write customized file reading abstractions (with support for various file-like objects, compressed files, array data, etc.) that keep getting written for each type of file format we have. io.fits has this. io.ascii seems to have something like this as well, as does the votable package. And of course also the routines in astropy.utils.data. pyasdf, although not yet part of Astropy, also has a fairly extensive GenericFile class.

I feel like at some point I want to to through all our code and find things like that can be further unified so that we can fix bugs like this all in one place :D

@embray

@embray - in a way, the get_readable_fileobj in astropy.utils.data is a first step at unifying, right? And the bug is there (not in io.ascii or in table as I originally thought).

I would see get_readable_fileobj is more of a frontend to that. I really like the GenericFile class in pyasdf, and it's not terribly dissimilar to the _File class in pyfits (though the latter still has a few bits in it particular to pyfits, and other dark corners).

Per discussion I've changed the label from io.ascii to utils.

I have addressed the original problem here in #3132. The driver to doing this in io.ascii is described there. I leave it to the rest whether this issue should remain open since it looks like it got a bigger scope.

Dropping this for 0.4.3 since #3132 added at least workaround for the original symptom of the issue. But leaving this open as a reminder to address the root issue in get_readable_fileobj...

Is this still a problem? You need to pass encoding='binary' into get_readable_fileobj if you are loading binary files.

The original example in the issue description now works so I'll go ahead and close.

Navigation Menu

Repository navigation

get_readable_fileobj behaves differently for file object input in Python 3 #3013

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

get_readable_fileobj behaves differently for file object input in Python 3 #3013

Description

Activity

astrofrog commented on Oct 2, 2014

astrofrog commented on Oct 2, 2014

astrofrog commented on Oct 2, 2014

astrofrog commented on Oct 3, 2014

astrofrog commented on Oct 3, 2014

embray commented on Oct 3, 2014

astrofrog commented on Oct 3, 2014

embray commented on Oct 3, 2014

taldcroft commented on Oct 9, 2014

taldcroft commented on Nov 24, 2014

embray commented on Dec 31, 2014

pllim commented on Oct 30, 2019

astrofrog commented on Oct 30, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions