Skip to content

Commit 12a693c

Browse files
committed
Merge pull request #483 from taldcroft/ascii-table
Improvements to io.ascii - Closes #307 : io.ascii - rework internals for Table support - Closes #335 : Formatted output in io.ascii can generate incorrect column type - Closes #330 : io.ascii doesn't know about table column formats - Closes #361 : Issue with null values in IPAC tables - Closes #469 : Daophot example doesn't keep units - Closes part of #463 : astropy.io.ascii and astropy.io.votable fail with this table
2 parents b7d5e37 + 0e9538e commit 12a693c

24 files changed

Lines changed: 557 additions & 767 deletions

‎CHANGES.rst‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,36 @@ see the "What's New" section of the documentation for more details.
3030
- In ``astropy.wcs``, the ``wcs.cunit`` list now takes and returns
3131
``astropy.units.Unit`` objects.
3232

33-
- Added SExtractor table reader to ``astropy.io.ascii`` [#420]
33+
- ``astropy.io.ascii``
34+
35+
- Improved integration with the ``astropy.table`` Table class so that
36+
table and column metadata (e.g. keywords, units, description,
37+
formatting) are directly available in the output table object. The
38+
CDS, DAOphot, and IPAC format readers now provide this type of
39+
integrated metadata.
40+
41+
- Changed to using `astropy.table` masked tables instead of NumPy
42+
masked arrays for tables with missing values.
43+
44+
- Added SExtractor table reader to ``astropy.io.ascii`` [#420]
45+
46+
- Removed the Memory reader class which was used to convert data input
47+
passed to the ``write`` function into an internal table. Instead
48+
``write`` instantiates an astropy Table object using the data
49+
input to ``write``.
50+
51+
- Removed the NumpyOutputter as the output of reading a table is now
52+
always a ``Table`` object.
53+
54+
- Removed the option of supplying a function as a column output
55+
formatter.
56+
57+
- Added a new ``strip_whitespace`` keyword argument to the ``write``
58+
function. This controls whether whitespace is stripped from
59+
the left and right sides of table elements before writing.
60+
Default is True.
61+
62+
- Fixed a bug in reading IPAC tables with null values.
3463

3564
- Added support for masked tables with missing or invalid data [#451]
3665

‎astropy/io/ascii/__init__.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -31,11 +31,11 @@
3131

3232
from .core import (InconsistentTableError,
3333
NoType, StrType, NumType, FloatType, IntType, AllType,
34-
Column, Keyword,
34+
Column,
3535
BaseInputter, ContinuationLinesInputter,
3636
BaseHeader,
3737
BaseData,
38-
BaseOutputter, NumpyOutputter, TableOutputter,
38+
BaseOutputter, TableOutputter,
3939
BaseReader,
4040
BaseSplitter, DefaultSplitter, WhitespaceSplitter,
4141
convert_numpy,
@@ -50,7 +50,6 @@
5050
from .ipac import Ipac
5151
from .daophot import Daophot
5252
from .sextractor import SExtractor
53-
from .memory import Memory
5453
from .fixedwidth import (FixedWidth, FixedWidthNoHeader,
5554
FixedWidthTwoLine, FixedWidthSplitter,
5655
FixedWidthHeader, FixedWidthData)

‎astropy/io/ascii/basic.py‎

Lines changed: 15 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
"""Asciitable: an extensible ASCII table reader and writer.
1+
"""An extensible ASCII table reader and writer.
22
33
basic.py:
44
Basic table read / write functionality for simple character
@@ -8,7 +8,7 @@
88
:Author: Tom Aldcroft ([email protected])
99
"""
1010

11-
##
11+
##
1212
## Redistribution and use in source and binary forms, with or without
1313
## modification, are permitted provided that the following conditions are met:
1414
## * Redistributions of source code must retain the above copyright
@@ -19,7 +19,7 @@
1919
## * Neither the name of the Smithsonian Astrophysical Observatory nor the
2020
## names of its contributors may be used to endorse or promote products
2121
## derived from this software without specific prior written permission.
22-
##
22+
##
2323
## THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
2424
## ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
2525
## WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
@@ -28,7 +28,7 @@
2828
## (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
2929
## LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
3030
## ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
31-
## (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
31+
## (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
3232
## SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
3333

3434
import re
@@ -42,7 +42,7 @@ class Basic(core.BaseReader):
4242
configurable.
4343
::
4444
45-
rdr = asciitable.get_reader(Reader=asciitable.Basic)
45+
rdr = ascii.get_reader(Reader=ascii.Basic)
4646
rdr.header.splitter.delimiter = ' '
4747
rdr.data.splitter.delimiter = ' '
4848
rdr.header.start_line = 0
@@ -52,7 +52,7 @@ class Basic(core.BaseReader):
5252
rdr.data.comment = r'\s*#'
5353
5454
Example table::
55-
55+
5656
# Column definition is the first uncommented line
5757
# Default delimiter is the space character.
5858
apples oranges pears
@@ -133,7 +133,7 @@ def __init__(self):
133133

134134

135135
class Tab(Basic):
136-
"""Read a tab-separated file. Unlike the :class:`Basic` reader, whitespace is
136+
"""Read a tab-separated file. Unlike the :class:`Basic` reader, whitespace is
137137
not stripped from the beginning and end of lines. By default whitespace is
138138
still stripped from the beginning and end of individual column values.
139139
@@ -148,7 +148,7 @@ def __init__(self):
148148
self.header.splitter.delimiter = '\t'
149149
self.data.splitter.delimiter = '\t'
150150
# Don't strip line whitespace since that includes tabs
151-
self.header.splitter.process_line = None
151+
self.header.splitter.process_line = None
152152
self.data.splitter.process_line = None
153153
# Don't strip data value whitespace since that is significant in TSV tables
154154
self.data.splitter.process_val = None
@@ -187,7 +187,7 @@ def get_type_map_key(self, col):
187187

188188
def get_cols(self, lines):
189189
"""Initialize the header Column objects from the table ``lines``.
190-
190+
191191
This is a specialized get_cols for the RDB type:
192192
Line 0: RDB col names
193193
Line 1: RDB col definitions
@@ -204,10 +204,10 @@ def get_cols(self, lines):
204204

205205
if len(self.names) != len(raw_types):
206206
raise ValueError('RDB header mismatch between number of column names and column types')
207-
207+
208208
if any(not re.match(r'\d*(N|S)$', x, re.IGNORECASE) for x in raw_types):
209209
raise ValueError('RDB types definitions do not all match [num](N|S): %s' % raw_types)
210-
210+
211211
self._set_cols_from_names()
212212
for col, raw_type in zip(self.cols, raw_types):
213213
col.raw_type = raw_type
@@ -217,9 +217,8 @@ def write(self, lines):
217217
lines.append(self.splitter.join([x.name for x in self.cols]))
218218
rdb_types = []
219219
for col in self.cols:
220-
if issubclass(col.type, core.NumType):
221-
rdb_types.append('N')
222-
else:
223-
rdb_types.append('S')
224-
lines.append(self.splitter.join(rdb_types))
220+
# Check if dtype.kind is string or unicode. See help(np.core.numerictypes)
221+
rdb_type = 'S' if col.dtype.kind in ('S', 'U') else 'N'
222+
rdb_types.append(rdb_type)
225223

224+
lines.append(self.splitter.join(rdb_types))

‎astropy/io/ascii/cds.py‎

Lines changed: 41 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
"""Asciitable: an extensible ASCII table reader and writer.
1+
"""An extensible ASCII table reader and writer.
22
33
cds.py:
44
Classes to read CDS / Vizier table format
@@ -131,11 +131,13 @@ def get_cols(self, lines):
131131
col.start = int(re.sub(r'[-\s]', '', match.group('start') or match.group('end'))) - 1
132132
col.end = int(match.group('end'))
133133
col.units = match.group('units')
134-
col.descr = match.group('descr')
134+
if col.units == '---':
135+
col.units = None # "---" is the marker for no units in CDS table
136+
col.description = match.group('descr').strip()
135137
col.raw_type = match.group('format')
136138
col.type = self.get_col_type(col)
137139

138-
match = re.match(r'\? (?P<equal> =)? (?P<nullval> \S*)', col.descr, re.VERBOSE)
140+
match = re.match(r'\? (?P<equal> =)? (?P<nullval> \S*)', col.description, re.VERBOSE)
139141
if match:
140142
if issubclass(col.type, core.FloatType):
141143
fillval = 'nan'
@@ -152,7 +154,7 @@ def get_cols(self, lines):
152154
cols.append(col)
153155
else: # could be a continuation of the previous col's description
154156
if cols:
155-
cols[-1].descr += line.strip()
157+
cols[-1].description += line.strip()
156158
else:
157159
raise ValueError('Line "%s" not parsable as CDS header' % line)
158160

@@ -196,8 +198,16 @@ class Cds(core.BaseReader):
196198
"""Read a CDS format table. See http://vizier.u-strasbg.fr/doc/catstd.htx.
197199
Example::
198200
199-
Table: Spitzer-identified YSOs: Addendum
201+
Table: Table name here
200202
= ==============================================================================
203+
Catalog reference paper
204+
Bibliography info here
205+
================================================================================
206+
ADC_Keywords: Keyword ; Another keyword ; etc
207+
208+
Description:
209+
Catalog description here.
210+
================================================================================
201211
Byte-by-byte Description of file: datafile3.txt
202212
--------------------------------------------------------------------------------
203213
Bytes Format Units Label Explanations
@@ -206,20 +216,38 @@ class Cds(core.BaseReader):
206216
5- 6 I2 h RAh Hour of Right Ascension (J2000)
207217
8- 9 I2 min RAm Minute of Right Ascension (J2000)
208218
11- 15 F5.2 s RAs Second of Right Ascension (J2000)
219+
--------------------------------------------------------------------------------
220+
Note (1): A CDS file can contain sections with various metadata.
221+
Notes can be multiple lines.
222+
Note (2): Another note.
209223
--------------------------------------------------------------------------------
210224
1 03 28 39.09
225+
2 04 18 24.11
226+
227+
**About parsing the CDS format**
228+
229+
The CDS format consists of a table description and the table data. These
230+
can be in separate files as a ``ReadMe`` file plus data file(s), or
231+
combined in a single file. Different subsections within the description
232+
are separated by lines of dashes or equal signs ("------" or "======").
233+
The table which specifies the column information must be preceded by a line
234+
starting with "Byte-by-byte Description of file:".
235+
236+
In the case where the table description is combined with the data values,
237+
the data must be in the last section and must be preceded by a section
238+
delimiter line (dashes or equal signs only).
211239
212240
**Basic usage**
213241
214-
Use the ``asciitable.read()`` function as normal, with an optional ``readme``
242+
Use the ``ascii.read()`` function as normal, with an optional ``readme``
215243
parameter indicating the CDS ReadMe file. If not supplied it is assumed that
216244
the header information is at the top of the given table. Examples::
217245
218-
>>> import asciitable
219-
>>> table = asciitable.read("t/cds.dat")
220-
>>> table = asciitable.read("t/vizier/table1.dat", readme="t/vizier/ReadMe")
221-
>>> table = asciitable.read("t/cds/multi/lhs2065.dat", readme="t/cds/multi/ReadMe")
222-
>>> table = asciitable.read("t/cds/glob/lmxbrefs.dat", readme="t/cds/glob/ReadMe")
246+
>>> from astropy.io import ascii
247+
>>> table = ascii.read("t/cds.dat")
248+
>>> table = ascii.read("t/vizier/table1.dat", readme="t/vizier/ReadMe")
249+
>>> table = ascii.read("t/cds/multi/lhs2065.dat", readme="t/cds/multi/ReadMe")
250+
>>> table = ascii.read("t/cds/glob/lmxbrefs.dat", readme="t/cds/glob/ReadMe")
223251
224252
**Using a reader object**
225253
@@ -231,15 +259,15 @@ class Cds(core.BaseReader):
231259
have header information for the given table.
232260
233261
>>> readme = "t/vizier/ReadMe"
234-
>>> r = asciitable.get_reader(asciitable.Cds, readme=readme)
262+
>>> r = ascii.get_reader(ascii.Cds, readme=readme)
235263
>>> table = r.read("t/vizier/table1.dat")
236264
>>> # table5.dat has the same ReadMe file
237265
>>> table = r.read("t/vizier/table5.dat")
238266
239267
If no ``readme`` parameter is specified, then the header
240268
information is assumed to be at the top of the given table.
241269
242-
>>> r = asciitable.get_reader(asciitable.Cds)
270+
>>> r = ascii.get_reader(ascii.Cds)
243271
>>> table = r.read("t/cds.dat")
244272
>>> #The following gives InconsistentTableError, since no
245273
>>> #readme file was given and table1.dat does not have a header.

0 commit comments

Comments
 (0)