Repository navigation
Improvements to io.ascii - #483
Conversation
|
Rats, looks like there are some numpy 1.4 and python 3 test failures that need investigation. |
|
I'm not familiar enough with this code to be able to comment very intelligently, but nothing stood out as odd in a cursory glance. I'll help investigate the Numpy 1.4 problems if need be--I think I may have seen something like this before. |
|
@iguananaut - As you can see I have just blocked out more of the io.ascii tests that are due to numpy 1.4 (which is formally not supported for masked tables). I got numpy 1.4 working OK on my mac as a workaround. When numpy 1.7 comes out I think we should really talk about dropping 1.4 support. |
Conflicts: astropy/io/ascii/core.py
Conflicts: astropy/io/ascii/core.py
Conflicts: astropy/io/ascii/core.py
Split _pformat_col into an iterator section that iterates over the string-formatted column values instead of collecting them into a list. The post-processing part then does stuff that is needed for pretty-printing. The driver for this was to use the iterator (and all the nice formatting machinery) to do the string formatting in io.ascii.write. Conflicts: astropy/table/pprint.py
Conflicts: astropy/io/ascii/core.py
Conflicts: astropy/io/ascii/core.py
Conflicts: astropy/io/ascii/__init__.py
This commit modifies the TableOutputter to copy any io.ascii column format and units attributes to the corresponding attrs in the output Table columns. The initial use case is Daophot, which includes both units and format info in the default specification. Some side effects of this change cropped up and are dealt with here. Reading the format and units info for Daophot required modifying the ContinuationLinesInputter to have a no_continue attribute that provides a regex which disables line continuation when the regex matches. It also required rewriting the header parsing for Daophot. The Daophot table is also the basis for all write tests, which is good because adding the format info highlighted a known issue that when writing tables with format specifiers like "%10.2f", the extra spaces generated by that output fooled the Python CSV writer into thinking the spaces had to be quoted. This is a little tricky to deal with because in theory a table could have values with significant whitespace on the ends which cannot be stripped. This forced adding a new strip_whitespace kwarg to ui.write and iu.get_writer that allows explicit control in case the default behavior is not correct. The latex splitter join method had to be modified to strip extra whitespace from the formatting. A number of test write outputs changed slightly because of the application of the proper value formatting. Conflicts: astropy/io/ascii/core.py
When 'null' occurs in the NULL row it now generates a fill_value entry that will create a True mask if 'null' occurs in the data. Closes astropy#361 (Issue with null values in IPAC tables) and addresses half of issue astropy#463 (astropy.io.ascii and astropy.io.votable fail with this table).
Since column units can now be a Unit object, it must be stringified before further processing.
…rom a reader object
|
Rebased because of a merge conflict in |
|
This looks good to me, and I can confirm it fixes #361. My only comment is that I think |
There was a problem hiding this comment.
Building the docs on this branch causes a warning which I trace back to this line. The problem is that the numpy sphinx extensions get confused and think that this is a documentation section name "Table: Table name here". Adding the space after the first line fixes that, which why it was in the exisiting code. There may be some other way to make it not think this is a section, or perhaps the numpydoc extension needs to be altered... But regardless, right now this gives a doc build warning.
|
Other than that one place where the doc build breaks, this seems great to me. |
|
By the way, I've created a mini-survey to find out what versions of packages people are currently using: I will send this out to the mailing list on Monday (for maximum impact) but I already advertised it by twitter and have ~50 replies. Out of those, 4 people use Numpy 1.4.x. Will be interesting to see how this evolves as more people reply. |
Improvements to io.ascii - Closes #307 : io.ascii - rework internals for Table support - Closes #335 : Formatted output in io.ascii can generate incorrect column type - Closes #330 : io.ascii doesn't know about table column formats - Closes #361 : Issue with null values in IPAC tables - Closes #469 : Daophot example doesn't keep units - Closes part of #463 : astropy.io.ascii and astropy.io.votable fail with this table
Improvements to io.ascii - Closes astropy#307 : io.ascii - rework internals for Table support - Closes astropy#335 : Formatted output in io.ascii can generate incorrect column type - Closes astropy#330 : io.ascii doesn't know about table column formats - Closes astropy#361 : Issue with null values in IPAC tables - Closes astropy#469 : Daophot example doesn't keep units - Closes part of astropy#463 : astropy.io.ascii and astropy.io.votable fail with this table
Update pyvo maintainers
This PR makes a number of substantial improvements to
io.ascii. This addresses:There is still some possible minor code and documentation cleanup, but the main changes are complete:
astropy.tableTable class so thattable and column metadata (e.g. keywords, units, description,
formatting) are directly available in the output table object. The
CDS, DAOphot, and IPAC format readers now provide this type of
integrated metadata.
astropy.tablemasked tables instead of NumPymasked arrays for tables with missing values.
passed to the
writefunction into an internal table. Insteadwriteinstantiates an astropy Table object using the datainput to
write.always a
Tableobject.formatter.
strip_whitespacekeyword argument to thewritefunction. This controls whether whitespace is stripped from
the left and right sides of table elements before writing.
Default is True.
This PR has a disproportionately long update to CHANGES.rst. I don't know where is the appropriate place to capture all the detailed changes to a subpackage. In this case many of these changes represent API breakage with
asciitableso users need to know.