Repository navigation
bug in io.ascii #3319
Description
I am finding that in certain cases, io.ascii struggles to correctly ignore comment lines, and also the start of the data rows. Here is some particularly simple data that illustrates the problem:
# some header info
#F header info beginning with 'F'
1 2 3
4 5 6
7 8 9
If I run ascii.read(fname), I get
AttributeError: 'NoneType' object has no attribute 'group'
The workaround seems to be to set guess=False. Based on an email chain with @eteq the problem seems to be that the DAOPHOT reader specifically looks for the "#F..." pattern, and for some reason even though this is not a DAOPHOT output file, the reader fails when it tries to actually interpret that line. Erik told me to call the attention of @taldcroft to this issue.
My workaround for the time being is s imply to set guess=False:
ascii.read(fname, guess=False).
This does not bail when reaching the #F line, but now the first row of data is being ignored. My workaround for that latter problem is to manually set data_start=0, but this should probably be the default behavior.
Activity
embray commentedon Jan 21, 2015
Probably could catch whatever is resulting in that exception and pass on it when guessing. Would help to have the full traceback though (though from the looks of it it sounds like something is trying to match to a regular expression, but not bailing if the match fails). So even if this is too much of a corner case for the guessing to work around, at least that much should be fixed.
hamogu commentedon Jan 22, 2015
Thanks for the report @aphearin ! In the end, the guessing just tries out data format and see which one works and if a table is very similar to such a a format, there is a chance that it the guessing fails (if you know the format of your table beforehand, it is always better to avoid guessing, because of performance).
However, in this case it looks like reading as a DAOphot tables failed and thus the reader should have tried the next alternative.
I would like to debug this and and fix it.
Unfortunately, I cannot see the exact format of your table because github renders that as markdown to me (I see bold font and an underline). I don't know how to reverse engeneer the actual ASCII from the rendered format.
Can you edit your post and enclose the table into triple backticks? Github will then show that as code and not parse it as markdown.
aphearin commentedon Jan 22, 2015
@hamogu - done. Hope this helps track down the bug, let me know if there's anything else I can do. If you'll be working on this in the short term, there's a possibly more serious performance bug in ascii.read() I have found that I will write up and submit shortly and reference in this thread when I do.
hamogu commentedon Jan 22, 2015
Thanks, I'll look at that this evening. I think I have a pretty good idea what might be causing this and it's probably a five line fix, so it's unlikely to interfere with anything else.
aphearin commentedon Jan 22, 2015
Almost certainly an unrelated problem with ascii.read(), but see also #3334
hamogu commentedon Jan 23, 2015
OK, I have a solution for this problem that I will push to github in a minute that causes the guessing to read this table without error. Note, however, that it will read it as a CommentedHeader table so that the result is:
with column names
['some', 'header' 'info']. That is not wrong, although it might not be the expected behaviour in this case. However, there is just no way to algorithmically decide that the table above should have columns that are named col1, col2, col3 automatically (because the first line is a comment) and that the table below should use the first line as column names:That is what I meant above when I said "In the end, the guessing just tries out data format and see which one works and if a table is very similar to such a a format" then it might or might not read correctly.
Tables that look similar to DAOPhot could cause guesing to fail -> fi…
aphearin commentedon Jan 23, 2015
Nice work @hamogu !