Repository navigation
Add io.ascii format to read a basic format input with additional header line #3099
Description
Starting from the BasicHeader class, parse a header that includes a second line which must consist of only -, =, and possibly whitespace. Here is a simple pure Python implementation to get started:
class HeaderLineHeader(BasicHeader):
def get_cols(self, lines):
"""Initialize the header Column objects from the table ``lines``.
Based on the previously set Header attributes find or create the column names.
Sets ``self.cols`` with the list of Columns.
:param lines: list of table lines
:returns: None
"""
def check_if_valid_header_line(line):
"""
Check if line has only -, =, and/or whitespace. Note that pure-whitespace lines
are already removed in processing.
"""
for header_char in ('-', '='):
header_line_re = r'[{0} \s]+ $'.format(header_char)
if re.match(header_line_re, line, re.VERBOSE):
break
else:
raise core.InconsistentTableError(
'Header line after column names must contain only -, = and whitespace')
start_line = core._get_line_index(self.start_line, self.process_lines(lines))
if start_line is None:
raise ValueError('start_line cannot be None for header_line format')
for i, line in enumerate(self.process_lines(lines)):
if i == start_line:
self.names = next(self.splitter([line]))
elif i == start_line + 1:
check_if_valid_header_line(line)
break
else: # No header line matching
raise ValueError('No header line found in table')
self._set_cols_from_names()
class HeaderLineData(BasicData):
start_line = 2
class HeaderLine(Basic):
_format_name = 'header_line'
_description = 'Basic table with an additional header line of - or = characters'
header_class = HeaderLineHeader
data_class = HeaderLineData
To do:
- Write the corresponding fast version
- Write the corresponding writer
- Add to guess list
- Tests
- Docs
- CHANGES.rst
- Question: should this format accept all the usual
Basicdelimiters? Probably, but it seems slightly odd
Will close #3025.
Activity
taldcroft commentedon Nov 12, 2014
@hamogu - I was curious what it would take to implement this at a basic level and let myself get distracted with the low-hanging fruit here. But as you noted I should stay focused on other 1.0 priorities, and as always the full PR always ends up being a lot of work. I've assigned you as requested and you can perhaps come up with something nicer. 😄
hamogu commentedon Nov 12, 2014
@taldcroft I did not want to take that away from you. I was just offering help ;-)
I am looking for an easy way to solve this without a new reader. This might be better addresed with some heuristic in the guessing process. After all, this is only a
BasicReaderwith a fixeddata_start=2.Or, tables like this can also be viewed fixed width table where the second line defines the width.
So, I don't have code to show, but I am thinking about ways to avoid YAR (Yet another reader). I see feature creep here ;-)
Pros for a reader:
format=ascii_YAR.Cons for reader:
embray commentedon Nov 12, 2014
Regarding
for header_char in ('-', '='), it might make sense to make the set of allowed header line characters a class attribute. I see no reason to include any additional characters for now, but it would make it easy for a subclass to do so if needed.hamogu commentedon Nov 12, 2014
We are all forgetting what our readers already do:
http://astropy.readthedocs.org/en/latest/io/ascii/fixed_width_gallery.html
The first example is exactly the kind of table we are talking about, and the second to last example shows how to write it with any character as
header_char.So , the REAL question is: Why was the table in #3025 not read correctly?
I guess that the order of readers in the guessing process is not quite ideal, but I will look at this.
embray commentedon Nov 13, 2014
I already know exactly why it isn't being guessed: Although I'm no expert, it's first trying a Basic something or other reader, and that returns good-looking results so it stops there--the heuristics don't check that the first item in each column happens to be a string like '---------' even if all other items in each columns are numbers, say.
taldcroft commentedon Nov 13, 2014
Fixed width is not in the guess list. Maybe that's an easy solution but I wonder if there might be some side effect or other regression. Anyway, avoiding YAR is good if possible.
Add FixedWidthTwoLine Reader to guess list
Add FixedWidthTwoLine Reader to guess list
Add FixedWidthTwoLine Reader to guess list