Add lazy parsing of FITS headers #5593

As a followup to #5065 and related issues, there's another lazy-loading enhancement that could greatly speed up reading of FITS files:

When seeking through a FITS file, in order to determine how to find the next HDU the header needs to be read and parsed. Depending on the size and complexity of the header, this is currently an expensive operation in Astropy, even considering that there's already some laziness--for example the values of header cards are not actually parsed until that value is needed (or running verify()).

However, when seeking through a FITS file, really only a few standard keywords, typically appearing (in a valid header) within the first 8 or so cards of the header, are needed. These include: BITPIX, NAXIS, the NAXISn keywords (of which there may be up to 99 but are often not more than 2 or 3), and in some cases PCOUNT and GCOUNT, as well as the extension type.

Lots of efficiency can be gained by reading only that first part of the header when seeking through the file (as well as searching for the END card of the header, which is unfortunately necessary), and deferring loading of the full header until later.

One might approach this in a couple different ways. One might be to use some kind of "minimal" header parser that only has as much functionality as one needs to parse those first few header cards. A lot of code that's already written in the Card class can be used for this--the idea would be to parse one Card at a time, without bothering to initialize a full Header object (an exception would be if the required keywords are not found in the positions they're suppose to be in--this of course makes the header malformed, but it's still nice to be able to read such malformed files if possible--in this case we would switch to reading the full Header).

Another possibility would be to add some more lazy-loading capabilities to the Header class itself, somewhat similar to #5065, though I think it can be done more simply in the case of headers. First I should explain how headers are currently read from a file (this is somewhat different from creating headers from scratch):

1. The Header.fromstring classmethod is responsible for reading the actual bytes of the header, from the first card through to the end card, as well as any trailing padding. This in turn calls the worker method Header._from_blocks which figured out the extend of the 2880 byte blocks that comprise a header, and is responsible for locating the end card. It looks complicated but really isn't--most of the barnacles in it have one of two sources:

   a) The same code is also used to read headers from a text file containing newlines. This adds a bit of complication, but most of the logic for understanding the extents of a header are otherwise the same.

   b) Fallbacks for handling slightly invalid / corrupted FITS files of the sort that have appeared in the wild in the past, such as headers that are missing an END card.

2. The 2880 byte header blocks are joined into a single string and passed to the classmethod Header.fromstring. So in other words fromfile is just responsible for finding the bytes of a header in an actual file and returning that header as a string. Then fromstring is responsible for parsing a header given the contents of the header as a string. Here already there's room for improvement: While most FITS files don't have massive headers in the first place, most of the keywords we need are going to only be in the first 2880 byte header block. The only cases where they wouldn't be is if there is some huge number of axes, or in pathological cases. So one might, say, implement a shortcut where only the first header block is returned and parsed at first, and only look at later blocks if necessary.

3. Header.fromstring reads through the string, extracting each 80-byte header "card". It also has hard-coded logic for handling "CONTINUE cards" which is a single keyword/value pair split over multiple cards. For each logical card, it then creates a Card object in stows its binary data away in that Card object's "._image" attribute. No parsing of the bytes themselves are done yet.

4. The Header instance itself is created by being passed a list of Card objects. This is normally how a Header is instantiated (though in practice it's common to see just Header() where it is instantiated empty). Incidentally, the list doesn't have to strictly contain Card objects--it may also contain (keyword, value) or (keyword, value, comment) tuples from which Card objects can be instantiated.

5. This is the part that probably contributes the most overhead: When instantiated with a Card list, Header.__init__ then loops over that list and calls self.append(card) one card at a time. This is slow for a number of reasons: the Header object is fairly complex and contains a lot of book-keeping and state that needs to be updated each time a card is added to the header. One could probably implement a faster method for instantiating a Header from a bulk existing list of Cards that were parsed right out of a file, rather than calling self.append() one at a time.

Another problem with Header.append() (or any method that adds a new card to a Header is that it necessarily must parse the card's keyword, since the keyword is an important part of the book-keeping (the value and comment do not need to be parsed at this time, as mentioned previously). Parsing the card keyword is actually surprisingly expensive. Normal FITS keywords are just up to the first 8 bytes of the card and can be extracted with ease. However, the generic path that the code currently goes through also supports special kinds of keywords from FITS extensions, such as ESO "HIERARCH" keywords, and the rare but not non-existent "record-valued keywords". Of course, when it comes to just seeking through a FITS file we only need to read simple, standard keywords. So for this case there could be a mode for temporarily ignoring non-standard cards.

Finally, one other possible optimization is to make Header.__init__ lazy--rather than calling self.append() on all the cards it is passed in (or adding them in a bulk operation as suggested above), it could just store off that list internally, and only look through it when a specific keyword is asked for. This is similar to what #5065 does. This again would be most efficient for the case of seeking through a FITS file, because in most cases it would only need to process the first 7 or 8 cards, and leave the rest until the full header is needed.