Read ASCII tables in chunks - #6458

Hi there @taldcroft 👋 - thanks for the pull request! I'm just a friendly 🤖 that checks for issues related to the changelog and making sure that this pull request is milestoned and labeled correctly. This is mainly intended for the maintainers, so if you are not a maintainer you can ignore this, and a maintainer will let you know if any action is required on your part 😃.

Everything looks good from my point of view! 👍

If there are any issues with this message, please report them here.

WIP for memory optimization, but this is not so easy so punt for now. This is for future reference.

    out = None
    for tbl in _read_in_chunks_generator(table, chunk_size, **kwargs):
        if out is None:
            out = tbl
            continue

        if out.colnames != tbl.colnames:
            raise ValueError('column name mismatch in chunked reading')

        for out_col, tbl_col in zip(out.itercols(), tbl.itercols()):
            try:
                dtype_str = common_dtype([out_col, tbl_col])
            except MergeConflictError:
                raise  # TO DO: handle this
            else:
                if dtype_str != out_col.dtype.str:
                    # Promote column and change output table reference
                    out_col = out_col.astype(dtype_str)
                    tbl[out_col.name] = out_col
                out_col.insert(len(out_col), tbl_col)

@taldcroft - No need to address Fix one range(..) to be six.moves.range(), as this will go into 3.0, won't it?

@bsipocz - Good point, WOOHOO for Py3 only!

I can reproduce failures on my Windows but they are not exactly the same ones as Appveyor:
testlog.txt

There are also a couple of ResourceWarning about unclosed sockets (not captured in log) but probably unrelated?

p.s. To run the tests, I rebased your branch here on top of master locally first before install.

@astrofrog @hamogu - this is ready for final review. The changes in ui.py (+ docs and tests) may be the place to focus.

One test still fails on Appveyor (which is an improvement over the two that I found a week ago).

@pllim - argh, it was passing appveyor before the last commit.

[EDIT] - well I thought it was, but maybe not? Sigh.

At least I now have a functional Windows dev environment on my mac.

It passed! There is a conflict though.

Ah yes, the inplace resize for non-masked columns is no good in the case of a string that changes size. This should be an easy fix.

@taldcroft - Could you add a whatsnew for this, too? http://docs.astropy.org/en/latest/whatsnew/3.0.html#what-s-new-in-astropy-3-0