ascii.read stalls out when reading large files

@hamogu

@hamogu - I am trying to use ascii.read() to reduce a large (~6Gb) ASCII dataset into a processed binary file in which I have applied some cuts. The strategy I have been taking is to the use data_start and data_end keywords in a loop over chunks of the file, so that only ~100Mb chunks are loaded into memory at a time. Then I make cuts on the chunk, append a Table with the rows passing the cuts, and move on to the next chunk. So, pretty classic 'big data' task.

What I am finding is that ascii.read() is failing to only load chunks at a time. In particular, any call to my ASCII file with ascii.read(fname, data_start=i, data_end=i+chunksize) results in an interminable stall of the python interpreter. So far as I am able to tell, I have no way to use astropy's ascii reader to process data this large. I have verified that this behavior does not happen for an artificially reduced version of the file (~100Mb) with an otherwise identical format.

I understand that ascii.read() has been re-vamped for v1 with 'big data' tasks like this in mind. However, I have verified that this stall-out occurs both with v0.4.x and also with the version of the master branch that is current at the time of this issue.

@mdmueller

You are right that originally this module was meant to be super-versitile at the cost of performance. Originally, we had those 20-500 line tables in mind that you copy and paste from papers or that may have been compiled by hand from the literature or are written as output by tools like DAOPhot. The main point of the io.ascii module is to guess the data format (Can I express column 1 as integer? OK, that fails, can I express it as a float? Can I parse the header as LaTeX? No? Maybe as IPAC? No? Maybe as plain ascii?)

Particularly the "guessing the format" bit requires reading the entire file into memory (although we've been working on the idea to use only e.g. the first 1000 lines for that in the hope that we don't have tables where the first 1000 lines are numbers and then line 1001 is a string). You are right, in recent times we've tried to increase performance for larger tables (maybe ~100 MB) and the main addition to the io.ascii module in recent times is the fast reader implemented in C by @mdmueller but I think even that reads the entire file into memory first (I don't know for sure, though). It was really him and @taldcroft that pushed for better performance, so I ask those two to comment.

However if your data is > 1 GB then io.ascii is probably not the right module to read it at this point. (And personally, I'm not sure if it should be. If you have tables this big, you presumably have a very well defined format and you don't need any of the fancy guessing and conversion and there are other packages out there that specialize on a simpler read routine and are thus much faster, such as np.readfomr txt, pandas, pytables, etc.).

@aphearin

@aphearin - if you are using ascii.read(fname, data_start=i, data_end=i+chunksize) then the issue is that it is trying to guess the format which causes severe slowdown and memory bloating. You should be specifying a format, e.g. ascii.read(fname, data_start=i, data_end=i+chunksize, format='basic') and that should make things a lot faster. Note also that some of the formats have a c backend and will be even faster (@taldcroft or @hamogu can provide more details here). So, to summarize:

Explicitly specifying a format will speed things up a lot
Using one of the 'fast' C-backed readers will speed things up even more (available in 1.0.dev)

I've read many-Gb ASCII tables in the past without issues, provided I've specified the format.

@hamogu

@hamogu - thanks for the rundown. I am happy to use pandas for this purpose instead. However, I should point out that my needs for this task are for developing a core feature of an astropy affiliated package. I don't at all mind if pandas becomes my go-to reader, but in that case I guess I'd want to have the "OK" from the astropy team that using pandas as a required dependency would not prohibit me from being an official affiliated package. This is something I've discussed with @eteq at some length, so maybe the two of you could weigh in on this?

I think we should look into this more - my understanding was that the new C readers in astropy should approach (and in some cases exceed) the performances in pandas. If turning off guessing doesn't help, can you provide a sample of the table so that we can look into it further?

@astrofrog

@astrofrog - it would be easy for me to specify a specific format. But will this specification bypass ascii.read() from loading the entire table into memory? That is a deal-breaker feature for my purposes, since I cannot assume that all halotools users will have enough memory on their laptop to load an entire, uncut halo catalog into memory.

@astrofrog

@astrofrog - the problem is not speed. The problem is loading the entire table into memory.

@aphearin

@aphearin - just to be clear, if you are using guessing, io.ascii will use 20x more memory than needed by the table (mabye an exageration, but a lot more). So if you try and read in just a chunk it would still look like it needed a large amount of memory. If you have disabled guessing and you can't read in just a section of the table using data_start and data_end, I would consider that a bug and we should definitely fix it.

So that we can examine the problem in more detail, can you provide a sample of the table (or even the whole table)? Note that if you want to make graphs of the memory and CPU usage when reading in the table, you can use https://github.com/astrofrog/psrecord.

Maybe @taldcroft and @mdmueller can comment on this?

@astrofrog

@astrofrog - it would be easy for me to specify a specific format. But will this specification bypass ascii.read() from loading the entire table into memory?

Well, it should, but it might not currently. I'm not familar enough with the code to understand whether io.ascii is first loading the whole file. I'm in agreement it shouldn't be loading the whole table if you are asking for a specific range of lines.

@aphearin

@aphearin - in the short term you may be able to use the following (untested) workaround to get reasonable performance (esp. in master). EDIT: fixed a couple of bugs and tested with a real IPAC format file.

import os
from astropy.io import ascii

def read_chunks(filename, n_chunk, format='basic'):
    lines = []
    for i, line in enumerate(open(filename, 'r')):
        lines.append(line.rstrip(os.linesep))
        if i % n_chunk == n_chunk - 1:
            if i < n_chunk:  # first
                chunk = ascii.read(lines, format=format, guess=False)
                names = chunk.colnames
                yield chunk
            else:
                chunk = ascii.read(lines, format='no_header', names=names, guess=False)
                yield chunk
            lines = []
    if lines:
        yield ascii.read(lines, format='no_header', names=names, guess=False)

for dat in read_chunks('ipac.dat', 20, format='ipac'):
    print(dat)

@mdmueller

Yes, io.ascii always reads the whole file. There are a few reasons for this, but for instance the top-level API allows specifying a negative value for data_end, and allows for comment lines anywhere within the file. So the fully-flexible version of io.ascii really wants the whole file in advance.

Having said that, the summer project by @mdmueller was a great success at speed but there wasn't time to fully focus on memory. The fast readers also use significant memory because they implement the full API, so for really big files we need to limit things. This is something we definitely can and should address in the context of io.ascii. Maybe another GSOC project??

BTW, in my code snip above you can also do better by using the data types derived from the first chunk for subsequent chunks so it doesn't waste time guessing the data type.

@taldcroft

@taldcroft - just a thought - it would be worth also including the vstack call in your generator

@astrofrog

@astrofrog - now there is 2 minutes of my life lost forever laughing and watching the animation.

@taldcroft

@taldcroft - you're welcome :)

@astrofrog

@astrofrog - the assumption is that there isn't enough memory for the full table.

So based on the above, I think the necessary functionality is the following. I can supply a single row as input to the reader. The reader uses that row to guess the format, and then it uses that format for the entire file, with no further guessing.

Yck, Offhand I don't think there is any way to get the values of the guessed format in order to use later. Would need to look through the code on this question, but there is always the possibility of manually replicating a limited bit of the format guessing code with an initial chunk of the file.

But in most cases with huge files don't you typically know the delimiter in advance?

Using data_start and data_end should exclusively read in those rows, under that format assumption.

data_start and data_end are predicated on having the whole file in memory (at least now), so you probably aren't going to end up using those. You need to just feed in chunks at a time.

So based on the above, I think the necessary functionality is the following. I can supply a single row as input to the reader. The reader uses that row to guess the format, and then it uses that
format for the entire file, with no further guessing.

Yck, Offhand I don't think there is any way to get the values of the guessed format in order to use later. Would need to look through the code on this question, but there is always the possibility of manually replicating a limited bit of the format guessing code with an initial chunk of the file.

The way to get this information is to keep the Reader object. Something like:

bigreader = ascii.get_reader(..., delimiter= ...)
tabchunck = bigreader.read(chunk1)

So, what we need is (1) a way to return the reader from the guessing process and then we can use that reader read the other chuncks, too.
Then (2) we need an option to re-use the converters and not to try finding new converters that work again. Of course, that will fail if something has be recognised as longint in chunk1, when it really is a float in chunk2.

@taldcroft

Got it. Now it makes perfect sense why this behavior is default.

Something similar to your snippet would be ideal. At the risk of needless repetition, the idea would be this:

Turn off the entire-file-load feature.
Provide the delimiter (yes, I think it is perfectly fine to require users to supply the delimiter in this large dataset case, this is nothing to worry about)
Provide any possible characters that should be interpreted as a comment
Maintain the feature to interpret the first line as providing column names
Pass something analogous to data_start and data_end, so that users can read in their own chunk and process it, one chunk at a time.

@taldcroft & @astrofrog , if there are only 30 hours left before a feature freeze, I realize this will very probably not make the cut. But there really is no way large data files can be robustly read with a laptop without something very much like the above. And this is functionality that halotools cannot do without, and I am rapidly approaching this being a bottleneck in package development. So if you don't mind, would you let me know whether y'all decide to 1) include this in the next release, or 2) include this functionality at all? Either option is fine, of course, but in the event of (2) I would go ahead and start structuring my package to use pandas more liberally, which would be good to know sooner rather than later. Thanks in advance.

@aphearin

@aphearin - this probably isn't going to make it for 1.0 since there are design discussions etc.

The astropy version would be a utility that is mostly outside of the core reading functionality (as illustrated roughly in the code snippet). Would you be willing to develop an astropy PR to do this (with any help we can provide)? If written as a standalone utility then you could put this routine into your package and then call either the local version or the astropy version (if it is available). Even if it got into 1.0 there might be users on 0.4 that you want to support.

@taldcroft

@taldcroft - I'm willing to give it a shot. My timeline for getting to it would be a couple of weeks though.

@aphearin

@aphearin - excellent. No worries on the timescale.

@hamogu

@hamogu - agree with your (1) and (2). How to do (1) in a nice clean way is stumping me at the moment. The best I can think of is a new read keyword like return_reader, but I'm not super excited about that.

@astrofrog

@astrofrog and @taldcroft - I'd forgotten about this discussion until the recent back-and-forth on the astropy mailing list today. I now have a TabularAsciiReader class in Halotools that does this chunked reading using a generator solution inspired by @taldcroft .

https://github.com/astropy/halotools/blob/master/halotools/sim_manager/tabular_ascii_reader.py

The class is reasonably well unit-tested and documented, and only requires numpy. The generator only yields the requested columns, and on-the-fly cuts on rows are made on the yielded chunks. If you'd have any interest migrating this into astropy, I'd be happy to have some version of this absorbed.

@aphearin

I had forgotten about this as well until that thread jogged my memory. This is still a good idea, and it will definitely help to have a working and tested implementation to reference / absorb when doing the astropy version. Thanks @aphearin.

FYI I fixed a couple of bugs in the original POC example of chunked reading and it now actually runs on at least one example.

There's plenty of things that will fit into memory, just not in ascii form.

If anyone here wants to have a go at adapting it you could look at the code I wrote in perl/PDL 10+ years ago for rcols(). One million lines? Not a problem! (I just timed it, took a few seconds)

https://github.com/PDLPorters/pdl/blob/master/IO/Misc/misc.pd

Around about line 600...

Basically you extend the binary arrays each time you read a suitable large chunk of ascii text in.

See #6458 for a mostly-there implementation. This leverages some initial work I did on this a couple years ago that got stalled out.

@karlglazebrook

@karlglazebrook - thanks for the reminder. This has been sitting on my to-do list for a while, hopefully for not much longer.

Navigation Menu

Repository navigation

ascii.read stalls out when reading large files #3334

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ascii.read stalls out when reading large files #3334

Description

Activity

hamogu commented on Jan 22, 2015

astrofrog commented on Jan 22, 2015

aphearin commented on Jan 22, 2015

astrofrog commented on Jan 22, 2015

aphearin commented on Jan 22, 2015

aphearin commented on Jan 22, 2015

astrofrog commented on Jan 22, 2015

astrofrog commented on Jan 22, 2015

taldcroft commented on Jan 22, 2015

astrofrog commented on Jan 22, 2015

taldcroft commented on Jan 22, 2015

astrofrog commented on Jan 22, 2015

taldcroft commented on Jan 22, 2015

astrofrog commented on Jan 22, 2015

taldcroft commented on Jan 22, 2015

12 remaining items

taldcroft commented on Jan 22, 2015

hamogu commented on Jan 22, 2015

aphearin commented on Jan 22, 2015

taldcroft commented on Jan 22, 2015

aphearin commented on Jan 22, 2015

taldcroft commented on Jan 22, 2015

taldcroft commented on Jan 23, 2015

aphearin commented on Feb 26, 2016

taldcroft commented on Feb 26, 2016

taldcroft commented on Feb 27, 2016

karlglazebrook commented on Aug 18, 2017

taldcroft commented on Aug 20, 2017

taldcroft commented on Aug 20, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions