Reading table with URL is doing many requests by taldcroft · Pull Request #2001 · astropy/astropy

taldcroft · 2014-11-25T14:10:49Z

If I read a web-based CSV file using requests it works straightaway and quickly:

In [62]: weather_url
Out[62]: 'http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1'

In [63]: dat = Table.read(requests.get(weather_url).text, format='ascii')

But if I do what I think is the equivalent directly with Table.read it's hitting the site many times and takes probably 10 times as long:

In [64]: dat = Table.read(weather_url, format='ascii')
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]

In [65]:

astrofrog · 2014-01-25T15:43:33Z

Huh, that's not good!

taldcroft · 2014-01-25T19:10:52Z

I haven't looked at how it works, but I ran the commands again (this time in master, not 0.3) and it seems like there is maybe a default timeout that is too short. Using master, after trying ~10 times it actually gave a timeout exception.

embray · 2014-01-27T16:43:54Z

Why would the log message read "[Done]" if the request was actually timing out, or is that an oversight too? :)

pllim · 2014-01-27T17:16:29Z

I did it a slightly different way and did not see such behavior. FYI.

>>> import astropy
>>> astropy.__version__
'0.3'
>>> from astropy.table import Table
>>> from astropy.utils.data import get_file_contents
>>> dat = Table.read(get_file_contents(weather_url), format='ascii')
Downloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statenamDownloading http://www.wunderground.com/history/airport/KLAS/2010/3/1/CustomHistory.html?dayend=28&monthend=4&yearend=2010&req_city=NA&req_state=NA&req_statename=NA&format=1 [Done]

embray · 2014-11-24T19:39:37Z

I double checked and it seems this is still an issue. My guess (pun not intended) is that it's due to the guessing process in io.ascii, except it's downloading the file every time instead of using a cached copy. @pllim's version works because by using get_file_contents it's just returning the table as a string and it's reusing that string each time.

taldcroft · 2014-11-25T14:16:04Z

Code attached. I haven't read the get_readable_fileobj code carefully, but my assumption is that if you don't pass something that can be turned into a readable file object (e.g. a list of data lines) that it just throws an exception. I'm using that to just always try to get a readable file object from the input table when guessing (except for CDS format). This passes io.ascii tests and seems reasonable.

@embray - you were right about this being related to guessing.

cc: @hamogu

embray · 2014-11-25T16:20:45Z

I don't really understand the fix, but it works for me so 👍

hamogu · 2014-11-27T19:43:05Z

This is a good way to do it (I think I can say that after spending significant time on get_file_contents and guessing last week, when I tried to debug why python3 and python2 gave different results when reading from a file object).

So 👍 to merging this.

Just for the record: Not all CDS tables have a separate readme object. Some might have the readme on top of the file and those would go through get_file_contents. However, for those CDS tables that is the right way to do it.

Reading table with URL is doing many requests

astrofrog modified the milestones: v0.4.1, v0.3.3 May 9, 2014

astrofrog modified the milestones: v0.4.1, v0.3.3 May 21, 2014

embray modified the milestones: v0.4.2, v0.4.3 Sep 23, 2014

embray added Affects-release Bug Effort-low 💤 Package-intermediate Effort-medium and removed Effort-low labels Nov 24, 2014

embray mentioned this pull request Nov 24, 2014

Improve io.ascii.read guess when inputting file objects #3132

Merged

Fix problem where URL is being downloaded for each guess

0703686

taldcroft self-assigned this Nov 25, 2014

embray added a commit that referenced this pull request Nov 28, 2014

Merge pull request #2001 from taldcroft/io-ascii-2001

68bd605

Reading table with URL is doing many requests

embray merged commit 68bd605 into astropy:master Nov 28, 2014

embray added a commit that referenced this pull request Dec 31, 2014

Merge pull request #2001 from taldcroft/io-ascii-2001

71462f5

Reading table with URL is doing many requests

taldcroft deleted the io-ascii-2001 branch February 25, 2019 20:23

Navigation Menu

Repository navigation

Reading table with URL is doing many requests - #2001

Reading table with URL is doing many requests#2001
embray merged 1 commit into
astropy:masterfrom
taldcroft:io-ascii-2001

taldcroft commented Nov 25, 2014

Uh oh!

astrofrog commented Jan 25, 2014

Uh oh!

taldcroft commented Jan 25, 2014

Uh oh!

embray commented Jan 27, 2014

Uh oh!

pllim commented Jan 27, 2014

Uh oh!

embray commented Nov 24, 2014

Uh oh!

taldcroft commented Nov 25, 2014

Uh oh!

embray commented Nov 25, 2014

Uh oh!

hamogu commented Nov 27, 2014

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

taldcroft commented Nov 25, 2014

Uh oh!

astrofrog commented Jan 25, 2014

Uh oh!

taldcroft commented Jan 25, 2014

Uh oh!

embray commented Jan 27, 2014

Uh oh!

pllim commented Jan 27, 2014

Uh oh!

embray commented Nov 24, 2014

Uh oh!

taldcroft commented Nov 25, 2014

Uh oh!

embray commented Nov 25, 2014

Uh oh!

hamogu commented Nov 27, 2014

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants