Get error when trying to download data using astroquery.eso

I am attempting to use astroquery.eso to grab data from the ESO archive. Unfortunately, even after following the examples, I find that I get an error when I try to grab the data. There isn't much to troubleshoot with this command so I quickly got to a point where I am stuck.

I have no issues connecting to the ESO archive and receive the notification emails that I am authorized to download all the files I have requested.

I am doing this in Jupyter notebooks using python version 3.7.2, and have attempted this with astroquery version 0.3.9 and 0.4.dev0 with the same error.

What I am doing:

eso = Eso()

eso.login(my_username)

tbl1 = eso.query_main(prog_id=program_id, columns= ('OBJECT','RA','DEC','Program_ID','Instrument',
                                                    'Category','Type','Mode', 'OB ID',
                                                    'TPL START','Exptime','Exposure','Filter',
                                                    'MJD-OBS','Airmass','DIMM Seeing at Start'))

files = eso.retrieve_data(tbl1["Dataset ID"][:2], destination=out_dir, continuation=False, with_calib="none")

Screen shot of traceback:

Please let me know if there is any other additional information that can be helpful when resolving this issue.

I'm trying to reproduce, and it would be easier to do it with using the same program id. Could you share it, or it's still proprietary data? (you can also e-mail it if you like)

The program_id is 094.C-0625(A).
Thanks!

@bsipocz

@bsipocz Is there an update on this? This is still an issue for me. Thanks!

@bsipocz

@bsipocz I am now getting this same error on other data as well. Here are three specific dataset IDs that fail when trying to use astroquery, but work if I try to download them from the archive directly:

NACO.2016-09-16T08:37:27.966
NACO.2016-09-19T01:02:27.449
NACO.2016-03-23T03:41:53.187

Cheers!

I get the same error when downloading MASCOT all sky images:


NFO: Staging request... [astroquery.eso.core]
INFO: Staging form is at https://dataportal.eso.org/rh/requests/Knusper2000/540172 [astroquery.eso.core]
INFO: Downloading 209 files... [astroquery.eso.core]
INFO: Downloading file 1/209: MASCOT.2020-01-30T00:16:29.004.fits.Z... [astroquery.eso.core]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-6277bc0f2761> in <module>
----> 1 download_mascot(dataset_id_list, '/home/ehere/work/test/')

<ipython-input-12-547b5d82d5a4> in download_mascot(dataset_id_list, output_directory)
     30 def download_mascot(dataset_id_list, output_directory):
     31     eso.login(eso_portal_username, store_password=False)
---> 32     eso.retrieve_data(dataset_id_list, destination=output_directory)

~/miniconda3/envs/astroconda/lib/python3.6/site-packages/astroquery/eso/core.py in retrieve_data(self, datasets, continuation, destination, with_calib, request_all_objects)
    811                          .format(i, nfiles, fileId))
    812                 filename = self._request("GET", fileLink, save=True,
--> 813                                          continuation=True)
    814 
    815                 if filename.endswith(('.gz', '.7z', '.bz2', '.xz', '.Z')):

~/miniconda3/envs/astroconda/lib/python3.6/site-packages/astroquery/query.py in _request(self, method, url, params, data, headers, files, save, savedir, timeout, cache, stream, auth, continuation, verify)
    229             self._download_file(url, local_filepath, cache=cache,
    230                                 continuation=continuation, method=method,
--> 231                                 auth=auth, **req_kwargs)
    232             return local_filepath
    233         else:

~/miniconda3/envs/astroconda/lib/python3.6/site-packages/astroquery/eso/core.py in _download_file(self, url, local_filepath, **kwargs)
    613         while trials <= 2:
    614             resp = super(EsoClass, self)._download_file(url, local_filepath,
--> 615                                                         **kwargs)
    616 
    617             # trying to detect the failing authentication:

~/miniconda3/envs/astroconda/lib/python3.6/site-packages/astroquery/query.py in _download_file(self, url, local_filepath, timeout, auth, continuation, cache, method, head_safe, **kwargs)
    298                 log.info("Continuing download of file {0}, with {1} bytes to "
    299                          "go ({2}%)".format(local_filepath,
--> 300                                             length - existing_file_length,
    301                                             (length-existing_file_length)/length*100))
    302 

TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

@kjbrooks

@kjbrooks @Knusper Does clearing your cache at ~/.astropy/cache/astroquery/Eso help?

I have tried other requests recently and it works fine, but I have not actively cleared any cache. Anyway, if cache is related, I might have requested a dataset here for which I previously did not complete the full download... I can try to recreate such a scenario...

When looking at this piece of code, it seems to me that the length = None case is badly handled and can certainly break as shown in the traceback above:

astroquery/astroquery/query.py

Lines 286 to 308 in 52b490c

    
           if 'content-length' in response.headers: 
        
               length = int(response.headers['content-length']) 
        
               if length == 0: 
        
                   log.warn('URL {0} has length=0'.format(url)) 
        
           else: 
        
               length = None 
        
           if ((os.path.exists(local_filepath) and ('Accept-Ranges' in response.headers) and continuation)): 
        
               open_mode = 'ab' 
        
               existing_file_length = os.stat(local_filepath).st_size 
        
               if length is not None and existing_file_length >= length: 
        
                   # all done! 
        
                   log.info("Found cached file {0} with expected size {1}." 
        
                            .format(local_filepath, existing_file_length)) 
        
                   return 
        
               elif existing_file_length == 0: 
        
                   open_mode = 'wb' 
        
               else: 
        
                   log.info("Continuing download of file {0}, with {1} bytes to " 
        
                            "go ({2}%)".format(local_filepath, 
        
                                               length - existing_file_length, 
        
                                               (length-existing_file_length)/length*100))

Hello there, I think I'm having a related issue. I am trying to retrieve the following files:

dpid_list = [
  'EFOSC.2008-11-22T19:10:54.794',
 'EFOSC.2008-11-22T19:11:52.755',
 'EFOSC.2008-11-22T19:12:51.768',
 'EFOSC.2008-11-22T19:13:49.740',
 'EFOSC.2008-11-22T19:14:47.742',
 'EFOSC.2008-11-22T19:24:21.792',
 'EFOSC.2008-11-22T19:25:24.736',
 'EFOSC.2008-11-22T19:26:27.749',
 'EFOSC.2008-11-22T19:27:30.743',
 'EFOSC.2008-11-22T19:28:33.747'
]

using the command eso.retrieve_data(dpid_list, unzip=False). But I get the following TypeError:

INFO: Detecting already downloaded datasets... [astroquery.eso.core]
INFO:astropy:Detecting already downloaded datasets...
INFO: Checking availability of datasets to download... [astroquery.eso.core]
INFO:astropy:Checking availability of datasets to download...
INFO: Downloading datasets... [astroquery.eso.core]
INFO:astropy:Downloading datasets...
INFO: Contacting retrieval server... [astroquery.eso.core]
INFO:astropy:Contacting retrieval server...
INFO: Staging request... [astroquery.eso.core]
INFO:astropy:Staging request...
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-239-42d115367052> in <module>
----> 1 eso.retrieve_data(dpid_list, unzip=False)

~/.env/lib/python3.8/site-packages/astroquery/eso/core.py in retrieve_data(self, datasets, continuation, destination, with_calib, request_all_objects, unzip)
    736                 # should be included too
    737                 # form name is "retrieve"; no id
--> 738                 data_download_form = self._activate_form(
    739                     data_confirmation_form, form_index=-1, inputs=inputs,
    740                     cache=False)

~/.env/lib/python3.8/site-packages/astroquery/eso/core.py in _activate_form(self, response, form_index, form_id, inputs, cache, method)
     66         # Construct base url
     67         form_action = form.get('action')
---> 68         if "://" in form_action:
     69             url = form_action
     70         elif form_action.startswith('/'):

TypeError: argument of type 'NoneType' is not iterable

I'm using astroquery version 0.4.1.

Any workaround?

Addressed by #2681

Navigation Menu

Repository navigation

Get error when trying to download data using astroquery.eso #1580

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	if 'content-length' in response.headers:
	length = int(response.headers['content-length'])
	if length == 0:
	log.warn('URL {0} has length=0'.format(url))
	else:
	length = None

	if ((os.path.exists(local_filepath) and ('Accept-Ranges' in response.headers) and continuation)):
	open_mode = 'ab'

	existing_file_length = os.stat(local_filepath).st_size
	if length is not None and existing_file_length >= length:
	# all done!
	log.info("Found cached file {0} with expected size {1}."
	.format(local_filepath, existing_file_length))
	return
	elif existing_file_length == 0:
	open_mode = 'wb'
	else:
	log.info("Continuing download of file {0}, with {1} bytes to "
	"go ({2}%)".format(local_filepath,
	length - existing_file_length,
	(length-existing_file_length)/length*100))

Get error when trying to download data using astroquery.eso #1580

Description

Activity

bsipocz commented on Nov 6, 2019

kjbrooks commented on Nov 6, 2019

kjbrooks commented on Nov 22, 2019

kjbrooks commented on Dec 2, 2019

Knusper commented on Feb 1, 2020

jwoillez commented on Apr 17, 2020

Knusper commented on Apr 17, 2020

saimn commented on Apr 17, 2020

masolimano commented on Oct 12, 2020

bsipocz commented on Dec 11, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions