You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Some of the data retrieval options in the ESO module lack a verbose flag and are excessively verbose in at least one reported case (see astropy mailing list). We need to make sure that all output can be silenced.
Actually those "Downloading URL..." lines are coming from the progress bar in query.py, so a way to easier silence it (and/or docs how to do it) is what we need.
Otherwise it's worth the time either to invest in setting up an IDE, or build up command line tools that help in searching in the codebase rather than guessing where certain lines are coming from.
This looks like an error where the progressbar_or_spinner class is producing a newline each time instead of overwriting the existing line; I suspect this has to do with your terminal settings but I'm not exactly sure how to fix it. We generally have a verbose boolean option that can be set to False, but it looks like we omitted it in this case; I consider that a bug so we'll note it and try to fix it.
-- Dated circa February 13, 2019
(Edited for clarity and brevity.)
It seems the issue is in astropy.utils where the spinner is drawn. I will
investigate as I didn't see an issue (neither open or closed) in their
bugtracker.
@nandita11 - yes, absolutely. However, I would advise to wait until the maintainers discuss whether the verbose kwarg or switching to logging is the more preferable.
@keflavich@ceb8 - what do you think, should we still at verbose kwargs, or rather use the logger? @andamian good arguments was pushing my thinking towards the logger only setup rather than verbose printing
Sorry, I think there is a misunderstanding here. This issue is not about turning verbose off and on (this was fixed). This is about an issue with the spinner spamming the output on some systems.
There appears also be a related issue opened by @bsipocz in the astropy bugtracker... So I really have no idea why this happens, in principle ProgressBarOrSpinner should not spam the output like this... Unfortunately I don't have the time to debug this further.
Ah, and by the way - I was wrong about the verbose switch being fixed. So this bug here is actually two bugs - if verbose is turned on (cannot be turned off at the moment), it should not be excessively and meaninglessly verbose...
I don't think astroquery should be using astropy progress bar. It is a long-standing bug and I don't see it getting fixed anytime soon. Better to replace it with something else more modern. Also see astropy/astropy#4738
For now I have monkey patched astroquery for my project to avoid the excessive verbosity, while still getting feedback on what is happening:
modified astroquery/query.py
@@ -411,8 +411,8 @@ class BaseQuery:
progress_stream = io.StringIO()
with ProgressBarOrSpinner(
- length, ('Downloading URL {0} to {1} ...'- .format(url, local_filepath)),+ length,+ f"Downloading {url} ...",
file=progress_stream) as pb:
with open(local_filepath, open_mode) as f:
for block in response.iter_content(blocksize):
I guess the idea for a permanent fix would be to use tqdm?
No, from what I understand, the progress bar is fine. But for downloads
from the ESO archive no file size is known - so here `ProgressBarOrSpinner`
will yield a spinner.
As I wrote in my bug report in astropy [1]
If the line is too long for the terminal, which is the case when the filename and the URL is displayed, then every spinner update will create a new line.
Thus I just made the line shorter for now 🤠
[1]: astropy/astropy#11556
Activity
kakirastern commentedon Mar 31, 2019
Will add this when I add some additional keywords to the ESO module as outlined in #1233...
kakirastern commentedon Apr 5, 2019
@keflavich Are these log messages from Requests?
keflavich commentedon Apr 5, 2019
I don't think so.
requestsdoesn't generate any output that I'm aware ofbsipocz commentedon Apr 5, 2019
Actually those "Downloading URL..." lines are coming from the progress bar in
query.py, so a way to easier silence it (and/or docs how to do it) is what we need.Otherwise it's worth the time either to invest in setting up an IDE, or build up command line tools that help in searching in the codebase rather than guessing where certain lines are coming from.
kakirastern commentedon Apr 16, 2019
Copying from original emails for reference by contributors...
Inquiry from user:
Response from repo maintainer:
-- Dated circa February 13, 2019
(Edited for clarity and brevity.)
Knusper commentedon Feb 3, 2020
This error still exists.
bsipocz commentedon Feb 3, 2020
@Knusper - If an issue is open, it's likely it's still relevant. PRs are welcome.
Knusper commentedon Feb 3, 2020
nandita11 commentedon Oct 18, 2020
Can I take this up @bsipocz ?
bsipocz commentedon Oct 18, 2020
@nandita11 - yes, absolutely. However, I would advise to wait until the maintainers discuss whether the verbose kwarg or switching to logging is the more preferable.
@keflavich @ceb8 - what do you think, should we still at verbose kwargs, or rather use the logger? @andamian good arguments was pushing my thinking towards the logger only setup rather than verbose printing
Knusper commentedon Oct 18, 2020
Sorry, I think there is a misunderstanding here. This issue is not about turning verbose off and on (this was fixed). This is about an issue with the spinner spamming the output on some systems.
ceb8 commentedon Oct 19, 2020
@Knusper That was my bad, I didn't read it closely enough.
@nandita11 Given the above clarification this isn't actually a novice ticket.
Knusper commentedon Oct 20, 2020
I think the bug title also should be changed.
Title: Downloads from ESO archive spam shell. Description: see #1357 (comment)
The relevant code that spams the output is around here:
astroquery/astroquery/query.py
Line 374 in e7ea01c
There appears also be a related issue opened by @bsipocz in the astropy bugtracker... So I really have no idea why this happens, in principle ProgressBarOrSpinner should not spam the output like this... Unfortunately I don't have the time to debug this further.
Knusper commentedon Oct 20, 2020
Ah, and by the way - I was wrong about the verbose switch being fixed. So this bug here is actually two bugs - if verbose is turned on (cannot be turned off at the moment), it should not be excessively and meaninglessly verbose...
Knusper commentedon Apr 14, 2021
OK - after hunting I am really sure the excessive spamming of the output comes from astropy #1357 (comment)
The requested verbose flag here should be easy to implement. I will try to file a PR soon.
pllim commentedon Apr 14, 2021
I don't think
astroqueryshould be usingastropyprogress bar. It is a long-standing bug and I don't see it getting fixed anytime soon. Better to replace it with something else more modern. Also see astropy/astropy#4738Knusper commentedon Apr 30, 2021
For now I have monkey patched astroquery for my project to avoid the excessive verbosity, while still getting feedback on what is happening:
I guess the idea for a permanent fix would be to use tqdm?
keflavich commentedon Apr 30, 2021
Yes,
tqdmwould be fine.Your monkeypatch just removes the filename reporting, though - is the issue not with the progressbar?
Knusper commentedon Apr 30, 2021
keflavich commentedon Apr 30, 2021
Ah, yes, that makes sense, and is a fair workaround. But indeed, the best solution is to avoid the spamming in the progressbar
bsipocz commentedon Dec 11, 2023
Addressed by #2681