warning control "once"

While ignoring warnings works well, I can't make the warning appear only once (as they actually should do as default behaviour)

I have a (potentially corrupt) fits file that outputs tons of warnings like:
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: ??????????????????????????????????????????????????????????????????????????????? [astropy.io.fits.card] , even if I explicitly set the warning filter to "once", they keep coming..

Is this wanted behaviour? If so, why, and can we not change it, or at least allow control over it?

Thanks for reaching out. Can you please give more info, following the "bug report" template? We need a minimal code snippet with relevant data (in your case, offending FITS header) that can be reproducible. Please also provide relevant version and OS info.

Sure thing. The fits is available in our public archive

then just:

from astropy.io import fits
h = fits.getheader("INPE_20120103_113000_58.fit.gz")

and you should see all the warnings.

Then using the linked example about warning control, suppressing it with ignore works, but "once" does the same as "default"...

warnings.filterwarnings('once', category=AstropyUserWarning, append=True)

the python docs state:

default: print the first occurrence of matching warnings for each location (module + line number) where the warning is issued

as far as I can tell all those warnings come from the exact same module+line number so IMO only one warning should be printed with default settings..?

Tested on Win10 with astropy4.0, python 3.7.2

There is no warning filter by defaut for UserWarning: https://docs.python.org/3/library/warnings.html#default-warning-filter

Running your code with python -W default::UserWarning (or defining the filter with the Python API) would do what you expect.

And this file is clearly invalid, you will get an error in the feature (#10895).

Running your code with python -W default::UserWarning (or defining the filter with the Python API) would do what you expect.

like in my example above?

warnings.filterwarnings('once', category=AstropyUserWarning, append=True)

once is even stricter than default and it still prints the warnings..?

Ah and yes. There are clearly some corrupt files,.. happens. That is what we received... unless the instrument operator doesn't correct or discard them they will continue to appear in the archive. Throwing an Error is absolutely fine ;-)

I tried your suggestion with python -W default::UserWarning too, does not solve it
edit: interestingly enough, reading the header twice it only prints all the warnings the first time. So I think for some reason the warnings module thinks the warning is different?

More unexpected behaviour: Processing a bunch of files WITHOUT using the ignore filter on AstropyUserWarning, I get exactly one of those warnings:
WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "time is after end of day (Note 5)" [astropy._erfa.core]

code:

for root, subdirs, files in os.walk(path):
    for f in files:
        fits_file = os.path.join(root, f)
        err_text = None
        try:
            #with warnings.catch_warnings():  # warning control
                #warnings.simplefilter('ignore', AstropyUserWarning)
            header = fits.getheader(fits_file)
        except zlib.error:
            err_text = "zlib.error failed gunzip"
        except OSError as e:
            err_text = f"OSError: {e}"

        if err_text == None:
            # convert date formats to ISO yyyy-mm-dd
            for k in ['DATE', 'DATE-OBS', 'DATE-END']:
                if k not in header:
                    err_text = "missing DATE fields"
                    break
                if len(header[k]) == 8:  # some files such as from PHOENIX use dd/mm/yy
                    header[k] = datetime.strptime(header[k], "%d/%m/%y").strftime("%Y-%m-%d")
                else:  # normal files use yyyy/mm/dd format
                    header[k] = header[k].replace("/","-")

        if err_text == None:
            try:
                Time( header['DATE-OBS'] + "T" + header['TIME-OBS'], format="isot" ).jd
                if header['TIME-END'][0:2] == "24":
                    header['TIME-END'] = "00" + header['TIME-END'][2:]
                Time( header['DATE-END'] + "T" + header['TIME-END'], format="isot" ).jd
            except Exception as e:
                err_text = f"{e} {header['TIME-OBS']} {header['TIME-END']}"

        if err_text != None:
            print(f"{path}: {err_text}")

WITH the ignore filter (commented lines), I get the same ErfaWarning multiple times
does using with warnings.catch_warnings(): reset the warnings-filter? It should not...

edit: interestingly enough, reading the header twice it only prints all the warnings the first time. So I think for some reason the warnings module thinks the warning is different?

Yes I think the warnings are all different, because the message differs (it includes the keyword).

The ErfaWarning is different, not sure what is unexpected.

The ErfaWarning is different, not sure what is unexpected.

Well why does the same ErfaWarning appear once if no filter is used, but multiple times when I ignore AstropyUserWarnings during getheader() ? Is that on purpose? My expectation would be the with warnings.catch_warnings(): block to have no effects to code before and after. The docs states the global state is altered, hence not thread-safe, but upon closing the previous state is restored... I'm not multi-threading anything here (am I?), so I don't see a reason for this..

Yes I think the warnings are all different, because the message differs (it includes the keyword).

Hmm yes, but without specifying a message to filter, all messages should match?
From https://docs.python.org/3/library/warnings.html#warnings.filterwarnings:

Omitted arguments default to a value that matches everything.

I'm a bit lost in what is the issue here. You are processing many files, which I cannot reproduce, so I have no idea why you would get one or more ErfaWarning. But about catch_warnings(), it modifies temporarily the warning filters and restore it at the end of the block, so it should have no effect on the ErfaWarning that is likely generated by the Time calls.

so it should have no effect on the ErfaWarning that is likely generated by the Time calls.

Yes, that's what I'm saying. It should not but it does have an effect.. at least on my end. Here is a sample script that reproduces it, using this file:

PS pls also let me know whether you can reproduce the initial problem, getting the other warning displayed just once still doesn't work on my end.

import os
from astropy.io import fits
from astropy.time import Time

import warnings
from astropy.utils.exceptions import AstropyUserWarning

path = os.path.join(os.path.dirname(__file__), r"2005\12\03")

print(" --- WITHOUT WARNINGS FILTER ---")

for f in ["BLEN5M_20051203_073000_59.fit.gz"]*5:
    fits_file = os.path.join(path, f)
    header = fits.getheader(fits_file)
    Time( header['DATE-OBS'].replace("/","-") + "T" + header['TIME-OBS'], format="isot" ).jd
    Time( header['DATE-END'].replace("/","-") + "T" + header['TIME-END'], format="isot" ).jd

print("\n --- WITH WARNINGS FILTER ---")

for f in ["BLEN5M_20051203_073000_59.fit.gz"]*5:
    fits_file = os.path.join(path, f)
    with warnings.catch_warnings():  # warning control
        warnings.simplefilter('ignore', AstropyUserWarning)
        header = fits.getheader(fits_file)  # , ignore_missing_end=True -> results in "Empty or Corrupt" or "missing date fields"
    Time( header['DATE-OBS'].replace("/","-") + "T" + header['TIME-OBS'], format="isot" ).jd
    Time( header['DATE-END'].replace("/","-") + "T" + header['TIME-END'], format="isot" ).jd

output:

 --- WITHOUT WARNINGS FILTER ---
WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "time is after end of day (Note 5)" [astropy._erfa.core]

 --- WITH WARNINGS FILTER ---
WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "time is after end of day (Note 5)" [astropy._erfa.core]
WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "time is after end of day (Note 5)" [astropy._erfa.core]
WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "time is after end of day (Note 5)" [astropy._erfa.core]
WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "time is after end of day (Note 5)" [astropy._erfa.core]
WARNING: ErfaWarning: ERFA function "dtf2d" yielded 1 of "time is after end of day (Note 5)" [astropy._erfa.core]

Finally had a more detailed look, and that's Python's warnings behavior:

In [1]: import warnings

In [2]: warnings.warn('hello')
<ipython-input-2-bba56b013a54>:1: UserWarning: hello
  warnings.warn('hello')

In [3]: warnings.warn('hello')
<ipython-input-3-bba56b013a54>:1: UserWarning: hello
  warnings.warn('hello')

In [4]: for i in range(5):
   ...:     warnings.warn('hello')
   ...: 
<ipython-input-4-3e5aa4640a88>:2: UserWarning: hello
  warnings.warn('hello')

In [5]: for i in range(5):
   ...:     with warnings.catch_warnings():
   ...:         pass
   ...:     warnings.warn('hello')
   ...: 
<ipython-input-5-4df48fc8ac58>:4: UserWarning: hello
  warnings.warn('hello')
<ipython-input-5-4df48fc8ac58>:4: UserWarning: hello
  warnings.warn('hello')
<ipython-input-5-4df48fc8ac58>:4: UserWarning: hello
  warnings.warn('hello')
<ipython-input-5-4df48fc8ac58>:4: UserWarning: hello
  warnings.warn('hello')
<ipython-input-5-4df48fc8ac58>:4: UserWarning: hello
  warnings.warn('hello')

There is some buffering when calling warnings.warn in a loop, but this is not the case if you call warnings.catch_warnings in the loop iterations. So this is not an Astropy issue, closing this.

Navigation Menu

Repository navigation

warning control "once" #10905

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

warning control "once" #10905

Description

Activity

pllim commented on Oct 21, 2020

Laubeee commented on Oct 21, 2020

saimn commented on Oct 21, 2020

saimn commented on Oct 21, 2020

saimn commented on Oct 21, 2020

Laubeee commented on Oct 21, 2020

Laubeee commented on Oct 21, 2020

Laubeee commented on Oct 21, 2020

Laubeee commented on Oct 21, 2020

saimn commented on Oct 21, 2020

Laubeee commented on Oct 22, 2020

saimn commented on Oct 23, 2020

Laubeee commented on Oct 23, 2020

saimn commented on Feb 20, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions