Repository navigation
astropy.wcs: presence of keywords like "PV1_5" causes error #299
Description
I have a FITS file with an SIP transformation, and also, a series of keywords of the form PV?_? (PV1_1, PV1_2, etc). The WCS constructor seems to think these are coordinate transformation parameters. Perhaps it is searching for the SIP coefficients by looking for anything of the form ???_? ?
To reproduce, grab
http://supernova.lbl.gov/~kbarbary/tmp/PTF_200906253621_i_p_mask_t084123_u012052003_f02_p002899_c07.fits (17MB) (This is the same example file I used in #257).
>>> from astropy.io import fits
>>> from astropy import wcs
>>> hdulist = fits.open('PTF_200906253621_i_p_mask_t084123_u012052003_f02_p002899_c07.fits')
>>> w = wcs.WCS(hdulist[0].header)
WARNING: FITSFixedWarning: 'celfix' made the change 'PV1_5 : Unrecognized coordinate transformation parameter'. This FITS header contains non-standard content. [astropy.wcs.wcs]
>>> w.wcs.print_contents()
ERROR: InvalidTransformError: ERROR 6 in wcsset() at line 1561 of file astropy/wcs/src/wcslib/C/wcs.c:
PV1_5 : Unrecognized coordinate transformation parameter.
[unknown]
... or perhaps there is something I don't know about these PV?_? keywords. But it looks like they are SCAMP-specific and not intended to interact with wcs or SIP transformations.
Activity
mdboom commentedon Jun 21, 2012
PVi_makeywords are used to store extra parameters for certain projection types. Their meaning is specific to the projection being used. For example, there is this note in the wcslib docs:What wcslib is doing here is seeing a greater value for
mthan what it expects for any of the projections it knows about. It's not clear to me whether it should be raising an error here, or it would make more sense to just silently ignore these keywords. On the other hand, theCTYPEin the wcs isTAN, which has noPVparameters, so it's not necessarily wrong that it should complain that there arePVparameters when none would be expected.mdboom commentedon Jun 21, 2012
Also -- do you know what these extra parameters are used for?
kbarbary commentedon Jun 21, 2012
Thanks for the explanation. I believe that these are keywords written and used by SCAMP and SWARP. From the SCAMP manual (available at http://www.astromatic.net/software/scamp ; bottom of page 4):
This header contains
PV_1_0throughPV_1_16andPV_2_0throughPV_2_16, in a section commented as "SCAMP distortion keywords". I interpreted this as a representation of distortion alternative to the SIP coefficients, mainly for the sole purpose of being read by SWARP.To be honest I'm not familiar with how the
PVparameters are used in the FITS standard. Perhaps this use is in conflict with the standard?By the way, this file is a data product from IPAC (for the PTF survey). I'm quite surprised they are creating headers that can't be parsed by wcslib!
mdboom commentedon Jun 21, 2012
This is getting curiouser. The SCAMP documentation refers to
PV_xx_xxkeywords (note the underscore after thePV), which would not conflict with WCS keywords, but of course also aren't defined in the "earlier WCS document" that it links to. Perhaps that's just a typo in the SCAMP documentation. The file you link to hasPVi_makeywords, which, of course, do conflict with WCS keywords. As far as I can tell, their use is in conflict with the WCS standard, since the projection type is declared asTAN, but the tangent projection should have noPVi_makeywords.Also telling in the SCAMP documentation is that they are using a very old copy of wcslib from 1999 that has then been patched to handle the
PVkeywords in a special way.In WCS Paper I, section 2.1.4, the WCS standard says that
PVi_mavalues are for extra parameters needed by certain projections. In WCS Paper II, where the projection types are actually defined, it says what they mean, and their meaning is dependent on the specific projection being used. The spec doesn't really say what should happen in the case of extra keywords. there's an argument to be made here that wcslib should be raising an error for allPVi_makeywords in this case (since the projection is declared asTAN), but at present wcslib always stores the first four, which is the most it would ever need for any of its builtin projections.I went back through the history of wcslib to sanity check that this isn't simply a new bug introduced when the better error handling was put in. Indeed, this behavior (to not accept
PVi_makeywords wherem > 4goes back to at least wcslib 4.7).Also, as an experiment, it's quite simple to remove the error check and just blindly continue when encountering one of the extra
PVkeywords. Unfortunately, you run into problems later because, even though it doesn't need them for this particular projection, wcslib (incelset) validates thePVi_makeywords numerically anyway, and that fails.But this is all somewhat naive conjecture on my part, having read the specs, but not written them and being somewhat late to the WCS party. I fired off an e-mail to Mark Calabretta to look at this issue and provide feedback if able.
kbarbary commentedon Jun 21, 2012
Yes, you're right, the SCAMP manual says
PV_xx_xxbut in fact, it usesPVx_xx. I would guess that parts of the manual are probably way out of date. Here is an example file that is the direct output of SWARP: http://supernova.lbl.gov/~kbarbary/tmp/PTF201105082378_1_o_48428_09.w.head (text, 4.4k). Indeed the keywords arePVx_xx.I suppose this comes down to the fact that in SCAMP and SWARP, Emmanuel Bertin has used a modified version of wcslib so that these keywords work, with the result being that they don't work with the standard version. This is really a shame - it seems that no file containing SCAMP keywords can be read with the standard wcslib! I suppose we could email Emmanuel to check that this is indeed the case.
mdboom commentedon Jul 3, 2012
From Mark Calabretta:
kbarbary commentedon Jul 3, 2012
Thanks for checking into this @mdboom. Hopefully this discussion will serve as guidance for others that run into this problem when using SCAMP in the future.
lpsinger commentedon Jan 29, 2013
I am encountering the same error messages trying to open PTF images using astropy and APLpy. How was this resolved? Do I need to manually chop out the SCAMP part of the FITS headers before reading them in?
kbarbary commentedon Jan 29, 2013
Personally I gave up on trying to read these images with astropy. But if you find a workaround, please do post it here.
eteq commentedon Jan 29, 2013
@mdboom - is this something that might be worth somehow working around? Seeing as how SCAMP is a pretty commonly-used utility, this seems like it would be a nice thing to make work, even if it isn't quite up-to-spec...
mdboom commentedon Jan 29, 2013
It's a tough one -- SCAMP appears to be reusing the
PVi_makeywords for a purpose completely different from how they were originally intended. Does anyone know where to find the documentation about these keywords in SCAMP? The SCAMP documentation refers briefly toPV_xxx_xxxkeywords (which are different and thus should work fine), but the code usesPVi_m.I looked at what
wcstoolsis doing (understanding we can't use it directly, because it's LGPL) -- it does have explicit SCAMP support, but it's a mess. It loads the SCAMP distortion coefficients, but only when theCTYPEisTPV, but all of the SCAMP files I've seen in the wild have aCTYPEofTAN, so wcstools is ignoring whatever distortion parameters might be in these files. Personally, I think thewcslibbehavior is better here because at least it isn't silently ignoring something in the file.I think we can work to try to support this, but in lieu of a clear standard, it will probably require some coordination with the SCAMP folks. At the very minimum, we need some sort of mathematical definition of what these keywords are storing, which may give us a running start at converting them to SIP.
eteq commentedon Jan 29, 2013
If it's not in the scamp manual (https://www.astromatic.net/pubsvn/software/scamp/trunk/doc/scamp.pdf) you might glance at the manual for swarp (https://www.astromatic.net/pubsvn/software/swarp/trunk/doc/swarp.pdf) - my understanding is that SCAMP was basically written to do intermediate work for SWARP.
If the info isn't there, I think your best bet is to contact Emmanuel Bertin - I don't know how responsive he is, but for some of the astromatic utilities (at least, that I've worked with) there simply isn't documentation on intermediate steps.
stargaser commentedon Jan 30, 2013
Hi all,
I'm afraid I have some culpability here. PTF asked me to convert PV to SIP and vice-versa, then decided they wanted both in their headers. We wrote an SPIE paper on the converter software for the conference in Amsterdam last June. Someone (@lpsinger) reminded me that it is posted here: http://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/SIP_To_PV_2012.pdf
I meant to find out if SPIE allows me to post it to astro-ph...then lost my nerve thinking maybe it is too provocative (the part about TPV being superfluous).
Anyway, to be clear, PTF decided they wanted both SIP and PV keywords in their headers so that's what drove this.
David
mdboom commentedon Jan 30, 2013
@stargaser: Thanks for the reference -- I think from that we should be able to move forward on this. I think for astropy we can probably get away with adding a
relaxmode that converts these non-standardPVkeywords intoSIPon the fly and then uses that. Whether we want to write non-standardPVback out, I think, is less important than just getting some way to read these extant files. I'm going to go ahead and reopen the issue for this.ijiraq commentedon Apr 14, 2013
New to this thread.
I am trying to make use of the astropy packages for a CFHT MEGACAM images. High precision WCS information in those images is created using a keyword mapping that matches swarp. They raise the PV1_5 error, due to swarps non-standard keywords, but we can't blame swarp entirely as the standard appears to have been flexible at the time of authorship.
Perhaps, when projection is 'TAN' and there are PV?_? values in the header the wcs could apply the swarp usage?
The keyword definitions appear in the CFHT MEGACAM headers:
http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/data/pub/CFHTSG/821543p.head
The use of the keywords is documented in the MegaPipe paper (Gwyn 2008)
"""The higher order part of the WCS has no standard representation as of yet but the most common is the PVn_n keywords of M. R. Calabretta et al. (2004, private communication2)."""
I know that Stephen Gwyn wrote an appendix for the paper that describes how to use the PV?_? keywords but that section was removed at the request of the reviewer.
embray commentedon Apr 15, 2013
I'm starting to think there needs to just be a "special case" registry for this.
wcs.pva_b_exception_registry.append(lambda header: header['ORIGIN'] == 'CFHT' and header['DETECTOR'] == 'MegaCam')(half tongue-in-cheek?)
ijiraq commentedon Apr 15, 2013
The documentation of how to use the PV keywords is well describe in Appendix A of Gwyn 2008, which is available from his web page:
http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/megapipe/docs/CD_PV_keywords.pdf
Provided in addition to information linked to paper from @stargaser.
Since this issue was reopened 3 months ago with the decision to allow this as a relax mode, is there a patch that might be available for this issue?
vterron commentedon Jun 13, 2013
I've just come across this explanation by Emmanuel Bertin in the Astromatic forums:
He also discusses some details of the implementation, in case they can be of any help.
lpsinger commentedon Jul 23, 2013
FYI, I now have a patch ready; I'm just checking with the PTF project to find out which image or header thereof could be released as a test case.
Remove redundant SCAMP distorition parameters when SIP is also present
Remove redundant SCAMP distorition parameters when SIP is also present
eteq commentedon Oct 18, 2013
@lpsinger - if we merge #1278, then this can be closed, right?
lpsinger commentedon Oct 18, 2013
Yes.
pkgw commentedon Oct 18, 2021
A little note on the exact 8th anniversary of the last comment on this issue (!): Astropy added support for TPV a while ago, but I've just discovered that you'll still get these wcslib complaints about an unrecognized
PV1_5parameter if your header specifies aTANprojection type (or presumably anything other than TPV). If that's the case, you need to changeRA---TANtoRA---TPV, etc.(Discovered with a FITS file from DASCH.)
Remove heuristic code to handle PTF files which is causing a bug