Repository navigation
Tool to get SFD dustmap values #733
Description
(This is based in part on the discussion in #700)
We should add a function/class that retrieves the Schlegel, Finkbeiner, and Davis 1998 dustmaps, caches them, and uses that to get E(B-V) at arbitrary locations (specified by objects from astropy.coordinates). We probably also should include the R_X coefficients from the SFD paper, so people can get out extinctions directly in their preferred bandpass.
We might also include the improved Schlafly & Finkbeiner 2011 calibrations, although that might not be "standard" enough yet.
Note that there's already a python/numpy implementation of the interpolation scheme in astropysics. So most of the work is already done between that and the astropy data/caching framework.
Activity
adrn commentedon Feb 6, 2013
This would be kickass, so I'd be happy to help!
kbarbary commentedon Feb 10, 2013
I'd also like to see this happen (although I'm no dust expert). To get the ball rolling, here's some initial thoughts I had about interface and implementation:
mwdust(),dustval()?astropy.utils? I didn't closely follow the removal ofastropy.tools.sourcekeyword argument that can specify to retrieve the values from the IRSA web interface instead of downloading the maps (e.g.source='irsa'orsource='maps'). In fact, we might want to make this the default so that users don't unwittingly run the function once and have to wait for a huge download to finish. That is, in order to download and cache the maps, you have to actually read the docs (where you'll be warned about the size of the download and cache).scipy.ndimagefor interpolation. I suppose we'll makeinterpolatean optional kwarg so that the function is usable without scipy?astrofrog commentedon Feb 10, 2013
Does this belong in
astroquery? and more generally, how do we decide what should live inastroquery, and what should get moved to the core?kbarbary commentedon Feb 10, 2013
@astrofrog I was wondering the same about
astroquery, but I'm not that familiar with it. I do think that if we are going to have two methods of retrieving values (from cached files or from IRSA), they should be accessible from the same function.eteq commentedon Feb 11, 2013
I think this does not belong in
astroquery, because in my mindastroqueryis about accessing science data, while this is primarily about getting the extinction value for a given point in the sky. In my mind this is a fundmental necessary tool/utility (although that may reflect my extragalactic bias). I guess that means the title of the issue wasn't quite right, but that's what I was thinking.I agree with @kbarbary that one function seems fine, as well as
interpolateshould only work when scipy is present (unless there's a way to simply re-cast those operations intonumpy-only code... I haven't really thought hard about it). I think it's also fine to pull straight from the princeton site. This may be the sort of thing one would always access from there anyway - in my mind the astropy data server was for data specific to the project, rather than for science data that we sometimes use (but that could change if bandwidth is an issue or something).As for where it should live, that's an excellent question: I don't think it belongs in
utilsbecause that's now mostly for code users are unlikely to want. One possibility might becoordinates, as this is all about getting the dust corrections in a given direction. But that seems a little un-intuitive, I admit.Maybe this needs to be in its own place? In the future there will probably be something for photometry, and this might make sense there. Alternatively, we could make something like
obstools- that's the package this is in in astropysics, and I made it basically because there were some things that are useful observation tools, but don't otherwise have an obvious place (the other thing that naturally sits there would be specific forms of extinction laws, and possibly things like distance modulus calculations. It might even be a good place to put classes for fluxes/magnitudes.)eteq commentedon Feb 11, 2013
Oh, and now that I look a bit more, its not obvious to me that there's a good way to get the IRSA form of the data - the intent of that site seems to be pretty much for people viewing things on the web, so parsing the output might be needlessly complex.
kbarbary commentedon Feb 11, 2013
@eteq I have the same thinking about this being a fundamentally necessary tool for doing other things (and also the same extragalactic bias).
Location-wise, I feel like this belongs in a grab-bag of tools (that's where I would look first as a user).
obstoolsis OK but that sounds like tools for observers or tools having to do directly with telescope observations, which this isn't, necessarily. What aboutmisclike scipy has? Or perhaps it should go inutilsafter all. In the reference docs forutils, we could clearly separate into different sections the generally useful functions and those mainly of interest to developers.There's a programmable interface for IRSA described here:
http://irsa.ipac.caltech.edu/applications/DUST/docs/dustProgramInterface.html
It returns an XML document, so you can parse the values out in about 5 lines of code.
astrofrog commentedon Feb 11, 2013
Small comment: as a non-extragalactic astronomer, I think we would need to make sure that it's made obvious this is the total extinction through the galaxy - obviously for us galactic observers, things are a lot more complex for the dust extinction.
Now for the bigger issue: to me, SFD isn't more fundamental than say SIMBAD, which is currently in
astroquery. SFD is itself science data (granted, that's used for other science), and it's certainly not exact, so it's not truly 'fundamental'. The reason I want to keep these online services separate for now (and including SFD) is because they are much more prone to breakage. If the service's API changes then suddenly all Astropy users have to update and we need to release an update just for this, or they have to go and edit a configuration file.I think there should be a bigger discussion on any kind of online querying in Astropy in future. We need to find ways to make it robust in the long term (e.g. developing an API layer we can control on a server we control, which translates the calls if the APIs ever change). Once we have a good idea how we can manage this, then I'm all for including things like SFD, Simbad, and Vizier (imagine
Table.from_vizier('J/A+A/518/L73')) in the core package. But in the mean time, they can be developed and tested in astroquery. The purpose of astroquery was a kitchen sink for all kind on online querying, science or not, until we figure out which ones to include in the core and how.astrofrog commentedon Feb 11, 2013
Naive question - is there a VO service for the SFD maps? If so, it could potentially go in
astropy.vo.astrofrog commentedon Feb 11, 2013
As to where this could belong once it did make it to the core - how about
astropy.query? Then any online service we move fromastroquerycan naturally go intoastropy.query.kbarbary commentedon Feb 11, 2013
@astrofrog Pretty reasonable. Just to be clear, you're only talking about the functionality for querying IRSA, not downloading and caching the maps, right? Seems like your comment also applies to downloading the maps directly from an externally hosted site, such as the Princeton site. But if we had the maps on an astropy-controlled server, you'd be OK with the download-and-cache function being in the core package?
kbarbary commentedon Feb 11, 2013
For location I'm more concerned with the function that uses the maps directly -- and I don't think
astropy.queryis right for that, of course.astrofrog commentedon Feb 11, 2013
@kbarbary - yes, if we hosted it ourselves, it would be easier to make sure the API doesn't break, but I'm not necessarily advocating storing everything ourselves, since in some cases the datasets are huge, but we can also provide API translation. Basically, I'm just suggesting that once 0.2 is out, we start a discussion on mailing list or google hangout about how to deal with online services in general.
astrofrog commentedon Feb 11, 2013
Regarding location, what do you mean by the function that uses the maps directly? If it's used to correct photometry, then it should go into
astropy.photometry. But the querying code itself, which could be used directly by users, probably belongs in something likeastropy.query(and trust me, we do not want to talk aboutastropy.miscandastropy.utils, norastropy.tools!)kbarbary commentedon Feb 11, 2013
I mean a function originally described in the issue which would be analogous to, e.g., DUST_GETVAL in IDL, but with automatic download and cache of the maps. I said
astropy.querydoesn't seem right because once the maps are downloaded and cached, you're not querying an external server. I suppose you're "querying" values in the maps on your hard drive... but that isn't how I'd normally think of it.In supernova measurements, we tend to separate "photometry" from these dust corrections: Photometry measurements are summing up flux on your images. Dust corrections are something you do after you've determined the total observed flux of the source. This distinction is useful: for example, I would use these dust corrections for simulations that don't even go to the image level, and otherwise wouldn't touch
astropy.photometry.15 remaining items
kbarbary commentedon Jul 16, 2014
There's still a need to get values of the full locally stored fits images. When you're doing thousands of positions, the web query can take a significant amount of time (dominant in a real use case I had).
eteq commentedon Jul 16, 2014
@astrofrog - It's not quite the same thing as this issue, I think.
astroqueryonly provides a way to use the IRSA interface to the dustmaps - it doesn't actually get the dustmaps for you or anything. The IRSA interface is good for many things, but if you want to do something like get the extinction for a bunch of objects, that's much more efficient to do with a local copy of the dustmap.EDIT: posted this before seeing @kbarbary's, but same answer 😉
AstroMike commentedon Jul 17, 2014
Yes, it all depends on your use: for a few objects, you don't want to download the SFD dust maps because they are relatively large (200 MB). On the other hand, for a large number of extinction calls, the query is slow.
sebaguro commentedon Oct 9, 2015
@eteq had an excellent method in "astropysics" to return E(B-V) values for multiple sources it was quick and one needed to first download the SFD_dust_4096_ngp.fits SFD_dust_4096_sgp.fits (granted)
I think astropy + affilated packages (incl. astroquery) currently do not efficiently support a E(B-V) query for a large number of sources. If true, would it not be worth reconsidering to include the etud astropysics code into astropy (as mentioned by Erik in one of the related tickets)?
kbarbary commentedon Oct 9, 2015
I haven't looked at the implementation in astropysics lately, but we have similar code in sncosmo that could be upstreamed:
http://sncosmo.readthedocs.org/en/v1.1.x/api/sncosmo.SFD98Map.html (class interface)
http://sncosmo.readthedocs.org/en/v1.1.x/api/sncosmo.get_ebv_from_map.html (functional interface)
It also requires the user to download the maps manually, just because I felt dodgy about automatically dumping 200MB+ into
~/.cache/.sebaguro commentedon Oct 13, 2015
I guess being able to access many E(B-V) values (for many sources)
is useful for many extra-galactic projects and will become increasingly
useful in LSST era. Maybe it is a good idea to have kbarbary "sncosmo" or etud
"astropysics" version in core because of this argument?
Do I understand correctly that the newer 2011 E(B-V) values are just
found by re-scaling the 1998 E(B-V) values by a constant value? Such that:
E(B − V)2011 = 0.86[E(B − V)1998]
kbarbary commentedon Nov 21, 2016
For anyone finding this issue: there are now a few stand-alone pip packages that provide this functionality:
Brief comparisons here.
bwinkel commentedon May 13, 2017
Please don't use LAB data anymore. Even the main author of the data release paper (Peter Kalberla, a colleague of mine) considers it as deprecated. There is a much better alternative now: HI4PI. HI column densities are provided in HEALPix and various other projections.
cdeil commentedon Jul 6, 2018
@eteq - Given that this is available in other packages now (see comment #733 (comment) by @kbarbary above), maybe close this issue from 2013? Or would you still prefer to add / maintain this feature in Astropy core?
kakirastern commentedon Feb 22, 2019
Hi, am triaging and would like to check if this Issue should be kept open... Would opening a new issue and continue the discussion there in light of the developments leading up to late Feb 2019 be more favorable?
Please advise. Thanks!
bsipocz commentedon Feb 22, 2019
I'm going ahead and close this issue. As @kakirastern pointed out, not much happened in core about it but in the meantime other packages and affiliated packages have been created to cover the topic.