Repository navigation
Relation to specutils #1263
Description
Hi,
I am currently reviewing Pypelt for the inclusion as an affiliated package. For me, the relation of the package to the specutils package (an Astropy coordinated package) is unclear. There seem to be some duplicated efforts, f.e. the OneSpec class seems to have a similar goal as the Spectrum1D class of specutils.
Is there a specific reason not to use specutils? I could even imagine that some routines (like spectrum coadding) may be better put into specutils.
Activity
profxj commentedon Sep 13, 2021
In short, PypeIt was developed in parallel with the new
Spectrum1D(there wasan original but it was deemed undesirable).
Second, the functionality we desire for
OneSpecis much lighter thanwhat is built into
Spectrum1D(e.g. uncertainty handling).All that said, I could imagine a day where the two paths merged,
but that's a bit hard to predict.
As a short-term solution, I presume it would be easy enough
to add a
to_spectrum1dfunction inOneSpecif users requested it.olebole commentedon Sep 15, 2021
Would it be useful to contact the specutils people to see how a common class hierarchy could look like? To me, having two independent classes in the Astropy ecosystem for basically the same thing seems not the best way forward.
profxj commentedon Sep 15, 2021
Well, there is at least even a 3rd in the ecosystem:
https://github.com/linetools/linetools/blob/master/linetools/spectra/xspectrum1d.py
which existed well before the newer development of
Spectrum1D.I think
specutilshad special needs in mind (JWST) and went off on theirown path.
olebole commentedon Sep 15, 2021
Maybe someone from specutils, @rosteen or @nmearl could comment here?
rosteen commentedon Sep 16, 2021
Any decisions on this front were made before my time, but as far as I'm aware the needs of
pypeitwere at least on the radar when the currentspecutilswas designed, so future merger/cross utility seems perfectly reasonable. We certainly didn't design specifically around JWST, there are data loaders for JWST included inspecutils(largely contributed by people on the JWST team) but there are also loaders for a bunch of other missions/datasets as well.@eteq probably has a lot more historical context on the decision making than I do.
nmearl commentedon Sep 16, 2021
specutils, at its core, is meant to be an analysis package. That is, spectral data should be reduced and properly formatted before being fed intospecutils. From there, the package modules represent ways to interact with and analyze the spectral data.Comparatively,
PypeItis more a reduction package: it can take raw observations and detector-specific information and reduce it to the level that would be useful in a scientific analysis. It's understandable thatPypeItwould need something to represent their 1D spectral data, but not necessarily something that would need to interoperate deeply with astropy-oriented analysis functionality (e.g. astropy model fitting, complex wcs operations, uncertainty propagation, etc).I feel that
PypeItandspecutilsdo operate in distinct spaces, but with some overlap that I think could be easily addressed in the future as @profxj points out.astronomeara commentedon Sep 16, 2021
eteq commentedon Sep 16, 2021
I mostly agree with that, but I think it is worth thinking about how/if the machinery in pypeit can use the specutils data structures internally (echoing what @rosteen said that this). That doesn't necessarily have to be the case for affiliated package inclusion, but it's something to think about (and historically in some cases has landed affiliated packages in "yellow" instead of "green" status, although I think @olebole can make his own decision there!)
Where I think there is more overlap is pypeit and the
specreducepackage. That said, I think those also have somewhat different aims -pypeitaims to be a all-in-one reduction process for specific (albiet many) spectrographs, whereasspecreduceis almost exactly the opposite in philosophy - intended to be a toolkit one can pull together pieces of to make a pipeline, but not be the pipepline itself. If we think of it that way, there's room for the two to grow together over time, as long as lines of communication stay open. And in the meantime, pypeit is certainly more immediately useful for spectrographs it supports thanspecreduceis at least as of right now!profxj commentedon Sep 16, 2021
I propose the moment we hear more people are using
specutilsto view/analyze theirPypeIt products than
linetoolsthen we abandon the latter. I surely have no desireto maintain its GUIs indefinitely. ;)
olebole commentedon Sep 16, 2021
@eteq specreduce isn't an Astropy affiliated package, is it?
To me, the situation currently looks as if there are two competing approaches to handle spectra: one is represented by specutils, specreduce and synphot (driven by STScI), and the other is linetools+PypeIt. And both seem to have their potential and their followers.
This is -- in my opinion -- not a very fortunate situation, as spectroscopy is one of the basics in Astrophysics, and I would expect from Astropy to evolve into a powerful, coherent solution here, where the astronomer (or science sw engineer) does not need to decide for on which of the packages they base their own tools, but to combine them seamlessly. What do you think, @profxj @rosteen @eteq (and all others) is there a chance (and an interest) to unify the packages in the way that they are part of a common (bigger thought) spectroscopy area in Astropy?
profxj commentedon Sep 16, 2021
I think the ideal answer is "maybe" to unification while the
reality is that I have little motivation to develop
linetoolsany further(beyond maintaining what is there).
And I suspect I'd be more likely to adopt
speclitein PypeIt if we were to use a different external package.
https://github.com/desihub/speclite
and DESI's tools for spectra viewing (although I admit I've
not used them much so far).
While I agree that spectroscopy is a "basic", I fear the
requirements of various users varies to the extent that
it is tough if not impossible for one size to fit all..
FWIW, here is an ongoing poll in the PypeIt Users Slack:
tepickering commentedon Sep 16, 2021
i'll chime in to state that i agree 100% with @eteq that
pypeitandspecreduceare opposite, yet highly complementary, in their goals and missions.i also want to state that unity in defining/handling 1D spectra does not currently exist within the astropy affiliated ecosystem. specifically,
synphotdoes not usespecutilsorSpectrum1Dat all and instead defines its own hierarchy of spectrum classes. as withpypeit, the reasons for that are both technical and historical. greater unity and interoperability is obviously desirable, but there needs to flexibility within that for differing needs.