You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As discussed at the astropy coordination meeting, a useful feature would be a class or function that can do the standard CCD reduction tasks that are exemplified by the ccdproc iraf task. Astropy should have that.
I have some code for this already (that does include variance frame and BPM handling) for doing basic CCD reductions and I would be happy to contribute this. However, it will require me to update it for using NDdata as well as integrating/updating the unit tests that I have for these as well.
(Also let me know if I should submit this as a pull request or be writing this in a different way--I'm still not always sure if I'm using git correctly)
@crawfordsm - I agree git PRs/issues aren't ideal for brainstorming. Perhaps either the astropy github wiki page or a google doc (as you suggested). You might want to put some of your ideas down, and then post to astropy-dev and see if there are others with strong opinons?
Once there's an actual API doc (e.g. python code showing how to use the object/function in sort of example form), issue a PR to astropy-api. Then once that's accepted (with suitable discussion), hopefully the PR against astropy itself will be much less controversial. (You can also use the API document as the testing suite - then you get your tests basically for free!)
That's a great idea, @astrofrog - @crawfordsm, perhaps try setting out a (free) Cloud9 account tied to your github account and we can explore this there?
In particular, I would be very interested in finding out which geometric transform tasks astropy (+ skimage?) can be used to replace specific IRAF tasks with which I am (perhaps unfortunately) familiar. I suppose if anyone wants to tell me where this belongs, I can give it a go.
Speaking of ccdproc-like functionality, I had raised with Erik and Tom the possibility of adapting our stpipe framework for building things like ccdproc on. It does seem to fit the pipeline model fairly well (i.e., ccdproc does). Mike Droettboom thinks that it would need a little work to make it compatible with astropy (and deal with the data model issue that it is layered on), but I can see advantages to using it for this and other common pipeline models, as well as providing a tool for people to build their own pipelines.
Here's a link to some documentation on how it is being used for JWST pipelines:
I did start the api for ccdproc and I haven't heard anything about it since updating it. So I'll submit it as a pull request and we can further build on it as needed.
Transformation are probably the biggest issue. Personally, I'd prefer never to transform the data and just use WCS information and keep that up dated throughout the process. However, at times, there is need for transformations and I can't say I've been happy with any of the transformation methods that I have been using so far. So @keflavich, if you have any good ideas it would definitely be welcome.
Thanks for the link @perrygreenfield, I think there is definitely somethings to look at it there. So far, I had only been focused on the processing steps, but setting up something for an easy pipeline would probably be very helpful as well.
My bad @crawfordsm - I remember now you asked me to look over it and I never did (mainly because I never set aside the time to figure out how best to use cloud 9...). Anyway, yes, it probably makes sense to PR it to astropy-api and we can have a discussion there.
Glancing at it, though, this is a slightly different approach (although not incompatible) relative to the "pipeline" approach @perrygreenfield is mentioning - your stuff is based on calling methods of a master object, while the pipeline in principal doesn't need all the methods on the same object. But these could be reasonably combined by just having the pipeline call the methods of the object (and users could add there own custom functions that operate on it, if they want).
Oh, one other thing. This is a definite case where we will want to use the NDData uncertainties framework - A CCD image has a fairly well-defined distribution (poisson in each pixel), so I think it makes sense for this use case to essentially drive the definition of that framework (another goal for 0.3)
Activity
crawfordsm commentedon Oct 25, 2012
I have some code for this already (that does include variance frame and BPM handling) for doing basic CCD reductions and I would be happy to contribute this. However, it will require me to update it for using NDdata as well as integrating/updating the unit tests that I have for these as well.
eteq commentedon Oct 25, 2012
That would be great, @crawfordsm - is it functional or object-oriented (in its current state)?
crawfordsm commentedon Oct 25, 2012
It is mainly functional in its current state but I think we can start to put together the api and start a further discussion on the design.
adrn commentedon Oct 25, 2012
I'm interested in following along / helping out with this -- I think we should start with an API document in astropy-api!
crawfordsm commentedon Oct 25, 2012
Okay I've started writing something here--very initial, but just to start brainstorming. It might be better to do this in a different format than git--maybe a google doc for the very initial brainstroming, but I've atleast started listing some of the things that are needed. I'm off right now, but if no one else does before then, I'll add some more detail tomorrow.
https://github.com/crawfordsm/astropy-api/blob/master/ccdproc_api.py
(Also let me know if I should submit this as a pull request or be writing this in a different way--I'm still not always sure if I'm using git correctly)
eteq commentedon Oct 26, 2012
@crawfordsm - I agree git PRs/issues aren't ideal for brainstorming. Perhaps either the astropy github wiki page or a google doc (as you suggested). You might want to put some of your ideas down, and then post to astropy-dev and see if there are others with strong opinons?
Once there's an actual API doc (e.g. python code showing how to use the object/function in sort of example form), issue a PR to astropy-api. Then once that's accepted (with suitable discussion), hopefully the PR against astropy itself will be much less controversial. (You can also use the API document as the testing suite - then you get your tests basically for free!)
astrofrog commentedon Oct 26, 2012
It might be worth trying out Cloud9 IDE for editing API documents in a GitHub repo - it allows collaborative editing and chat too: http://c9.io
eteq commentedon Oct 30, 2012
That's a great idea, @astrofrog - @crawfordsm, perhaps try setting out a (free) Cloud9 account tied to your github account and we can explore this there?
keflavich commentedon Feb 14, 2013
I think this discussion is ongoing, but I think a useful tangent to this would be an "IRAF->astropy" on par with https://www.cfa.harvard.edu/~jbattat/computer/python/science/idl-numpy.html for IDL->numpy translation.
In particular, I would be very interested in finding out which geometric transform tasks astropy (+ skimage?) can be used to replace specific IRAF tasks with which I am (perhaps unfortunately) familiar. I suppose if anyone wants to tell me where this belongs, I can give it a go.
eteq commentedon Feb 14, 2013
@keflavich - I agree this is a very good idea. Can you make a separate issue about this?
perrygreenfield commentedon Feb 15, 2013
Speaking of ccdproc-like functionality, I had raised with Erik and Tom the possibility of adapting our stpipe framework for building things like ccdproc on. It does seem to fit the pipeline model fairly well (i.e., ccdproc does). Mike Droettboom thinks that it would need a little work to make it compatible with astropy (and deal with the data model issue that it is layered on), but I can see advantages to using it for this and other common pipeline models, as well as providing a tool for people to build their own pipelines.
Here's a link to some documentation on how it is being used for JWST pipelines:
http://ssb.stsci.edu/doc/jwst_dev/jwstlib.stpipe.doc/html/index.html
I wondered if this is worth considering for ccd-proc in astropy...
(we can take this to the developer list...)
Perry
crawfordsm commentedon Feb 18, 2013
I did start the api for ccdproc and I haven't heard anything about it since updating it. So I'll submit it as a pull request and we can further build on it as needed.
Transformation are probably the biggest issue. Personally, I'd prefer never to transform the data and just use WCS information and keep that up dated throughout the process. However, at times, there is need for transformations and I can't say I've been happy with any of the transformation methods that I have been using so far. So @keflavich, if you have any good ideas it would definitely be welcome.
Thanks for the link @perrygreenfield, I think there is definitely somethings to look at it there. So far, I had only been focused on the processing steps, but setting up something for an easy pipeline would probably be very helpful as well.
eteq commentedon Feb 18, 2013
My bad @crawfordsm - I remember now you asked me to look over it and I never did (mainly because I never set aside the time to figure out how best to use cloud 9...). Anyway, yes, it probably makes sense to PR it to astropy-api and we can have a discussion there.
Glancing at it, though, this is a slightly different approach (although not incompatible) relative to the "pipeline" approach @perrygreenfield is mentioning - your stuff is based on calling methods of a master object, while the pipeline in principal doesn't need all the methods on the same object. But these could be reasonably combined by just having the pipeline call the methods of the object (and users could add there own custom functions that operate on it, if they want).
eteq commentedon Feb 18, 2013
Oh, one other thing. This is a definite case where we will want to use the
NDDatauncertainties framework - A CCD image has a fairly well-defined distribution (poisson in each pixel), so I think it makes sense for this use case to essentially drive the definition of that framework (another goal for 0.3)astrofrog commentedon May 3, 2014
This is being developed in the
ccdprocaffiliated package, so re-tagging this as 'Future', and we can close after the first ccdproc release.astrofrog commentedon Jul 12, 2014
ccdproc 0.1 has been released, so closing this!
Merge pull request astropy#448 from kelle/drop-interim