Skip to content

Fermi LAT data access #37

New issue
New issue
Closed
Closed
Fermi LAT data access#37

Description

@cdeil
cdeil
opened on Jan 7, 2013

I would like to add access to the Fermi LAT data to astroquery.

There's a bunch of different data products, most are FITS files with tables, which I guess is easy:
http://fermi.gsfc.nasa.gov/ssc/data/access/

However the main data server is via this web form where the user enters some parameters:
http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi

With acceptable parameters one gets a page that contains info on the status of the data preparation on their data server and after a minute or so the download links. Here's an example:
http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/QueryResults.cgi?id=L1301071121485C26556A03

@keflavich Is it possible to interface this data server from astroquery?
(I'll also ask their helpdesk as well if this is allowed and if there is a documented web API I don't know about)

Activity

  1. keflavich commented on Jan 7, 2013

    @keflavich
    keflavich
    on Jan 7, 2013
    Contributor

    Yes, it's possible. I will probably try to unify the interface before
    implementing new query services, but adding this one shouldn't be hard.
    On Jan 7, 2013 8:33 AM, "Christoph Deil" [email protected] wrote:

    I would like to add access to the Fermi LAT data to astroquery.

    There's a bunch of different data products, most are FITS files with
    tables, which I guess is easy:
    http://fermi.gsfc.nasa.gov/ssc/data/access/

    However the main data server is via this web form where the user enters
    some parameters:
    http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/LATDataQuery.cgi

    With acceptable parameters one gets a page that contains info on the
    status of the data preparation on their data server and after a minute or
    so the download links. Here's an example:

    http://fermi.gsfc.nasa.gov/cgi-bin/ssc/LAT/QueryResults.cgi?id=L1301071121485C26556A03

    @keflavich https://github.com/keflavich Is it possible to interface
    this data server from astroquery?
    (I'll also ask their helpdesk as well if this is allowed and if there is a
    documented web API I don't know about)

    —
    Reply to this email directly or view it on GitHubhttps://github.com//issues/37.

  2. cdeil commented on Jan 7, 2013

    @cdeil
    cdeil
    on Jan 7, 2013
    MemberAuthor

    Great!
    I'm curious, are you thinking screen scraping, or is there a proper solution?

  3. keflavich commented on Jan 8, 2013

    @keflavich
    keflavich
    on Jan 8, 2013
    Contributor

    Essentially scraping... that's what we've done for most projects, I think.

  4. keflavich commented on Jan 8, 2013

    @keflavich
    keflavich
    on Jan 8, 2013
    Contributor

    Upon closer look, the POST queries are not so simple, and the data is returned in a slightly different way than most others. The besancon model will serve as a decent model for the data retrieval process, but I'm not so sure how to make the POST query just yet.

  5. kialio commented on Jan 8, 2013

    @kialio
    kialio
    on Jan 8, 2013

    Interesting idea. I have one request: let us know how we (at the FSSC at Goddard) can locate these queries in our logs (probably easy to find since the user agent would be pretty unique but would need to know what it actually is). This way, if there are issues on our end responding to these we can find the problem.

  6. keflavich commented on Jan 8, 2013

    @keflavich
    keflavich
    on Jan 8, 2013
    Contributor

    @kialo - is there a description of the POST keywords available somewhere? Perhaps we could add one identifying the query source by astroquery version?

  7. kialio commented on Jan 8, 2013

    @kialio
    kialio
    on Jan 8, 2013

    Let me talk to our software developers tomorrow at our weekly meeting and see if I can get this information for you.

  8. kialio commented on Jan 22, 2013

    @kialio
    kialio
    on Jan 22, 2013

    I talked to the folks around here and the word is that we won't be able to provide you with the POST keywords (our IT security folks would frown on that). Other than that, we encourage you to do this and will help you if we can (just email me, chat here or send a query via the helpdesk).

  9. cdeil commented on Apr 13, 2013

    @cdeil
    cdeil
    on Apr 13, 2013
    MemberAuthor

    @keflavich Can you give this a try and sketch a solution (preferably in Python :-)? I would very much like this but don't know how to get started.

  10. keflavich commented on Apr 13, 2013

    @keflavich
    keflavich
    on Apr 13, 2013
    Contributor

    @cdeil - I let this one slide because of the last respons from kialio. However, let's see if we can piece together what basic queries look like. I think we need to encode the query in json (which can be done without too much difficulty, though I can't tell you how off the top of my head) and then send that json bundle to the server.

    This site supposedly says which http methods (POST, GET, etc.) allow you to do that:
    http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-14#page-17

    I'm investigating further, but - while I have some idea of where to start - I'm not sure I'll get anywhere.

  11. keflavich commented on Apr 13, 2013

    @keflavich
    keflavich
    on Apr 13, 2013
    Contributor

    This gives a useful explanation:
    http://stackoverflow.com/questions/12385179/how-to-send-a-multipart-form-data-with-requests-in-python

  12. keflavich commented on Apr 13, 2013

    @keflavich
    keflavich
    on Apr 13, 2013
    Contributor

    Woo! got it! Requires a new dependency, requests, which looks pretty mature.

    payload = {'shapefield':1,'coordsystem':'J2000', 'coordfield':'m31', 'destination':'query', 'timefield':'', 'timetype': 'Gregorian', 'energyfield':'', 'photonOrExtendedOrNone': 'Photon', 'spacecraft':'on'}
result = requests.post(url, data=payload)

# if you want to see the result:
with open('/var/tmp/test.html','w') as f: f.write(result.content)

    OK, now have to scrape the link on that page.

  13. astrofrog commented on Apr 13, 2013

    @astrofrog
    astrofrog
    on Apr 13, 2013
    Member

    I'm not sure I understand - you can just do the query with urllib, right? (it handles POST requests). Or am I missing some subtlety?

  14. keflavich commented on Apr 13, 2013

    @keflavich
    keflavich
    on Apr 13, 2013
    Contributor

    You're missing a subtlety: http://bugs.python.org/issue3244 (urllib/urllib2 currently doesn't support the type of requests FERMI/LAT requires)

  15. keflavich commented on Apr 13, 2013

    @keflavich
    keflavich
    on Apr 13, 2013
    Contributor

    (just fyi, I wasn't sneaky enough to find that issue directly; got linked from here http://stacyprowell.com/blog/2009/05/29/handling-multipartform-data-in-python/)

  16. astrofrog commented on Apr 13, 2013

    @astrofrog
    astrofrog
    on Apr 13, 2013
    Member

    Ok, I see now :)

  17. cdeil commented on Apr 13, 2013

    @cdeil
    cdeil
    on Apr 13, 2013
    MemberAuthor

    I think using the requests package should be fine.
    It looks much easier than the Python build-in web modules.

  18. keflavich commented on Apr 14, 2013

    @keflavich
    keflavich
    on Apr 14, 2013
    Contributor

    Further discussion on #68

  19. keflavich
    closed this as completedon Apr 14, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions