Developer documentation: write "HOW TO" about package data in affiliated packages.

Affiliated packages for various subfields of astronomy are likely to include package data specific to the subfield. It would be nice to have a step-by-step "HOW TO" (with an example) in the developer documentation on the proper way to include such data.

There are currently guidelines on package data and configuration here. The process is fairly obvious for small data (<100kb), but it is not exactly clear how to extend the outlined process to a third-party server for the affiliated package. One might be able to figure it out by reading the docs for astropy.config and astropy.utils.data, but it would be nice to have a simple example that shows the following:

Adding a default configuration file for the affiliated package (wherein the dataurl for the affiliated package is set)
How to appropriately name your data files on the server side.
A simple class or function in the affiliated package that uses the data.

The documentation might also address some related questions that came up for me regarding data file sizes:

Related question 1: How big is too big? (for even the externally hosted data scheme) Specifically, I'm thinking here of the SFD dust maps, which are ~200MB.

Related question 2: Is/should/could there be a way to install or cache files somewhere other than the user's home directory? (1) For a system-wide install used by multiple users, you might want to only have one copy of the package data. (2) Also, on some systems, disk quotas in home directories are quite small and data is meant to reside elsewhere (or should the user just set XDG_CACHE_HOME accordingly in this case?)

@kbarbary

I like this idea - although the main hang-up is that we do not yet have the data server running. That's mainly because no one has wanted it yet. @kbarbary, are you working on something that needs it? (I speculate something related to the dust maps?)

To answer the questions:

For including in the package, I think ~1MB is about the limit we were thinking (although it's case-by-case). For the external server, I suppose it depends on the server, but I don't imagine 200MB would be much of an issue. Astronomy is more and more about "big data", after all...
There is not a way to share cache across users. Again, if there's call for it, we certainly could implement that. It wouldn't be too hard to access the cache to try multiple locations, one of which would be the users', and another of which could be a shared location.

For situations where user data is supposed to live elsewhere, setting XDG_CACHE_HOME is probably the best choice.

@eteq

I'm starting to develop an astropy-based package for supernova cosmology (could eventually be officially "astropy affiliated" but for now it's just me) that has some built-in data for supernova models. These aren't too big (~14MB right now) but they do need to be on an external server.

Since this is pretty specialized, I was imagining the data would reside on a 3rd-party server (at my own institution, or even kbarbary.github.com for now) rather than the astropy data server. For the "HOW TO" I was specifically thinking of 3rd-party servers so getting the astropy data server up and running is not a necessary steup.

The SFD dust maps is a separate issue, but its something that I imagine might evenually be in the astropy core or an affiliated package. It is also necessary for my package, so I was thinking about how to implement it. (@eteq I was looking at your implementation in astropysics).

@kbarbary

@kbarbary - ahh, I understand now. So let me try again: For external hosting: "too big" is only set by a users' hard drive. I suppose we might want to put a guideline that you should at least include a notice in the documentation if the cached file is going to be relatively large (which is moving target due to Moore's law... but I would think >100 MB might reach the point where people would wonder what's taking up all that space...)

So I think SFD dust maps is fine space-wise, as long as it says somewhere in the docs "this will cache ~200 MB on your hard drive" or similar. Or am I still misunderstanding which part of the docs you're referencing?

As for how to implement it, yeah, I tried to cross-check the astropysics implementation with what's provided with the SFD maps (and http://irsa.ipac.caltech.edu/applications/DUST/ ), so I think it should be fairly solid. Note that I created #733 to make sure we remember to do something like this in astropy. If you want to take a crack at it, @kbarbary, feel free.

@eteq

@eteq That perfectly answers my "related question 1", thanks. And I agree about the HD space warning in the docs! Thanks for opening #733. Maybe someone who is more familiar than me with those maps will take a crack at it (or maybe I will eventually).

Just to be clear though, the main reason I opened the issue was for a scenario like this: I'm developing some astronomy package that uses astropy (affiliated or whatever) and uses package data. I want to distribute it to other users but maybe I don't want to bother putting my data on data.astropy.org; I just want to put it on my own server. My first instinct is to tar up my package data, mypackagedata.tar.gz, put it on a server, and then have instructions that say "download and unpack mypackagedata.tar.gz and point $MYPACKAGE_DATA to it. However, I feel like I should be using the functions in astropy.utils.data instead of all that.

How do I use those functions to get data from my server without messing up other functions that might want to still get data from the data.astropy.org server? Can I just change the astropy DATAURL configuration item at runtime? Or should I have a separate DATAURL configuration item in my own package?

@kbarbary

Ah, I understand now @kbarbary - it's actually even simpler than that in your use case. Instead you could just put this in your code:

def myfunc(...):
    from astropy.utils.data import get_file_contents

    data = get_file_contents('http://www.kbarbary.org/mypackagedata.dat', cache=True)
    ... do whatever you wanted with data ...

or if its actually a .tar.gz file, this should work:

def myfunc(...):
    import tarfile
    from astropy.utils.data import get_readable_file_obj

    with get_readable_file_obj('http://www.kbarbary.org/mypackagedata.tar.gz', cache=True) as f:
        t = tarfile.TarFile(f)  # f is already un-gzipped by the function above.
        d1 = t.extractfile('dat1.dat')
        d2 = t.extractfile('dat2.dat')
... use them appropriately ...

You could also use a ConfigurationItem or something to set the URL, but it amounts to the same thing as the snippet above.

Does that answer the question? If so, I can update the docs to reflect this.

I think the coding guidelines you reference here represent an earlier thinking that there would be a lot of package data, something that hasn't really come to pass. (Although that of course may change).

Ah I didn't realize you could specify the entire URL for the astropy.utils.data functions. I probably didn't read the docstrings closely enough. In my case I would probably want to do something like your first snippet, but in multiple different functions. It would probably make sense to use a ConfigurationItem for this, so that the server address is only in one place? If you could show that in the docs example, that would be great.

Alright sounds good - it definitely makes sense to update the docstring to reflect this use case. I've milestoned this to 0.3, as I don't think we want it holding up 0.2, but it can certainly go there if it gets done in time.

Is have a package (maybe someday an astropy affiliated package) where I'd like to have some example data files.

For now I've started to put them on dropbox.com:

$ wget --no-check-certificate https://www.dropbox.com/s/ac6x3tlvs3ga1gs/GC_FOV_large.fits.gz
$ wget --no-check-certificate https://www.dropbox.com/s/7epv6bqltoyimpv/emin_0010_emax_1000.fits.gz

Is there any way to download them from Python using the astropy functions?
https://tevpy.readthedocs.org/en/latest/tevpy/data/index.html
With wget and web browsers I can download them, but from Python I can't figure out how to do it.

I guess it's not possible to share them via an astropy data server at this point?

Does http://docs.astropy.org/en/latest/_generated/astropy.utils.data.download_file.html not work for this? That's what I'm using for my external data files.

No, the problem is with the certificate ... when you call astropy.utils.data.download_file it puts a html page into the file, not the real file content: https://gist.github.com/cdeil/6037712

This is rather old... And shouldn't this be in package-template?

@pllim

@pllim - The package template is still in the middle of the big overhaul. I suggest to keep this one open here at least until we move the template related docs over to the template.

@cdeil

@cdeil @bsipocz @pllim I bumped into this exact issue today. Are there any news?

@pllim

No news other than it should be written up by someone.

Maybe this can live in the best practices part of the FAQ @pllim envisioned the other day?

cc @astropy/coordinators

Should we move this issue to the package-template repo? (using the new issue move feature)

Should we move this issue to the package-template repo?

Yes, I think it would be great to move it there.

I can't see the magic "move issue" button and I don't have patience to wait for someone to do it, so here it is: astropy/package-template#398

Navigation Menu

Repository navigation

Developer documentation: write "HOW TO" about package data in affiliated packages. #700

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Developer documentation: write "HOW TO" about package data in affiliated packages. #700

Description

Activity

eteq commented on Jan 29, 2013

kbarbary commented on Jan 29, 2013

eteq commented on Feb 5, 2013

kbarbary commented on Feb 6, 2013

eteq commented on Feb 7, 2013

kbarbary commented on Feb 10, 2013

eteq commented on Feb 11, 2013

cdeil commented on Jul 14, 2013

kbarbary commented on Jul 18, 2013

cdeil commented on Jul 19, 2013

2 remaining items

pllim commented on May 11, 2017

bsipocz commented on May 11, 2017

matteobachetti commented on Dec 14, 2018

bsipocz commented on Dec 14, 2018

pllim commented on Dec 17, 2018

astrofrog commented on Dec 18, 2018

bsipocz commented on Mar 18, 2019

pllim commented on Mar 18, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions