Repository navigation
hash/ URLs not working/available #4
Description
When downloading a file using a hash/ URL, such as hash/34c33b3eb0d56eb9462003af249eff28, if that file has been previously downloaded and is in your local cache that works fine. However, if you've never downloaded it the hash URL isn't resolved on the data.astropy.org server.
Whenever we add a data file to the server we should also add the appropriate link so that the hash URL works. Or perhaps even better, all files on the server should actually be stored under there hash primarily, with the actual filenames being symlinks to the correct hash for the latest version of that file.
We should also have a database on data.astropy.org (just a JSON file is fine for now) mapping hashes to filenames and vice-versa (the latter may be one-to-many, with the hashes in chronological order for files that have had multiple versions).
Activity
Add optional ``package`` argument to get_pkg_data_* functions so that
Add optional ``package`` argument to get_pkg_data_* functions so that
Add optional ``package`` argument to get_pkg_data_* functions so that
Add optional ``package`` argument to get_pkg_data_* functions so that
eteq commentedon Oct 5, 2015
@embray - I agree with this idea, but I think its probably best done as the hash symlinking to the real file, rather than the other way around. That's because this repo is not used only for the hash, but also for files that are sometimes referenced by name (e.g. in the tutorials). And that makes it much easier for submitters to understand how to add new files.
Either way, though, I think the hash-generating code needs to be done on the server-side (i.e. the server that's currently at STScI mirroring the repository), rather than in the github repo.
embray commentedon Oct 5, 2015
But a single filename can map to multiple files with different hashes, not the other way around.
eteq commentedon Oct 7, 2015
It could be done either way by having something that copies the real (named) file over to its hash before it gets replaced by a new one (which then gets a hash symlinked to it). But now that you say that, you're right it's simpler to just re-symlink the name to the hash (which is the "real" file).
My bigger point here is that we don't want PRs in this repository to have to worry about any of this because it's an error-prone enterprise for anyone who's not a hardcore dev. The hashing should be a thing taken care of by the underlying data server (which can't be done in github, at least until we get Github LFS access).
embray commentedon Oct 8, 2015
Yes, that I agree with--it should be automated.
embray commentedon Oct 8, 2015
And we should look into what we can do with Git LFS. It's been a few months since I've looked at that so I forget how it works in detail.
Add optional ``package`` argument to get_pkg_data_* functions so that
pllim commentedon Dec 29, 2020
This issue is very old and a lot has changed. I am closing this.